We started to see the impact of the “ISO” era about eight years ago, when clients first started asking if we could measure the improvement of their trainees after an English course. We met this requirement by giving the trainees a test on the course content both at the beginning and at the end of the course and comparing the results. Nowadays, this practice of pre- and post-testing has become commonplace. Around 70% of our courses now involve some form of pre-/post-testing.
In recent years, we’ve seen an increased focus on the meaning of the test results. Employers want to know what it means if a trainee scored 85%, or got a B grade, or achieved Level 2. How does the result relate to their communication skill on the job? Companies are looking for more and more sophisticated measurements of their trainees and their training programs. This presents challenges for both training providers and for the companies themselves because measuring the impact of language training is no simple matter.
HIERARCHY OF TRAINING MEASUREMENT
There is a kind of hierarchy in evaluating training programs, from the simplest measures (1) to the most sophisticated (4).
1. Trainee reaction. Collect the trainees’ feedback about the program.
2. Trainee performance on the course. Test how well the trainees learn the training content.
3. Transfer to job performance. Evaluate the impact on real work communication.
4. ROI (return on investment). Assess the financial benefit of improved performance compared to the cost of training.
Let’s look at how these measures apply to language training.
1. Measuring trainee reaction. This is the simplest evaluation of an English training program. Using a questionnaire, trainees are asked to rate the program on criteria such as instructor’s knowledge and methods, learning materials, interest level, etc.
A good questionnaire will also include open questions like “What is the most useful part of the course?” and “How could this course be improved?” Answers to these open questions are usually the most useful feedback. However, the numerical results from questionnaires are not always a reliable indicator of training performance. For one things, trainees may hesitate to give low scores to a teacher they like, even if his/her teaching performance is poor. In addition, the trainees may have different objectives than the company – for example, they may be more focused on having fun together than doing challenging assignments, and this will affect how they rate a course.
2. Trainee performance on the course. The way to measure how much trainees have learned or improved during a course is to conduct pre- and post-training assessments. (A final test alone may show what the participants know or can do at the end, but cannot show how much impact the course had on it.) There are two approaches you can take to pre-/post-testing: curriculum-based testing or proficiency-based testing.
§ Curriculum-based testing. In its most basic form, the pre-/post-test is a progress test – often right from a textbook– which is administered both before and after training. The good thing about this type of evaluation is the results will clearly show an impact from the training. And it will be in statistical form, so that performance of different trainees (or teachers) can be compared.
The drawback is that the result doesn’t tell you much about the trainees’ ability to use English in the real world. She improved her score by 50%….but can she deal with an English caller on the phone?
§ Proficiency-based testing. If you want a measurement that means something in the real world, you should consider proficiency-based testing. This type of pre-/post-test is not linked to the content of any specific training course. Instead, it measures language skills on a proficiency scale. TOEIC is a well-known example of a proficiency test that measures reading, grammar and listening skills. Other types of proficiency tests can measure speaking skill or writing skill. This type of testing provides a more meaningful measurement of both individual performance and overall program performance. Using a proficiency scale, you can set performance targets for different job positions, and you can estimate training hours for staff to achieve targets.
However, there are two important considerations. Firstly, language proficiency improves slowly. It may take 80-120 hours of training to produce a measurable improvement in proficiency scores, so using proficiency tests involves setting longer range goals. Secondly, proficiency testing must be conducted by experts in order to ensure consistency and reliability. Using proficiency testing will certainly add cost and complexity to your training program. A professional training provider can help you do this in the most cost-effective and efficient way.
3. Transfer to job performance. The next step up from measuring trainee performance on a training course is measuring how they apply it to their work performance. This is very difficult to measure directly. If a trainee writes emails in English, you could analyse a sample of their email writing before and after training. But just collecting the samples poses admin challenges, not to mention who would evaluate them and how. Trying to evaluate telephone skills or meeting skills presents even greater challenges. The only practical option is to measure performance indirectly through work-related assessment tasks, which are scored by a trained evaluator. For example, trainees who receive customer enquiries by phone could perform authentic role-play tasks as an assessment. Trainees who make presentations can be evaluated on a presentation. It is possible to devise a proficiency scale for any of these skills and then set performance targets for trainees.
This type of task-based measurement has very high validity, but it requires a lot of development to create authentic tests, and it requires expert evaluators in order to ensure consistency and reliability. A professional training provider can provide this service.
4. Return on investment. ROI is the ultimate measure of any business investment. Wouldn’t it be great if you could say to your top management, “This year we invested 250,000 baht in English training. The resulting improvement in language skill added 1.8 million baht to our revenue through added sales and customer retention.”
Unfortunately, that’s not going to happen. It is difficult to measure the financial impact of a training program because you usually cannot separate that impact from other factors (e.g., market forces, the economy, changes in staff, other training programs, etc). Moreover, many of the staff who participate in English training are not frontline sales or customer service staff, so their language skills do not directly affect sales or customer retention – they may only use English for internal contacts within your company. The impact of their language skill has more to do with how much confusion or frustration occurs in communication within your company, rather than making or losing money. Can we measure confusion, or put a money value on it? Probably not.
CONCLUSIONS
Training is an investment, and like other investments it must provide measurable benefits to justify the cost. While it may not be possible to measure the financial benefits of language training, it is certainly possible to measure the practical impact of language training on the knowledge and skills of the trainees. At a minimum, a curriculum-based pre- and post-training test will show you that the participants learned something on a course. If you have a longer range goal of improving the language skills of your staff over several years, then proficiency-based testing is the way to go. If your people perform specific communication tasks, such as making sales presentations, receiving telephone calls from customers, participating in project meetings, then task-based assessments will give you the most meaningful results.
I welcome any questions you may have about your particular situation or requirements. I hope you will feel free to contact me.
