Agency Training Plan (ATP) Resource Kit
Appendix 10
Description of Levels of Evaluation
Level One: Reaction Evaluation
Reaction evaluation checks the participants' responses to
- The session or course
- The trainer's style or presentation
- The content of the course
- Comfort of the venue
- Quality of the session materials, handouts, exercises
- Catering, back-up services such as phones, copiers etc.
Ways to evaluate Reaction level include
- Ask participants their views
- Fill out a questionnaire
- List comments and impressions
- Rank the value of the course on a score sheet
- Have a report from the trainer
Level Two: Learning Evaluation
Learning evaluation checks what participants know as a result of the session that they didn't before the training started. This includes:
- Factual information
- Interpretation of information
- Applying information to situations
- Using practical or technical skills in different situations
Ways to evaluate participants' learning level include
- Pre and posttest to compare what participants understood about the themes to be addressed in the training and what they understood after the training
- Short quizzes throughout the session to check understanding of new material
- A case study or role play so participants can demonstrate what they have learned
- Participants' summaries of the key points at the end of each stage of the training session or program
If trainers and the agency cannot organise on-site visits so participants can demonstrate new learning in real situations, then role plays, where possible involving clients can be organised as part of the training program.
Participants should be involved in devising how they will be assessed so they feel it is fair and recognise the results as valid.
If the trainer cannot verify the participants' application of their learning on-the-job then they can ask a manager or other reliable person to do so, using the statement of objectives provided by the trainer.
Level Three: Behaviour Evaluation
Evaluation of Behaviour should occur two-three months after a training session or program. The purpose is to find out whether participants:
- Carry out tasks more effectively since the course
- Treat people better
- Are more knowledgeable, skillful, (or other competencies the training was intended to improve)
Evaluation techniques for behaviour include
- Self-assessment
- Assessment of the participants' work by their supervisors
- Comments from clients, colleagues and other stakeholders about observed new behaviours of the participants
To ensure fairness, the pro-forma used to assess the participant's performance two months after the training should be used before the training event. This will ensure the evaluation rests more on observation at the time than memory of what they were like several months ago and comparing that to what they do now.
Behaviour Evaluation
Participants, supervisors, colleagues, clients, carers, family members complete this form together. You can best evaluate the immediate, long-range, and lasting effects of the program. Now is a good time as it has been several months since the completion of the training program. You have had time to reflect on the material, attempt to put it into practice and make a more informed evaluation of the program.
Level Four: Results Evaluation
The purpose of Results evaluation is to identify any significant improvement in the way an individual carries out their role as a result of the training.
Results Evaluation
- What is the area of competency our agency is evaluating?
- Why is your agency evaluating this competency?
- How will your agency evaluate this?
- Who will evaluate the outcomes?
- Instructors? Managers? Directors?
Training Providers? Consultants?
Clients? Carers?
- Instructors? Managers? Directors?
- When will your agency collect data?
- Prior to the training?
Indicate period of data collection -- 1 week? 2 weeks? 1 Month? Other? ;
- After the training? ;
- 6-12 months after the training.
- Prior to the training?
- What will you measure?
- Number of reported incidents?
Improved communication?
Observed behaviour?- Ask: what specifically are we looking for? How will you know it is happening?
What are the indicators?
- Ask: what specifically are we looking for? How will you know it is happening?
- Number of reported incidents?
- How will you apply a continuous quality improvement strategy?
- (Plan, Act, Review)
- if the training has not provided the desired outcome, what were the barriers that prevented this?
- (Plan, Act, Review)
Evaluation techniques for results include
- Documented improvements in the standard of work or services delivered to clients and colleagues by the participant
- Service targets being met more effectively
- Work being accepted by others, with repeat business as a demonstration, or complimentary letters, testimonials etc
- Greater measurable productivity, improved safety records, etc.