Evaluation/Monitoring Of Training

Contents

• Why evaluate training?
• What do you want out of training?
• Approaches
• The Kirkpatrick model
  • Level one: reaction evaluation
  • Level two: learning evaluation
  • Level three: behaviour evaluation
  • Level four: results evaluation
• Summary

Why Evaluate Training?

Agencies have a vested interest in making sure the training they purchase achieves what was intended. The more an agency knows about what has been effective in the past the more strategic they can be about future investment in training. They can also be more informed about the most effective ways for training delivery.

With the information from past evaluations of training, agencies can assert their needs more effectively and make sure what they pay for is customised to their particular needs.

What Do You Want Out of Training?

It is important for both the agency and the training provider to have a shared, clear understanding of:

• What is the purpose of the training; what skill, attitude, behaviour or knowledge is the training intended to develop?
• What is the agency's preferred form of delivery of the training? E.g. on-site/off-site, classroom, on-the-job, action learning, mentoring, competency based to match the Community Services Training Package, etc….
• How will participants be assessed, if at all; against stated competencies, statement of attendance, on-the-job application of the skills learned, review of application of skills four weeks after training, etc.?

When the agency and the provider are clear about what was expected from the training, the evaluation is fairer and more realistic. Otherwise, unstated expectations surface during the evaluation and the training provider may be blamed for not having foreseen them. It is a similar situation to employing someone without having a clear job description, then after four weeks criticising them because they have not done the job properly: the employee did not know exactly what she was expected to do so was set up to fail.

Approaches

It is important to place all of the following in the context of a collaborative model of assessment:

• The staff should know exactly how they will be assessed and have some say in how the process will work
• What is a fair way to measure increased productivity?
• How should new skills, knowledge, behaviours be assessed?
• The trainer should also have the opportunity to provide feedback on the training from their perspective.

The Kirkpatrick Model

The Kirkpatrick Model outlines four levels of evaluation:

1. Reaction to the program and/or the trainer.
2. Learning as a result of the training program.
3. Behaviour changed as a result of the program.
4. Results - the changes that are noticeable in the participant's skills, knowledge and/or attitudes over the longer term; the effects these have begun to have on the organisation, or on colleagues, clients, quality of service delivery etc…

The first two are typically what you will be involved in unless you have an on-going relationship with a training provider. A long-term training relationship could occur if you use a Registered Training Organisation (RTO) to access qualifications under the National Qualifications System through the Community Services Training Package.

Level One: Reaction Evaluation

Reaction evaluation checks the participants' responses to:

• The session or course
• The trainer's style or presentation
• The content of the course
• Comfort of the venue
• Quality of the session materials, handouts, exercises
• Catering, back-up services such as phones, copiers etc.

Ways to evaluate Reaction level

• Ask participants their views
• Fill out a questionnaire
• List comments and impressions
• Rank the value of the course to them on a score sheet
• Have a report from the trainer.

Level Two: Learning Evaluation

Learning evaluation checks what participants know as a result of the session that they did not before the training started.

• This includes:
• Factual information
• Interpretation of information
• Applying information to situations
• Using practical or technical skills in different situations.

Ways to evaluate participants' Learning level

Pre and post-test to compare what participants understood about the themes to be addressed in the training and what they understood after the training

• Short quizzes throughout the session to check understanding of new material
• A case study or role-play so participants can demonstrate what they have learnt
• Participants' summaries of the key points at the end of each stage of the training session or program.

If trainers and the agency cannot organise on-site visits so participants can demonstrate new learning in real situations, then role-plays, where possible, involving clients can be organised as part of the training program

Participants should be involved in devising how they will be assessed so they feel it is fair and recognise the results as valid.

If the trainer cannot verify the participants' application of their learning on-site then they can ask a manager or other reliable person to verify this, using the statement of objectives provided by the trainer.

Level Three: Behaviour Evaluation

Evaluation of Behaviour should occur two-three months after a training session or program. The purpose is to find out whether participants:

• Carry out tasks more effectively after the course
• Provide better care to clients
• Are more knowledgeable or skilful relative to the training.

Evaluation techniques for Behaviour

• Self-assessment
• Assessment of the participants' work by their supervisors
• Comments from clients, colleagues and other stakeholders about observed new behaviours of the participants.

To ensure fairness, the proforma used to assess the participant's performance two months after the training should be used before the training event. This will ensure the evaluation rests more on observation at the time than memory of what they were like several months ago and comparing that to what they do now.

Level Four: Results Evaluation

The purpose of Results evaluation is to identify any significant improvement in the way an individual carries out their role as a result of the training. Thus, Results evaluation involves an analysis of levels one to three listed above.

Evaluation techniques for Results include

• Documented improvements in the standard of work, care delivered to clients or teamwork with colleagues shown by the participant
• Service targets being met more effectively
• Work being accepted by others, with repeat business as a demonstration, or complimentary letters, testimonials etc
• Greater measurable productivity, improved safety records, etc.

Summary

The level of evaluation undertaken by an agency is entirely their choice. What is described above provides ideas for agencies to implement at whatever level they wish. The staff, management and trainer all need to be clear about the evaluation process and involved in its development and application.