1/16/2010

Know your Trainer---! • Name: Sultan Mohammad • Qualification: MBA (Peshawar University) • Work Experience: 18 Years of field work and managerial experience in different organizations • Specialty: Human Resource Management, Training & Development • e-mail address: [email protected]

The ISD Model of Training Step-4: Evaluating Effectiveness of Training

Assessment

Design/ Develop

Implement

Assess needs Prioritize needs

Select evaluation criteria

Define objectives

Determine evaluation design

Develop lesson plan Develop material Select trainer Select methods

Evaluate

Deliver the HRD program

Conduct evaluation Interpret result

Schedule training

1

1/16/2010

What is an HRD Evaluation? • “The systematic collection of descriptive and judgmental information necessary to make effective training decisions related to the selection, adoption, value, and modification of various instructional activities”

Descriptive and Judgmental Data • Descriptive data provides a picture of what is happening or has happened e.g. – “40% of first-line managers attended a budgeting workshop” – “20% fewer managers attended this workshop than in the previous 12 months”

Descriptive and Judgmental Data • Judgmental information is someone’s opinion based on facts, e.g. – “The turnout for the budgeting workshop over the last six months is disappointingly low compared to last year’s turnout”

2

1/16/2010

Evaluating HRD Programs • Some judgments are made by those involved in the program while, individuals not involved in the program make others. • This collection of information is pre -planned • Evaluation helps managers, employees, and HRD professionals make informed decisions about contents and methods of any training program

Purposes of HRD Evaluation • Whether a program is accomplishing its objectives • Identifying strengths and weaknesses of HRD programs, which can lead to changes, as needed • Determine the cost-benefit ratio of an HRD program • Decide who should participate in future programs

Purposes of HRD Evaluation • Identify which participants benefited the most or the least from the program (and why) • Gather data to assist in marketing future programs • Establish a database to assist managers in making decisions

3

1/16/2010

How often are HRD Programs Evaluated? • Surprisingly, HRD programs are not evaluated regularly and properly • Most commonly used form of evaluation is participants’ reaction, which is not a good measure of assessing the benefits ( ---why?) • It is the ethical responsibility of HRD professionals to prove to the organization whether their programs are indeed beneficial

Reasons---? • Not an easy task: – Requires time, skills, resources and willingness

• Other factors can affect performance: – Economy, equipments, policies and procedures

• HRD professionals are afraid of criticism, accountability and losing funding for HRD activities

Evaluating HRD Programs • Various models have been suggested for HRD evaluation • There is no single model which can be referred to as perfect or ideal • The Donald Kirkpatrick 4 -levels framework is the most commonly used model for HRD evaluation

4

1/16/2010

The Four Levels of Training Evaluation • Kirkpatrick’s evaluation framework provides four levels or criteria for evaluation • Reaction (Level-1) • Learning (Level-2) • Performance or Job Behavior (Level-3) • Results or Impacts (Level-4)

Level-1: Reaction • It is the learners response to training program • Did they like it and feel it was valuable? • Focus is on the learners perception of the course and their satisfaction • A positive feedback will encourage future participation of other employees as well • A negative perception may discourage others and may be reluctant to use the knowledge and skills obtained in the program

Level-2: Learning • What new knowledge or skills were acquired? • Did it produced any change in their attitude? • Did the participants learned what the training program promised them? • Allows a judgment to be made about the learner's capability for performance* • May involve assessment thru a test or quizzes

5

1/16/2010

Level-3: Job Behavior • Does the trainee use what was learnt in training back on job? • Testing the students capabilities to perform learned skills while on the job, rather than in the classroom. • This evaluations can be performed formally (testing) or informally (observation) • Indicators may include reduced customers complaints, decrease in rejections, and so on

Level-4: Results • "What impact has the training achieved?“ • Has the HRD effort increased organization’s effectiveness? • These impacts can include such items as monetary, efficiency, moral, teamwork, etc. • Collecting, organizing and analyzing level four information can be difficult, time-consuming and more costly than the other three levels, but the results are often quite worthwhile when viewed in the full context of its value to the organization

The 5th Level: Return on Investment Some researchers have added a 5 th level to assess training effectiveness beyond results-----!

6

1/16/2010

The 5th Level: Return on Investment • This specifically addresses the issue of organization’s return on investment (ROI) i.e. assessing training effectiveness in dollar terms • Some others include a 5 th level to assess the societal contribution and outcomes created by an HRD program

Employees Training and Return on Investment ROI Productivity Show Performance Develop Skills Employees acquire new Knowledge

Data Collection Methods for HRD Evaluation 1. Interview: to asses individuals’ opinions 2. Questionnaire: a standardized set of questions 3. Direct Observation: watching employees perform in actual work settings 4. Tests and Simulations: structured environment 5. Archival Performance Data: reports and files

7

1/16/2010

Ethical Issues in HRD Evaluation • Confidentiality: – Asking questions about trainee’s and other’s performance, mistakes they make on job, supervisor’s behavior and their feelings about it – Trainees may want such information collected during an evaluation to be kept confidential – A good way is to use group results, unnamed questionnaires

Ethical Issues in HRD Evaluation • Informed Consent: – Participants of an evaluation study should know that they are participating in a study and know its purpose, what they will be expected to do and the potential risks and benefits of participating – Participants are required to sign a consent form – Wherever possible, informed consent should be obtained from employees who participate in an evaluation study

Ethical Issues in HRD Evaluation • Withholding Training: – In control group method, a group of employees is kept under observation without providing any training for comparison and evaluation purpose – Issue: If training is really enhancing employees’ productivity or can bring some promotional opportunity, is it ethical to train some employees and not others just for evaluation purpose---? – Solution:

8

1/16/2010

Ethical Issues in HRD Evaluation • Use of Deception: – Sometimes, a disclosure of the study purpose and evaluation can lead to a positive or negative bias – To eliminate a probable bias, the researcher may want to hide some or give false information about the study – Employees who are deceived can become angry with management, damaging the trust – Must be informed as soon as the study ends

Ethical Issues in HRD Evaluation • Pressure to Produce Positive Results: – Professionals and Managers in HRD may feel pressurized to produce positive results and show that the program they prepared or purchased is really effective – The possibility of a fraud in reporting results cannot be ruled out – It is important to decide on how to demonstrate that an HRD effort has had a positive impact on important organizational measures

Evaluation of Training Costs • Evaluation of a training compares the costs incurred in conducting an HRD program to the benefits received by the organization

9

1/16/2010

Evaluation of Training Costs • This can be done in two categories, namely: – Cost-Benefit Analysis: compares the cost of training to the nonmonetary benefits received such as improvement in attitude, safety, and health – Cost-effective Analysis: financial benefits received like increase in quality and profits, reduction in waste and rejection, reduction in processing time

Categories of Training Costs (Table 7-5, page 25)

• Direct costs: – Course materials, instructional aids, rooms and equipments rent, travel, food, trainer’s salary

• Indirect costs: – Time and resources spent in preparation by the trainer, time spent in planning and implementing, mailing and marketing, other clerical support

Categories of Training Costs (Table 7-5, page 25)

• Development costs: – Development of videos, CDs/DVDs, costs of evaluation and tracking

• Overhead costs: – Costs necessary for the smooth running of training like maintenance of audiovisual equipments, rooms

• Compensation for participants: – TA/DA if specially paid to employees for training, wages of part time workers, or per daily average salaries of participants x no. of days

10

1/16/2010

Calculating Return on Investments • Return on Investment =

Results Training Costs

• An ROI above one means benefits received are more than the costs incurred • An ROI less than one indicated that training costs are more than the benefits received • Example: an ROI of 1.15 or 115% means 115 dollars earned for each 100 dollars spent on training, other than the benefits received in non-monetary terms

HRD Outcomes other than ROI • It is not always necessary (possible) to determine the ROI in dollar terms • Other simpler approaches can be used like: – Changes in turnover rates, Reduced absenteeism – Fewer employees needed to staff various shifts and departments – Increased customer satisfaction and brand loyalty – Customers recommendations of your goods or services to other customers – Improved image of the organization in the industry – Better social image, corporate citizenship

It Was Great Being With You

11