ROI ARTICLES

CREATIVITY AND INNOVATION ARTICLES

MANAGEMENT SKILLS ARTICLES

7 METHODS FOR EVERYTHING

OTHER ARTICLES

TURKISHTIME ARTICLES

Measuring ROI-The Fifth Level of Evaluation / Jack Phillips

Download

Training departments are increasingly being asked to justify every dollar spent. Here’s an approach to calculating return on investment using a time-honored training formula

For too long, the training and development process has escaped the scrutiny of accountability. While expenditures have grown, many training departments have not taken the extra step to show the payoff of their efforts—particularly the more elaborate, comprehensive, and expensive programs. Tools and techniques to measure return on investment (ROI) are now available, and the process has become reliable and acceptable. Measuring the return on training investment should be a requirement in most organizations, at least for some programs.

Before dismissing the idea of measuring your company’s return on training investment, consider the rewards of such an evaluation. An example of how valuable training can be is evident in the literacy training program at Magnavox Electronics Systems Company, West Coast Division, in Torrance, California.

Many literacy programs are undertaken because they are needed, but little attention is paid to the program’s economic returns. Magnavox wanted to know if this type of training returned an economic dividend. After its 18week literacy program was covered verbal and math skills, it was shown to have a significant payoff for the company. The benefits of the program (reduced scrap, rework, and increased productivity) were converted to dollar values. The program yielded a cost-benefit ratio of 8.4:1 and a 741 percent return on training investment! And while these numbers are impressive, the program designers consider them conservative.

In every corner of the training and development field, the pressure to measure the return on investment is increasing. At some time or another, virtually every organization will face this important issue. Many progressive organizations are taking a logical and methodical approach to developing ROI for a limited number of programs, using a sampling basis. Six trends have increased the interest in, and use of, ROI measurement in training and development:

▼ Training and development budgets are continuing to grow, which creates more pressure for accountability.

▼ Training and development are linked to competitive strategies, which make them important areas for measuring program contribution. 

▼ Many programs have failed to deliver what was expected; consequently, program sponsors have requested ROI calculations/ justifications.

▼ The concern for accountability in all functions in an organization is increasing; thus, the training and development function becomes one of many support efforts under scrutiny.

▼ Top executives in a large number of organizations now require ROI information.

▼ To justify their contribution, trainers have increased their interest in the ROI process.

A NEW MODEL FOR EVALUATION LEVELS

Developing ROI for training requires a key modification of a classic model. The four-level framework developed by Donald Kirkpatrick in 1959 does not focus directly on the ROI issue. As shown in figure 1 (page 12), Kirkpatrick defines Level 4 evaluation as the results linked to training. These results could take the form of reduced absenteeism and turnover, quality improvement, productivity, or even cost reduction. But this level of evaluation does not require a specific monetary value (cost savings) to be determined. To obtain a true ROI evaluation, the monetary benefits of the program should be compared to the cost of implementation in order to value the investment. In effect, this process moves evaluation to the next level—Level 5 in our revised Kirkpatrick model. Thus, the fifth level of evaluation is developed by collecting Level 4 data, converting the data to monetary values, and comparing them to the cost of the program to represent the return on training investment.

In practice, many organizations are taking evaluation to this new level for a few selected courses, often using some form of sampling. When the ROI formula is developed, evaluation is conducted at all five levels. For example, in the Magnavox case, Level 1 evaluation— reaction of employees—was measured by post- course surveys. Level 2 learning was measured by TABE (Test of Adult Basic Education) scores before and after training. At Level 3, changes in the behavior of employees were measured by daily efficiency ratings. At Level 4, business results were measured through improvements in productivity and reductions in scrap and rework. Finally, at Level 5, ROI was calculated by converting productivity and quality improvements to monetary values and comparing these to the full program costs to yield an ROI value.

THE ROI PROCESS

Calculating ROI requires a process model, as depicted in figure 2 below. The various elements of evaluation (design instruments, levels, and purposes) form the specific data collection plans. A variety of data collection tools, ranging from questionnaires and surveys to monitoring on-the-job performance, are available to trainers.

Once data is collected, the next step of the ROI analysis begins with deliberate attempts to isolate the effects of training on the data items. At least ten strategies have been used to accomplish this: 

▼ use of controls
▼ trend line analysis (time series)
▼ forecasting methods
▼ participant estimates of training impact
▼ supervisor estimates of training impact
▼ management estimates of training impact
▼ customer input
▼ expert estimates of training impact
▼ subordinate input on training impact
▼ calculations/estimations of the impact of other factors.

The next step is to convert collected data to monetary values. This requires a direct conversion of hard data, such as quantity, quality, cost, or time, which is an easy task for some programs such as technical training. For “soft” data, the task is more difficult, although a variety of techniques are used to place values on the improvements. Among the techniques used are

▼ historical costs
▼ supervisor estimation
▼ management estimation
▼ expert opinion
▼ participant estimation
▼ external studies.

The next step is to calculate the costs for the program. Although there has always been a need to capture training costs, the need is amplified with more attention on accountability and the ROI calculation. The ROI formula is the annual net program benefits divided by program costs, where the net benefits are the monetary value of the benefits minus the costs of the program. The ROI formula is as follows:
 
ROI (%) = (Benefits – Costs x 100) / Costs
 
This model also recognizes that there should be intangible benefits that will be presented along with the ROI calculation.
 
ROI STRATEGIES/BEST PRACTICES
Although it is difficult to uncover companies’ precise strategies, a recent search identified over 2,000 organizations that could be contacted in an effort to determine the nature and status of the ROI process. Although this was not a carefully designed research project, it presented a review of the efforts of many progressive organizations. Several common strategies began to emerge that can be considered best practices for calculating an ROI in training and development. A review of such best practices is as follows:
 
Set targets for each evaluation level.
Recognize the complexity of the evaluation levels described earlier. Some organizations attempt to manage the process by setting targets for each level. A target is the percentage of training programs measured at that level. For example, at Level 4, where it is difficult to measure, organizations have a low level of activity—usually less than 20 percent. Level 5 evaluation—ROI—is even less likely to occur—usually around five percent, reflecting the complexity of a process that commands significant resources and budgets.
 
​The process of establishing evaluation targets has two important advantages. First, it provides measurable objectives for the training staff to clearly measure progress for all programs or any segment of the process. Second, adopting targets focuses more attention on the accountability process, communicating a strong message to the training staff about the commitment to measurement and evaluation. 
 
Evaluate at the micro level. Training measurement and evaluation usually focuses on an individual program or a few tightly integrated courses. The ROI process is more effective when assessing the direct payoff of an individual program. Attempting to evaluate a group of courses conducted over long periods of time is quite difficult. The cause and effect relationship becomes more confusing and complex. Also, it is inappropriate to attempt to evaluate an entire function such as quality training or technical training. For this reason, evaluation must be a micro-level activity.
 
Use sampling for ROI calculations.
Determining the desired level of ROI calculations is an important issue. There is no prescribed formula and the number depends on many variables, including
▼ staff expertise on evaluation
▼ nature and type of training programs
▼ resources that can be allocated to the process
▼ support from management for training and development
▼ organization’s commitment to measurement and evaluation. 
 
Other variables specific to the organization may enter the process. Most organizations settle on evaluating one or two sessions of their most popular programs. For example, the federal government’s Office of Personnel Management has developed an ROI calculation for one of its most popular courses—Introduction to Supervision. Still others may select a program from each of their major training segments. In a large bank with six training academies, a program is selected from each academy, each year, for an ROI calculation.
 
For organizations implementing the ROI concept for the first time, only one course should be selected for a calculation
as part of the ROI learning curve. In the final analysis, the selection of programs for ROI calculations should yield a level of sampling where top management is comfortable in its assessment of the training and development function. 
 
Editor’s Note: The Magnavox case is just one of 18 cases reported in Measuring Return on Investment, recently published by the American Society for Training and Development as volume one of its In Action series.


Download