Lack of Progress in Measuring Return on Learning Investment (ROI) Hampers Future Implementations.
Companies spend an average of $1,202 per employee on training, according to a 2009 report released by Bersin & Associates. What’s surprising is that even in today’s tough economic climate, many companies are still tracking completion rates to evaluate the value of their investment in e-learning. And this is 40 years after Donald Kirkpatrick first published his ideas on evaluation! Why has there been such a lack of progress in measuring the return on investments (ROI)?
The slow progress has something to do with the strong link that has been made between evaluating training and proving its value. After all, when the measurement of a course’s ROI is discussed, it’s with the intent to prove that the training was worth the investment to design, develop and deliver it.
This scenario is a little like closing the barn door after the horse has escaped. Does anyone really want to know if training didn’t work? Worse yet, who wants to be the bearer of bad news to senior leadership?
The purpose of evaluation is to identify and correct problems that hinder learners’ ability to apply new knowledge and skills on the job. In other words, evaluation allows you to course correct to ensure that you get results from training and to capture lessons learned to apply to future projects. In contrast, calculating the ROI of training must happen before the investment in training is made — before it’s too late.
ENSURING TRAINING ADDS VALUE: ROI OR ROE?
Still, looking for a return on investment on every training program doesn’t make sense.
For example, in the case of end-user systems training, the ROI the company realizes should be on the millions of dollars spent to purchase, customize and implement the system. The training program is simply a piece of the system implementation process. In fact, this is true anytime a company introduces something new such as a new system, process, product, service standard or job responsibilities. In this case, to prove the value of training, you need to make sure that the training works so that it does have value.
This is where evaluation comes into play. And I don’t mean knowledge checks. You’ll need to assess whether employees are using their newly acquired knowledge and skills on the job. You can gather this data from secret shoppers, help-desk reports, customers, managers, peers and employees themselves.The goal is to find out what knowledge and skills they’ve been able to implement and what, if anything, is standing in the way of applying everything they’ve learned. In other words, it’s not about ROI but about return on expectations (ROE).
For example, several years ago we developed a class to teach employees how to use a new system. To help ensure that end-user adoption was on track, we both interviewed employees who were identified as “shining stars” in adopting the inventory system as well as those who were floundering. We wanted to determine what we could do to close the skill gap between the two groups.We knew they had all been through the same training session. So, were the shining stars receiving additional support or coaching back on the job?
There was one major difference between the two groups: accountability.
During weekly staff meetings, the managers of the shining stars asked for inventory reports based on the metrics provided by the new system. Rather than look foolish in front of their peers, the shining stars put in the extra time and effort to master the new system so that they could show up for those meetings prepared. The fact that their managers held them accountable for using the new system was the extra push they needed to apply what they had learned in training to their jobs.
The secret to success: Senior leadership needed to hold managers accountable for
generating reports from the new system. Of course, the only way the system would
generate these reports was for recently trained employees to actually use it — that’s how the company would realize its ROE (and ROI) for the new system.
AN ABSOLUTE MUST
So, when must you calculate the ROI of training? When training is the solution,
in whole or in part, to a specific business challenge. Such challenges could include
an increase in customer complaints or employee turnover, a drop in sales or profit margin, or even a lack of leadership bench strength. Here are the steps to take to figure out the ROI of training before you start:
NO. 1: Determine the benefits of the training.
First, you’ll need to determine the potential monetary benefits so that you can compare them to the anticipated cost of the training. The manager requesting the training often can help you land on a number for the monetary value. Look for cost savings or increased profit over a specific time period. Some ideas include:
>> Probability of making a mistake times the cost of the mistake equals additional savings as a result of employees making fewer mistakes;
>> Additional sales times average value per sale equals additional profit as a result of employees selling more;
>> Reduced turnover times cost of new hire acquisition equals additional savings as a result of the company having to spend less money recruiting and training new hires.
For example, a training course we developed helped one of our clients reduce its new employee training period by 78 percent. In other words, employees were fully productive rather than sitting in training 78 percent faster. This was a huge cost savings.
NO. 2:Determine the cost of the training.
Next, you’ll need to figure the cost of designing and implementing the training.
If you plan to buy the training, costs to consider include:
>> Design and development of the training;
>> Customization of a generic course and;
>> Licensing fees.
If you plan to develop the training inhouse,
costs can include:
>> Salaries and benefits for instructional designers and others involved in the design and development of the training;
>> Travel, lodging andmeals for instructional designers to gather information and;
>> Purchase of books or reprints of articles needed to research the course topic.
A simple way of estimating the cost of a particular training course is to check the costs involved in the design and delivery of a similar, recent course.
NO. 3: Calculating the value of the training.
The ROI calculation is fairly straightforward. Let’s say you expect a certain program to provide benefits of $800,000 at a cost of $200,000. Since ROI uses only the net benefits, you need to start by subtracting the cost from the benefits. The net benefits in this example would be $600,000 ($800,000 – $200,000), so the ROI calculation would be:
ROI (%) = ($600,000/$200,000) x 100 = 300%
In this example, the there is a $3 return in net benefits for each dollar invested.
IF IT’S NOT WORTH IT…
Sometimes, the cost-benefit analysis may reveal that the training is not worth it. If this is not an acceptable outcome, you might need to get creative about the type of training you offer. I have found that a simple, low-cost solution will often do the job just as well as a sophisticated and expensive offering.
Take the example of a surgical checklist that saves lives. A report on the results of using this simple checklist was published in the Jan. 14, 2009, online edition of the New England Journal of Medicine. At first, surgeons thought the checklist was juvenile. Then the results rolled in. During the year of the study, the rate of major complications in operating rooms dropped by more than a third when the checklist was used. More importantly, deaths dropped by more than 40 percent when the checklist was introduced. Not bad for a lowly job aid.
Of course, downloadable job aids are not always the answer. But if you ask, “What is the simplest, least expensive tactic we can use to solve this training problem?” you could discover a range of affordable — yet equally effective — options that you hadn’t considered.
To get ideas for other affordable options, consider looking to the Web, a content-rich environment with a multitude of methods for sharing information. For example, Ryan Yee is the founder and star of Nerd Boy TV. He uses simple, low-cost video to explain the features and benefits of all sorts of interesting electronic
gadgets. You could borrow from Ryan by filming subject-matter experts demonstrating the features and benefits of a new product or even how to perform a new process and then making it available online in a YouTube-type format. The production values don’t have to be high for it to be effective.
One idea that I have seen backfire is to try to contract work out to a low-cost vendor. Unfortunately, with instructional design and development you get what you pay for. So if you do pay for instructional design, you should at least get something considerably better than what you could have produced inhouse by subject-matter experts. By “better,” I mean specifically both instructionally sound and high production values.
COVER YOUR ASSETS
Once you’ve calculated the anticipated ROI for the training, you are ready to cover your assets (CYA). To do this, you present the manager requesting the training with the following:
>> A summary of your mutual understanding of the situation;
>> Agreed-upon anticipated benefits of the training;
>> Options for training design and delivery;
>> Associated costs, including implementation, for each option;
>> Pros and cons of each option in terms of learning effectiveness and efficiency;
>> Resources, in addition to budget dollars, required.
Next, sit back and let the manager decide how to move forward. Make sure to document the decision the manager made and the rationale for making it. By involving the manager in making the decision, you’ve positioned yourself in such a way that you will not be called on the carpet to prove the value of the training — again. And, if you are, you’ve got notes to back you up.
THE BOTTOM LINE
Training, like any investment, is a risk. There is no guarantee it will pay off. Learning programs that drive business results are not easy to develop, and sometimes it’s darn near impossible to get the support you need to implement them properly.
However, ensuring that you calculate the ROI before the investment is made, and using evaluation techniques to identify issues and course correct, will go along way to helping you speak the language of business and get the respect you deserve.
—Diane Valenti has more than 20 years of experience in performance consulting. She is president of Applied Performance Solutions, Inc. Her clients include Genentech, Nike and Starbucks. She is also the author of “Training Budgets Step-by-Step” and has written a professional booklet for ASTD. Contact her at (415) 701-7600 or firstname.lastname@example.org.