There remains much confusion about how to measure the return on investment (ROI) of e-learning. Because e-learning is simply another method of instructional delivery, the process of calculating ROI is
There remains much confusion about how to measure the return on investment (ROI) of e-learning. Because e-learning is simply another method of instructional delivery, the process of calculating ROI is the same as with any other delivery method.
ROI is calculated by comparing the net benefits of a program to the costs. Instructional delivery, be it face-to-face or electronic, is captured as a cost in the denominator. The outcomes resulting from the use of the content, whether delivered via in-person or electronically, make up the benefits. These benefits may be based on increase in profit, reduction of costs or avoidance of costs.
However, in far too many cases, e-learning investments are justified by simply comparing the cost of the face-to-face delivery to the cost of electronic delivery of learning content. While reducing the denominator (program costs) does contribute to increasing the ROI, if the benefits of the e-learning are not as expected, the ROI will be unacceptable to key stakeholders. So the key to ensuring an acceptable payoff for e-learning is to position the learning content so that it drives improvement more than the cost — enough to achieve an acceptable ROI.
Business alignment is the process of ensuring that investment drives relevant business results. Achieving business alignment requires that programs, projects, and initiatives be positioned for success by:
>>Clarifying stakeholder needs prior to investing in the program
>>Developing measurable objectives that represent the outcomes associated with stakeholder needs
>>Evaluating the success of e-learning at all levels, including the ROI
>>Using the chain of impact as the basis for data categorization, clear alignment between needs, program objectives, and evaluation exists.
An acceptable ROI for an e-learning initiative may be one that is higher than the ROI for the same program delivered in a face-to-face format. However, it is not always feasible to run two programs at the same time using different delivery methods, then conducting an ROI study on both to make the comparison. Even still, to know whether or not the ROI in the e-learning investment is a good return, there must be a target. If the ROI of the e-learning program falls below the target, the program was not successful in terms of financial contribution. If it meets the target, it was successful. If the program exceeds the hurdle rate, it was better than expected.
SETTING THE ROI TARGET
Techniques for setting sample targets:
>>At the same level as other investments e.g. 15%
>>Slightly above other investments e.g. 25%
>>At break-even, 0%
>>Based upon client expectations
By comparing ROI to an objective or an alternative solution, you can define whether or not the ROI is acceptable. But the ROI is not going to explain how much job performance has improved or how much learning occurred or even how effective the e-learning solution is in terms of relevant content, ease of use and appropriateness.
While the ROI is the ultimate measure of e-learning success, it will not help you explain how you got to that ROI; and it certainly won’t answer questions about how to improve the program in the future. For these reasons, additional measures of success are taken. The most logical and most useful measures are represented in the chain of impact.
CHAIN OF IMPACT
The real payoff of e-learning begins with how successfully participants apply what they learn back n the job (Level 3). Given that there are elements in the organizational universe that can get in the way of application, this level of measurement also requires that the capture of barriers and enablers to application. These data describe how well the organization system is supporting the transfer of learning. If the organization (e.g. supervisors, peers, processes or other functions) doesn’t support the transfer of learning, then application will not occur. If there is no application, there is no improvement in business measures (Level 4).
Improvement in business measures of output, quality, cost, time, customer satisfaction, employee satisfaction, work habits and innovation are the critical results that describe the impact e-learning has on the organization. Knowing that these measures can improve as a result of any number of causes, the chain of impact accounts for these other factors by isolating the effects of the e-learning program on the improvement in these measures.
However, an improvement at Level 4 is no guarantee there is a payoff for the elearning solution. It is only when we bring in additional data and take additional steps that the real payoff of e-learning is known.
Level 5, ROI, on the chain of impact is the ultimate measure of success. This metric describes just how well the benefits of a program compare to the costs. By converting business measures to money, a direct, mathematical comparison to costs can be made, which gives a more accurate accounting of how well the program contributed to the business than by reporting benefits as intangible only. Because data are collected at all levels of the chain of impact, the result is:
>>Evidence of program success important to all stakeholders
>>Explanations for how you arrived at the ROI
>>Information to help improve the program To ensure an acceptable ROI in e-learning, success at all levels in the chain of impact must occur. By aligning your elearning programs to the business, you can increase your chances of this occurring.
CLARIFY STAKEHOLDER NEEDS
Initial alignment occurs when stakeholder needs are identified, which begins with potential payoff for opportunities or problems. These payoff opportunities represent an organization’s opportunity to make money, save money and/or avoid costs. Some payoff opportunities are obvious, such as a $1 million cost due to unwanted employee turnover. Other payoff opportunities are not so obvious, like the desire to become a great place to work. The payoff opportunity represents the potential monetary impact that can be achieved by investing resources in various programs and projects, including e-learning. These are the Level 5 needs.
Once clear about the potential payoff opportunity, the next step is to identify the specific business measures that, if improved, will position a program to take advantage of the payoff opportunity. Business needs (Level 4) represent measures of output, quality, cost, and time as well as measures of customer satisfaction, employee satisfaction, work habits, and innovation.
Next, identify the performance needs. Performance needs (Level 3) are those behaviors or actions taken, or not, that if changed will address the business needs. A performance need may be a people issue, a process issue, or even a policy issue.
For example, a large pharmaceutical manufacturer was incurring excessive costs associated with pre-filled syringes (payoff need). Upon further investigation, managers found that the specific business measure that needed the most attention was false rejects of good syringes (business need). By meeting with inspectors, management found that they were inconsistent in their process. This resulted in some inspectors throwing out perfectly good syringes. This inconsistent process represents the performance need (Level 3).
Once the performance need is identified, the next step is to determine what people need to know (Level 2, learning needs) in order to change their performance. In the example, the learning need is that inspectors need to know the difference between acceptable and unacceptable syringes. In addition inspectors needed to recognize the importance of a consistent approach to inspecting syringes.
After learning needs are clarified, the next step is to determine preference needs (Level 1) by asking: How do we deliver the knowledge, skill or information people need to know in order to change performance so that business measures can improve and we can take advantage of the payoff opportunity?
The final needs that must be addressed are the input needs (Level 0). These needs represent the investment required for implementing the solution.
Program objectives correspond to stakeholder needs. Developing powerful program objectives at all levels represented in the chain of impact positions the investment for success.
Objectives describe to designers and developers the intent of an initiative, giving them direction as to what components to include. They provide facilitators, team leaders and program managers with direction as they assist participants in preparing to change behavior, apply knowledge and drive business outcomes. Objectives representing stakeholder needs communicate the what and the why of an e-learning program to participants. They keep impact objectives in constant focus, reminding participants of the ultimate reason for investing in the program. Developing objectives that reflect stakeholder needs communicates to stakeholders that the program owner “gets it” — that he or she has paid attention to what is needed to support organization success.
Objectives also set the stage for program evaluation, ensuring the right measures are taken and that results important to stakeholders are developed. They set the target to which results at all levels will be compared, including the ROI. Developing powerful objectives for your e-learning program ensures the right measures are addressed during implementation and that the right questions are asked during evaluation.
EVALUATE THE PROGRAM
This last phase in the alignment process is collection and analyis of data. All too often, when an e-learning program is up for evaluation, the project team will ask: What measures do we take? How do we measure performance? What questions do we ask on the follow-up survey? Answers to these questions lie in the objectives. Powerful program objectives give you guidance for all steps in the assessment, design, development, implementation and evaluation of an e-learning program. They are central to its success.
The outcome of aligning elearning programs with the business is a reporting of the chain of impact that occurs as people are involved in an e-learning program. They react to the content and the delivery and they acquire the requisite knowledge, skill or information. Most important, however, they apply what they learned and, as a consequence, key business measures improve.
This improvement is credible because a step is taken to isolate the effects of the program. Isolation techniques include control group arrangement, trend-line analysis, forecasting methods and expert estimations. This step makes the direct connection between business needs and the contribution of the program to those needs.
The chart below presents the ROI Institute’s Business Alignment Model. Alignment is important not only at the program level but also as you develop your overall e-learning strategy. By using the chain of impact as your guide for clarifying stakeholder needs, developing program objectives and evaluating the program, you can be confident that every effort was made to drive a positive ROI in the e-learning investment.
THE VALUE OF E-LEARNING
Electronic delivery of learning content can increase convenience and reduce costs, but it is the content and how the content is positioned that drives the benefits to the organization. An ROI in elearning is only as good as that to which is compared, and to achieve that target ROI, the program must be aligned with the business needs. Aligning e-learning with the business can ensure the right programs are positioned to drive improvement in the right measures at the right time.
Dr. Patti Phillips is president and CEO of the ROI Institute. She also serves as Professor of Practice for The University of Southern Mississippi’s Ph.D. in Human Capital Development program and as Principal Research Fellow at TheConference Board. Reach her via email at email@example.com. Dr. Jack Phillips is chairman of the ROI Institute and developer of the ROI Methodology. His work has been featured in the Wall Street Journal, BusinessWeek and Fortune magazine. Reach him via e-mail at firstname.lastname@example.org.