Measuring and evaluating learning is one of the most critical issues in the learning and development field.
The value of learning is inherent in today’s business environment where leaders understand that it takes a trained workforce to drive business results in an increasingly competitive, global environment.
As learning becomes more integrated with business strategy, human capital is seen as a key differentiator and driver of growth, transformation and productivity. In this knowledge-based economy, investments in employee learning and development continue to accelerate, with technology-based training solutions penetrating all areas of corporate education and training. At a cost over $3 billion a year, high-quality training courses, tutorials, and multimedia simulations that keep up with the pace of change and rapid technological advancement are intrinsically expensive and time-consuming to create.
While most executives intuitively realize learning is a basic necessity and that e-learning can provide greater flexibility for workplace learning to occur, there is increased frustration from the lack of evidence to show that learning programs of any modality really work. As executives and managers watch budgets grow without appropriate accountability, return-oninvestment (ROI) calculations and evidence of bottom-line value are now being routinely required from learning departments and e-learning functions. Sponsors need to see how major investments of time, money and resources are paying off and contributing to strategic business goals.
In today’s dynamic, cost-competitive, resource-constrained business climate, measuring and evaluating learning has earned a place as one of the most critical issues in the learning and development field and is largely driven by the needs of the clients and sponsors of learning projects, programs, initiatives and solutions.
THE E-LEARNING PERSPECTIVE
Despite heightened interest in measuring the value of learning, there seems to be much confusion about measuring the ROI of e-learning methods. From the practical perspective, e-learning is simply another method of instructional delivery and that the process of calculating ROI is the same, regardless of the delivery method.
However, in far too many cases, e-learning investments are justified by simply comparing the cost of the instructional-led approach to the e-learning approach. This can result in radically different cost amounts because of the savings generated with travel, lodging, classroom facilities and instructor cost. For such a comparative analysis to be accurate, the outcomes of the instructor-led learning must be the same as those generated with e-learning — and that is not always the case.
THE RECOMMENDED APPROACH
An accurate ROI calculation of an elearning program requires data collection at four levels: reaction, learning, application and impact. The impact data is isolated from other influences and converted to monetary value. This monetary value is then compared to the cost of the elearning program. The result is an incredible ROI study.
To fully compare the value of an e-learning program to the value of an instructor-led program requires two comparable studies with the same calculation process in which a string of benefits and costs is developed for both scenarios. If the ROI is higher for the e-learning delivery alternative, then it can be proven to be a good approach from the economic perspective.
AN ALTERNATIVE APPROACH
Due to the extra effort to capture four levels of data and conduct a comprehensive ROI study, it is not always possible to complete an instructor-led comparison. Consequently, the ROI based on cost savings alone may be appropriate. The key issue is determining program success with targeted learning outcomes.
For instance, if there is data from the instructor-led program that shows the extent of desired learning gains, then the same measures can be used to compare learning gains from the e-learning perspective. In other words, if the participants learn just as much or more with the e-learning delivery method as they do with an instructor-led option, then it may be proper to compare the two projects on cost savings alone — with maybe one more adjustment. At level three, when the evaluation follows what participants do back on the job with what they have learned, our studies have shown that participants in e-learning programs are less likely to follow through than in an instructor-led program. In an instructor-led process there is often a commitment made between the participant and the instructor that might increase the likelihood of participants applying what was learned and increase the likelihood that performance improvement and impact data is provided in a follow-up survey. An e-learning participant is generally disconnected from the instructor and can conveniently ignore the computer, which reduces application of learning. This factor will then tend to reduce the monetary value or business impact of an e-learning option, because the monetary benefits will not be the same as those from an instructor-led solution. In this case, the complete ROI study would be conducted using benefits and costs for the e-learning, along with the cost savings generated by the cost differential of presenting the program two different ways.
This should also include a footnote suggesting that the e-learning results might not be quite as impressive, because there will typically be less application. Essentially, the payoff is expressed as a difference in the cost two different ways, but with a qualifier statement.
SUMMARY AND NEXT STEPS
The complexities of this situation have caused some professionals to steer ROI calculations away from cost comparisons alone.
For example, in a recent article in Training magazine that focused on ROI in e-learning, readers were asked to submit ROI studies for inclusion in a future article. The request included a notation that studies must not be based on cost savings alone, but must also show the full ROI calculation. This was intended to underscore the importance of a complete, accurate analysis, as described here.
For designers and developers, there are some lessons on conducting credible ROI studies. Showing the cost-versus-benefit of an e-learning approach is no different than any other delivery method. Many e-learning programs are developed on the assumption that converting facilitator-led programs to e-learning platforms are more cost-effective and practical. To ensure that this is the case, the ROI of the facilitatorled program should be compared with the ROI of the e-learning version to determine its true cost savings and utility.
Essentially, running the two programs parallel may be the most preferred way to gather this data. The good news is that cost reductions are so significant that even if the instructor-led and e-learning benefits are not always on the same level, the e-learning approach may still result in a much more positive ROI. In fact, many organizations have shown that investments in learning technologies are paying off with clear cost savings, efficiency gains, increased content reuse, and decreased costs of learning development and delivery. The key is to ensure that the e-learning project or solution is connected to one or more business measures that will reflect its overall success.
In reality, most learning or e-learning projects are undertaken to deliver business value. It is important to remember that positioning an e-learning project to add value requires a more broad-based view than a simple ROI calculation.
Using the ROI process to collect results data at multiple levels of impact and show how results are linked to targeted business measures is an effective way to demonstrate the worth of e-learning projects and add credibility to the e-learning function.
Since e-learning is an investment that impacts multiple areas of an organization, a strategic “line-of-sight” on the organizational use of e-learning is critical. Key questions include:
>> How does your e-learning function provide strategic value?
>> How does your strategic activity translate into business results?
>> Do your e-learning programs make you more responsive to your customers?
>> Is your organization able to innovate more effectively as a result of e-learning investments? >> Do those who need information have greater access to it?
>> Do e-learning services help your organization achieve its core capacities more effectively? More evaluation work is needed in the e-learning arena, so give us your input, conduct your own impact studies, share lessons learned and best practices … and stay tuned.
—Jack Phillips, Ph.D., is chairman of the ROI Institute and a world-renowned expert on accountability, measurement and evaluation. Using more than 20 years of professional experience, he has developed a sound, tested methodology for calculating true financial ROI for training, leadership, marketing, e-learning and performance improvement initiatives. Visit the ROI Institute at www.roiinstitute.net.
—Holly Burkett, MA, SPHR, CPT, Principal of Evaluation Works, is a certified ROI professional with more than 18 years experience in performance improvement and measurement. Editor-in-chief of ISPI’s Performance Improvement Journal, she recently co-authored “The ROI Fieldbook” with Jack and Patti Phillips and Ron Stone (2006). Reach her via e-mail at firstname.lastname@example.org
Making the Business Case for E-learning
You’ve heard the traditional benefits of how organizations are harnessing the power of e-learning to scale to global audiences, reduce travel costs and deliver just-in-time training. But as e-learning itself has evolved, so have the ways in which it is used. From driving organizational change to addressing the needs of the multigenerational workforce, here are some additional ways that corporate America is making the case for e-learning as a valuable tool.
A December 2006 survey by BlessingWhite focusing on career development reported that 41 percent of respondents felt their employer’s approach to career development failed to meet their personal needs. What employees really want to know is where to go, what to take, and how to squeeze it in without taking too much time away from the office.
As a result, job-role-specific learning roadmaps are now a hot topic in talent management. What you’ll find within many of these corporate roadmaps is e-learning. Including custom e-courses developed internally and off-the-shelf products, e-learning is being used to conduct employee orientations, develop soft skills and provide the foundation for technical certification programs. From a management perspective, e-learning demonstrates a commitment to employee growth, is a critical component in an overall human capital strategy, and is a solid retention tool to keep employees loyal in an economy where the Society for Human Resource Management estimates turnover costs at $5 trillion annually.
In many cases, the development of new training offerings is the result of a change impacting an organization or its employees. Whether it’s training designed to increase awareness about a new initiative, a new technology or a new process, ensuring that the company’s target audience receives the information needed to support the change is critical.
The goal of any change management initiative is adoption. Many organizations are using e-learning as a change management vehicle to minimize resistance and ensure employees have the information needed to rapidly move up the adoption curve.
According to a 2005 National Business Ethics survey, the most common types of unethical employee misconduct range from abuse or intimidating behavior to discrimination and sexual harassment. The Conference Board/Ethics and Compliance Officers (ECOA) survey shows company anti-corruption practices becoming significantly widespread. So many corporations are under pressure to demonstrate compliance with legal statutes and minimize risk.
E-learning can be used to set expectations from the beginning as a means to communicate the parameters, requirements and consequences of not following company policy. Whether it’s a Sarbanes-Oxley course or an interactive simulation on sexual harassment, e-learning has become an effective means for many corporations to demonstrate compliance through mandated training.
With 75 million “Millennials” entering the U.S. workforce in the years to come, the pressure to understand, retain and cater to their needs is immense. Millennials — with an extreme amount of technical savvy and comfort with Web 2.0 tools — have shorter attention spans than their generational peers and prefer bite-sized education they can access whenever they want it.
Whether e-courses are delivered on a PC or an iPod, the modular nature of e-learning not only helps to fulfill the needs of these knowledge-hungry workers with an appetite for technology, it also helps their “Gen-X” and “Baby-Boomer” peers as well. Whether taken in an office or commuting on a bullet train, elearning can be an educational compromise to meet the conflicting needs of all members of the multigenerational workforce.
With the expansion of e-learning into so many facets of their businesses, it’s clearly evident why more C-level executives and higher are leveraging e-learning to enable their strategic initiatives. —Tom Floyd is the CEO of management consulting firm IEC (www.ieconsulting.biz) in San Jose, Calif. Reach him at email@example.com.