Learning Impact Research: Can Serious Games Measure Up?
BY Kathy Heldman and John R. Mattox II
In today’s “big data,” metrics-driven business world, talent development professionals are challenged with demonstrating a tangible return on investment from their learning solutions. Traditional measures like participant feedback surveys and course completion rates are not enough anymore. Instead, C-Suite executives are keenly interested in how programs improve specific business outcomes like increasing sales, improving workforce performance, or maximizing employee engagement.
TALENT DEVELOPMENT THE SAVVIS WAY
Jim Sokolowski is the chief learning officer and director of Global Learning and Leadership Development for Savvis, an outsourcing provider of managed computing and network infrastructure for I.T. applications. He joined Savvis in 2010 to build a learning organization from the ground up, and during the past three years has done just that with an approach to learning that is strategic and laser-focused on impacting results. In the three-year transformation process, the Savvis learning team has been awarded the Training Magazine Top 125 Award in 2012 and 2013 and also the B.E.S.T. award from the American Society for Training and Development in 2012 and 2013.
“We have come a long way in our rapid from-the-ground-up build and transformation,” says Sokolowski. “I am very proud of what the team has accomplished, and I am honored to be part of this award-winning team. The question we continue to ask ourselves, ‘Now what?’ To keep our seat at the table requires that we maintain relevance by staying connected to the most critical strategic imperatives. We must also continue to improve the services we provide and the solutions we deliver.
” Interdependence is a key element of the Savvis way. At the beginning of each year, Sokolowski and his team meet with business-unit leaders to better understand their strategies, goals and talent development needs, with the ultimate outcome of being able to link all learning solutions and efforts to the most critical organizational strategic imperatives. In addition to the annual connection point, Sokolowski’s team also uses a strategic consulting process along with monthly enterprise learning dashboards and quarterly program level dashboards to regularly review learning solution progress and share the results with business-unit leaders. These strategies reinforce a continuous connection with the business, while also providing a mechanism to continuously improve the learning solutions the team is providing.
A SHARED CHALLENGE
In order to ensure profitable revenue growth, the Savvis sales organization is a key internal customer. Two key areas of focus for this internal customer are improving sales communication and negotiation skills with prospects and clients to accelerate bookings and improve customer satisfaction.
Savvis sales representatives engage in a complex selling cycle that often results in a highly customized solution, so it is imperative that contract terms and conditions are negotiated clearly and favorably early in the sales cycle.
Effective sales communication is essential throughout the process in order to accurately identify the needs and interests of clients and to gain their trust. These key competencies — part of the Savvis sales success profile — are linked to the sales employee lifecycle.
According to Sokolowski, “The importance of linking the organization’s talent development initiatives with its priority of growing revenue at or above market rates made it clear that a heightened focus on improving sales communication and negotiation skills is a critical success factor for Savvis.”
THE GAMIFICATION SOLUTION
Finding a transformational learning program that provides measureable results was essential. Sokolowski has always been intrigued by gamification, in part because he has observed the positive impact that computer games have on their players in terms of learning, engagement and having fun.
Acknowledging that this was a bit of a hypothesis, Sokolowski adds. “I thought this methodology could translate to impact work performance, particularly with the sales organization which is often an early adopter of technology.” The L&D team was excited by gamification, because it was an avenue to learning that fit the sales team’s results driven, competitive culture.
Savvis ultimately selected Merchants, an online sales communication and negotiation game-based program designed and developed by Madrid-based Gamelearn S.L., and licensed by Savvis from Game On! Learning, the U.S. distributor of the program.
The decision to implement Merchants was based on the following factors:
>> Skills application – The skills application focus of the training would help ensure that sales representatives mastered the key competencies. In serious learning games like Merchants, learners spend more than 90 percent of their time practicing and applying new skills.
>> Engagement and innovation – Savvis wanted something that was highly interactive. In fact, the organization has conditioned its workforce to expect engaging learning.
>> Cost – Savvis wanted to deploy a solution that would be cost effective and not break the bank.
>> Scalability – If the solution demonstrated positive impact, Savvis would want to quickly scale the approach to other areas of skill development.
“We have been fortunate to be able to foster and operate in an environment where we can innovate with learning while taking risks in trying new things,” says Sokolowski. “This is an environment we have created, and we accomplished this through the ongoing showcase of the value and impact of learning, while maintaining laser focus on flawless execution to deliver on commitments to the business.”
IMPLEMENTING THE SOLUTION
Savvis began the implementation with an initial cohort to fine tune the approach before rolling it out to a broader audience. The cohort consisted of about 30 sales representatives in two U.S. cities. Each works on the phone to prospect and develop sales opportunities that will result in sales.
The cohort was constructed to create a diverse demographic of tenure and experience. The manager of the sales team also participated in the training, as did Sokolowski. The players are extremely competitive in their selling activities, and the learning game has brought that same competitiveness to the training.
Prizes were awarded to the players with the greatest accumulated wealth and highest trust scores in the game. Players who accumulate more than $1 million in wealth were admitted to a “Millionaires Club.”
Measurement of learning transference and performance improvement is essential. The Savvis team identified the following success drivers for the training:
>> Increased confidence in selling skills
>> Shortened sales cycles
>> Improved win/loss ratios
>> Increased revenue per transaction
>> More favorable agreement terms between buyer and seller
>> Improved sales behaviors in the specific skill areas of (a) Identifying prospect or client needs and interests; (b) Structuring the sales conversation to build trust; (c) Reaching mutually beneficial agreements in each conversation; and (d) Managing each conversation to develop a long-term relationship.
MEASURING SUCCESS, DEMONSTRATING ROI
The initial implementation of Mer-chants began this past July, and so did the measurement research. Savvis and KnowledgeAdvisors designed an evaluation approach to determine whether knowledge and skills were acquired from the program and if they were being applied on the job to improve individual sales performance and improve revenue. In addition to the game-based learning cohort, Savvis launched a parallel cohort of salespeople completing negotiations training using traditional e-learning courses. A third cohort will complete a
classroom-based negotiation program during the fall of this year.
Explains John R. Mattox, director of Research for KnowledgeAdvisors: “It is impossible to implement a randomized experimental design to evaluate the impact of game-learning, but this comparison group design is the next-best approach.”
As this story went to press, the pre-assessments had been deployed, the train-ing completed, and post-training measurement had begun. The early returns are quite encouraging, with 80 percent of those starting the 8- to 10-hour program finishing it during the six weeks allotted. According to Sokolowski, “Getting 80 percent completion on a comprehensive
sales learning intervention is no small accomplishment, since training time takes away from their time selling. This high completion percentage indicates a high perceived value to the sales force, since this became a priority for the team.” The response to the training has also been quite enthusiastic, including comments from learners such as:
>> “Once I got through the first level, I found myself wanting to play more and more. It is addictive and very fun talking smack with my peers.”
>> “It is impossible to simply click through this game without stopping and making critical decisions based upon real negotiation techniques. Merchants presents the user with challenging scenarios and demands creative thinking for successful outcomes. But the best part is, it’s fun. This is learning gamification done right.”
Enthusiastic feedback, while important, is not enough. Sokolowski’s plan is to benchmark game-based learning against both traditional e-learning and classroom training that maps to the same competencies and learning outcomes.
The approach employs multiple assessments, evaluations and business data. Pre- and post-course knowledge and skill assessments are being used to quantify the knowledge and skills gained. All assessments and evaluations are deployed using Metrics That Matter, a proprietary learning analytics system from KnowledgeAdvisors. Savvis has used Metrics That Matter for the past two years to automate its standard training evaluation process.
Upon completion of the training, learners receive a web-based evaluation that asks if they have learned new skills and to what extent they expect to apply them. Learners
are asked to estimate their expected performance improvement and estimate how much of the improvement is due to training alone. KnowledgeAdvisors’ research has shown these predictive measures provide reliable “leading” indicators of the success of the program.
Follow-up evaluations will be sent to learners and their managers 60 days after completing the program. Questions in these evaluations will focus on how much performance has improved and how much of the improvement was due to training. These measures will be correlated with the predicted performance improvement measures and
will be compared to industry benchmarks maintained by KnowledgeAdvisors. Learners will also rate whether performance has improved for specific sales behaviors (e.g., number of client contacts, reaching mutually beneficial agreements, etc.).
Ninety days after the program, KnowledgeAdvisors and Savvis will conduct interviews with selected learners. Following the process outlined in Brinkerhoff ’s Success Case Method, half of the interviews will be with learners who provided the highest ratings on the evaluations and half will be with learners who provided the lowest ratings. The interviews will
focus on the strengths of each program and which aspects provide the most useful knowledge and skills. The interviews will also allow learners the opportunity to provide detailed examples of how they have used their knowledge and skills to improve their performance.
Figure 1 shows the evaluation approach with key performance measures across each group.
According to KnowledgeAdvisors’ Matox: “To demonstrate the impact of game-based learning programs on the business, it is essential to gather business data such as the number of sales, sales margins and other indicators of success.” The final data sources will therefore come from Savvis’ sales CRM system, which will provide results about sales cycles, win/loss ratios, revenue per transaction, and customer satisfaction. Information will be gathered for each learner for the 12 months prior to the program and six months after to control for seasonal fluctuations.
The road to success often requires innovation. The Savvis L&D group has embraced innovation — gamification of learning —with the intent of transferring knowledge and skills more efficiently and effectively to its sales force. This bold move was matched with an equally bold approach to evaluation. Only measurement can truly show how effective learning gamification is for Savvis compared to traditional forms of training delivery. Stay tuned. Savvis, Game On! Learning and KnowledgeAdvisors will report their evaluation results in 2014 with another case study.
—Kathy Heldman co-authored this article for Game On! Learning. John R. Mattox II co-authored it for KnowledgeAdvisors. To receive a copy of the completed research findings, send an email to email@example.com.