The Defense Acquisition University (DAU) is the primary training organization for the Defense Acquisition Workforce. It is committed to providing the training — both formal and informal — to improve the professionalism of the more than 160,000 members of the acquisition workforce by engaging them both in the class- room and on the job. DAU plays a vital role for our nation by ultimately developing fully qualified acquisition professionals who deliver cost-effective systems, equipment, and services to meet warfighter requirements.
It’s an exciting time in both of Defense Acquisition University’s worlds: learning and acquisition. We couldn’t be more optimistic about how we will thrive in this environment. People all over the learning world are busy applying new technologies to engage a workforce that learns differently. DAU is no different. Our innate desire to improve drives us to use these technologies in innovative and powerful ways. We are doing this by increasing our focus, efforts and resource investments on our clients’ business results and making that our measure of success.
STRATEGIC PLANNING AT DAU
This year, we developed a completely revised DAU strategic plan that ensures alignment with business strategy and our vision and mission. There will always be more changes, new technology, and new possibilities. We are well positioned to identify these, adapt, and remain a world-class learning leader. This is reflected in our New Strategic Plan, which includes our comprehensive learning strategy — Acquisition Learning Model (ALM) — and in our goal of improving acquisition outcomes via business results.
There are three domains of our new ALM: foundational, workflow and performance learning. One of our strategy’s primary goals is to ensure cross-domain integration and repurpose learning assets among domains. For example, “train like you work, work like you train.” By doing so, we have focused all activities primarily on job and organizational success.
The ALM “links” our training (both in classroom and online); acquisition resources and job support tools; mission assistance and customized workshops. This allows us to be current, connected, and innovative in every area. Our students need current information the latest policies, guidelines, and lessons learned relevant to their increasingly diverse and fast-paced work. Likewise, DAU must have current and direct knowledge of what’s going on in the acquisition enterprise so that we can anticipate the requirements of our students, bring them the assets they need, and teach in a way that is relevant to their work and their learning style.
DAU must also be tightly connected to its customers and aligned with their priorities and challenges. This helps make our work meaningful and is also a key function of a corporate university. Our students need to be connected quickly and easily to the information required for their success on the job. In addition, the domains of learning in our ALM are most powerful when they are connected to each other integrated and reinforcing.
Our customers also need innovative approaches to problem solving and the ability to think critically. Likewise, we must be innovative in our methods for transferring knowledge to the workforce whether in the classroom, online or through mission assistance.
To implement our learning strategy, we incorporated its three domains into our performance-based strategic plan. Our strategic planning process is DAU’s engine for change and transformation this is the perfect vehicle for implementing the new learning strategy. This required the creation of three new strategic goals that incorporate the ALM (foundational learning, workflow learning, performance learning) as well as two additional supporting goals of people and infrastructure. All are focused on business results (acquisition outcomes).
The entire strategic planning process is a deliberate, planned, measured, iterative and integrated cycle that continuously moves DAU toward its organizational goals and vision for the workforce. The ALM’s primary purpose “improve acquisition outcomes” is a now a pivotal part of DAU’s vision statement, ensuring all supporting objectives and tasks in our strategic plan drive to that goal.
As implemented in the new Strategic Plan, the ALM extends the concept of learning beyond the classroom. DAU is now delivering more in-context consumption learning on the job anytime, anyplace. All learning assets (e.g., courses, how-to videos, self-service portals and job-support tools) are integrated and shared among the three domains. Moreover, the University is achieving these results in a cost-effective manner while maintaining high standards of quality. By implementing the ALM, the university aligns with senior leadership, and continuously modernizes its business and learning infrastructure. The world-class learning architecture supports continuously updated curricula, talent development and rewards. The scale and scope of the ALM:
Foundational Learning – DAU offers more than 400 technical training courses supporting the 14 Defense Acquisition Workforce Improvement Act (DAWIA) career fields, which require that Defense Acquisition workforce members be certified for their positions. Workforce members can fulfill their training requirements through DAU’s core certification and core plus training courses.
Workflow Learning – Access to acquisition knowledge outside traditional learning environments improves efficiency, innovation and effectiveness, enhancing job performance. It also augments the foundational learning that occurred in the classroom and gives individuals quick, easy access to information, connects them to other acquisition professionals, and provides in-context consumption learning opportunities.
Performance Learning – Extends help beyond the classroom into the workplace with mission assistance services. This program places seasoned faculty onsite at organizations ranging from smaller acquisition teams to larger acquisition programs to provide advice, consulting, rapid-deployment training on new initiatives, and training targeted to address unique mission needs. An organization’s complex problems often require face-to-face and high-impact support.
MEASURING LEARNING IMPACT
DAU’s learning measurement strategy is to provide the key performance metrics that are credible to our senior leadership team. Our executive team values DAU as an enterprise when determining impact and performance by both growth and multi-year metric trends aligned to business goals within our strategic and annual performance plans.
To assess the effectiveness of our learning strategy and solutions, DAU measures success by the value-added contributions across the three domains of our learning strategy. This total enterprise view of our contributions determines if the learning needs of the Defense Acquisition workforce are being met.
DAU’s increases in capacity and throughput did not come at the expense of learner satisfaction. The university’s customers consistently give top ratings to DAU’s learning assets and faculty who deliver them. DAU uses the four-level Kirkpatrick training assessment model to evaluate student perceptions, learning outcomes, job performance, and customer impact. DAU consistently receives high marks (80 percent and above) in student surveys from more than 1,756 classroom course offerings per year, delivered at DAU’s five regional campuses and more than a dozen satellite locations. Of the surveys completed by university students, DAU’s classroom courses received an average rating of 6.3 (90 percent) on the seven-point Likert Scale. This exceeded DAU’s target of 80 percent by 10 percent and is 5 percent above the Metrics that Matter corporate benchmark of 85 percent.
DAU also has a robust benchmarking program where we seek out our peer organizations with transferable best practices to adopt and adapt. We average four visits per year and, in return, we share our best practices with others. In the last ten years, more than 80 organizations have benchmarked DAU. Benchmarking is a powerful tool to gain and maintain leadership in our field.
DAU continues to implement Kirkpatrick’s Four Levels of Evaluation, compiling a multi-year database of millions of surveys. DAU has contributed several chapters and case studies to several of the Kirkpatricks’ recent books. However, we knew we must continue to build on this success and raise the bar to take the next big step: to measure the impact of our learning strategy on business outcomes.
IMPACT METRICS AT DAU
The most urgent challenge facing DAU was to help improve business results especially with a new, inexperienced and challenged workforce. We had to better understand learning and development’s relationship, impact, and contribution. To accomplish this, we invested in more in-depth learning analytic capabilities to focus our attention on business results as our key measure of success. What we needed was a more comprehensive learning analytics capability that directly targeted impact metrics: job performance and business outcomes.
Learning analytics must be incorporated into the context and be in direct support of our overarching learning strategy that focuses on the right impact measures. With the ALM, DAU’s primary measure of success (or KPI) for all our courses, Web assets and performance support became the impact metrics. DAU ‘s approach to learning analytics complements this by focusing on measuring and analyzing level III/IV evaluation feedback preferably when students are back on the job.
IMPACT METRICS STRATEGIC OUTCOMES
Foundational learning: Although students respond to impact metrics questions on the end of course surveys, we have found the Level III questions on the 60-day follow-up evaluations are more insightful. At this point, learners (and in special cases their supervisors) indicate whether training contributed to improved job performance.
We also found some interesting dynamics in survey scores on both instructor-led (ILT or classroom) and distance learning (DL) courses regarding impact metrics.
ILT courses score higher than DL. Both drop significantly end of course (t=0) and when back on the job 60 days later (t=60). The difference is less significant between ILT and DL when back on the job.
This has contributed to resource discussions and decisions by the leadership team on which delivery is more appropriate for each course. When analyzing hundreds of thousands of surveys for “impact metrics,” we believe that the 60-day follow-up when back on the job is the more accurate gauge as a KPI for DAU’s contribution to the workforce.
In looking at our surveys regarding the utility of courses to job relevance 60 days later, we find some courses do well and some not as well. We’ve added more “use on the job” survey questions to more than 400 courses. Additionally, we analyze a 60-day follow-up for “utility to the job” questions.
The good news is more than 89 percent are or will use DAU course content on the job. However, another view of the same data would be that 11 percent never use the content (compared to industry standard of 40 percent). This level of analyses has changed the conversation during curricula and new course development reviews.
Text mining helps complete the foundational learning picture. Text mining is an analytic tool we use for interpreting the “meaning” or “semantic space” described by the words extracted from the documents analyzed, to create a mapping of words and documents into a common space, computed from word frequencies or transformed word frequencies, identifying the latent semantic space that organizes the words and documents in the analysis. In some way, once such dimensions can be identified, you have extracted the underlying “meaning” of what is contained (discussed, described) in the thousands of comments in our survey base by course.
We use Statistica’s Text Mining Module for these analyses. DAU now analyzes the root cause(s) of low performing courses through “text mining” of open-ended student comments that accompany the survey scores and takes appropriate action to improve their results through periodic curriculum updates and new course development. In another text mining example, our business and finance courses show the most frequent themes derived from thousands of comments by course include, “Not applicable, good general info, more practical, good course, useful on the job, improved job performance, etc.” The text mining helps with the root cause analyses and supplements Likert and percent scores to ascertain the cause of low-performing courses.
Deep Diver Learning Analytics: The learning analytic “deep-dive” capability has proven invaluable during curricula reviews and prioritizing course update funding. DAU keeps more than 400 existing technical courses current and relevant, while developing 40 new courses per year. New course designs have now improved job impact and business results scores to over 6.75 for business impact.
Our strategic analysis and review have changed the level and scope of discussions during senior leader meetings and curricula reviews. This impact has changed policy level decisions that drive requirements on who must attend which course based on data, not irrational needs.
Measuring Workflow Learning: To measure impact metrics in the informal space, we use all the Google Analytics’ capabilities available to us. We measure our workforce uses, sources, technologies used, dwell times, time spent, access, etc. For example, of 170,000 users, 84 percent found what they wanted in one click; 3 clicks rose it to 97 percent. Other usage data is also tracked here.
Feedback from our workforce on the quality of Web-based assets is tracked through informal survey questions in MTM, supplementing the Google Analytics utilities.
Measuring Performance Learning: High job impact is measured with quantitative interviews, supplementing our qualitative analyses through carefully structured interviews to gain personal feedback on the impact of our Mission Assistance (MA) and consulting efforts.
We primarily conduct Level III/IV executive interviews to assess DAU’s total impact of job performance and business; more than 100 were conducted with senior leadership in FY16. This yielded strategic qualitative feedback from senior business leaders.
ENTERPRISE-WIDE LEARNING ASSETS
We are putting more emphasis on learning assets and less on courses in our overall L&D approach. This has changed who, where and how we develop, deliver and deploy our assets. We moved from curricula and asset development upstream in our process to drive more impact. This minimizes bias toward a course-only solution. We can also leverage technologies that best suit the material and students’ needs. We are also better able to translate learning objects into asset building blocks to use discretely and re- purposing across the ALM.
The future is uncertain. Many successful organizations have failed to keep reinventing themselves and fall trap to the “S” curve into extinction. In this environment, that is a death sentence. Technology is changing the business. Good for customers; good for innovators; tough on incumbents! Beware of “the way we’ve always done it.”
The heart and future for DAU will remain an evolving strategy. Through it, we will touch every professional in the Defense Acquisition workforce at every stage of their career and help them improve acquisition outcomes. Finally, in everything we do, we must always continue to stay current, connected and innovative. At the same time, we must to be focused on executing our critical mission and achieving our vision: helping the workforce succeed and improve acquisition outcomes.
Only in this way, will DAU can prepare the Defense Acquisition workforce to provide America’s warriors the best in weapons and equipment in defense of our nation, now and in the future.
—Dr. Christopher Hardy is Global Strategic Director of Defense Acquisition University. DAU is a seven-time Learning! 100 Award winner.