Solving the 2 Sigma Problem

Solving the 2 Sigma Problem

One of the “holy grails” of education has been the quest to solve the “2 Sigma Problem.” This was a problem originally posed in 1984 by Benjamin enjamin S. Bloom, the famous educational educational psychologist, sychologist, college professor rofessor and researcher. esearcher. In a series of dissertations issertations and subsequent ubsequent studies performed performed by Bloom and his students, students, he observed bserved drastically rastically different different student achievement achievement scenarios, cenarios, depending depending on the type of instructional nstructional methods used.

Between 1982 and 1984, a series of studies were performed by Bloom and some of his students. In 1984, Bloom used those study results to publish a paper that described student achievement results obtained from the following three forms of learning: conventional ventional classroom; lassroom; mastery learning; earning; and tutoring. Below is a brief summary of these three learning methodologies.

Most of us are familiar with conventional classroom training. This is where 20 to 30 students are taught by a teacher or instructor. Tests are given periodically to grade each student’s performance, and the pace of the class moves at a pre-determined rate established by the teacher, instructor, or educational educational institution. nstitution. This form of teaching was established stablished as the baseline aseline control group for his study.

Mastery learning is a bit different than typical classroom lassroom training. training. This method of instruction instruction was first defined by Bloom in 1968, and it was used in class sizes of 20 to 30 students. tudents. It specified that the students tudents as a group must achieve achieve a level of mastery of 90-percent 0-percent on a knowledge nowledge test, before the class moved on to subsequent ubsequent topics.

Tutoring is a one-to-one teaching method where a good tutor is assigned ssigned to each student (or in some cases, the tutor is assigned ssigned to work with a small group of two or three students, tudents, who are then taught simultaneously). simultaneously). These students tudents were individually individually taught, and measured easured with formative formative tests and feedback-corrective eedback-corrective procedures procedures similar similar to the tests and prescriptive scriptive solutions solutions used in mastery learning classes. lasses. It should be noted however, owever, that the need for corrective orrective work was minimal minimal using this teaching eaching methodology, methodology, according according to Bloom.

When compared with the control group — the conventional classroom — the results were very significant. The average student who was taught using the tutoring method performed higher than 98 percent of the students who were trained via the conventional classroom. That was two standard deviations, or sigmas, higher. Thus the naming of the “2 Sigma Problem” came into being.

But it is also noteworthy that even the average student taught using the mastery learning methodology performed one standard deviation above those students taught in the conventional classroom environment. That was still 84-percent above the students in the conventional classroom.

So armed with the knowledge of the absolute best way to teach, we’re now turning to EdTech in our search for solutions to the 2 Sigma Problem. In turn, that search is fueling a lot of investment speculation but not with the inside knowledge that Bloom’s study provides.

Many educational outliers have already started to solve the problem with technology — many with only their gut instinct leading the way. When one thinks about mastery mastery learning earning and the use of educational educational technology, he or she should think of Sal Khan and the Khan Academy. Khan was the evangelist who pointed to the shortcomings of the “keep-on-going” classroom model. Sal knew intuitively that students learn at different rates, so why should every student be expected to keep up with on predefined learning pace?

Khan Academy’s video learning snippets help to solve that problem. If a student doesn’t understand a lesson, he or she can continue to retake that lesson, until he or she finally masters that particular topic. This teaching methodology continues to be a very successful approach for millions of Khan Academy students, and it approximates the mastery learning teaching methodology. Perhaps the only missing piece is to provide provide each student student with a forum where additional clarity on a specific topic can be found. If the same video is watched four or five times, chances are that a fifth review is not going to help the student understand the concept being covered. That’s why tutoring has an advantage over both the conventional ventional classroom lassroom and mastery mastery learning earning teaching methods.

Many people have come forward to tell how their lives were changed by Khan Academy’s approach to learning. In turn, how their lives were changed by Khan Academy’s Academy’s approach pproach to learning. learning. In turn, this is a great testimonial estimonial for how we can use simple, existing existing technology echnology to help solve at least the first leg of this battle, the mastery learning earning teaching eaching methodol- ethodology. Kahn’s work has helped to throw the whole e-learning -learning industry ndustry into high gear. Kahn understood understood the mastery learning earning pedagogy, pedagogy, intuitively, ntuitively, and he used YouTube technology echnology and simple e-learning -learning teaching teaching and illustration llustration tools to enable his solution. olution.

This same level of mastery would be very difficult i in a conventional onventional classroom, lassroom, without without the use of technology. echnology. You would have to focus your pace on the slowest slowest learner, learner, by topic, to avoid leaving learners earners behind, behind, which isn’t fair to fast learners. earners. So the solution olution has always defaulted efaulted to the “keep-on-going” “keep-on-going” conventional onventional classroom lassroom pace that’s too fast for one-third ne-third of the class, too slow for another third, and about right for the remaining emaining third.

These classroom shortcomings have caught the attention ttention of many political olitical leaders and that added attention ttention and discussion is also fueling the EdTech investment nvestment space, as the industry ndustry seeks to find better answers answers to improve teaching eaching and learning. earning.

Better solutions can be created if we think through our solutions olutions a bit more carefully, carefully, armed with work like Bloom performed. formed. Just because we can do something something with technology, echnology, doesn’t mean that it’s going to work immediately. mmediately.

Consider how a live MOOC (Massive Open Online Course) Course) operates. perates. Isn’t that simply the classroom lassroom model on technological logical steroids? teroids? One-to-thousands ne-to-thousands seems to completely ompletely sidestep idestep the need for the one-to-one one-to-one tutoring utoring that really helps students thrive. Could this lack of individual- ndividualized attention attention be one of the elements lements that added to the high dropout rate? That’s likely the case.

MOOC providers and proponents are figuring iguring it out. Micro-credentialing is probably the biggest incentive for a student to continue in a MOOC course. That credential is useful for résumés and lays the groundwork in case a series of MOOC courses can be used toward an eventual degree. If the student signs up to learn a particular skill that propels him or her into a new career, like the initial app development courses offered in MOOCs, then the need for credentialing is not as strong. But what we have observed is that the industry is adding hundreds of courses, just to show that they have a full and robust set of offerings. That is where the micro-credential can play a role. Otherwise, a student is likely to drop a course if he or she doesn’t doesn’t feel it’s germane to their personal needs. If they at least can point to a certificate, then that might be worth keeping up with it.

Another solution solution is the notion of massive massive office hours. Although this can be a very effective approach, there are other options. Udemy, uses a large base of instructors to teach an equally large range of courses. These courses run the gamut from photography to web development, languages, and even soft skills like management best practices. There are practically no restrictions on topic matter, so Udemy is addressing the general marketplace for consumers and professionals.

Udemy puts a lot of the responsibility on the instructors to provide that one-toone support when a student gets stuck. It utilizes a text-based question and answer capability throughout each course that directs questions to the instructor for that particular course, which can put a lot of pressure on instructors. But the instructors actually do take on that added role, to avoid receiving a bad course rating from a student. Several bad ratings will ultimately affect their sales for any given course so there’s an inherent reward for doing the extra one-on-one mentoring. That coupled with the fact that the course is already developed and delivered electronically, the instructor’s time is actually freed up to provide that additional tutoring. As more questions are asked and answered, they become a database of knowledge that the rest of the students can access. The more savvy instructors use those questions to revise their courses going forward, thus preventing venting the same questions uestions from being asked. It can take students 24 to 72 hours to receive an answer, and it may not be one that clicks immediately.

The notion starts to change the instructor’s role: “If I don’t have to teach every day, then I guess my role is going to move more toward the tutoring and mentoring side of the equation.” But as instructors, we like that scenario — right? Consider the flipped classroom where students listen to recorded video lectures at night and then spend the class time implementing and discussing what they learned. That has proven to be an excellent teaching methodology, more fun for everyone, and a good use of video technology technology (read EdTech) EdTech) to help address address the one-on-one tutoring benefits.

It looks like Udemy and Udacity are both trying to improve on the one-to-many model by using partners and formal call-in centers that are continuously staffed. Udemy announced in December 2015 that it was creating a partnership with Codementor to provide one-on-one live tutoring for its coding classes. Udacity had already introduced a call center team to provide coaching for their technical offerings back in 2013 and even relied on text chat, video calls, and even phone calls to solve the oneto-one tutoring problem.

The downside of the call center approach to tutoring will be whether the people providing that coaching understand the course’s content, as well as the instructor’s method of teaching. That becomes part of the requirement to provide this type of tutoring solution.

Tools are now being developed for alternative learning reinforcement, and other areas that technology can enable. Take reinforcement provided by the teacher which can be challenging when faced with a large class size and the burden falls onto a single instructor.

An example of an alternative learning reinforcement tool is a product, Trivie, which is based on the game Trivial Pursuit In the corporate learning world, Trivie uses technology to send out questions to students both during and after a training scenario. As the student gets each answer right, he or she can be rewarded with points, positive comments or other incentives  that can be stipulated in the app. And of course, the notion of points has a special meaning because of the world of gamification. The accumulation of points can also be made to have an impact on another list item — peer group influence.

This kind of game and learning reinforcement technology  can use leaderboards, prescriptive paths when wrong answers are given, or predefined coaching prods if the student doesn’t get the right answer. Some implementations even employ an electronic “great job!” to reinforce a student’s learning. But most importantly, reinforcement provides a 1.2 sigma impact on student achievement.

Pepper is a novel tool that begins to address the feedback-corrective variable is a novel tool under development that was designed to help train sales people to overcome the top three objections to his or her company’s products. If a sales person is able to overcome these objections, there is a lot of data that says it will translate into significantly larger sales pipelines and order volumes.

The way this tool works is by randomly sending out a pre-recorded telephone call to each sales team member during their normal work day. Each message poses a recorded objection that a customer might verbalize, verbalize, and the sales representative is required to respond with an explanation that mitigates that objection. That verbal response is stored in a database, and can then be passed to that sales representative’s manager or to a peer for review. At that point, the sales manager or peer can suggest corrective tutoring to help that sales person overcome that objection better. That same type of technology can easily become a tool for determining a student’s grasp of a particular subject. Now that mobile phones are so prevalent, it would be possible to send out a verbal microtest question, in order to see how the student might answer it when put on the spot. If the student gets it wrong, the teacher or professor could use that data to provide the necessary corrective measure to help that student. This same technology might also provide a way to measure critical thinking on the part of a student. Professors often find it difficult to encourage and test critical thinking amongst their students. Using this tool to prompt a judgment may provide a way to test whether an objective analysis and evaluation of an issue is being made by a student.

The two best solutions on the horizon for solving the 2 Sigma Problem are AI (artificial intelligence) and the Internet of Things (IoT).

Back in 2010, a social robot named Bina48 was a keynote at the Enterprise Learning Conference and she was the most advanced social robot in existence at that point. Questions could be posed verbally, to her, and her AI framework allowed her to interact with the person in a conversation directed to her, and her AI framework allowed her to interact with the person in a conversational mode. She represented the precursor of the personal tutor — especially when Bina48’s caretaker mentioned that she would one day sell for less than an iPhone at a store like Sharper Image. That price point could potentially mean a personal tutor in every home.

Social robotics is not the only avenue for AI. It’s being used successfully in devices as ubiquitous as our mobile phones, as well as in products like Alexa that Amazon successfully brought to market. Using either of these devices, we can verbally ask questions that we don’t know the answers to. All of this tutoring-like technology is thanks to AI algorithms. And considering its rapid proliferation by major software and hardware producers, it appears that in the next decade we’re going to be able to solve the elusive 2 Sigma Problem using AI. But if a Bina48-lookalike is not by our side tutoring  us or answering our every informational need, maybe it’ll be a wearable device that will provide us with coaching, tutoring and mentoring that can help us with our form and tell the wearer when he or she is at the gym and isn’t in the correct position to lift as much weight as they would like. These latter devices are now in early stages of funding and development in the form of wearable vests, and the investment backing is coming from the insurance industry, which certainly has a vested interest in helping people avoid injuries. Another possibility is that tutoring might just be integrated into our work environment or in our work equipment, much like navigation is now integrated into our cars.

To ultimately solve the entire 2 Sigma Problem, we’re going to need to consider another teaching conundrum. Part of any teacher’s role is to present new information to students, even when a student didn’t ask for it. The notion divides knowledge into three categories:

1. Things we know (known knowns);

2. Things we don’t know (known unknowns); and

3. Things we don’t know, we don’t know (unknown unknowns).

The concept of “known unknowns” and “unknown unknowns” has largely been attributed to NASA in its work. The known unknowns are generally categorized into risks that can be measured for compliance, such as the operation of various various sensors in a space craft. It’s the unknown unknowns that are a result of unexpected or unforeseeable conditions, such as a properly installed heat panel failing during re-entry.

We encounter similar problems when we learn more about a topic. There are things we know, and things we know that we don’t know. But how can we be tutored by a technology, unless that technology can introduce the things that we don’t know we don’t know?

That will be the final obstacle to overcome when we use AI technology to create an EdTech tutoring device. Just as a human tutor would be able to suggest that a student should consider an additional topic he or she might not be aware of, we need to build that “suggestive capability” into our AI tutors.

Additional Reading: Benjamin S. Bloom, “The 2 Sigma Problem: The Search for Methods ofGroup Instruction asEffective asOne-to-One Tutoring,” EducationalResearcher,American EducationalResearchAssociation, Vol. 13, No. 6. (Jun. – Jul., 1984), pp. 4-16. URL:

Leave a reply