Online Learning: An Evaluation

Blended Learning is More Effective Than All On-Line or All Face to Face Instruction

Online learning—for students and for teachers— is one of the fastest growing trends in educational uses of technology. It overlaps with the broader category of distance learning, which encompasses earlier technologies such as correspondence courses, educational television and videoconferencing.

Studies of distance learning conclude that these technologies are not significantly different from regular classroom learning in terms of effectiveness. Policy-makers reason that if online instruction is no worse than traditional instruction in terms of student outcomes, then online education initiatives could be justified on the basis of need or cost efficiency to provide access to learners in settings where face-to-face instruction is not feasible.

The question of the relative efficacy of online and face-to-face instruction needs to be revisited, however, in light of today’s online learning applications, which can take advantage of a wide range of Web resources including multimedia,Web-based applications and new collaboration technologies. These forms of online learning are a far cry from the televised broadcasts and video conferencing that characterized earlier generations of distance education.

Moreover, interest in hybrid approaches that blend in-class and online activities is increasing. Policy-makers and practitioners want to know about the effectiveness of Internet-based, interactive online learning approaches and need information about the conditions under which online learning is effective.

Our office conducted a systematic search for empirical studies of the effectiveness of online learning and a meta analysis of those studies from which effect sizes that contrasted online and face-to-face instruction could be extracted or estimated. This analysis and review distinguish between instruction that is offered entirely on line and instruction that combines online and face-to-face elements.

The first of the alternatives to classroom based instruction, entirely online instruction, is attractive on the basis of cost and convenience—as long as it is as effective as classroom instruction. The second alternative, which the online learning field generally refers to as “blended” or “hybrid” learning, needs to be more effective than conventional face-to-face instruction to justify the additional time and costs it entails.

STUDY EFFECTS

A meta-analysis of 51 study effects — 44 of which were drawn from research with older learners — found that:

>> Students who took all or part of their class on line performed better, on average, than those taking the same course through traditional face-to-face instruction.

Learning outcomes for students who engaged in online learning actually exceeded those of students receiving face-to-face instruction. Interpretations of this result, however, should take into consideration the fact that online and face-to-face conditions generally differed on multiple dimensions, including the amount of time that learners spent on task. The advantages observed for online learning conditions, therefore,may be the product of aspects of those treatment conditions other than the instructional delivery medium per se.

>> Instruction combining online and face-to-face elements had a larger advantage relative to purely face-to-face instruction than did purely online instruction.

However, many studies did not attempt to equate (a) all the curriculum materials, (b) aspects of pedagogy and (c) learning time in the treatment and control conditions. Indeed, some authors asserted that it would be impossible to have done so. Hence, the observed advantage for online learning in general, and blended learning conditions in particular, is not necessarily rooted in the media used and may reflect differences in content, pedagogy and learning time.

>> Studies in which learners in the online condition spent more time on task than students in the face-to-face condition found a greater benefit for online learning.

>> Most of the variations in the way in which different studies implemented online learning did not affect student learning outcomes significantly.

Analysts examined 13 online learning practices as potential sources of variation in the effectiveness of online learning compared with face-to-face instruction. Of
those variables, (a) the use of a blended rather than a purely online approach and (b) the expansion of time on task for online learners were the only statistically significant influences on effectiveness. The other 11 online learning practice variables that were analyzed did not affect student learning significantly. However, the relatively small number of studies contrasting learning outcomes for online and face-to-face instruction that included information about any specific aspect of implementation impeded efforts to identify online instructional practices that affect learning outcomes.

>> The effectiveness of online learning approaches appears quite broad across different content and learner types.

Online learning appeared to be an effective option for both undergraduates and for graduate students and professionals in a wide range of academic and professional studies. Though positive, the mean effect size is not significant for the seven contrasts involving K–12 students, but the number of K–12 studies is too small to warrant much confidence in the mean effect estimate for this learner group. Three of the K–12 studies had significant effects favoring a blended learning condition, one had a significant negative effect favoring face-to-face instruction, and three contrasts did not attain statistical significance. The test for learner type as a moderator variable was not significant.No significant differences in effectiveness were found that related to the subject of instruction.

>> Effect sizes were larger for studies in which the online and face-to-face conditions varied in terms of curriculum materials and aspects of instructional approach in addition to the medium of instruction.

Six methodological variables were tested as potential moderators: (a) sample size, (b) type of knowledge tested, (c) strength of study design, (d) unit of assignment to condition, (e) instructor equivalence across conditions, and (f) equivalence of curriculum and instructional approach across conditions.Only equivalence of
curriculum and instruction emerged as a significant moderator variable. Studies in which analysts judged the curriculum and instruction to be identical or almost identical in online and face-to-face conditions had smaller effects than those studies where the two conditions varied in terms of multiple aspects of instruction.

Instruction could differ in terms of the way activities are organized (for example, as group work in one condition and independent work in another) or in the inclusion of instructional resources (such as a simulation or instructor lectures) in one condition but not the other.

MAJORITY RULES

The narrative review of experimental and quasi-experimental studies contrasting different online learning practices found that the majority of available studies suggest the following:

>> Blended and purely online learning conditions implemented within a single study generally result in similar student learning outcomes.

When a study contrasts blended and purely online conditions, student learning is usually comparable across the two conditions.

>> Elements such as video or online quizzes do not appear to influence the amount that students learn in online classes.

The research does not support the use of some frequently recommended online learning practices. Inclusion of more media in an online application does not appear to enhance learning. The practice of providing online quizzes does not seem to be more effective than other tactics such as assigning homework.

>> Online learning can be enhanced by giving learners control of their interactions with media and prompting learner reflection.

Studies indicate that manipulations that trigger learner activity or learner reflection and self-monitoring of understanding are effective when students pursue online learning as individuals.

>> Providing guidance for learning for groups of students appears less successful than does using such mechanisms with individual learners.

When groups of students are learning together on line, support mechanisms such as guiding questions generally influence the way students interact, but not the amount they learn.

CONCLUSIONS

In recent experimental and quasi-experimental studies contrasting blends of online and face-to-face instruction with conventional face-to-face classes, blended instruction has been more effective, providing a rationale for the effort required to design and implement blended approaches. Even when used by itself, online learning appears to offer a modest advantage over conventional classroom instruction.

However, several caveats are in order:

1) Despite what appears to be strong support for online learning applications, the studies do not demonstrate that online learning is superior as a medium.

2) In many of the studies showing an advantage for online learning, the online and classroom conditions differed in terms of time spent, curriculumand pedagogy. Itwas the combination of elements in the treatment conditions (which was likely to have included additional learning time andmaterials aswell as additional opportunities for collaboration) that produced the observed learning advantages.

3) At the same time, one should note that online learning is much more conducive to the expansion of learning time than is face-to-face instruction.

4) In addition, although the types of research designs used by the studies in the meta-analysis were strong (i.e., experimental or controlled quasi-experimental),
many of the studies suffered from weaknesses such as small sample sizes; failure to report retention rates for students in the conditions being contrasted; and, in many cases, potential bias stemming from the authors’ dual roles as experimenters and instructors.

5) Finally, the great majority of estimated effect sizes in the meta-analysis are for undergraduate and older students, not elementary or secondary learners. Although there was not a significant effect by learner type, when learners’ age groups are considered separately, the mean effect size is significantly positive for undergraduate and other older learners but not for K–12 students.

Another consideration is that various online learning implementation practices may have differing effectiveness for K–12 learners than they do for older students. It is certainly possible that younger students could benefit more from a different degree of teacher or computer-based guidance than would college students and older learners.Without new random assignment or controlled quasi experimental
studies of the effects of online learning options for K–12 students, policymakers will lack scientific evidence of the effectiveness of these emerging alternatives to face-to-face instruction.

—The authors wrote this 2009 paper as employees of the Center for Technology in Learning as part of the U.S. Department of Education’s Office of Planning, Evaluation and Policy Development.

Leave a reply