Augmented reality (AR) offers a new approach to exploration and learning by blending real-world conditions with digital data. While clearly suited to encourage play and to entertain, AR also has appeal to marketers wishing to engage with consumers. These same elements can engage learners in their quest for insights and experiences.
AR is already in use in many curricula and programs for specialized instruction. Tools permitting tech-savvy educators to explore and implement AR in their materials are available today. With easier to use development/design tools, one day any educator will be able to design new interaction for learning in situ.
In the technology portfolio of educators, many tools are designed to meet a specific objective — to support inquiry, to provide practice and repetition without risk to a valuable resource or subject, to encourage creativity and collaboration. AR is not one of those targeted tools — but maybe it will have the ability to inspire educators and learners to use technology fully for every day experiences, including learning.
Imagine you are helping a group of learners who have never seen an X-ray discover the human skeleton. Just like a circus sideshow in the 1930s, where anyone prepared to pay a few pennies could stand near a radioactive source and expose a film to produce an “X-ray,” today, students can stand in front of a Microsoft Kinect system and see a skeleton superimposed on themselves. Turn around and use the same basic principles with a smartphone to learn the names and heights of summits. What do these learning experiences have in common?
WHAT IS AR?
In simple terms, AR brings users information that exists in the digital world and presents it in tight association with things in the real, or physical, world — automatically and intuitively.
AR creates, makes explicit and displays the relationships between the real and
virtual worlds. At the highest level, AR can be seen as the latest stage in information search, viewing and, ultimately, to its easy manipulation by the user. With AR, digital information can be automatically “connected,” in context, to real-world objects. The past, the future, the facts and the myths, can now be part of what you see and hear.
AR can take many different forms, but at the core an experience is produced in three stages.
Sensors in the user’s device (such as camera, GPS and/or compass, microphone,
even a thermometer) detect a “condition” in the local environment—the visual
image of a recognizable geographic feature like a mountain or famous building, for
instance. The same sensors detect the user’s pose relative to the scene or objects.
The application or “system” finds the digital data (any text, image, 3-D model,
video, URL, sound, etc.) that has previously been associated with the specific condition in the local environment identified by the sensor.
The digital information is presented visually (or aurally) to the user in such a
way that it is synchronized with the real world. It is only presented as long as the
condition remains the same. When the user’s condition changes (for example, he
moves away from the object), the digital information disappears or is replaced with
new information if the sensors are triggered by another condition.
LEARNING APPLICATIONS FOR AR
Creating immersive and engaging experiences consistently increases the learner’s
retention of a solution or stimulates deeper understanding of facts or issues. AR has been shown to accelerate comprehension and increase engagement with learners of all ages and in numerous disciplines, from geography and physics to culture and language.
AR projects have been developed to enhance unusual objects, such as an aircraft
engine, as well as a very traditional learning tool: a book.While examples of AR interacting with print are valuable, they are only one of the many materials that surround learners and educators. Let’s examine a few disciplines that have been made more interesting using AR.
Science and physics: An early showcase of using AR for learning geography was
developed by Metaio in mid-2008, using the company’s Unifeye platform for AR
and the “BigWorld Atlas.” The project produced an interactive atlas that was made available during the Frankfurt Book Fair, an international annual book-buyer
event. This particular work is only available in German, but the concept is easy to
understand and could be implemented with the same models in any language. Connecting the spatial world with digital information can also help when discovering
a new place using a map.Microsoft’s Bing Maps director, Blaise Aguera y Arcas
used Augmented Reality to show how maps might appear in the future using a combination of visual recognition and the user’s onboard compass and GPS.
Cultural heritage: The exploration of the physical world is not limited to concepts.
AR is also used to help the discovery of cultural heritage that is tied to objects that have since disappeared. In one example, the BerlinWall has been recreated and
exists in virtual space when a user of the Layar BerlinWall layer is in proximity of the former barrier between East and West Germany. Many other historical landmarks — from the Summer Palace of the Ming Dynasty on the outskirts of Beijing to the barn in which the Wright Brothers built their first flying machine —have been enhanced with AR.
Language: Perhaps one of the most fundamental skills a person can learn is the
mastery of a new language. There are Augmented Reality applications that are
helping this process for young learners as well as people of any age. The game Put a Spell, developed for the iPhone by Ogmento, uses an animated panda bear to
help a child learn to spell. The Google Goggles andWord Lens applications provide
the user the ability to instantly “read” a sign in a foreign language simply by holding the print in the field of view of the camera phone running the applications. It’s not difficult to imagine how these real-world applications will help us to better understand our surroundings as well as those of different cultures and language groups.
Kinesthetic learning: As a result of having AR be integral with the real world, it is
clear that it can assist with the learning of physical skills. One example, undertaken
by researchers in the Computer Vision Laboratory at the ETH in Zurich, has explored the use of haptic interfaces in combination with real and virtual objects
to study and train “manipulative” skills in a surgical environment.While not in daily
use, these tools are promising for practicing high-risk procedures before undertaking them on real patients. The U.S. Marine Corps also has been testing the
use of AR to train mechanics to repair vehicles in the field. In the scenario developed at the Columbia University Computer Graphics and User Interfaces laboratory of Steve Feiner, parts of the engine and virtual representations of tools
become animated and have labels associated with instructions hovering over them
when viewed through a head mounted display. Studies conducted with Marines demonstrated that AR was superior to using an electronic manual presented on an LCD screen (see figure 1).
TOOLS FOR CREATING AR EXPERIENCES
How do all these applications migrate from the graphics laboratory and the Marines in the field to the world of everyday learners? It’s not as easy as building a Web page, but it isn’t as difficult as bringing the Pyramids to your middle-school classroom.
As in the early days of the Web, to begin implementing AR in your class and
courses today requires some programming skills. But the threshold is rapidly reaching a technically astute educator and should be well within reach of most within 18 to 24 months.
Open-source: The first and most common entry point for AR experimentation
has been the AR Toolworks Toolkit. This open-source software can be downloaded
from theWeb and, after several tutorials, a simple AR experience with a fiducial marker is possible. The software supports all levels of expertise and can become the basis of new applications which “stand alone” and can be installed by a user to
run on a Macintosh or Windows personal computer. The system has been adopted by thousands of developers, so finding a company to hire to create a learning application for you will not be difficult should you find that your ambitions exceed your programming and 3-D design skills.
Currently, the AR Toolworks platform relies entirely on the use of visual recognition
and tracking to locate objects in the user’s environment and enhance them with still or moving digital data. The platform does not have an AR “browser,” so it is not possible to have one application with many different types of learning experiences, as is the case with other platforms.
Platforms for publishing AR channels and layers: Another option available to
create enticing AR experiences quickly is to use one of theWeb-based platforms
that support the creation of dedicated information “channels” or “layers” that
then appear in one of the Augmented Reality Browsers available for Android and
iPhones. AR browsers are dedicated applications that detect the user’s environment using the GPS and compass, and, optionally, in the case of the Junaio browser,
some object in the camera’s field of view. Then, by matching the situation with digital data, overlays a digital file on the smartphone screen in alignment with the user’s focus of attention.
Popular AR browsers are available at no cost on the application stores such as the
Android Marketplace and the Apple iTunes Application Store. The Layar Content Management System and the junaio publishing platform are both available
at no cost to developers. Creating a developer account is just a matter of completing an on-line form.
Zooburst: There are also dedicated platforms really targeting educators. One such
platform is provided by Zooburst, a New York-based start-up. The ZooBurst program is, like the platforms of the AR Browsers, all on-line. Already, more than
750 educational institutions around the world have signed up for accounts to develop their own interactive AR books.
Augmented reality will continue to get more popular in daily life and professional
settings, making it a natural extension of the educator’s portfolio of approaches
to helping learners discover and explore new topics in context and at the most natural levels.
As the number of people familiar with AR creation tools increases, educators will
also be able to partner with subject matter experts in various disciplines to bring digital experiences to learners with AR using geospatial as well as objects detected using machine learning and computer vision. The computational requirements of AR
are quite high by comparison with other learning tools so the computers on which these applications are installed must be powerful, similar to “game” PCs today. In
the future, as the processors available on mainstream computing platforms increase, the needs of AR applications will be in line with the commonly available student computers.
—Christine Perey is an independent consultant and industry evangelist for augmented reality services and technologies. She is the founder of several community organizations for accelerating the adoption of AR through education and the development of open and interoperable interfaces for technology platforms. For more information, visit www.perey.com. E-mail her at firstname.lastname@example.org.
The U.K.’s Learn AR Program
Learn AR is a new learning tool that brings investigative, interactive and independent learning to life using augmented reality. It is a pack of 10 curriculum resources for teachers and students to explore by combining the real world with virtual content using a Webcam. The resource pack consists of interactive learning activities across English, math, science, RE, physical education and languages that bring a “wow” factor to the curriculum. From James Alliban’s blog:
Learn AR – Augmented Reality for Schools
Augmatic the British company founded by James Alliban (you may remember him from that augmented reality business card) has launched a new tool, called Learn AR.
The Geiger counter and physiology demos seem to provide real value when coming to teach such subjects. On the other hand, I really don’t find any benefit of using the English application over a computer game. In a sense, Learn AR is showing how gimmicky and how useful AR can be at the same time.
For another perspective on using AR for educational purposes, you should really check out Gail Carmichael’s blog. Since she’s doing her Ph.D. on this subject and has blogged about some very interesting concepts: https://www.ssatrust.org.uk/achievement/future/Pages/AugmentedReality.aspx
The AR Cover Demo: How We Made It
hen we heard from Elearning! magazine earlier this year about a special issue
focusing on augmented reality (AR), we were excited to say the least. While AR
technology is in its infancy, it has huge potential for use in workplace learning. At
Upside Learning, we have been experimenting with AR since the technology first emerged on the Web. So this was a perfect opportunity to show the world some of our experiments with AR.
After our initial discussions with Elearning!, we finalized how the demonstration would look and function. Essentially, the team wanted to create a marker that the magazine could print on its cover, which could then be held up to a computer camera and serve as the entry point into some user interaction and display of media elements. The idea was simple enough: hold up a marker, play/control a video using the marker, and end up with a user interface that plays videos of AR in action.
An initial analysis revealed that we’d need the application to include routines for camera hardware detection, marker recognition, video decoding and rendering, interaction using the marker, and an embedded browser object. We set out by identifying application libraries that offered appropriate functionality for what we were trying to accomplish. For detecting the marker and how the user might manipulate it in their workspace, we chose FLARManager. (http://words.transmote.com/wp/flarmanager/). FLAR works within Flash, which is our preferred development environment. It’s also sufficiently lightweight and an
easy-to-use development tool. Additionally, it supports a host of tracking libraries, of which we use FLARToolkit (http://saqoo.sh/a/en/flartoolkit/start-up-guide).
We also needed a 3-D library to create a 3-D environment to render the video elements. Since we use Flash, we used PaperVision3D (http://code.google.com/p/papervision3d/) to provide that functionality. After we identified these critical libraries, we simply started to activate the individual components.
Once these routines were working individually, we then integrated them into a single application. The development platform we used was Windows XP, coupled with Flash CS4, and Flash Developer. For the Mac version, it was a matter of moving the code base to a Mac and doing a recompile. Then, after some simple bug fixing, we got it to work on that platform as planned. Once the application was working on both platforms, we began to integrate the media elements. This lets the user view the video as soon as the application recognizes the marker.
Although it sounds complicated, overall, it was quite simple to do. The application
libraries we used are free and easily available at the URLs mentioned. Simply download them and try your own hand at AR.
With this understanding of AR, you will be well ahead of the curve. As mobile technology and persistent Internet connectivity become home and workplace realities, more and more AR applications will invade our lives. The challenge to learning designers and developers will be to use these new technologies effectively.
—Abhijit Kadle, Upside Learning
Augmented/ Alternate Reality Games
The economics of professional learning requires ever-increasing use of technology,
even though current e-learning technologies just don’t support learn-by doing
very well. This may be changing with the impending “Reality Revolution.”
Both augmented and alternate reality games offer compelling new opportunities
for technology aided learning. They both empower learning-by-doing through technology.
Augmented reality and alternate reality games both provide new opportunities
for enterprise learning producers to integrate training into the real life reality of their learners.
AUGMENTED REALITY (AR)
AR supplements real life with additional streams of relevant data. These streams
can be location, time, action, task-based or anything adds to the production of useful knowledge.
For example, blind intersections are dangerous to drivers. There is an increased risk for serious traffic accidents when drivers can’t see further than their breaking distance. This risk is usually mediated with traffic management tools like low speed limits, stop and yield signs. Reducing the speed of a moving vehicle reduces the distance needed to stop that vehicle. Problem solved for normal traffic conditions — but what about in emergency conditions? First responders often need to travel faster than posted speed limits.
AR can provide solutions for emergency vehicles. If equipped with a video camera
facing forward from the driver’s point-of-view and a dashboard screen, the driver
would be able to see the upcoming road from both the windshield and video
screen. This dashboard video image can be augmented with a data stream from a
camera located behind a wall at a problematic blind corner. Fig. 1 is exactly this.
The problematic wall appears in the dashboard video image is semi-transparent, because the real image from the driver’s point of view is augmented with data from
the second location-specific data stream from a camera on the other side of the
wall. In this example, AR supplements the data available to the driver, providing
information that allows a better decision. This particular application could be immensely useful to drivers hoping to avoid head-on collision.
However, if training is defined as teaching people to do the right things at the
right times, this application of AR functions as an Electronic Performance Support System (EPSS), not training.
AR has great potential for government and military just-in-time training and EPSS.
Often people need to do the right things at the right times only once. They will never need to complete the task more than once in a lifetime. Significant time and money is wasted training people on tasks that they will only do once, if ever. AR is a brilliant solution to this problem.
What if a repair technician works on systems in which components are sourced from multiple vendors over many decades? Should the technicians be required to maintain training on hundreds of devices that they may or may not ever encounter?
Take the challenge of training to soldiers complete complex and unusual repairs on
critical equipments in a war zone. Are they trained on every possible repair, or is the equipment assumed to be disposable?
A Marine completes a complicated task with the aid of augmented reality EPSS.
There are two hardware components to this system. The learner wears head mounted video display goggles that display video images overlaid on the view of
the natural environment. The second is a smart phone equipped with an EPSS navigation application. The hardware is shown in Fig. 2.
The Marine views the interior of a vehicle through the head mounted video display
goggles. An additional data stream provides an animated task repair sequence. The Marine is walked through the repair process. Through the goggles, the Marine sees the inside of the vehicle with overlaid animation providing step by step instructions for the repair (Fig. 3). The animation can be controlled by the learner using the smart phone navigation worn on the wrist.
ALTERNATE REALITY GAMES (ARG)
ARGs are played in real life. The experience is not mediated by special hardware
or software but normal communication modalities. Clues, in the form of story pieces, are delivered to players via e-mail, twitter, TV or any other information
source. These interactive games require players to manipulate story lines by constructing cogent narratives from fragments.
Training uses for ARGs are very promising, because they accomplish something amazing that has alluded e-learning. ARGs effectively model complex learner
behaviors in cognitively engaging and emotionally compelling ways. Imagine
your learner population has been effectively trained on systems, processes and soft-skills. However, until now there has not been a way to model use all of these
skills in the real world of the learners. Before ARGs, the closest we could get was very expensive “serious games” that play like video games in an artificial world
accessed from a computer screen. Would it be more effective to simulate all the
information tools that law enforcement may need to use in an emergency situation
as a video game or to be able to train them using these actual tools? Modeling behavior with the real communication modalities of learners is the amazing power of ARGs.
Augmented reality and alternate reality games are both revolutionary new and
powerful tools for enterprise learning and development producers. It’s important to find the correct partners and learning opportunities.
—Tim Martin has worked on the forefront of technology enabled education and training for the past 25 years. He has been at the table as an e-learning producer,
P&L manager, vendor and media analysis as the industry has matured from its infancy. Reach him via e-mail at email@example.com.