Augmented Reality: Forget the Glasses

While we waited for a Magic Leap, Pokémon Go walked off with the AR crown

8 min read
Illustration of a Pokemon
Photo-illustration: Edmon de Haro

In mid-2014, Magic Leap began teasing us with visions of realistic baby elephants playing in the palms of our hands, promising to soon unveil a mind-blowing augmented reality technology that would dramatically change the worlds of both entertainment and computing. Investors have ponied up an astounding US $1.39 billion so far to own a piece of this AR future, according to Crunchbase.

We’re still waiting. For a while, it seemed that 2017 was going to be the year of Magic Leap, but the company’s technology does not appear ready for prime time, though AR fans are hoping for at least one public demo.

Meanwhile, a funny thing happened on the way to beta testing. While we were waiting for Magic Leap to show us what’s behind its curtain, another startup, Niantic, working with the Pokémon Company and Nintendo, launched a free mobile app in July 2016 that inserted the little critters of the decades-old Pokémon franchise into live scenes on the screens of mobile devices. Pokémon Go challenged fans old and young to go out and “catch ’em all!” And it worked: CEO John Hanke, speaking at an Apple event in September, reported 500 million downloads worldwide in just two months, with players collectively walking 4.6 billion kilometers while playing the game. In the United States, Pokémon Go quickly beat Candy Crush Saga, becoming the most popular game ever.

iPhone Use, in Minutes

bar chartAn analysis for 11 July 2016 shows that the average iPhone user spent more time on Pokémon Go that day than on any of the most popular social-networking apps. Chart Source: SensorTower

Since then, many companies have been scrambling in stealth to develop competing AR game apps, and dozens of these apps will likely roll out throughout this year. This is the start of something big, a new form of mass entertainment. It’s a watershed moment as significant as the early days of video games.

Consider the episode widely referred to as the birth of today’s video-game industry. In August 1972, a little startup called Atari put a prototype coin-op video game, Pong, in a bar in Sunnyvale, Calif. A few days later, the engineer who built the game went back to the bar to check out reports that Pong was broken. The problem turned out to be an overstuffed coin box.

Now consider last summer. For the first few days after Niantic put Pokémon Go out on app stores, its servers around the world crashed regularly, and new users struggled to open accounts on the overloaded systems. Speaking at San Diego Comic-Con the month of the launch, Hanke admitted that the company hadn’t been at all prepared for the app to become such a hit.

In the year after the introduction of Pong, companies like Sega and Taito announced that they, too, were getting into the video-game business, and with Atari, they built a new industry out of a revolution in entertainment technology.

That, in a PokéBall, is likely to be the story of AR in 2017.

Let’s start with a brief explanation of Pokémon Go, just in case you’ve been stranded on a desert island or something for much of the past six months. The game puts you in the role of a trainer, charged with catching little virtual critters—Pokémon—and then bringing them to virtual gyms where they can battle other critters. The virtual creatures are linked to real-world locations, and not randomly: You’ll need to be near a body of water to find aquatic Pokémon; an amusement park crammed with flashing lights is a good place to hunt the electric type. To find them, you walk around using a sort of Pokémon “radar” that appears on your smartphone screen. Get close enough and you can see the Pokémon itself; the app combines the virtual image with whatever your phone’s camera is seeing, so it looks like the creature is right in front of you, perhaps sitting on your car—or your dog—leading to hilarious photos shared on social media. You catch one by using your finger to toss a virtual ball at it, and it’s not as easy as it looks.

That’s the gist. There are also more complexities, but the main takeaways are that you “see” virtual creatures in the real world by looking at your smartphone’s screen and that you can’t do much sitting at home. You have to get out and walk and walk and walk.

Pokémon Go and its coming cohort represent a type of AR technology that some are calling “augmented reality lite.” On the opposite end of the spectrum is “augmented reality heavy.” That’s where you’ll find Magic Leap. Here, the ultimate goal is an augmented reality that passes the Turing test for AR: If you see two identical objects sitting on a table, you won’t be able to tell which is real and which is virtual. This approach requires some kind of head-mounted display and lots of computing power.

Magic Leap has stated that its technology uses a “dynamic digitized light-field signal” to “generate images indistinguishable from real objects and…place those images seamlessly into the real world.” Generally, people interpret this as meaning that the company is projecting images directly onto the retina in the form of a light field—that is, the technology models the direction from which each ray of light from the virtual objects would travel to the observer if the virtual objects were actually at the real-world locations they are supposed to occupy. A light-field approach for AR would allow a virtual image to be more realistically mixed with real images than in a conventional display.

photo from magic leap videoSeeing Pink Elephants: Magic Leap’s mysterious artificial-reality technology is the elephant in the room—or in the palm of your hand, as this early demo suggests—but it’s unlikely to reach consumers this year.Image: Magic Leap

I have not tried out a Magic Leap prototype (and if I had, I couldn’t admit it in print because the company’s nondisclosure agreements are that tight). But I have gotten to one degree of separation: I talked to someone who has. He couldn’t be specific but admitted that he was indeed blown away. “Before I saw it,” he said, “I thought it was much further out. As soon as you see it you say, ‘Oh, yeah, how could this not be…” He trailed off, leaving what it could be to my imagination. A huge success? The future of computer games? The end of computing as we know it?

Also in the AR-heavy category, and already shipping to developers and some business customers, is Microsoft’s $3,000 HoloLens, an AR headset with all its necessary computing performed on board. The trade-off Microsoft made to make this technology portable is in its field of view: At less than 45 degrees, it’s like looking through a small window. One of Microsoft’s competitors, Meta, is taking preorders from developers for a $950 headset with a 90-degree field of view—but the Meta headset requires tethering to an external computer to operate. Both project images outward, not directly on your retina, as Magic Leap is expected to do. But Magic Leap may not end up being the only AR retina display out there. Kartik Hosanagar, professor of technology and digital business at the Wharton School of the University of Pennsylvania, says he believes that MicroVision, a pioneer in retina displays, may move into commercial augmented reality.

And then, inevitably, there is Apple. CEO Tim Cook told The Washington Post in August that the company was doing a lot of things “behind the curtain” with augmented reality. Apple last year purchased the AR startup Metaio, and now the company reportedly has hundreds of engineers working on the technology, including some hotshot researchers recently hired away from Oculus and Magic Leap.

But in spite of all of these efforts, for the average person without a big budget and the willingness to tinker with a technology still under development, AR heavy remains somewhere beyond the horizon. “Magic Leap is not going to be an order of magnitude lower in price than the $3,000 HoloLens,” says Hosanagar, “which will make it prohibitive for the mass market.”

Niantic chief technology officer Phil Keslin agrees. “I’m not going to wait for Magic Leap,” he says in an interview. “I have products to deliver to the world, experiences to get out there. I’m going to use whatever tech is available.” Right now, he adds, that’s the phone, although he’s also keenly anticipating improvements in smart-watch technology. Moving to that platform would let AR users move more seamlessly between the virtual and the real worlds, he says. He’s also doing a lot of thinking about audio technology, looking at earbuds and other wearables (like the Bragi earbuds and the Oakley Radar Pace glasses) that have accelerometers and other sensors built in. “We haven’t explored an audio-augmented experience as much as we could,” he says.

That brings us back to Pokémon Go and the soon-to-come wave of AR-lite competitors vying to be the next big hit. What exactly will that hit be? Wouldn’t you like to know?

Janet Murray, professor of digital media and associate dean of research at Georgia Tech, whose students seem to be ahead of the curve—they demonstrated their Harry PottAR mobile phone game a semester before the launch of Pokémon Go—thinks the next big thing to exploit mobile-phone AR could be a time-travel application: As you walk through the real world, you will use your phone screen to peek back at the past. What did that building look like 10, 20, or 100 years ago? Who lived there? Can you follow a fictional character from another era through this parallel world, catching glimpses of intriguing past events, or maybe find the forgotten site of an old bank and collect a few virtual gold coins? I’d download that.

Georgia Tech’s Harry PottAR mobile phone game Video: Georgia Institute of Technology

Niantic’s Keslin is proud of what his company has achieved with Pokémon Go. “We exposed AR technology to a broader base of people. People had seen it in movies, but this made it real. And we demonstrated that there is a market for this, that people want this experience. That will help move the technology forward.

“The vision, when we started, was to get people off their couches and into the real world and meeting other people. We did that. But we were overwhelmed by its popularity this summer. It has been a humbling experience in many ways.”

And a personally rewarding one for Keslin. “My wife has a good friend whose 14-year-old son is autistic, and this has changed his life. He used to have to wear noise-canceling headphones out in the world; now he doesn’t have to do that—he can focus on the game. He is motivated to go out and play with others. And I’ve heard that this has had the same effect on other kids.”

Revenue in 2020

bar chartBy that year, the market for augmented or virtual reality is exected to be half the size of the film market. Chart Sources: Statistica (Film); International Data Corp. (AR/VR)

The number of people playing Pokémon Go daily dropped off dramatically in late summer. The app lost some 15 million—about a third—of its daily users, according to a report by Axiom Capital Management. Keslin attributes part of that drop to the start of school. “School took a huge portion of our player base and sucked it into buildings,” he says. Some of it is simply boredom. Keslin recognizes that and hopes to keep delivering “new and shiny” experiences to regularly draw players back in.

“People have said that Pokémon Go has peaked and is flaming out. I don’t think so,” says Matthew Szymczyk, CEO of AR content maker Zugara. “It’s on track to make $1 billion, and that’s just direct revenue, not including what retailers make by having Pokémon Go sites at their locations. It’s a multibillion-dollar ecosystem.”

“It’s like a TV show,” points out Tawny Schlieski, director of desktop research at Intel. “It’s absurd to ask if it’s lasting; that’s not how content works. Content evolves. Pokémon Go will move forward in the same way as a TV show that comes back for another season. It is the start of a new form of gaming that is contextual for your world, that you can’t play on the couch,” she concludes.

“AR and geolocation is a whole frontier that is waiting to be mapped out and settled,” adds Szymczyk. “This is just the beginning.”


This article is for IEEE members only. Join IEEE to access our full archive.

Join the world’s largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum’s articles, podcasts, and special reports. Learn more →

If you're already an IEEE member, please sign in to continue reading.

Membership includes:

  • Get unlimited access to IEEE Spectrum content
  • Follow your favorite topics to create a personalized feed of IEEE Spectrum content
  • Save Spectrum articles to read later
  • Network with other technology professionals
  • Establish a professional profile
  • Create a group to share and collaborate on projects
  • Discover IEEE events and activities
  • Join and participate in discussions