Augmented Reality: Forget the Glasses

While we waited for a Magic Leap, Pokémon Go walked off with the AR crown

8 min read
Illustration of a Pokemon
Photo-illustration: Edmon de Haro

Illustration of a Pokemon Photo-illustration: Edmon de Haro

In mid-2014, Magic Leap began teasing us with visions of realistic baby elephants playing in the palms of our hands, promising to soon unveil a mind-blowing augmented reality technology that would dramatically change the worlds of both entertainment and computing. Investors have ponied up an astounding US $1.39 billion so far to own a piece of this AR future, according to Crunchbase.

We’re still waiting. For a while, it seemed that 2017 was going to be the year of Magic Leap, but the company’s technology does not appear ready for prime time, though AR fans are hoping for at least one public demo.

Meanwhile, a funny thing happened on the way to beta testing. While we were waiting for Magic Leap to show us what’s behind its curtain, another startup, Niantic, working with the Pokémon Company and Nintendo, launched a free mobile app in July 2016 that inserted the little critters of the decades-old Pokémon franchise into live scenes on the screens of mobile devices. Pokémon Go challenged fans old and young to go out and “catch ’em all!” And it worked: CEO John Hanke, speaking at an Apple event in September, reported 500 million downloads worldwide in just two months, with players collectively walking 4.6 billion kilometers while playing the game. In the United States, Pokémon Go quickly beat Candy Crush Saga, becoming the most popular game ever.

iPhone Use, in Minutes

bar chart An analysis for 11 July 2016 shows that the average iPhone user spent more time on  Pokémon Go that day than on any of the most popular social-networking apps. Chart Source: SensorTower

Since then, many companies have been scrambling in stealth to develop competing AR game apps, and dozens of these apps will likely roll out throughout this year. This is the start of something big, a new form of mass entertainment. It’s a watershed moment as significant as the early days of video games.

Consider the episode widely referred to as the birth of today’s video-game industry. In August 1972, a little startup called Atari put a prototype coin-op video game, Pong, in a bar in Sunnyvale, Calif. A few days later, the engineer who built the game went back to the bar to check out reports that Pong was broken. The problem turned out to be an overstuffed coin box.

Now consider last summer. For the first few days after Niantic put Pokémon Go out on app stores, its servers around the world crashed regularly, and new users struggled to open accounts on the overloaded systems. Speaking at San Diego Comic-Con the month of the launch, Hanke admitted that the company hadn’t been at all prepared for the app to become such a hit.

In the year after the introduction of Pong, companies like Sega and Taito announced that they, too, were getting into the video-game business, and with Atari, they built a new industry out of a revolution in entertainment technology.

That, in a PokéBall, is likely to be the story of AR in 2017.

Let’s start with a brief explanation of Pokémon Go, just in case you’ve been stranded on a desert island or something for much of the past six months. The game puts you in the role of a trainer, charged with catching little virtual critters—Pokémon—and then bringing them to virtual gyms where they can battle other critters. The virtual creatures are linked to real-world locations, and not randomly: You’ll need to be near a body of water to find aquatic Pokémon; an amusement park crammed with flashing lights is a good place to hunt the electric type. To find them, you walk around using a sort of Pokémon “radar” that appears on your smartphone screen. Get close enough and you can see the Pokémon itself; the app combines the virtual image with whatever your phone’s camera is seeing, so it looks like the creature is right in front of you, perhaps sitting on your car—or your dog—leading to hilarious photos shared on social media. You catch one by using your finger to toss a virtual ball at it, and it’s not as easy as it looks.

That’s the gist. There are also more complexities, but the main takeaways are that you “see” virtual creatures in the real world by looking at your smartphone’s screen and that you can’t do much sitting at home. You have to get out and walk and walk and walk.

Pokémon Go and its coming cohort represent a type of AR technology that some are calling “augmented reality lite.” On the opposite end of the spectrum is “augmented reality heavy.” That’s where you’ll find Magic Leap. Here, the ultimate goal is an augmented reality that passes the Turing test for AR: If you see two identical objects sitting on a table, you won’t be able to tell which is real and which is virtual. This approach requires some kind of head-mounted display and lots of computing power.

Magic Leap has stated that its technology uses a “dynamic digitized light-field signal” to “generate images indistinguishable from real objects and…place those images seamlessly into the real world.” Generally, people interpret this as meaning that the company is projecting images directly onto the retina in the form of a light field—that is, the technology models the direction from which each ray of light from the virtual objects would travel to the observer if the virtual objects were actually at the real-world locations they are supposed to occupy. A light-field approach for AR would allow a virtual image to be more realistically mixed with real images than in a conventional display.

photo from magic leap video Seeing Pink Elephants: Magic Leap’s mysterious artificial-reality technology is the elephant in the room—or in the palm of your hand, as this early demo suggests—but it’s unlikely to reach consumers this year. Image: Magic Leap

I have not tried out a Magic Leap prototype (and if I had, I couldn’t admit it in print because the company’s nondisclosure agreements are that tight). But I have gotten to one degree of separation: I talked to someone who has. He couldn’t be specific but admitted that he was indeed blown away. “Before I saw it,” he said, “I thought it was much further out. As soon as you see it you say, ‘Oh, yeah, how could this not be…” He trailed off, leaving what it could be to my imagination. A huge success? The future of computer games? The end of computing as we know it?

Also in the AR-heavy category, and already shipping to developers and some business customers, is Microsoft’s $3,000 HoloLens, an AR headset with all its necessary computing performed on board. The trade-off Microsoft made to make this technology portable is in its field of view: At less than 45 degrees, it’s like looking through a small window. One of Microsoft’s competitors, Meta, is taking preorders from developers for a $950 headset with a 90-degree field of view—but the Meta headset requires tethering to an external computer to operate. Both project images outward, not directly on your retina, as Magic Leap is expected to do. But Magic Leap may not end up being the only AR retina display out there. Kartik Hosanagar, professor of technology and digital business at the Wharton School of the University of Pennsylvania, says he believes that MicroVision, a pioneer in retina displays, may move into commercial augmented reality.

And then, inevitably, there is Apple. CEO Tim Cook told The Washington Post in August that the company was doing a lot of things “behind the curtain” with augmented reality. Apple last year purchased the AR startup Metaio, and now the company reportedly has hundreds of engineers working on the technology, including some hotshot researchers recently hired away from Oculus and Magic Leap.

But in spite of all of these efforts, for the average person without a big budget and the willingness to tinker with a technology still under development, AR heavy remains somewhere beyond the horizon. “Magic Leap is not going to be an order of magnitude lower in price than the $3,000 HoloLens,” says Hosanagar, “which will make it prohibitive for the mass market.”

Niantic chief technology officer Phil Keslin agrees. “I’m not going to wait for Magic Leap,” he says in an interview. “I have products to deliver to the world, experiences to get out there. I’m going to use whatever tech is available.” Right now, he adds, that’s the phone, although he’s also keenly anticipating improvements in smart-watch technology. Moving to that platform would let AR users move more seamlessly between the virtual and the real worlds, he says. He’s also doing a lot of thinking about audio technology, looking at earbuds and other wearables (like the Bragi earbuds and the Oakley Radar Pace glasses) that have accelerometers and other sensors built in. “We haven’t explored an audio-augmented experience as much as we could,” he says.

That brings us back to Pokémon Go and the soon-to-come wave of AR-lite competitors vying to be the next big hit. What exactly will that hit be? Wouldn’t you like to know?

Janet Murray, professor of digital media and associate dean of research at Georgia Tech, whose students seem to be ahead of the curve—they demonstrated their Harry PottAR mobile phone game a semester before the launch of Pokémon Go—thinks the next big thing to exploit mobile-phone AR could be a time-travel application: As you walk through the real world, you will use your phone screen to peek back at the past. What did that building look like 10, 20, or 100 years ago? Who lived there? Can you follow a fictional character from another era through this parallel world, catching glimpses of intriguing past events, or maybe find the forgotten site of an old bank and collect a few virtual gold coins? I’d download that.

Georgia Tech’s Harry PottAR mobile phone game. Video: Georgia Institute of Technology

Niantic’s Keslin is proud of what his company has achieved with Pokémon Go. “We exposed AR technology to a broader base of people. People had seen it in movies, but this made it real. And we demonstrated that there is a market for this, that people want this experience. That will help move the technology forward.

“The vision, when we started, was to get people off their couches and into the real world and meeting other people. We did that. But we were overwhelmed by its popularity this summer. It has been a humbling experience in many ways.”

And a personally rewarding one for Keslin. “My wife has a good friend whose 14-year-old son is autistic, and this has changed his life. He used to have to wear noise-canceling headphones out in the world; now he doesn’t have to do that—he can focus on the game. He is motivated to go out and play with others. And I’ve heard that this has had the same effect on other kids.”

Revenue in 2020

bar chart By that year, the market for augmented or virtual reality is exected to be half the size of the film market.  Chart Sources: Statistica (Film); International Data Corp. (AR/VR)

The number of people playing Pokémon Go daily dropped off dramatically in late summer. The app lost some 15 million—about a third—of its daily users, according to a report by Axiom Capital Management. Keslin attributes part of that drop to the start of school. “School took a huge portion of our player base and sucked it into buildings,” he says. Some of it is simply boredom. Keslin recognizes that and hopes to keep delivering “new and shiny” experiences to regularly draw players back in.

“People have said that Pokémon Go has peaked and is flaming out. I don’t think so,” says Matthew Szymczyk, CEO of AR content maker Zugara. “It’s on track to make $1 billion, and that’s just direct revenue, not including what retailers make by having Pokémon Go sites at their locations. It’s a multibillion-dollar ecosystem.”

“It’s like a TV show,” points out Tawny Schlieski, director of desktop research at Intel. “It’s absurd to ask if it’s lasting; that’s not how content works. Content evolves. Pokémon Go will move forward in the same way as a TV show that comes back for another season. It is the start of a new form of gaming that is contextual for your world, that you can’t play on the couch,” she concludes.

“AR and geolocation is a whole frontier that is waiting to be mapped out and settled,” adds Szymczyk. “This is just the beginning.”

The Conversation (0)

Biggest Tech Companies Now Building the Biggest Data Pipes

Facebook will lay a record-capacity submarine cable across the Atlantic

4 min read

Google's Grace Hopper subsea cable landing in the seaside town of Bude in England

Google

Old-fashioned telecommunication carriers are falling behind in the global bandwidth race as global giants of content and cloud computing are building their own global networks. Facebook has commissioned electronics and IT giant NEC Corporation to build the world's highest capacity submarine cable. When finished it will carry a staggering 500 terabits—some 4000 Blu-Ray discs of data—per second between North America and Europe on the world's busiest data highway.

For decades, transoceanic cables were laid by consortia of telecommunication carriers like AT&T and British Telecom. As cloud computing and data centers spread around the world, Google, Amazon, Facebook and Microsoft start joining cable consortia, and in the past few years Google began building its own cables. The new cable will give Facebook sole ownership of the world's biggest data pipeline.

Transoceanic fiber-optic cables are the backbones of the global telecommunications network, and their change in ownership reflects the rapid growth of data centers for cloud computing and content distribution. Google has 23 giant data centers around the globe, each one constantly updated to mirror the Google cloud for users in their region. Three years ago, flows between data centers accounted for 77 percent of transatlantic traffic and 60 percent of transpacific traffic, Alan Mauldin, research director at TeleGeography, a market-research unit of California-based PriMetrica, said at the time. Traffic between data centers is thought to be growing faster than the per-person data consumption, which Facebook says increases 20 to 30 percent a year.

Vying for maximum bandwidth at the intersection of Moore's Law and Shannon's limit

Fiber-optic developers have worked relentlessly to keep up with the demand for bandwidth. For decades, data capacity of a single fiber increased at a faster rate than the number of transistors squeezed onto a chip, the definition of Moore's Law. But in recent years that growth has slowed as data rates approached Shannon's limit, a point at which noise in the transmission system overwhelms the signal. In 2016 the maximum data rate per fiber pair (each fiber carrying a signal in one direction) was around 10 terabits per second, achieved by sending signals at 100 gigabits per second on 100 separate wavelengths through the same fiber.

Developing more sophisticated signal formats offered some improvement, but not enough to keep pace with the demand for bandwidth. The only way around Shannon's limit has been to open new paths for data delivery.

In 2018, Facebook and Google placed bets on broadening the transmission band of optical fibers by adding signals at a hundred new wavelengths to squeeze 24 terabits through a single fiber. Each bought one pair of fibers on the Pacific Light Cable stretching from Hong Kong to Los Angeles. The leader of the consortium, Pacific Light Data Communications, of Hong Kong, retained four other pairs in the six-pair cable. Although the cable was soon laid, the U.S. Federal Communications Commission has refused to license its connection to the U.S. network because of security concerns arising from its Chinese connections.

Keep Reading ↓ Show less

Study: Recycled Lithium Batteries as Good as Newly Mined

Cathodes made with novel direct-recycling beat commercial materials

3 min read
iStockphoto

Lithium-ion batteries, with their use of riskily mined metals, tarnish the green image of EVs. Recycling to recover those valuable metals would minimize the social and environmental impact of mining, keep millions of tons of batteries from landfills, and cut the energy use and emissions created from making batteries.

But while the EV battery recycling industry is starting to take off, getting carmakers to use recycled materials remains a hard sell. "In general, people's impression is that recycled material is not as good as virgin material," says Yan Wang, a professor of mechanical engineering at Worcester Polytechnic Institute. "Battery companies still hesitate to use recycled material in their batteries."

Keep Reading ↓ Show less

Why Adopt PAM4 Signaling for PCIe 6.0

Learn about the advantages of adopting PAM4 signaling for PCIe® 6.0

1 min read

Data-intensive developments such as AI/ML applications and the enterprise workload shift to the cloud are fueling explosive data traffic growth. To address demands for data bandwidth, silicon providers are racing to adopt the latest high-speed interface technologies such as PCI Express 6.0. Read about the advantages PAM4 signaling brings to PCIe® 6.0.

Trending Stories

The most-read stories on IEEE Spectrum right now