The July 2022 issue of IEEE Spectrum is here!

Close bar

Augmented Reality and the Surveillance Society

Our future AR gadgets will be monitoring our lives more closely than anything we face now

2 min read
Photo of a person in glasses surrounded by eyeballs look at them.
Illustration: Dan Page

First articulated in a 1965 white paper by Ivan Sutherland, titled “The Ultimate Display," augmented reality (AR) lay beyond our technical capacities for 50 years. That changed when smartphones began providing people with a combination of cheap sensors, powerful processors, and high-bandwidth networking—the trifecta needed for AR to generate its spatial illusions. Among today's emerging technologies, AR stands out as particularly demanding—for computational power, for sensed data, and, I'd argue, for attention to the danger it poses.

Unlike virtual-reality (VR) gear, which creates for the user a completely synthetic experience, AR gear adds to the user's perception of her environment. To do that effectively, AR systems need to know where in space the user is located. VR systems originally used expensive and fragile systems for tracking user movements from the outside in, often requiring external sensors to be set up in the room. But the new generation of VR accomplishes this through a set of techniques collectively known as simultaneous localization and mapping (SLAM). These systems harvest a rich stream of observational data—mostly from cameras affixed to the user's headgear, but sometimes also from sonar, lidar, structured light, and time-of-flight sensors—using those measurements to update a continuously evolving model of the user's spatial environment.

For safety's sake, VR systems must be restricted to certain tightly constrained areas, lest someone blinded by VR goggles tumble down a staircase. AR doesn't hide the real world, though, so people can use it anywhere. That's important because the purpose of AR is to add helpful (or perhaps just entertaining) digital illusions to the user's perceptions. But AR has a second, less appreciated, facet: It also functions as a sophisticated mobile surveillance system.

This second quality is what makes Facebook's recent Project Aria experiment so unnerving. Nearly four years ago, Mark Zuckerberg announced Facebook's goal to create AR “spectacles"—consumer-grade devices that could one day rival the smartphone in utility and ubiquity. That's a substantial technical ask, so Facebook's research team has taken an incremental approach. Project Aria packs the sensors necessary for SLAM within a form factor that resembles a pair of sunglasses. Wearers collect copious amounts of data, which is fed back to Facebook for analysis. This information will presumably help the company to refine the design of an eventual Facebook AR product.

The concern here is obvious: When it comes to market in a few years, these glasses will transform their users into data-gathering minions for Facebook. Tens, then hundreds of millions of these AR spectacles will be mapping the contours of the world, along with all of its people, pets, possessions, and peccadilloes. The prospect of such intensive surveillance at planetary scale poses some tough questions about who will be doing all this watching and why.

To work well, AR must look through our eyes, see the world as we do, and record what it sees. There seems no way to avoid this hard reality of augmented reality. So we need to ask ourselves whether we'd really welcome such pervasive monitoring, why we should trust AR providers not to misuse the information they collect, or how they can earn our trust. Sadly, there's not been a lot of consideration of such questions in our rush to embrace technology's next big thing. But it still remains within our power to decide when we might allow such surveillance—and to permit it only when necessary.

This article appears in the January 2021 print issue as “AR's Prying Eyes."

The Conversation (0)

The Future of Deep Learning Is Photonic

Computing with light could slash the energy needs of neural networks

10 min read
Image of a computer rendering.

This computer rendering depicts the pattern on a photonic chip that the author and his colleagues have devised for performing neural-network calculations using light.

Alexander Sludds

Think of the many tasks to which computers are being applied that in the not-so-distant past required human intuition. Computers routinely identify objects in images, transcribe speech, translate between languages, diagnose medical conditions, play complex games, and drive cars.

The technique that has empowered these stunning developments is called deep learning, a term that refers to mathematical models known as artificial neural networks. Deep learning is a subfield of machine learning, a branch of computer science based on fitting complex models to data.

Keep Reading ↓Show less