Augmented Reality and the Surveillance Society

First articulated in a 1965 white paper by Ivan Sutherland, titled “The Ultimate Display," augmented reality (AR) lay beyond our technical capacities for 50 years. That changed when smartphones began providing people with a combination of cheap sensors, powerful processors, and high-bandwidth networking—the trifecta needed for AR to generate its spatial illusions. Among today's emerging technologies, AR stands out as particularly demanding—for computational power, for sensed data, and, I'd argue, for attention to the danger it poses.

Unlike virtual-reality (VR) gear, which creates for the user a completely synthetic experience, AR gear adds to the user's perception of her environment. To do that effectively, AR systems need to know where in space the user is located. VR systems originally used expensive and fragile systems for tracking user movements from the outside in, often requiring external sensors to be set up in the room. But the new generation of VR accomplishes this through a set of techniques collectively known as simultaneous localization and mapping (SLAM). These systems harvest a rich stream of observational data—mostly from cameras affixed to the user's headgear, but sometimes also from sonar, lidar, structured light, and time-of-flight sensors—using those measurements to update a continuously evolving model of the user's spatial environment.

For safety's sake, VR systems must be restricted to certain tightly constrained areas, lest someone blinded by VR goggles tumble down a staircase. AR doesn't hide the real world, though, so people can use it anywhere. That's important because the purpose of AR is to add helpful (or perhaps just entertaining) digital illusions to the user's perceptions. But AR has a second, less appreciated, facet: It also functions as a sophisticated mobile surveillance system.

This second quality is what makes Facebook's recent Project Aria experiment so unnerving. Nearly four years ago, Mark Zuckerberg announced Facebook's goal to create AR “spectacles"—consumer-grade devices that could one day rival the smartphone in utility and ubiquity. That's a substantial technical ask, so Facebook's research team has taken an incremental approach. Project Aria packs the sensors necessary for SLAM within a form factor that resembles a pair of sunglasses. Wearers collect copious amounts of data, which is fed back to Facebook for analysis. This information will presumably help the company to refine the design of an eventual Facebook AR product.

The concern here is obvious: When it comes to market in a few years, these glasses will transform their users into data-gathering minions for Facebook. Tens, then hundreds of millions of these AR spectacles will be mapping the contours of the world, along with all of its people, pets, possessions, and peccadilloes. The prospect of such intensive surveillance at planetary scale poses some tough questions about who will be doing all this watching and why.

To work well, AR must look through our eyes, see the world as we do, and record what it sees. There seems no way to avoid this hard reality of augmented reality. So we need to ask ourselves whether we'd really welcome such pervasive monitoring, why we should trust AR providers not to misuse the information they collect, or how they can earn our trust. Sadly, there's not been a lot of consideration of such questions in our rush to embrace technology's next big thing. But it still remains within our power to decide when we might allow such surveillance—and to permit it only when necessary.

This article appears in the January 2021 print issue as “AR's Prying Eyes."