The December 2022 issue of IEEE Spectrum is here!

Close bar

Augmented Reality and the Surveillance Society

Our future AR gadgets will be monitoring our lives more closely than anything we face now


2 min read
Photo of a person in glasses surrounded by eyeballs look at them.
Illustration: Dan Page

First articulated in a 1965 white paper by Ivan Sutherland, titled “The Ultimate Display," augmented reality (AR) lay beyond our technical capacities for 50 years. That changed when smartphones began providing people with a combination of cheap sensors, powerful processors, and high-bandwidth networking—the trifecta needed for AR to generate its spatial illusions. Among today's emerging technologies, AR stands out as particularly demanding—for computational power, for sensed data, and, I'd argue, for attention to the danger it poses.

Unlike virtual-reality (VR) gear, which creates for the user a completely synthetic experience, AR gear adds to the user's perception of her environment. To do that effectively, AR systems need to know where in space the user is located. VR systems originally used expensive and fragile systems for tracking user movements from the outside in, often requiring external sensors to be set up in the room. But the new generation of VR accomplishes this through a set of techniques collectively known as simultaneous localization and mapping (SLAM). These systems harvest a rich stream of observational data—mostly from cameras affixed to the user's headgear, but sometimes also from sonar, lidar, structured light, and time-of-flight sensors—using those measurements to update a continuously evolving model of the user's spatial environment.

For safety's sake, VR systems must be restricted to certain tightly constrained areas, lest someone blinded by VR goggles tumble down a staircase. AR doesn't hide the real world, though, so people can use it anywhere. That's important because the purpose of AR is to add helpful (or perhaps just entertaining) digital illusions to the user's perceptions. But AR has a second, less appreciated, facet: It also functions as a sophisticated mobile surveillance system.

This second quality is what makes Facebook's recent Project Aria experiment so unnerving. Nearly four years ago, Mark Zuckerberg announced Facebook's goal to create AR “spectacles"—consumer-grade devices that could one day rival the smartphone in utility and ubiquity. That's a substantial technical ask, so Facebook's research team has taken an incremental approach. Project Aria packs the sensors necessary for SLAM within a form factor that resembles a pair of sunglasses. Wearers collect copious amounts of data, which is fed back to Facebook for analysis. This information will presumably help the company to refine the design of an eventual Facebook AR product.

The concern here is obvious: When it comes to market in a few years, these glasses will transform their users into data-gathering minions for Facebook. Tens, then hundreds of millions of these AR spectacles will be mapping the contours of the world, along with all of its people, pets, possessions, and peccadilloes. The prospect of such intensive surveillance at planetary scale poses some tough questions about who will be doing all this watching and why.

To work well, AR must look through our eyes, see the world as we do, and record what it sees. There seems no way to avoid this hard reality of augmented reality. So we need to ask ourselves whether we'd really welcome such pervasive monitoring, why we should trust AR providers not to misuse the information they collect, or how they can earn our trust. Sadly, there's not been a lot of consideration of such questions in our rush to embrace technology's next big thing. But it still remains within our power to decide when we might allow such surveillance—and to permit it only when necessary.

This article appears in the January 2021 print issue as “AR's Prying Eyes."

The Conversation (0)

Why Functional Programming Should Be the Future of Software Development

It’s hard to learn, but your code will produce fewer nasty surprises

11 min read
Vertical
A plate of spaghetti made from code
Shira Inbar
DarkBlue1

You’d expectthe longest and most costly phase in the lifecycle of a software product to be the initial development of the system, when all those great features are first imagined and then created. In fact, the hardest part comes later, during the maintenance phase. That’s when programmers pay the price for the shortcuts they took during development.

So why did they take shortcuts? Maybe they didn’t realize that they were cutting any corners. Only when their code was deployed and exercised by a lot of users did its hidden flaws come to light. And maybe the developers were rushed. Time-to-market pressures would almost guarantee that their software will contain more bugs than it would otherwise.

Keep Reading ↓Show less
{"imageShortcodeIds":["31996907"]}