Meta’s Flamera Has a New Vision for Augmented Reality

Meta’s latest prototype headset, Flamera, looks ripped straight from a sci-fi action flick—and it quickly turned heads at Siggraph 2023, where Flamera won the coveted Best in Show award.

The Flamera headset, bug-eyed and glowing ominous red, is an intimidating piece of kit that looks radically different from its peers. And the look is not just for show. The headset debuts a new lens design that could solve augmented reality’s most pressing problem: “passthrough,” or the use of external cameras to provide a headset user with a view of the world around them.

“I didn’t feel the ‘swimming’ quality of the Meta Quest Pro,” says Anshel Sag, vice president and principal analyst at Moor Insights & Strategy. “It was definitely among the best quality [augmented reality] passthrough I’ve ever seen.”

Giving Reality the Bug Eye

For obvious reasons, it’s physically impossible to place a headset’s cameras in precisely the same location as a user’s eyes. That displacement leads to a shift in perspective I’ve personally experienced: I’ve bumped into walls and tripped over chairs while using AR/VR headsets. Cutting-edge headsets, like the Meta Quest Pro, leap that hurdle by reprojecting the view of your surroundings from the correct perspective, but the solution can lead to visual distortion—the “swimming” Sag mentioned.

Meta’s bug-eyed Flamera presents a novel solution. It ditches the array of external cameras favored by current headsets for a unique “lightfield passthrough” design that pairs image sensors with apertures that physically control the light reaching the sensors. Light that would contribute an incorrect perspective is blocked, while light that provides an accurate perspective is allowed to reach the sensors.

Meta Flamera: Reprojection-Free Light Field Passthrough OverviewMeta

The result is strange when viewed directly through the lens: It’s a bit like watching the world through holes poked through paper. The headset rearranges the raw image to remove gaps and reposition the sensor data. Once the process is complete, the headset delivers an accurate view of the world around the user.

“It was definitely a prototype, but I was impressed by the image quality and the resolution,” says Sag. “And the frame rate was good.” Edgar Maucourant, Director of Software Development at ModiFace, also demoed the headset and was similarly impressed. “There was no latency and no differences between what my eyes would see and where my hand was positioned, versus where my hand was really located.”

Maucourant thinks Flamera’s accuracy could be a boon for AR applications in which users directly interact with the world around them. “If we thought about remote assistance, for example, where people have to manipulate objects, today it’s done with AR glasses like HoloLens and Magic Leap.... We can imagine using AR passthrough for that.”

Meta’s Answer to the Apple Vision Pro?

The comparison to Microsoft’s HoloLens and Magic Leap is important. They bypass the perspective problem with a transparent display that leaves a user’s vision unobstructed. That approach felt like the future when HoloLens was released in 2016, but issues remain with its display quality, field of view, and brightness.

Meta’s Quest Pro and Apple’s upcoming Vision Pro underscore this shift in direction. Although marketed as AR/VR headsets (or, in Apple’s case, as “spatial computers”), they’re clearly a branch of virtual reality’s family tree. They fully obstruct the user’s field of view with an opaque display. Augmented reality is provided through a video feed that projects the outside world through to the display.

The Vision Pro’s solution to the passthrough problem favors brawn over brains. It uses an array of external cameras similarly to other AR/VR headsets but pairs them with Apple’s custom R1 chip, which operates in tandem with an Apple M2 chip (like that found in the company’s laptops). The R1 is a “vision processor” that helps the Vision Pro’s 12 cameras computationally correct the perspective (among other tasks). It’s much like the approach Meta tried with the Quest Pro, but Apple cranks the power up to 11.

Flamera: Through the LensMeta

Meta’s Flamera tosses the computational approach in the bin and instead corrects the perspective with its lenses. This drastically lowers the raw compute power required for accurate passthrough AR.

But that’s not to say Meta has cracked passthrough AR wide open. Flamera’s technology is a long way from shipping in a headset available for purchase, and currently suffers some drawbacks relative to traditional passthrough AR. Sag says the headset had “a fairly limited field of view” and noticed its depth of field “wasn’t continuous,” meaning distant objects seem closer than they really are. Maucourant warned that “the color was not great” and felt the headset’s resolution was lackluster.

Still, Flamera shows the shape of the coming conflict between Meta, Apple, and others looking to enter the AR arena. The biggest names in tech seem convinced tomorrow’s best AR/VR headsets will look more like the original Oculus Rift than like Microsoft’s HoloLens—but their top engineers still have a few surprises up their sleeves.