VR Headsets Are Approaching the Eye’s Resolution Limits

The Chinese consumer electronics company TCL Technology recently unveiled a monstrous, 163-inch 4K Micro-LED television that one home theater expert described as “tall as Darth Vader.” Each of the TV’s 8.3 million pixels is an independent, miniscule LED, a feat for which TCL charges over $100,000.

But here’s the real surprise: TCL’s new TV isn’t the most pixel-dense or exotic display ever produced. That honor goes to the emerging frontier of Micro-OLED and Micro-LED displays built for AR/VR headsets. Mojo Vision, a leader in micro-LED displays, recently demonstrated a full-color Micro-LED display frontplane with a density of 5,510 pixels per centimeter (14,000 pixels per inch) at CES 2024. That display, if blown up to the size of TCL’s television, would pack over 220 billion pixels.

“Why so many pixels? Well, who wouldn’t want more pixels? Now the question becomes, how do you do it?” —Patrick Wyatt, Varjo

Pixel densities that high may seem absurd, but absurd density is key for the next generation of augmented, mixed, and virtual reality headsets. Stuffing more pixels inside each centimeter allows not only lifelike visuals but also smaller, more compact displays that achieve a necessary level of visual resolution. But building displays at this scale isn’t easy, and it leads to unique technical hurdles the AR/VR industry is still learning to leap.

“Why so many pixels? Well, who wouldn’t want more pixels?” asks Patrick Wyatt, Chief Product Officer at VR headset maker Varjo. “Now the question becomes, how do you do it?”

More pixels, more problems

Resolution remains a focus for the latest fully immersive AR/VR headsets. Apple’s Vision Pro, which has two displays packing over 23 million pixels, is undoubtedly the most well-known example, but it’s far from alone. Varjo’s XR4 has two 3,840 x 3,744 displays and Pimax, a company that has long pushed the envelope on pixel count, is working on a headset with two 6K QLED mini-LED displays.

Nordic Ren, CEO of Pimax, says the goal is not total pixel count but instead pixels-per-degree (PPD), a measurement of the pixels in each degree of the user’s field of vision. “PPD is pivotal in defining visual clarity,” says Ren. “Even for industry front-runners like Pimax and Apple, the immersion quality still isn’t optimal. Every incremental enhancement to resolution elevates the experience.”

High-resolution displays must be driven by hardware capable of rendering to them at a high refresh rate. Apple’s Vision Pro effectively sticks a MacBook Air on the user’s face.

Even so, the latest headsets with the best display technology are close to achieving what Wyatt calls the “human resolution bar”—the limit past which a person with 20/20 vision will no longer see an improvement. Future displays will eventually exceed it.

Yet achieving a crystal-clear image is just a quarter of the battle. High-resolution displays must be driven by hardware capable of rendering to them at a high refresh rate. Pimax and Varjo headsets tackle this by connecting to a desktop PC with a dedicated graphics card, while Apple’s Vision Pro effectively sticks a MacBook Air on the user’s face.

Both approaches have problems. A desktop PC is powerful and adaptable, but typically requires a wired connection (though wireless add-on are sometimes available). The Vision Pro is wireless out of the box, but it’s somewhat heavy, struggles with meager battery life which, and can’t match the fidelity of Varjo or Pimax headsets.

“The shortfall in raw graphical computing power remains a challenge, and it’s likely to deepen with time,” observes Ren.

High resolutions are demanding, but it’s not without a fix

Fortunately, solutions are on the horizon.

Foveated rendering offers the most immediate fix. This technique uses the human eye’s limited peripheral vision to its advantage. Only the portion of the display a user is focused on is rendered at its full resolution—the rest is reduced. The user should never notice, as the fidelity of our peripheral vision is much lower than the center of our focus.

It’s not a new idea: I first encountered it while touring Microsoft’s R&D labs in 2013. However, it’s only recently become practical to implement in mass-produced headsets. Foveated rendering works best alongside eye-tracking cameras that can precisely measure where a user is looking and adjust the display resolution accordingly, a task that requires both excellent cameras and processing power to analyze the data with minimal latency. That’s why the Vision Pro’s M1 chip is aided by a second, the R1, which is dedicated to handling the headset’s many sensors.

“Our quantum dots are engineered from scratch for this application. They’re not TV quantum dots.” —Nikhil Balram, Mojo Vision

Only a handful of competitors, including the Varjo XR-4 Focal Edition, Pimax Crystal, and Oculus Quest Pro, support foveated rendering. And there’s still plenty of room for improvement. I’ve noticed the Vision Pro’s implementation is sometimes delayed or inaccurate when my eyes dart quickly across the display. Using better cameras, or dedicating more processing power to them, can help with this issue. But it also adds cost, weight, and complexity—which engineers must solve for in future headsets.

Ren believes artificial intelligence is another promising option. “Intelligent frame-interpolation algorithms are increasingly important,” he says. “And the algorithms themselves are evolving swiftly.” AI denoising, which removes grain or noise from an image, may prove vital not just for visual fidelity in virtual reality but also for improved image quality and performance of mixed-reality video captured by a headset’s cameras.

If this sounds familiar, there’s good reason for it. Nvidia uses frame interpolation to add synthetic frames to real-time rendering and enhance the quality of ray-traced reflections and shows, techniques that can improve the performance of computer graphics five-fold (or more). There’s still more work to be done here, as Nvidia has yet to officially offer these techniques for VR, but Wyatt thinks it’s just a matter of time.

“I don’t see why it wouldn’t work,” says Wyatt. “And the other exciting thing is the up sampling you can do through neural accelerators on the hardware in the headset itself. We could have a headset with built-in hardware for [AI].”

The smallest displays face big obstacles

Improved visual fidelity is the most obvious reason to boost pixel density, but companies looking to build lightweight AR glasses need low power consumption in a small package just as badly as enhanced resolution.Micro-LED is the only viable option here; other technologies can’t compete on pixel density, brightness, and power. And while this implies that Micro-LED will look sharper and more vibrant, the technology’s incredible pixel density also provides flexibility. If 14,000 ppi is possible, then so too is 10,000, or 5,000. Similarly, if it’s possible to build a micro-LED that’s just 450 microns across (the size of Mojo Vision’s latest), it should also prove possible to manufacture larger micro-LEDs for different applications.

The challenge is in building them at all. Micro-LCD and micro-OLED displays are relatively mature technologies that manufacturers can build at scale. Micro-LED isn’t there yet. Mojo Vision’s research remains focused on fundamental problems that are just now being addressed.

“What we had said was that [others'] quantum dots were not reliable enough. The amount of emission degrades rapidly, which is kind of what happens with OLED,” says Mojo Vision CEO Nikhil Balram. “Our quantum dots are engineered from scratch for this application. They’re not TV quantum dots. So, while [others] expect the light level hitting the quantum dot to be about four milliwatts per centimeter, squared, we’re hitting it with four watts.”

And Mojo Vision’s decision to pursue quantum dots underscores other problems faced by micro-LED. The company to this approach to side-step other fundamental problems, like the difficulty of obtaining the desired shade of red from a microscopic LED.

“The problem is that Indium is a big atom,” says Balram, referencing the use of Indium Gallium Nitride (InGaN), a common semiconductor material used for LEDs. “How do you control the amount of Indium from one 1.3 or 1.2-micron emitter, to the next? It’s really hard.” Mojo Vision instead uses quantum dots to convert blue light to red.

Challenges like these remain an obstacle for wide-spread adoption, but there’s no shortage of effort spent on finding a solution. Mojo Vision is joined by a range of other companies. Jade Bird Display showed its first full-color micro-LED displays in 2023, though they are currently limited to a resolution of 640 x 480. And Vuzix, a leader in lightweight AR glasses, has partnered with micro-LED firm Atomistic to design a 2K x 2K micro-LED display for next-gen headsets.

Still, advancements like Mojo Vision’s quantum-dot micro-LED frontplane show the technology is viable and can achieve pixel densities that would have seemed impossible a decade ago. They imply a future where pixel density is no longer a limitation for any display—though it may still take years, even decades, to fully come to fruition.

“Getting the breakthrough was important,” says Balram. “Now we can really optimize, optimize, optimize. But it already looks quite nice.”