Turn Any Surface Into a Touchscreen

A clever optical trick can help project interactive touchscreens onto any surface

3 min read
A projector-camera system that can display a touchscreen on any surface.

The projector-camera system can display a touchscreen on any surface.

Nara Institute of Science and Technology

This article is part of our exclusive IEEE Journal Watch series in partnership with IEEE Xplore.

Since the advent of the smartphone, the use of touch to interact with digital content has become ubiquitous. So far though, touchscreens have mainly been limited to pocket-sized devices. Now researchers have come up with a low-cost way to turn any surface into a touchscreen, opening a host of new possibilities for interacting with the digital world.

Previous attempts to allow people to manipulate projected images with touch have relied on special input devices, multiple sensors, or image processing algorithms that struggle with cluttered or confusing visual content. The new system only requires a single camera attached just below the projector, and the system's developers say it works well no matter what you want to project.

The key to the approach is a clever optical trick that ensures only movements just above the surface of the projection are detected. This makes it possible to register users fingers as they press buttons, while ignoring everything else in the camera's field of view. Its designers hope the technology could be used to create large, interactive displays almost anywhere.

"You can project whatever you want and make the system interactive for games or for different types of user experiences," says Suren Jayasuriya—an assistant professor at Arizona State University's schools of Arts Media and Engineering and Electrical, Computer and Energy Engineering—who helped design the system alongside colleagues from Nara Institute of Science and Technology (NAIST) in Japan.

Projected touchscreen with fingers touching yes and no blocks. The camera only picks up movements just above the surface of the projection.Nara Institute of Science and Technology

"The infrastructure required is very minimal, its just this one projector camera," Jayasuriya says. "Other systems, to try to get to this level of interactivity, would need either multiple cameras or additional depth sensors."

The approach relies on a laser scanning projector, which rapidly scans up and down, drawing the projected image line by line. This is paired with a rolling shutter camera, which exposes its sensor's pixels one row at a time so that it scans the visual scene line by line, much like the projector.

By synchronizing the two devices, the horizontal plane of light emitted by the projector can be made to intersect with the horizontal plane that the camera is receiving from. And because the two devices are slightly offset, the principle of triangulation lets you calculate the depth of this point where they overlap.

That makes it possible to calibrate the setup so that the camera only picks up light at a specific distance from the projector, which can be set to hover just above the projected image. That means that while the camera can pick up users' fingers as they press on areas of the projected image, it ignores the rest of the visual scene.

In a paper in IEEE Access, the researchers describe how they paired this set-up with a simple image-processing algorithm to track the position of the users' fingers relative to the projected image. This tracking information can then be used as input for any touch-based application.

The system is already fairly simple to set up, says Jayasuriya, but the team is now working on an auto-calibration scheme to make things even easier. It's also relatively low-cost, he adds. The initial prototype took about $500 to build, but that could be reduced significantly if commercialized.

Diagram of a camera and projector turning surface into a touchscreen Touchscreens on phones, tablets and smart devices are at the core of digital life today. But why can't *any* surface be made into a touchscreen? With a projector and single camera, US and Japanese researchers did just that.Nara Institute of Science and Technology

"The idea of combining a scanning laser with a scanning camera to create a virtual sensing plane is very clever," says Chris Harrison, an associate professor at Carnegie Mellon University's Human-Computer Interaction Institute.

The accuracy of the approach is unlikely to match physical touchscreens, he says. But being able to project them on any surface could enable a variety of novel functionalities, like watching an interactive sports match on your coffee table or projecting TV controls onto a nearby surface so you don't need to use a remote. "It is similar to the vision of augmented reality in many ways, but no one has to wear headsets," says Harrison.

The researchers think the technology could be particularly useful in situations where sticky fingers might be a problem, for instance when dealing with young children or while trying to use a touchscreen while cooking. "With this device we can search recipes or look up directions on how to make the dishes without having to wash our hands first," says Takuya Funatomi, an associate professor at NAIST.

At present the device can only register one finger at a time, but enabling multi-touch would simply require the researchers to swap out the image processing algorithm for a smarter one. The team also hopes to enable more complex gesture recognition in future iterations of the device.

This article appears in the November 2021 print issue as "Any Surface Can Be a Touch Screen."

The Conversation (4)
Takafumi Matsumaru 16 Sep, 2021
SM

Our laboratory has already realized a similar system:

https://www.youtube.com/watch?v=9XTLCkm031I

# Lixing Zhang, Takafumi Matsumaru: "Near-field Touch Interface Using Time-of-flight Camera", Journal of Robotics and Mechatronic (Fuji Technology Press) (ISSN: 0915-3942(Print) / 1883-8049(Online)), Vol. 28, No. 5, pp. 759-775, (2016.10.20).

https://www.fujipress.jp/jrm/rb/robot002800050759/

Gregory Bunt 08 Sep, 2021
INDV

MicroVision already has made this and it’s available for sale to OEMs,

https://youtu.be/X_3iyuE7-bI

Gotowls Inmyhouse 08 Sep, 2021
INDV

Microvision already invented this tech years ago...

https://www.youtube.com/watch?v=k7I_q1hCN3c

https://www.youtube.com/watch?v=j1KlJgT-eNY&t=8s

Can This DIY Rocket Program Send an Astronaut to Space?

Copenhagen Suborbitals is crowdfunding its crewed rocket

15 min read
Vertical
Five people stand in front of two tall rockets. Some of the people are wearing space suits and holding helmets, others are holding welding equipment.

Copenhagen Suborbitals volunteers are building a crewed rocket on nights and weekends. The team includes [from left] Mads Stenfatt, Martin Hedegaard Petersen, Jørgen Skyt, Carsten Olsen, and Anna Olsen.

Mads Stenfatt
Red

It was one of the prettiest sights I have ever seen: our homemade rocket floating down from the sky, slowed by a white-and-orange parachute that I had worked on during many nights at the dining room table. The 6.7-meter-tall Nexø II rocket was powered by a bipropellant engine designed and constructed by the Copenhagen Suborbitals team. The engine mixed ethanol and liquid oxygen together to produce a thrust of 5 kilonewtons, and the rocket soared to a height of 6,500 meters. Even more important, it came back down in one piece.

That successful mission in August 2018 was a huge step toward our goal of sending an amateur astronaut to the edge of space aboard one of our DIY rockets. We're now building the Spica rocket to fulfill that mission, and we hope to launch a crewed rocket about 10 years from now.

Copenhagen Suborbitals is the world's only crowdsourced crewed spaceflight program, funded to the tune of almost US $100,000 per year by hundreds of generous donors around the world. Our project is staffed by a motley crew of volunteers who have a wide variety of day jobs. We have plenty of engineers, as well as people like me, a pricing manager with a skydiving hobby. I'm also one of three candidates for the astronaut position.

Keep Reading ↓ Show less