The December 2022 issue of IEEE Spectrum is here!

Close bar

New AR System Alters Sight, Sound and Touch

Tactile Echoes allows users to freely interact with any surface

2 min read
tactile echoes application
In the tabletop application, different Tactile Echoes are assigned to each region of the projected surface. The finger is tracked via camera, allowing different echoes to be assigned to different mapped regions on the surface.

There’s the typical pitter-patter sound that comes from a person drumming their fingers along a tabletop. But what if this normal pitter-patter sound was perceived as a series of hollow echoes? Or rolling thunder? A new augmented reality system called Tactile Echoes provides users with experiences like this, and could be used for a wide range of gaming, entertainment and research purposes.

Notably, the system does not require any equipment between the user’s fingertips and the contact surface, meaning users can enjoy the real sensation of their environment along with the visual, haptic and auditory augmented enhancements.

“Tactile Echoes is possibly the first programmable system for haptic augmented reality that allows its users to freely touch physical objects or surfaces augmented with multimodal digital feedback using their hands,” says Anzu Kawazoe, a PhD candidate at the University of California, Santa Barbara who co-designed Tactile Echoes.

It accomplishes this using a sensor that is placed on the top of the user’s fingernail, which detects the vibrations that are naturally produced within the finger as it touches a surface. The vibrational signals are processed and translated into programmed sounds.

Different tactile feedback and sounds can be played with each interaction, because the vibrational patterns in our fingers change depending on what surface we touch, or the intensity of pressure applied. For example, Tactile Echoes may play a light, fun echo when you tap an object lightly, or play a sudden thud if you jab the object with force.

“We were motivated by the idea of being able to almost magically augment any ordinary object or surface, such as a simple wooden table, with lively haptic and effects that playfully respond to how or where we touch,” explains Kawazoe.

Her team took the system one step further by integrating the wearable device with virtual environments created by a smart projector or a VR or AR headset. In this way, users can “touch” virtual objects in their real environment, and experience enhanced graphic, sound and haptic feedback.

The researchers tested Tactile Echoes through a number of user experiments, described in study published May 26 in IEEE Transactions on Haptics. First, study participants were asked to describe different sounds, and what perceptions and associations are evoked with each one. In a second experiment, participants used Tactile Echoes to complete an interactive, augmented reality video game that was projected onto an ordinary desktop surface. Users reported that the Tactile Echoes feedback greatly enhanced the responsiveness, and their level of engagement and agency in playing the game.

While Tactile Echoes is still a prototype, Kawazoe says her team is interested in collaborating with companies to commercialize this tech. “Some of the most promising applications we can envisage include augmented reality games that can be played on any table top, musical devices whose interface can be projected wherever needed, and educational systems for enlivening learning by K-12 students,” she says.

She also notes that her team’s research so far with Tactile Echoes has revealed some interesting perceptual phenomena that occur when haptic feedback is delayed through the system.  In particular, they believe that perceptual masking is happening, whereby the perception of one stimulus affects the perceived intensity of a second stimulus.

“We are thinking that this tactile masking effect is working on the Tactile Echoes system. Specifically, time-delayed tactile feedback is as perceived stronger,” explains Kawazoe. “We are preparing new experiments to investigate these effects, and plan to use the results to further improve the Tactile Echoes system.”

The Conversation (0)

Deep Learning Could Bring the Concert Experience Home

The century-old quest for truly realistic sound production is finally paying off

12 min read
Image containing multiple aspects such as instruments and left and right open hands.
Stuart Bradford

Now that recorded sound has become ubiquitous, we hardly think about it. From our smartphones, smart speakers, TVs, radios, disc players, and car sound systems, it’s an enduring and enjoyable presence in our lives. In 2017, a survey by the polling firm Nielsen suggested that some 90 percent of the U.S. population listens to music regularly and that, on average, they do so 32 hours per week.

Behind this free-flowing pleasure are enormous industries applying technology to the long-standing goal of reproducing sound with the greatest possible realism. From Edison’s phonograph and the horn speakers of the 1880s, successive generations of engineers in pursuit of this ideal invented and exploited countless technologies: triode vacuum tubes, dynamic loudspeakers, magnetic phonograph cartridges, solid-state amplifier circuits in scores of different topologies, electrostatic speakers, optical discs, stereo, and surround sound. And over the past five decades, digital technologies, like audio compression and streaming, have transformed the music industry.

Keep Reading ↓Show less