Creating Text Out of Thin Air

New approach allows smartwatch users to write text in mid-air with their fingers

3 min read
hand writing in the air

One possible text entry scenario for smartwatches: handwriting in the air without rotating the user’s wrist.

Wei Dong et al

This article is part of our exclusive IEEE Journal Watch series in partnership with IEEE Xplore.

Smartwatches are growing in popularity thanks to their compact size and wide-ranging abilities, useful for anything from texting to tracking fitness. Yet the convenience of having a smartwatch close on hand (literally) comes with a caveat—namely, that only one hand can be used to input text on smartwatches.

But, one group of researchers in China are pointing to a novel solution, whereby smartwatch users could simply use their finger to write their texts in mid-air in front of them, with no need of a surface. The researchers developed and tested their proposed approach, called AirText, with several volunteers and describe it in a study published November 23 in IEEE Transactions on Mobile Computing.

While speech-to-text input may seem like an obvious solution to the issue of inputting text into smartwatches, relying on audio input involves a number of problems. For example, speech-to-text approaches are not ideal when users are in environments with a lot of background noise, or when users wish to keep the content of their message private from other people within earshot. At the same time, inputting text with just one hand can be slow and cumbersome.

Wei Dong is a professor at the College of Computer Science at Zhejiang University in China. Dong and his colleagues envisioned AirText as a simpler approach and sought to create the app.

“The goal of AirText is to infer the texts written by the fingertip in the air, using only the [inertial measurement unit (IMU)] readings, for example the accelerometers, gyroscopes, and magnetometer, from the smartwatch on the wrist as input,” explains Dong.

However, a major challenge in developing AirText lay in understanding how the user’s wrist movements correspond with the movement of their fingertips as they spell out letters. “[When you are writing] a character in the air using a fingertip, the movements of the wrist and the fingertip are not necessarily the same,” notes Dong. “In fact, as we show in the study, they are quite different.”

To overcome this issue, Dong and his colleagues used a program called Leap Motion, which is able to track the movement of fingers using infrared sensors. Eight volunteers spelled out more than 25,000 characters using five different kinds of smartwatches, while Leap Motion collected data on their wrist and hand movements. The data was then fed into an AI model to infer the relationship between users’ wrist movements and the characters they are spelling out with their fingertips.

The results show that AirText can be effective for writing text in the air, regardless of the type of smartwatch worn or the unique writing style of the user. The volunteers used AirText to achieve an average typing speed of 8.1 words per minute, and their average word error rate ranged from 3.6 percent to 11.2 percent.

table of letters, the hand-gestures necessary to generate those letters both on a surface and in the air (with both non-rotating wrist and rotating wrist)The trajectories of a smartwatch in the three different scenarios when writing four different characters in the air. We see that the trajectories in the rotating wrist scenario are completely different from the written characters. Wei Dong et al

To speed up the writing process, AirText can try to predict the word that a user is trying to spell, just like current word prediction software programs on smartphones. Users can tilt the watch to the right or left to select a suggested word, or shake their watch to indicate a backspace.

One limitation, however, is that AirText users must do a short pause in between spelling out individual characters. “This approach slows down the input speed,” says Dong, noting that his team is exploring ways to eliminate the need for this pause.

Dong notes that his team is also interested in commercializing AirText at some point in the future. He says, “We will talk to smartwatch manufactures and smartwatch application developers to see how to apply out technologies to their products.”

The Conversation (0)

Deep Learning Could Bring the Concert Experience Home

The century-old quest for truly realistic sound production is finally paying off

12 min read
Image containing multiple aspects such as instruments and left and right open hands.
Stuart Bradford

Now that recorded sound has become ubiquitous, we hardly think about it. From our smartphones, smart speakers, TVs, radios, disc players, and car sound systems, it’s an enduring and enjoyable presence in our lives. In 2017, a survey by the polling firm Nielsen suggested that some 90 percent of the U.S. population listens to music regularly and that, on average, they do so 32 hours per week.

Behind this free-flowing pleasure are enormous industries applying technology to the long-standing goal of reproducing sound with the greatest possible realism. From Edison’s phonograph and the horn speakers of the 1880s, successive generations of engineers in pursuit of this ideal invented and exploited countless technologies: triode vacuum tubes, dynamic loudspeakers, magnetic phonograph cartridges, solid-state amplifier circuits in scores of different topologies, electrostatic speakers, optical discs, stereo, and surround sound. And over the past five decades, digital technologies, like audio compression and streaming, have transformed the music industry.

Keep Reading ↓Show less