Sign Language by Cellphone

Software tricks will let the deaf sign over smart phones

2 min read

In the past, engineers working on technology to aid the deaf had focused primarily on hearing devices, such as hearing aids and cochlear implants, but recently they’ve been getting into what’s known as deaf technology: applications designed to make the day-to-day lives of the deaf and hearing-impaired easier. Now engineers from the University of Washington, in Seattle, and Cornell University, in Ithaca, N.Y., have taken a big step toward developing a mobile phone that allows real-time conversations in sign language.

Of course, many in the deaf community already use mobile phones to communicate via text messaging and e-mail, but deaf people almost always prefer sign language: It’s faster and more natural, just as speaking is easier than writing for most hearing people. Laptops are getting smaller and more portable, making video chats outside the home possible, but Wiâ¿¿Fi–enabled cellphones would provide even more freedom. When cellphones became capable of video sharing a few years ago, Eve Riskin, Sheila Hemami, and Richard Ladner, all newly minted IEEE Fellows, felt the time seemed right to develop a sign-language-capable phone. ”Today’s world is more connected by cellphones than by any other device,” says the University of Washington’s Ladner, whose parents were deaf.

From the beginning, the researchers knew that their project, which they named mobileASL (for mobile American Sign Language), would be a challenge. The low bandwidth available on wireless networks in the United States forced them into the balancing act between speed and quality that’s familiar to anyone who works with video, but there was an added twist. Most compression algorithms don’t focus on the aspects of video that would make ASL easily understandable, says Riskin, an electrical engineering professor at the University of Washington.

Hemami studies how the human visual system understands video at Cornell University. To help solve the problem, she has been working on integrating an intelligibility metric into the team’s video-compression software that would enable mobileASL phones to maximize comprehension. It accomplishes this, in part, by recognizing which areas of the image need to be in high resolution—such as the signers’ hands and faces—and which areas, such as the signers’ torsos, can be in low resolution.

The team also had to figure out how to preserve the phone’s battery life in the face of the power-draining compression and decompression that conversing by video requires. They tackled this problem by implementing a variable frame-rate system that oscillates between high and low frame rates depending on whether the user is signing or watching the other person sign.

Now, nearly four years after they began, the researchers are finally close to a functional prototype. A few months ago, Riskin and her lab at the University of Washington figured out how to increase the frame rates to more than 10 frames per second, a critical step for making mobile video conversations clear and realistic. The mobile phones they were working with weren’t capable of processing full-size images at that rate, but by sampling only a quarter of the pixels in each frame, the group was able to make the video-compression process about four times as fast. Fortunately, the interpolation feature in the Microsoft Windows Mobile operating system automatically expanded the resulting videos back to full size without a significant decrease in quality.

The team still faces one big challenge, which is finding the best way to get the mobileASL software into the hands of the people who want it. The group wants the application to be as broadly usable as possible. They are testing it over a Wi-Fi connection but are also experimenting with the data services of several wireless carriers.

This article is for IEEE members only. Join IEEE to access our full archive.

Join the world’s largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum’s articles, podcasts, and special reports. Learn more →

If you're already an IEEE member, please sign in to continue reading.

Membership includes:

  • Get unlimited access to IEEE Spectrum content
  • Follow your favorite topics to create a personalized feed of IEEE Spectrum content
  • Save Spectrum articles to read later
  • Network with other technology professionals
  • Establish a professional profile
  • Create a group to share and collaborate on projects
  • Discover IEEE events and activities
  • Join and participate in discussions