The December 2022 issue of IEEE Spectrum is here!

Close bar

AI System Warns Pedestrians Wearing Headphones About Passing Cars

A safety system for pedestrians detects nearby cars based on the sounds they make

4 min read
A man wears headphones and look at a screen in front of him while a car passes on a nearby street.
The Pedestrian Audio Warning System is designed to alert headphone wearers to passing vehicles.
Photo: Electrical Engineering and Data Science Institute/Columbia University

How can headphone-wearing pedestrians tune out the chaotic world around them without compromising their own safety? One solution may come from the pedestrian equivalent of a vehicle collision warning system that aims to detect nearby vehicles based purely on sound.

The intelligent headphone system uses machine learning algorithms to interpret sounds and alert pedestrians to the location of vehicles up to 60 meters away. A prototype of the Pedestrian Audio Wearable System (PAWS) can only detect the location but not the trajectory of a nearby vehicle—never mind the locations or trajectories of multiple vehicles. Still, it’s a first step for a possible pedestrian-centered safety aid at a time when the number of pedestrians killed on U.S. roads reached a three-decade high in 2018.

“Sometimes the newer vehicles have sensors that can tell if there are pedestrians, but pedestrians usually don’t have a way to tell if vehicles are on a collision trajectory,” says Xiaofan Jiang, an assistant professor of electrical engineering and member of the Data Science Institute at Columbia University.

The idea first came to Jiang when he noticed that a new pair of noise-cancelling headphones was distracting him more than usual from his surroundings during a walk to work. That insight spurred Jiang and his colleagues at Columbia, the University of North Carolina at Chapel Hill, and Barnard College to develop PAWS and publish their work in the October 2019 issue of the IEEE Internet of Things Journal.

An assortment of images shows a man wearing headphones next to a busy street and a close-up of the circuit board of a pedestrian warning system.The Pedestrian Audio Wearable System detects nearby cars by using microphones and machine learning algorithms to analyze vehicle sounds.Photo: Electrical Engineering and Data Science Institute/Columbia University

Many cars with collision warning systems rely upon visual cameras, radar, or lidar to detect nearby objects. But Jiang and his colleagues soon realized that a pedestrian-focused system would need a low-power sensor that could operate for more than six hours on standard batteries. “So we decided to go with an array of microphones, which are very inexpensive and low-power sensors,” Jiang says.

The array of four microphones is located in different parts of the headphone. But the wearable warning system’s main hardware is designed to fit inside the left ear housing of commercial headphones and draws power from a rechargeable lithium-ion battery. A custom integrated circuit saves on power by only extracting the most relevant sound features from the captured audio and transmitting that information to a paired smartphone app.

The smartphone hosts the machine learning algorithms that were trained on audio from 60 different types of vehicles in a variety of environments: a street adjacent to a university campus and residential area, the side of a windy highway during hurricane season, and the busy streets of Manhattan.

However, relying purely on sound to detect vehicles has proven tricky. For one thing, the system tends to focus on localizing the loudest vehicle, which may not be the vehicle closest to the pedestrian.The system also still has trouble locating multiple vehicles or even estimating how many vehicles are present.

A pair of headphones rest on a table with one earpiece opened to show the circuitry inside.The hardware for the Pedestrian Audio Wearable System can fit inside the ear housing of commercial headphones.Photo: Electrical Engineering and Data Science Institute/Columbia University

As it stands, the PAWS capability to localize a vehicle up to 60 meters away might provide at least several seconds of warning depending on the speed of an oncoming vehicle. But a truly useful warning system would also be able to track the trajectory of a nearby vehicle and only provide a warning if it’s on course to potentially hit the pedestrian. That may require the researchers to figure out better ways to track both the pedestrian’s location and trajectory along with the same information for vehicles.

“If you imagine one person walking along the street, many cars may pass by but none will hit the person,” Jiang explains. “We have to take into account other information to make this collision detection more useful.”

More work continues on how the system would use noises or other signals to alert headphone wearers. Joshua New, a behavioral psychologist at Barnard College, plans to conduct experiments to see what warning cue works best to give people a heads up. For now, the team is leaning toward either providing a warning beep on one side of a stereo headphone or possibly simulating 3D warning sounds to provide more spatially-relevant information.

Beyond ordinary pedestrians, police officers performing a traffic stop on a busy road or construction workers wearing ear protection might also benefit from such technology, Jiang says. The PAWS project has already received US $1.3 million from the National Science Foundation, and the team has an eye on eventually handing a more refined version of the technology over to a company to commercialize it.

Of course, one technology will not solve the challenges of pedestrian safety. In its 2019 report, the Governors Highway Safety Association blamed higher numbers of pedestrian deaths on many factors such as a lack of safe road crossings, and generally unsafe driving by speeding, distracted, or drunk drivers. A headphone equipped with PAWS is unlikely to prevent even a majority of pedestrian deaths—but a few seconds’ warning might help spare some lives.

This post was updated on 8 January 2020. 

A version of this post appears in the February 2020 print issue as “Smart Headphones Warn of Nearby Cars.”

The Conversation (0)

Chinese Joint Venture Will Begin Mass-Producing an Autonomous Electric Car

With the Robo-01, Baidu and Chinese carmaker Geely aim for a fully self-driving car

4 min read
A black car sits against a white backdrop decorated with Chinese writing. The car’s doors are open, like a butterfly’s wings. Two charging stations are on the car’s left; two men stand on the right.

The Robo-01 autonomous electric car shows off its butterfly doors at a reveal to the media in Beijing, in June 2022.

Tingshu Wang/Reuters/Alamy
Purple

In October, a startup called Jidu Automotive, backed by Chinese AI giant Baidu and Chinese carmaker Geely, officially released an autonomous electric car, the Robo-01 Lunar Edition. In 2023, the car will go on sale.

At roughly US $55,000, the Robo-01 Lunar Edition is a limited edition, cobranded with China’s Lunar Exploration Project. It has two lidars, a 5-millimeter-range radar, 12 ultrasonic sensors, and 12 high-definition cameras. It is the first vehicle to offer on-board, AI-assisted voice recognition, with voice response speeds within 700 milliseconds, thanks to the Qualcomm Snapdragon 8295 chip.

Keep Reading ↓Show less