Making Driverless Cars More Expressive

Judging whether it’s safe to cross the open road involves a complex exchange of social cues between pedestrian and driver. But what if there’s no one behind the wheel? As self-driving cars become more common, helping them communicate with human road users is crucial. Autonomous vehicle company Motional thinks making the vehicles more expressive could be the key.

When he’s waiting at a crosswalk, Paul Schmitt, chief engineer at Motional, engages in what he calls the “glance dance”—a rapid and almost subconscious assessment of where an oncoming driver is looking and whether they’re aware of him. “With automated vehicles, half of that interaction no longer exists,” he says. “So what cues are then available for the pedestrian to understand the vehicles’ intentions?”

To answer that question, his team hired animation studio CHRLX to create a highly realistic virtual reality experience designed to test pedestrian reactions to a variety of different signaling schemes. Reporting their results inIEEE Robotics and Automation Letters, they showed that exaggerating the car’s motions—by braking earlier or stopping well short of the pedestrian—was the most effective way to communicate its intentions.

The company is now in the process of integrating the most promising expressive behaviors into its motion-planning systems, and it has also open-sourced the VR environment so other groups can experiment. Getting these kinds of interactions right will be essential for building trust in self-driving cars, says Schmitt, as this is likely to be most people’s first encounter with the technology.

Self-driving car behaviors included having the car brake earlier and harder than the baseline, stopping the car a vehicle’s length away, adding exaggerated braking and low-revving sounds, and finally combining these sounds with an exaggerated dipping of the nose of the car as if it was braking hard.

“That motivates a lot of the work that we’re doing, to ensure that those first interactions go well,” he says. “We want to make sure that people feel comfortable with this new technology.”

The study carried out by Motional saw 53 participants don a VR headset that transported them to the corner of a four-way intersection in an urban area. Each participant faced 33 trials of a car approaching the intersection as they tried to cross, with the vehicle exhibiting a variety of different behaviors and appearances. While they were able to look around, they could not move and instead had to indicate when they felt it was safe to cross by pressing a button on a handheld controller.

Three baseline scenarios mimicked the way a human driver would come to a halt at a stop sign, but one featured a human driver behind the wheel, another had no driver and conspicuous sensors dotted around the car, and the third featured a large LED display that indicated when the vehicle was yielding—an approach popular among makers of driverless cars.

The company then designed various expressive behaviors meant to implicitly signal to the pedestrian that the car is stopping for them. These included having the car brake earlier and harder than the baseline, stopping the car a vehicle’s length away, adding exaggerated braking and low-revving sounds, and finally combining these sounds with an exaggerated dipping of the nose of the car as if it was braking hard.

To keep the participants honest, the team also included a control scenario where the car didn’t stop, and Schmitt says their reactions were testimony to the realism of the simulation. “I literally had people in our VR lab on the third floor of this office building raise the middle finger at a virtual car that just cut them off,” he says.

The team then measured how quickly participants decided to cross and also gave them a quick survey after each trial to find out how safe they felt, how confident they were of their decision to cross, and how clearly they understood a car’s intention. Both early, hard braking, and stopping short led to a considerably higher proportion of participants crossing the street before the vehicle had come to a complete stop. But in the surveys, stopping short elicited the highest ratings for sense of safety, decision confidence, and intention understanding.

The fact that stopping short elicited the best response isn’t surprising, says Schmitt, as this approach was inspired by the way human drivers behave when slowing down for pedestrians. What was more surprising was that there was little difference in reactions between the baseline scenarios with and without a driver, which suggests pedestrians are paying more attention to the movement of the vehicle than to the driver behind the wheel, he adds.

That’s backed up by other research, says Wilbert Tabone, a doctoral student at Delft University of Technology in the Netherlands who works on robot-human interaction. While most attempts to solve this problem have focused on displays that stand in for explicit cues like eye contact or hand gestures, he says studies keep showing that the implicit behavior of the car is what most people are looking out for.

Nonetheless, he thinks a combination of explicit and implicit signaling will ultimately be the most effective. One promising avenue is augmented reality, and he has developed a system that would allow driverless vehicles to communicate their intention directly to a pedestrian’s smart glasses, which would then indicate visually whether or not its safe to cross. The downside, he admits, is that it first requires widespread adoption of smart glasses, which is no sure thing.

One potential challenge for integrating these expressive behaviors could be driver acceptance, says Catherine Burns, a professor in systems design engineering at the University of Waterloo in Canada. “To what degree would people purchase a car that made exaggerated sounds? Or compressed its suspension to make an expressive nose dive?” she writes in an email.

Nonetheless, the study opens up an interesting new line of research, she says, and shows that making self-driving cars more expressive could significantly improve their interactions with pedestrians.

This article appears in the March 2022 print issue as “Can Autonomous Cars Show Pedestrians Their Intentions?.”