Autonomous Vehicles Need In-Cabin Cameras to Monitor Drivers

In-cabin camera monitors driver in self-driving car
Image: EDGE3 Technologies
Self-driving cars should adopt cameras and other technologies to monitor drivers' attention, behaviors, and cognitive load to know when it is safe to hand over control.

When Elon Musk unveiled his “Master Plan” for Tesla on the company’s blog, he argued for the electric car’s controversial Autopilot mode in stark ethical terms. It would be “morally reprehensible,” he said, to scale back or disable Tesla’s partially autonomous driving feature because, on balance, Autopilot still saves lives.

There is no doubt that Autopilot and other similar driver-assistance technologies improve safety. But as CEO of EDGE3 Technologies, a vision company developing driver-monitoring systems for vehicles, and as a former professor and head of the Machine Vision Lab at Embry-Riddle Aeronautical University, my experience suggests something else too. Namely, in the rush to achieve fully autonomous driving, we may be side-stepping the proper technology development path and overlooking essential technologies needed to help us get there.

Tesla’s Autopilot, although a great pioneering effort, is in fact a driver-assist feature, and not quite the fully autonomous capability we all dream of. In technical terms, it is a NHTSA-Level 2 autopilot system, defined as “automation of at least two primary control functions.” Such systems require you to keep your hands on the steering wheel at all times. In August, Tesla removed the Chinese words “autopilot” and “self-driving” from its China website—on the heels of an accident in Beijing in which the driver alleged Tesla misrepresented its cars’ capabilities.

There is a clear disconnect between drivers’ expectations and what the reality of Autopilot today looks like. The updated “hands on the wheel” requirement and notification that Tesla recently released does not equate to eyes on the road. Many Tesla and other car owners may wander off visually or mentally, even with their hands on the wheel. They instead hope or believe that their Autopilot enables limited self-driving (Level 3), or even full autonomous driving (Level 4), when in reality they are driving a Level 2 system.

Vehicle owners can surely be forgiven for not knowing the level of automation that their vehicles are equipped with. Since it is possible to let a vehicle drive itself on a highway for hundreds of miles, it is easy to understand how a driver may be lulled into a false sense of security believing the car is fully autonomous. This is problematic. We are starting to see the results with fender benders, and at least one deadly crash earlier this year caused when the driver was not actively engaged in watching the road. Joshua Brown’s Tesla was traveling down the freeway in Autopilot mode with no way for him to be alerted as to how dangerous the situation was. It was arguably an avoidable tragedy, had the vehicle known that Brown was watching a movie on a DVD player, as some reports have suggested.

So how do we get to the next stage of automation—that is, limited self-driving, or Level 3 automation?

NHTSA instructs that, for Level 3, vehicles have to be intelligently aware of their surroundings, understand when a problem is going to occur, and know when/how to cede control of the vehicle to the driver. Cars are certainly getting better at seeing and understanding everything around them, but they are still blind to the one factor that a limited self-driving system needs the most in order to know when/how to cede control: the driver. 

What is missing is technology inside the cockpit that not only ensures a driver is available, but also that the vehicle is aware of the cognitive state of the driver and his or her ability to take control of the vehicle. The only way to enable such a feature is through the use of cameras, pointing inward at the driver.

Measuring the driver’s state of awareness can be very nuanced and often challenging. It is not just about monitoring where the driver is looking at a given point in time, but also about identifying and measuring the driver’s state of awareness, or cognitive load. Systems, currently under development by various automotive equipment manufacturers—Tier 1 and Tier 2 suppliers, including EDGE3—use a variety of hardware technologies and designs, but share one commonality: one or more in-cabin cameras that are monitoring drivers and where they are looking.

Images are then processed by on-board embedded processors to extract critical information for a driver-handoff. A dominant design for driver monitoring, to be integrated to Level-3 autonomous driving, has yet to emerge, with a variety of techniques vying for production, including stereo cameras and time-of-flight cameras. Most solutions that will be first to market seem to focus solely on tracking the eyes, as Harman and Delphi showcased at CES earlier this year.

Although monitoring the eyes is useful, zooming out to monitor the entire face, head position, and facial expressions, as well as layering in telematics data, would yield the most accurate assessment of a driver’s state of awareness and overall cognitive load. Driver monitoring has to combine visual input from the in-cabin camera(s) with input from the car’s telematics and advanced driver-assistance system (ADAS) to determine an overall cognitive load on the driver. Level 3 (limited self-driving) cars of the future will learn about an individual’s driving behaviors, patterns, and unique characteristics. With a baseline of knowledge, the vehicle can then identify abnormal behaviors and equate them to various dangerous events, stressors, or distractions. Driver monitoring isn’t simply about a vision system, but is rather an advanced multi-sensor learning system. 

Monitoring the driver is not a trivial task. To date, it has been a crucial area of focus that has been overlooked by the industry. For instance, Google, in its effort to automate driving, has decided to remove the driver from the decision loop altogether, essentially jumping over Level 3 limited self-driving, believing it safer to do so. Setting aside the social implications of no longer having a steering wheel, skipping Level 3 automation might actually leave us in Level 2 limbo for a long time, because, as we are learning, error-free autonomous driving might well be somewhat of a red herring, or as one expert argued in a recent Wall Street Journal article, still another 15 to 20 years away.

Even the most sophisticated self-driving technologies have yet to prove themselves under complex circumstances. Looking at Google’s filings with the California Motor Vehicle Division (PDF) reveals that their vehicles have failed hundreds of times—and human intervention was needed repeatedly. In fact, under one filing, it seems as if human drivers initiated taking over control of the vehicles thousands of times over a span of 400,000 miles. Self-driving vehicles will only improve and become more aware of their surroundings over time, but they will still get into many everyday circumstances that would perplex even the most advanced systems that are going to be deployed in the near future.

This is the nature of technology. If the vehicle’s ADAS cannot resolve a potential hazard on the road ahead, or if the vehicle cannot recognize a police officer directing traffic, the driver must be expected to take over. Fully autonomous vehicles raise other ethical dilemmas. For instance, who is to blame in the event of a crash? What decisions would an autonomous vehicle make when faced with the choice of saving a driver’s life or saving someone else’s life? These decisions do not just involve technology. They involve a more ethical understanding and meaning of the impact that such technologies have at a societal level, and will surely involve policy makers, ethicists, and others.

Uber recently launched a similar effort to Google’s, with an “autonomous” fleet in Pittsburgh. Yet, in spite of Uber’s claims, they made sure that a professional driver is behind the wheel, and that the drivers take control of these vehicles any time a challenging circumstance arises, which is almost every ride.

So, clearly, Uber knows that they are not at fully autonomous driving, and they’re mitigating risk by adding a driver. What happens, however, if the driver zones out (as studies show to happen quite often), or the driver’s intervention is too late? Unfortunately, Uber’s fleet is not actively monitoring the drivers to know how or when to hand off control of the vehicle. There is a deep flaw in Uber’s design—a missing link with the driver.

What is essential to the outcome in all of these scenarios is the vehicle’s knowledge of the driver’s cognitive capacity, at any given moment, and their ability at handling driving responsibilities. If the driver is unable to take over some control of the vehicle, when necessary, other contingencies need to be pursued, especially if the window for any decision is in seconds or fractions of a second.

Having a deep understanding into where a driver’s attention is will be a crucial step in getting us to Level 3, the next level of vehicle autonomy. Enabling the vehicle to become intelligently aware of the degree of associated risk will allow it to make better decisions on when and how to hand off control to the driver. Level 3, in my opinion, is where we will be for quite some time before true and fully autonomous Level 4 driving is in place. If so, we need to put the technologies in place to embrace Level 3 driving as the reality of the road ahead.

Tarek El Dokor is the founder and CEO of EDGE3 Technologies in Phoenix, Ariz.

Advertisement

Cars That Think

IEEE Spectrum’s blog about the sensors, software, and systems that are making cars smarter, more entertaining, and ultimately, autonomous.
Contact us:  p.ross@ieee.org

Senior Editor
Philip E. Ross
New York City
Assistant Editor
Willie D. Jones
New York City
 
Senior Writer
Evan Ackerman
Berkeley, Calif.
Contributor
Lucas Laursen
Madrid
 

Newsletter Sign Up

Sign up for the Cars That Think newsletter and get biweekly updates, all delivered directly to your inbox.

Advertisement