Last year, a self-driving car failed about every 3 hours in California, according to figures filed with the state’s Department of Motor Vehicles.
Every January, carmakers testing self-driving cars in California have to detail how many times their vehicles malfunctioned during the preceding year. These so-called disengagement reports detail every time a human safety driver had to quickly take control of their car, either due to hardware or software failure or because the driver suspected a problem.
The reports—detailing 2,578 failures among the nine companies that carried out road-testing in 2016—give a unique glimpse into how much testing the different companies are doing, where they are doing it, and what is going wrong. None of this year’s disengagements resulted in an accident.
Image: Mark Harris; Source: California DMV
Alphabet’s spin-out company Waymo [Google in chart above] still has by far the biggest testing program—its 635,868 miles of testing accounted for over 95 percent of all miles driven by self-driving cars in California in 2016. Waymo’s fleet of 60 self-driving cars reported a total of 124 disengagements, 51 of them due to software problems. That represents a sharp reduction in disengagements from the previous year, from 0.8 disengagements for every 1,000 miles of autonomous driving down to just 0.2.
Bosch, by contrast, reported over 1,400 disengagements while covering just 983 miles in three vehicles—equivalent to a hefty 1,467 disengagements for every 1,000 miles of driving. But that doesn’t mean Waymo’s cars are 8,000 times safer than Bosch’s, as every company has its own way of counting disengagements.
For instance, Waymo does not count every single time the driver grabs the wheel to take over from the robotic chauffeur, which it admits happens many thousands of times annually. Instead, the company later simulates what would have happened if the human had not jumped in—and only reports disengagements where the car would have done something unsafe. It calculates that if its drivers had taken no action at all, nine disengagements in 2016 would have led to the car hitting an obstacle or another road user. That is down from 13 the previous year, despite covering 50 percent more miles.
“Waymo’s report would seem to suggest substantial improvement,” says Bryant Walker-Smith, a professor at the University of South Carolina. “But I’d want to know whether Waymo’s system could handle any of the system-initiated disengagements by achieving a minimal risk condition, say by pulling off to the side of the road, rather than immediately disengaging.”
The other problem with comparing disengagement rates is that different companies are using California’s testing permits for different things. Only Waymo and Cruise Automation, now owned by General Motors, have large, general-purpose testing programs. In its first year on the state’s roads, Cruise’s two dozen cars went from covering less than 5 miles in June 2015 to over 2,000 miles in September 2016. Its disengagement rate also plummeted over the same period, from over 500 to under 3 per 1,000 miles.
No other company drove more 5,000 miles in 2016, and some of the world’s biggest carmakers, including BMW, Ford, and Mercedes-Benz, covered less than 1,000. “The low number of miles, combined with high number of disengagements, suggests that R&D engineers are occasionally using a local vehicle to get real-world performance data useful for a specific project,” says Walker-Smith.
Despite holding testing permits, Honda and Volkswagen drove no autonomous miles at all last year on public roads in California, preferring to test on private courses or out of state.
Once more, the most mysterious disengagement report is from Tesla. In 2015, the company reported no disengagements at all, suggesting that it either carried out no public testing in California or that its cars were flawless. This year, its report admits 182 disengagements in 550 miles of autonomous driving during 2016.
However, all but a handful of those disengagements happened in just four cars over the course of a single long weekend in October, possibly during the filming of a promotional video. Tesla does much of its testing out of state and on test tracks, although it also benefits from receiving millions of miles of road data from thousands of AutoPilot-equipped vehicles owned by its customers.
Companies that began autonomous vehicle testing in California in 2016, including startups Zoox, Drive.ai, Faraday Future, and NextEV, will not have to report their disengagement data until this time next year. Uber, which abandoned its pilot program of self-driving Volvos in San Francisco rather than apply for a testing permit, is currently testing in Arizona and Pennsylvania, states that do not require companies to report disengagements or failures.
