Have Self-Driving Cars Stopped Getting Better?

Every January, the California Department of Motor Vehicles (DMV) releases data from companies that operated highly automated vehicles on the state’s public roads the previous year. By law, each company must report how many times a safety driver took control from an autonomous vehicle, either because the system had failed or because the human was worried it had.

Companies get to decide how to record these so-called disengagements. In 2017, for instance, relative newcomer Nvidia logged every single time a human touched the steering wheel of its test vehicle, even at the planned end of a test. Waymo, on the other hand, ran complex computer simulations after each disengagement, and only reported to the DMV those where it believed the driver was correct to take charge, rather than being overly cautious. GM chose not to report at least one instance where an autonomous car was about to block an intersection.

Such variety in reporting makes for widely disparate data. Nvidia told the DMV that its test runs were typically less than 5 miles in length, and thus it is not surprising that its car traveled an average of just 4.6 miles between disengagements. Waymo said its 75 test vehicles, in comparison, suffered disengagements only about once every 5,600 miles.

The disengagement reports are thus probably best viewed as marketing documents, indicative of neither the safety of a company’s technology nor its readiness for real-world deployment. However, as the DMV now has several years and over a million and a half miles’ worth of data, covering a growing list of companies (19 in 2017, compared with 11 the previous year), the reports do permit some analysis.

Despite more companies in the mix, the shine appears to be wearing off the Golden State for autonomous vehicle testing, with total reported mileage down by more than 20 percent from 2016. BMW, Ford, and Tesla all stopped testing there in 2017, reporting zero autonomous miles along with Honda and Volkswagen and startups Nio and Wheego. Delphi, now called Aptiv, only covered a tenth of the autonomous miles it reported in 2015. One explanation could be a shift in testing to other locations in the United States, such as Michigan and Florida, and around the world that allow fully driverless operation without any public disclosure.

Seven companies—Aptiv, Baidu, Bosch, Mercedes-Benz, Nvidia, Telenav, and Valeo—covered fewer than 2,000 miles each in autonomous mode in 2017. Such modest distances indicate limited research programs, where frequent failures might be expected. These companies averaged a disengagement every 4.5 miles.

Three companies, Drive.ai, Nissan, and Zoox, drove between 2,000 and 7,000 miles on California roads, corresponding to more ambitious research efforts. They managed more than 100 miles between disengagements.

The two remaining companies, Waymo and GM (with its Cruise subsidiary), accounted for over 95 percent of the autonomous miles traveled in 2017, tallying more than 480,000 miles. Their average disengagement rate was once every 2,900 miles—although the most recent data for GM suggests that it is approaching Waymo’s level of competence.

While a single disengagement every 5,000 or 6,000 miles appears to be thousands of times better than some startups, both Waymo and GM have announced the imminent start of commercial autonomous ride-sharing services. Waymo recently put in an order for “thousands” of self-driving Pacifica minivans with Chrysler, to start operations later this year, while GM said it will mass-produce autonomous Chevrolet Bolts without steering wheels or manual controls by 2019.

In a fleet of a thousand driverless vehicles, each driving a few hundred miles a day, even Waymo’s best-ever performance—a single disengagement in over 30,000 miles of driving last November—would correspond to multiple failures daily.

Waymo needs to improve its average disengagement rate by two or three orders of magnitude to avoid having a crowd of disappointed customers. However, even by its own figures, Waymo’s performance barely improved year on year. In 2016, it had a disengagement every 5,130 miles. In 2017, it averaged 5,600 miles. And remember that by Waymo’s definition, each of those disengagements represents a failure of its system to cope with real-world conditions. (Unlike last year, Waymo did not report how many of its 63 disengagements in 2017 might have led to an actual accident.)

It is possible that Waymo put its technology into more challenging scenarios in 2017, thus generating a higher level of disengagements. But it is also conceivable that, about every 6,000 miles or so, today’s technology encounters a situation or combination of problems that it simply cannot handle without human assistance.

One solution could be to deploy a teleoperation system such as the one being developed by Phantom Auto, where a remote human driver can take control for a short time to navigate an unexpected obstacle or assist passengers. Another possibility is that the road to our driverless future is going to be bumpier than expected.