The Unknown Start-up That Built Google’s First Self-Driving Car

Two of Google’s signature innovations, Street View cameras and self-driving cars, were actually developed by 510 Systems, a small start-up that the tech giant quietly bought in 2011

8 min read

Anthony Levandowski founded 510 Systems and Anthony's Robots to build a self-driving Prius
Photo: Conor O'Brien

Robotized Toyota Prius, built by 510 SystemsThe Original: A robotized Toyota Prius, built by Berekely, Calif.-based startup 510 Systems, was the foundation of much of Google’s autonomous car program. In a stunt for TV “PriBot” delivered pizza.Photo: Conor O'Brien

One of technology’s time-honored traditions is getting intellectual property by buying companies rich in ideas but poor in cash or connections. Burroughs Corp., for example, got the Nixie tube in 1955 by buying Haydu Brothers Laboratories. And Apple famously acquired a smart new operating system (and “reacquired” Steve Jobs) in 1996, when it bought NeXT Computer. Twitter got a search engine when it bought Summize in 2008.

Google has embraced this trend with a vengeance, buying more than 170 companies over the past 13 years. Voice over Internet Protocol, video hosting, Web analytics, mobile devices, GPS navigation, and visual search are just a few of the examples of technologies that were absorbed into the Google empire. Most of these purchases were trumpeted with press conferences, press releases, and ample news coverage.

And yet, one of Google’s most strategic acquisitions has mysteriously been actively blocked from public view. An investigation by IEEE Spectrum has uncovered the surprising fact that Google’s innovative self-driving car and the revolutionary Street View camera technology that preceded it were largely built by 510 Systems, a tiny start-up in Berkeley, Calif.

If you’ve never heard of 510 Systems, that’s exactly the way Google wants it. The purchase of 510 Systems and its sister company, Anthony’s Robots, in the fall of 2011 was never publicly announced. In fact, Google went so far as to insist that some 510 employees sign agreements not to discuss that the acquisition had even occurred. Google’s official history of its self-driving car project does not mention the firm at all. It emphasizes the leadership of Sebastian Thrun, the German computer scientist whose Stanford team won the autonomous-driving Grand Challenge in 2005, sponsored by the Defense Advanced Research Projects Agency (DARPA).

Why has Google worked so hard to keep this one acquisition a secret?

In April 2012, Google was about to make history. In just a few weeks’ time, its experimental autonomous Prius was due to take the world’s first self-driving test in Nevada. The company applied for a driver’s license in the state, and a sharp-eyed official at the Nevada Department of Motor Vehicles noticed something strange. “The proof of ownership on VIN JTDKB20U987806293 is not in Google’s name, and the insurance card is in Google’s name,” she wrote to Google engineer Anthony Levandowski. “Normally we would require these to both be in the same name.”

photo of applicationSecret Ownership: In May 2012 one of Google’s vehicles became the first robocar to pass a state license test. The car’s proof of ownership was still in the name of 510 Systems CEO Suzanna Musick, but Google had secretly purchased the startup by then.

One of the three self-driving 2008 Toyota Priuses that Google wanted to license for testing was, in fact, registered to a company called 510 Systems and a person by the name of Suzanna Musick. Levandowski’s explanation was simple: “510 Systems is part of Google, as Google purchased the company six months ago,” he replied. “Suzanna is/was their CEO.”

Though Google has portrayed Thrun as its “godfather” of self-driving, a review of the available evidence suggests that the motivating force behind the company’s program was actually Levandowski. In 2005, he was a 25-year-old graduate of the University of California, Berkeley, with a master’s in industrial engineering and operations research. That year, with the help of a group of engineers that included Berkeley undergraduate Bryon Majusiak, Levandowski entered a self-driving 90-cc motorcycle called Ghostrider into the DARPA Grand Challenge.

Levandowski and Thrun were actually competing against each other in the 2005 DARPA Grand Challenge. Ghostrider was the only two-wheeled autonomous vehicle in the contest. It relied on superaccurate GPS signals and a stereo camera rather than the expensive 3-D lidar units used on Thrun’s Volkswagen SUV, which was dubbed “Stanley.” Nevertheless, Ghostrider managed to balance, navigate, and even right itself after falling over, beating dozens of its four-wheeled rivals—although not Stanley. Ghostrider failed to proceed beyond the semifinals of the contest, but Levandowski had found the thing he wanted to do with his life.

Ghost Rider team led by Anthony LevandowskiMotorcycle Gang: 510 Systems founder Anthony Levandowski (right, white shirt) and coworkers built a robotic motorcycle to compete in the DARPA Grand Challenge. It was the only two-wheeled entrant.Photo: DARPA

After several short-lived business ventures, Levandowski founded 510 Systems with two engineering colleagues from Berkeley: Andrew Schultz and Pierre-Yves Droz. When Levandowski went to work on Google’s mapping technology in 2007, it was this start-up that would make one of the biggest contributions.

Majusiak was one of 510’s first hires, even though he was still in the middle of his undergraduate degree. “We were working on a smart machine-controlled camera that eventually evolved into the initial Street View systems,” he recalls. The company designed a processing board that could take inputs from digital cameras, high-end GPS units, and inertial sensors, and then integrate data from those systems so that the camera images were coded with positional data. The camera developed by 510 Systems was then manufactured and sold to Google by Topcon Positioning Systems, a GPS company that had previously sponsored the Ghostrider motorbike. “Google was our only customer for a year and a half,” says Majusiak.

Inevitably, 510 Systems became involved with lidar scanners. Lidar is the light version of radar; it uses lasers to measure the distance to nearby objects, typically steering the beam with mirrors to scan the surroundings in three dimensions. The 510 Systems team pioneered the use of lidar for mobile mapmaking, and the company’s systems were adopted by the biggest digital cartographers in the world.

One device it developed was deployed by utility companies for automatic surveying. Mounted on a van, the system could track individual wires strung between utility poles by the side of the road and even calculate whether they were too tight or too loose.

“At one point, we had one of the most incredibly detailed maps of Berkeley that I’ve ever seen,” says Majusiak. The company’s lidar expertise even caught the attention of Hollywood. When director James Frost wanted to use lidar images of Radiohead for a music video of the band’s song “House of Cards,” he asked Droz at 510 to process the raw data. The video was later nominated for a Grammy Award.

But while mapmaking and music-video imagery kept 510’s engineers occupied, there was a sense of unfinished business. “Robots were always in the back of our minds,” says Majusiak. “We wanted to do a better robot motorcycle: full size, full speed, and doing all those things we learned from the Grand Challenge.” So when the Discovery Channel contacted Levandowski early in 2008 and asked him to build a self-driving pizza delivery vehicle for a show called “Prototype This!,” he agreed immediately.

All that experience in mobile mapping, it turned out, was perfect for rapidly building a robot car. “It was a side project, but we had a technology pipeline that gave us essentially all the data that a self-driving car would need,” says an ex-510 employee who did not want to be named. Software engineer Jack Tisdale, for example, had written GPS filters for high-accuracy surveying that tied GPS into motion-sensor data for centimeter-level location accuracy. All that remained was to build a control system for their chosen vehicle, a 2008 Toyota Prius, VIN number JTDKB20U987806293.

“Since everything is electric in that car, you can do a man-in-the-middle type of deal,” says Majusiak. “We figured out what the signals were supposed to be, then spoofed them to make the car do what we wanted.” Within just a few weeks, the car, now dubbed “Pribot,” was ready to roll. Its route from San Francisco’s waterfront over the Bay Bridge to Treasure Island was planned meticulously in advance. Levandowski used one of 510’s lidar-equipped cars to map the 25-minute journey beforehand, just as Google does today in its hometown of Mountain View, Calif. These 3-D maps, stored in the Pribot, would be used with its centimeter-perfect positioning and a home-brewed control system to navigate the approximately 8-kilometer route. But although a roof-mounted lidar gave the Pribot basic collision-avoidance abilities, it had no way of predicting the behavior of pedestrians or other road users. So for the Discovery Channel shoot, the route was cleared of traffic and a squad of police cars and motorcycles escorted the robotic Prius from start to finish. The Pribot set off from San Francisco, crossed the Bay Bridge flawlessly, and then, in true pizza-delivery-car style, scraped against a wall on a tight exit ramp.

510 Systems founder Anthony Levandowski and Prototype This hosts

510 Systems lidar

510 Systems founder Anthony Levandowski led the team that tricked out a Prius to drive itself across the San Francisco Bay Bridge to deliver a pizzaPizza by Pribot: For the TV show “Prototype This!” 510 Systems founder Anthony Levandowski (top photo, middle) led the team that tricked out a Prius to drive itself across the San Francisco Bay Bridge to deliver a pizza. The vehicle relied on 510’s lidar system (center photo) to avoid collisions. But that didn’t stop the car from hitting the wall on an exit ramp (damage, bottom photo, rear of car).Photos: Conor O'Brien

“It was an incredible push to get the Pribot together, and nobody thought to tell the robot how big it was,” remembers Majusiak. But as a technology demonstrator, the Pribot had done its job. Levandowski had proven that safe, capable self-driving cars were not just possible but possible on a shoestring budget. Imagine what could be achieved with the resources of a company like Google. Within months, Larry Page and Sergey Brin had given Thrun and Levandowski a blank check to set up their own driverless car project. And the first place they turned to was 510 Systems.

“From then on, we started doing a lot of work with Google,” says Majusiak. “We did almost all of their hardware integration. They were just doing software. We’d get the cars and develop the controllers, and they’d take it from there.” A couple of years and five autonomous Priuses later, the inevitable happened: Google offered to buy the company. The 40-odd 510 Systems employees crowded into a meeting room and sat down to decide their future. “I absolutely believe 510 could have gone public,” says Majusiak. “It was about a 50-50 split between people who wanted to go forward with that and people who wanted to give the Google buyout a shot.”

Google’s bottomless purse won the day. In October 2011, 510 Systems quietly joined Google as a key part of the company’s semisecretive Google X “moon shot” division. The Pribot, long since bolstered by Google’s powerful software, written by Thrun’s team, came along with the acquisition. The company’s Priuses were now looking rather long in the tooth. They helped Google secure its testing license in Nevada early in 2012, but by the summer they were already being phased out for newer Lexus SUVs. “When 510 really got folded into Google, we did a major hardware spin and got everything much more to a production style rather than a college research project,” says Majusiak.

Google’s secrecy over the 510 Systems acquisition might be best understood through the lens of publicity. In 2010, a journalist at The New York Times, John Markoff, discovered the existence of Google’s self-driving car program. He was given a ride in one of 510’s Priuses and told that the project was the brainchild of Sebastian Thrun, director of the Stanford Artificial Intelligence Laboratory and winner of the 2005 DARPA Grand Challenge.

Maybe Google thought a famous prize-winning professor would be a more credible leader than the entrepreneurial runner-up Levandowski. Or perhaps the company simply wanted to avoid awkward questions about how the robot cars it had been secretly testing on public roads had actually been built in a Berkeley start-up. Either way, 510 Systems was neatly written out of the creation myth of Google’s self-driving cars long before it was acquired.

Most of 510 Systems, including all three founders—Levandowski, Schultz, and Droz—are still working on self-driving cars at Google. Levandowski remains the overall product lead, Schultz oversees embedded systems and electronics, and Droz manages 10 engineers. Majusiak eventually left Google in January to work at Blue River Technology, a start-up bringing robotics to agriculture.

The original Pribot, as far as anyone knows, is still floating around the workshop at Google X. “It might very well end up in a museum,” muses Majusiak.

If it does, the label should read: “First developed at 510 Systems, Berkeley.”

About the Author

Contributing editor Mark Harris has been delving into the history of Google’s self-driving car project for IEEE Spectrum and other publications. Before that he investigated the reason that Kodak’s patent portfolio fetched such a pittance.

The Conversation (0)