Gill Pratt Discusses Toyota’s AI Plans and the Future of Robots and Cars

The former DARPA program manager discusses what he's going to do next

15 min read

Erico Guizzo is IEEE Spectrum's Digital Innovation Director.

Gill Pratt Discusses Toyota’s AI Plans and the Future of Robots and Cars
Dr. Gill Pratt, the DARPA program manager who oversaw the DARPA Robotics Challenge, will lead Toyota's AI and robotics efforts.
Photo: Evan Ackerman/IEEE Spectrum

At a DARPA Robotics Challenge press conference earlier this year, Gill Pratt was asked about his post-DARPA plans. He politely declined to comment, saying he couldn’t discuss it at that point. There was speculation that Google, Apple, Uber, or other tech giant interested in robotics would try to lure him, and they probably did. The company that succeeded, though, comes as a bit of a surprise. Toyota, the world’s largest automaker, announced last week a big push into AI and robotics, and Pratt accepted to lead that effort.

“It’s going to be a big deal,” he told IEEE Spectrum about the Japanese firm’s plans. Pratt explained that a US $50 million R&D collaboration with MIT and Stanford is just the beginning of a large and ambitious program whose goal is developing intelligent vehicles that can make roads safer and robot helpers that can improve people’s lives at home.

In these further excerpts from an interview last week, Pratt gives more details about Toyota’s plans and what we have to look forward to over the next few years. What follows has been condensed and edited for clarity.

IEEE Spectrum: As part of your job as DARPA program manager, you oversaw some of the most advanced robot projects in existence and met with top roboticists in labs and companies around the world. Why did you decide to accept this position at Toyota?

Gill Pratt: There were a few things. Toyota, in our conversations, made it very clear that they are interested not only in developing new technologies for cars but for robotics as well. If you think about the use of robotics within the home, it is the same as the use of vehicles when we travel on the road, except that instead of moving goods and people outdoors, you move them indoors. A lot of the same technology can be brought to bear. Toyota is also convinced that there should be a strong relationship between people and the machines that are helping them to move. I’m going to serious meetings about technologies and budgets and timelines, and the one word we use the most to describe how we want people to feel about these technologies is “love,” and that struck me. And of course, the company is extremely successful and very, very big—it’s roughly 250 billion dollars in revenue per year. It became clear to me that this was the place to have the greatest impact of applying the technology that I’ve been working on for the last few decades, both in my academic career and at DARPA, to really improve the human condition. And I should add one last thing. When I was younger I used to fix my own cars and lots of friends’ cars. Toyotas had a very good design and last the longest. They were my favorite cars to fix.

imgToyota wants to find out how robotic helpers like the HSR (above) can assist people with tasks at home.Photo: Toyota

What is the current state of robotics at Toyota? And specifically, what about previous Toyota robot technologies like the semi autonomous car developed by Lexus/Toyota; the famed violin and trumpet-playing Partner robots; the personal mobility and healthcare robotic systems; and more recently, the Human Support Robot [pictured right], designed to assist people at home—will they be part of this program?

I recently visited each one of those labs, and I saw just tremendous work there. I’m certain that they’ll be very closely linked to what we’re planning to do. In terms of the cars, there’s a fundamental difference between the Toyota approach and what some of the other companies are pursuing: we think that the feeling of amplified mobility that a car can give you—when you drive you feel that a car is an extension of yourself—that’s something we want to preserve and enhance even more. And so the question is, how can AI and robotic technology be used not only to make cars safer but also more fun to drive, not less? We understand that in some cases you want the car to drive by itself and just take you wherever you want to go, or perhaps not even have a person inside. But the primary focus of what we want to work on first is actually to enhance the collaboration between the person and the machine.

“There’s a fundamental difference between the Toyota approach and what some of the other companies are pursuing: we think that the feeling of amplified mobility that a car can give you . . . that’s something we want to preserve and enhance. And so the question is, how can AI and robotics be used not only to make cars safer but also more fun to drive, not less?”

Can you put the $50 million in perspective and explain what you plan to accomplish with it? Compared to what companies like Google, Apple, Uber, Tesla, and others are investing in autonomous vehicle technology, or what DARPA put into its autonomous vehicle and robotics challenges, or even what some robotics and AI startups have raised, $50 million is a modest sum.

The $50 million is being focused on two particular groups within two schools, so when you consider the number of folks who’ll be impacted by this, I think this is a very strong effort in terms of R&D. And actually for a company to give two school labs this amount of money, it’s not something that happens often. It’s also just the beginning. We’re announcing this now because we want the work at the universities to start this term, but we hope very much to be announcing more in the future.

Can you describe what involvement will be coming internally from Toyota, as opposed to the centers at Stanford and MIT?

We’re actually working on that now. So it’s a work in progress to figure out just exactly how the universities are going to interact with the company. I can’t give more specifics because it hasn’t been fully decided.

There are 77 autonomous cars registered in the State of California alone; 48 belong to Google, which has nearly 200 registered human drivers to test its robot cars. Also investing hugely in robot vehicle technology are Apple, Uber, Tesla, Bosch, Baidu, Delphi, as well as several auto makers (Nissan, Mercedes, Audi, Volkswagen, to name a few) and a host of startups like Zoox, Cruise, and Mobileye. Will Toyota have to play catch up?

“I think that the race has just begun…No matter where the different companies are right now, and how they’re positioning themselves, the fact is, we haven’t gotten very far yet at all. Most of the miles that have been tested so far have been very easy, with only a few cases where there were really close calls, which is where AI needs to perform extremely well.”

First of all, I think that the race has just begun. And 77 cars—that’s nothing in this race. The million miles that Google has passed in terms of the driving that they’ve done sounds like a lot. But I want you think about this: there are around 10 million Toyota cars produced every year. They last around 10 years. So there’s on the order of—this isn’t a precise number, so order of magnitude—around 100 million Toyota cars in service around the globe. Each one of those on average drives around 10,000 miles per year, so you multiply that out and you come to 1 trillion miles driven by Toyota cars every single year. If you compare a million miles to a trillion miles, it’s a factor of a million. So when you think about the number of autonomous miles that have been driven by anybody so far, with their handful or few dozen cars or whatever the number is, it’s almost nothing. In fact, the trillion miles that Toyota cars drive—that’s around 10 percent of total miles driven by all cars on planet earth. So you have to keep in mind what the goal is: in this so-called race, the track is very, very long and it’s going to be very, very hard to get to the end. No matter where the different companies are right now, and how they’re positioning themselves, the fact is, we haven’t gotten very far yet at all. Most of the miles that have been tested so far have been very easy, with only a few cases where there were really close calls, which is where AI needs to perform extremely well. Our goal, which is a little different than the approach that others take, is to build intelligence to help the car be really a guardian angel for you and keeping you from having a wreck. That’s the hardest part of this whole thing, but that’s the part that we’re going to do first. We want to enhance the fun of driving for the human being while making it far more safe.

And how does a traditional automaker plan to compete with fast-paced, risk-taking tech giants and startups, especially when Japanese companies are typically slow and cautious in everything they do?

While it’s true that Japanese firms, including Toyota, are careful and try to reach internal consensus about the right thing to do, my experience, even just recently as I interacted with many people here, is once they make the decision to do it, they go whole hog. They just decide, “okay this is what we’re going to do, and we’re in this to win.” If you look at the semiconductor field, in the past, the Japanese companies, despite the carefulness and being very methodical, they ended up winning. I think it’s possible, perhaps even likely, that the same thing will happen here with cars and robots. Toyota really wants to do this well.

imgA semi-autonomous test vehicle that Lexus, a Toyota division, developed in 2013.Photo: IEEE Spectrum

Right now it seems that most of the companies investing in autonomous vehicle technology are trying to build actual commercial systems. But the Toyota AI and robotics effort seems to emphasize R&D. Are you mostly planning to fund and advance research or is there an expectation to develop commercially viable systems that Toyota can market?

There is a desire to do real-world commercial systems, for sure. We’re starting by establishing a collaboration with the two schools, but keep in mind again that this is by no means the size of the whole effort. We hope to talk about that in the future, but we’re beginning this first move by funding longer term R&D at the schools, because of the desire for that to feed the pipeline of people and ideas that are good for the field. But by all means, the end of that pipeline is vehicles and robots for the home, without a doubt.

Can you describe in more detail how the program is separated in vehicles and home robots? Sounds like the most emphasis is on vehicles but we’re also interested in what you have in mind for the home robots.

The answer is that we’re going to do both. We’re not completely sure internally exactly how we’ll phase things, but both will be going on from the very start. And of course cars is the biggest part of the business of the company now, but really the desire is to support human beings with their mobility needs. Consider that a car outdoors is really doing very much the same job as a robot indoors: it’s trying to move both people and things from place to place and to support and improve the human condition as a part of doing so. The amount of autonomy that’s required varies a little bit, but the technology for sensing, perception, and planning is very much the same. So particularly with regard to the R&D that’s going to occur at the schools, we think it will have very broad applications to both fields.

Cloud robotics and deep learning, which you recently mentioned in an article about the future of robots, are certainly important advances, but what about innovations in robotic hardware—soft materials and new kinds of actuators, for example. Will they be part of this program at all?

“The idea of lowering cost and raising the effectiveness of robotic hardware is absolutely key. We still need better and cheaper sensors and actuators. That’s part of why robots haven’t made it big in the marketplace yet. Here’s the really neat thing: I don’t know of a company on Earth that’s been better at manufacturing hardware than Toyota, and so I think it’s the perfect firm to try to achieve that goal of both raising the capability and lowering the cost.”

The AI programs that we’re starting at the two schools do not have hardware as part of the research scope, but the idea of lowering cost and raising the effectiveness of hardware is absolutely key. We still need better and cheaper sensors and actuators. That’s part of why robots haven’t made it big in the marketplace yet. Here’s the really neat thing: I don’t know of a company on Earth that’s been better at manufacturing hardware than Toyota, and so I think it’s the perfect firm to try to achieve that goal of both raising the capability and lowering the cost. I’ll give you the example of the incredibly courageous move that they made with the Prius when that first came out. If you know how that car works or any of the cars that use the drive system that came from it, there are actually two motor generators on the inside of the drive system, and it basically uses an electronic version of a transmission, so there aren’t any gearshifts within it. For a company to say, “let’s go with the high complexity way of doing these things, because after very careful analysis we decided that that’s actually the one with the greatest promise,” ended up being the right thing to do, and of course, it captured most of the market with that system. Because of the incredibly strong history in design and manufacturing of the firm, they could pull it off. And I think that the exact same thing is necessary in the robotics field, because you need that kind of strength in order to lower the price and raise the quality and effectiveness of what comes out. So that’s part of my hope for what we can bring to the field.

At IROS in 2012, you made the somewhat controversial comment that “grasping is solved.” Post-DRC and beginning research into robotics for improving quality of life, do you still believe this is true?

[Laughs.] I don’t regret saying that. I think that it’s important for any field that, if you’re working on a particular thing for a while and you become mostly successful, you need to realize that maybe it’s time to stop doing that so much and to move on to something else where there’s a whole lot more things to be solved and good to be done. What I had meant is that the program that I ran [at DARPA] was on grasping and manipulation, where once you grasp the thing, so you can pick it up, you need to have skills to turn the key, or to operate the tool, or to do anything else. The latter, the manipulation at that point, was still very much not solved—all the different dynamic behaviors that you need to do with the grasper in order to do a complete task in time. But the fundamental problem of picking this thing up had become easier, and people had mostly figured out how to do it. So I felt that it was time for us to refocus our efforts and start looking at the hard stuff. I think that in the autonomous driving or intelligent car field, the idea of trying to follow a map and register the road with respect to the map, that’s the easy part. Now we have to do the hard part. We have to figure out what to do when there is not map; what to do when the road differs from the map; what to do when the unexpected occurs—when the child chasing the ball runs in front of the car, or when somebody changes lanes very fast. These are situations where the dynamics are very hard, or when we have to make sudden decisions about changing control between the machine and the human driver. And so those are the things that we need to work on next. And that’s part of the reason that we’re investing so much in state of art schools that are going to push the frontiers not where it’s easy but where it’s hard.

Where do you hope Toyota’s robotics program will be in five years?

“A car that is never responsible for a crash, regardless of the skill of the driver, will allow older people to be able to drive, and help prevent the one and a half million deaths that occur as a result of cars every single year around the world. And so I hope to exponentially decrease that number over time, kind of a Moore’s Law but on the way down.”

Our long-term goal is to make a car that is never responsible for a crash. And of course, it’s impossible to make a car that cannot be hit by some other car that’s doing something bad. So I want to be very careful to say that we know that if you’re standing at a light and somebody whacks into you and you don’t have a chance to move out of the way, you’ll have a crash. But a car that is never responsible for a crash, regardless of the skill of the driver, will allow older people to be able to drive, and help prevent the one and a half million deaths that occur as a result of cars every single year around the world. And so I hope to exponentially decrease that number over time, kind of a Moore’s Law but on the way down. I don’t know now what the slope of that is going to be, but I want it to be true that every so many years, we go down by a certain factor of that number. And we just keep doing it until we get below the levels that occur because all sorts of freak acts of nature—lightning, for example, which kills a certain number of people every year. If we can make it so that the chance of getting struck dead by lightning is more than the change of getting killed in a car accident, then I declare that we’ve succeeded. We’re nowhere near that now. But that’s what my goal is. Will it be in 10 years, 15 years, 20 years? I don’t know. You asked me what’s going to happen within five years—it’s very hard to say. What I do know is that I’m going to try as hard as I can, and this first phase of the announcements, showing that we’re going to start by investing in basic R&D, I think that’s a crucial thing to do, because they are the feeders of the pipeline of the talent and the ideas that we need most.

Will you develop test vehicles to see how the technologies created as part of this effort perform in the real world? If so, where and how do you plan to test your vehicles?

Trying to develop really good test technology is going to be, I think, one of the most important parts of the whole field [of autonomous and intelligent vehicles]. I mentioned that Toyota vehicles drive a trillion miles every year. And that, even the million miles that Google has done, which is amazingly good, is still a tiny fraction. So we’re nowhere close to actually doing that, and we need to focus on the hard cases—the near misses, the bad weather, the fog, the rain, the snow, the dirty windshields, all of these things. We need to focus on testing hand-off between the person and the machine, making sure that in no case you’d make things worse—it’s a sort of a do-no-harm principle. It’s all going to be extremely difficult, and so one of the wonderful basic research areas that MIT is going to work on is going to be how they do that kind of testing: they’ll use computational methods to take the physical tests that we do and to boost them to be representative of much more equivalent testing. How do we get to trillion-mile reliability when it’s impossible to drive a trillion miles during the development? That’s the key question. I don’t want to go deeper into sort of the particular techniques that we’ll use to do that, but I can say that testing is a key question that all of us in this field have to address.

It seems that the United States and Japan are very different markets for both vehicles and support robots. Is that going to manifest itself at all in the research that you’re doing?

I don’t think so. Toyota is a global company, and our desire is to develop these technologies for the whole world. That said, the elder care needs in Japan are going to become much more severe than in the U.S. The number of people over age 65 in the U.S. will go from 15 percent to about 20 percent in 15 years, which is amazingly bad, but in Japan, it’s going to go from 25 percent right now to nearly 40 percent, an extraordinarily high number, and because the immigration in Japan is so low, they have a real crisis on their hands. So one of the things that we want to change is that you’ll never have to take away the car keys from our parents—which I had to do, my wife had to do, and that’s an awful thing to do. And another thing we want to improve is how we provide care for people who grow old. The solution the world has right now is essentially nursing homes and the like, where often strangers are taking care of you in ways that are quite invasive to your privacy. So the question is, if you had the choice of whether to age in place, at your own home, with a machine to help you do things—say, change your diapers or get you from the bed to the shower—or to move to a nursing home and have a series of strangers do these things to you, which do you prefer? I think the desire to live in a much more dignified way is very common between the U.S. and Japan, and I think throughout the whole world. And so I see us being able to apply this everywhere.

“I’m looking forward to the machine that helps clean up the house, that puts away the groceries, that can remember where all our things are . . . And when you get so old that it’s very difficult to get out of bed, I’d love to see a machine that helps me to do that, and actually helps me to walk, helps me to run, helps me to travel around, so I can still enjoy the freedom of mobility as if I were young.”

What are some of your favorite robots, real or fictional, and how do you think they might influence the direction of this Toyota program?

The favorite one I had—I think I may have said this before—was called Gigantor, and it was a dubbed version of a cartoon imported from Japan in the 1960s in the U.S. It was a Japanese robot named Tetsujin 28, and there was a little boy who got to tell it what to do, and the robot always came and saved the day. So that’s my favorite robot. [Laughs.] Now, the robots that I think are going to do the best, other than cars—because the cars of the future, and I think Rod Brooks made this point, are essentially going to become elder care robots that move us around out of doors—I’m looking forward to the machine that helps clean up the house, that puts away the groceries, that can remember where all our things are, and that basically does all of these things that are a kind of pain to do when you’re young and really hard to do when you’re old. And then, in particular, when you get so old that it’s very difficult to get out of bed, I’d love to see a machine that helps me to do that, and actually helps me to walk, helps me to run, helps me to travel around, so I can still enjoy the freedom of mobility as if I were young. It’s an amazing thing when a little child—and my wife and I have raised four of them—first learns to walk, and you see that smile on their faces that says “oh, I can get from here to there myself and I don’t need to wait for mom or dad to pick me up.” That’s a very different view of the word “autonomy” than we in the robotics field usually use. We always use it to describe the autonomy of the machine—but I like to think about the autonomy of the person, and that’s what the purpose of these machines should be: improving the autonomy of human beings, particularly when we get old.

The Conversation (0)