DARPA Robotics Challenge: Amazing Moments, Lessons Learned, and What's Next

With the DARPA Robotics Challenge over, here's what we learned, and what we have to look forward to

15 min read

Erico Guizzo is IEEE Spectrum's Digital Innovation Director.

ATLAS humanoid robot during DARPA Robotics Challenge Finals
WPI-CMU’s ATLAS was the only top-scoring robot that didn't fall or need a restart during the DARPA Robotics Challenge Finals.
Photo: Evan Ackerman/IEEE Spectrum

The DARPA Robotics Challenge is over. It will certainly be remembered as one of the defining robotics competitions of the decade—full of drama, hardship, and inspiration. We brought you as much of it as we could, including a detailed look at the winning robot, a fun compilation of robots falling, and our impressions of the first day of the competition. And we still have a lot more to come! But for now, to cap it all off, here are things that stood out to us about the final day of the DRC Finals, and a hint of what to look forward to in the future.

People Love Robots

When Team KAIST won the DARPA Robotics Challenge.Video: DARPA Robotics Challenge

We know people love robots, but seeing thousands of eager spectators filling the Fairplex’s grandstand, cheering for a bunch of machines when they succeeded, lamenting when they fell or failed, and celebrating their human creators, gave us a new perspective on how much people love robots and seem to connect with them. As DARPA program manager and DRC organizer Dr. Gill Pratt beautifully put it: 

“Why would anyone sit in the sun and heat, watching a machine take up to an hour to go through eight simple tasks that you could do in 5 minutes? The new discovery we made here is that there’s some incredible untapped affinity between people and robots that we saw for the first time today. Ordinary people, not roboticists, felt this identity, sympathy, empathy for the robot. The thrill of victory, the agony of defeat. . . . When the robot succeeded, they felt they succeeded. . . . I see potential for robots to connect with people and create a society where people actually feel better. Some people will say ‘That’s creepy, it’s a machine,’ except it’s a fact of life, we saw it today, people connecting with machines.”

Software Is a Hard Problem

MIT Atlas robot competes in the DARPA Robotics ChallengeOperator view of MIT’s ATLAS on the valve task.Image: MIT

At an event like the DARPA Robotics Challenge Finals, it’s easy to focus on the robotic hardware that’s out there actually doing stuff. What we don’t get to see is everything going on in the garages hundreds of meters away from the course, where the operators (“robot drivers”) are receiving data from the robot’s sensors, trying to interpret it, and telling the robot what to do. Given enough time, drivers could teleoperate their robots through every single task at the joint level, but that would take hours, and teams didn’t have that much time—they had to complete all eight tasks in less than 60 minutes.

One of DARPA’s main goals with the DRC was to advance how robots and their human operators can work in concert to perform difficult tasks. The key to achieve that is not only better sensors and actuators—what we need just as much is better software. At the DRC Finals, we saw some progress in that direction. We saw the operators relying on tools that made the most of minimal perception data and allowed them to command the robots to execute actions with some degree of autonomy.

We must say we wished we had seem even greater perception and planning autonomy. We felt that there was still too much teleoperation, in part because the course was not too hard (more on that later), and operators could just manually control the robots and tell them what to do step by step. For example, in the rubble task, we wanted to see teams push a “Go Over Rubble” button that would make the robot scan the terrain, compute a viable path, and then walk over the obstacles—all done autonomously. Instead, we saw the human operators looking at images from the robot’s camera and lidar and telling their robots exactly where to place their feet. The same happened with other tasks. Some may call that a semi-autonomous operation, but others may say it’s pure teleop.

Robot control interface for Atlas during DARPA Robotics ChallengeTeam IHMC control interface. The green shapes indicate where ATLAS should step on.Image: DARPA

So, there’s still much more we need to do. Autonomy is really hard, but it is key to the future of disaster robots, because we don’t want to have to rely on experienced robot drivers to get tasks like these done. Relatively untrained users will need to be able to interface with robots in a way that they can understand and readily use, and that means letting the robot (or more accurately, the software) deal with as many complex tasks as it can.

Falling Is Usually Okay . . .

Video: IEEE Spectrum
Apparently robots hate gravity.

There were far fewer falls on Day 2 than Day 1, and with a few exceptions, most robots were able to reset in under 10 minutes and keep right on going. From what we could tell, none of the ATLAS teams expected that they’d survive falling as well as they did, and for our part, we were expecting shattered limbs and hydraulic fluid explosions. There were only a few of those, and apart from some cosmetic damage, the hardware stood up pretty well. MIT’s ATLAS robot fell on egress on the first day and broke its right arm. The team switched some stuff in software and was able to continue on with only their left arm on that day. Repairing the arm later turned out to be tricky, and they had to work all night ripping most of the guts out of their robot. They made it, though, and then fell once more on Saturday. Considering that approximately zero of these robots were really designed to fall down ever, the uptime on them was an impressive achievement.

Robots falling during DARPA Robotics ChallengeMIT’s ATLAS falls while trying to operate a drill on the wall task.Photo: Evan Ackerman/IEEE Spectrum

That said, it was a bit disappointing that, with one exception, no robots fell over and got back up again on their own. With that same exception, no robots fell over and even attempted to get back up on their own. From the sound of things, teams figured that if their robot fell, it would be rendered unrecoverable, so nobody bothered trying to figure out how to do it.

Part of the problem here is that it terms of hardware, it doesn’t make much sense to design a robot to be able to get up from a fall and then do all kinds of other tasks, because you risk massively over-engineering your robot. But, falling happens, and it’s going to continue to happen, because it happens even to us humans, and we’re pretty reliable at walking. If legged robots are ever going to be truly effective, falling and getting up is something that they’re absolutely going to have to crack.

Maybe we need a new challenge where a humanoid is nudged, falls over on a hard surface, and has to get up again all by itself without having sustained major damage. We would watch that!

. . . Except When It’s Catastrophic

More robots falling during DARPA Robotics ChallengeTeam NEDO-JSK removes their robot on a stretcher.Image: DARPA

There were a few unlucky robots that hit the ground hard and couldn’t recover. We saw Team HKU’s ATLAS falling in the first hours of Day 1 and it seems that it never recovered. 

Robot falls to the ground during DARPA Robotics ChallengeTRAC Labs’s ATLAS bleeds hydraulic fluid after falling on egress task.Photo: Evan Ackerman/IEEE Spectrum

The most disastrous fall happened to Team TRAC Labs right at the end of their egress on Day 2. The TRAC Labs ATLAS got out of the vehicle and onto its little egress platform, and then lost its balance and slowly toppled over. On impact, there was a two-meter spray of hydraulic fluidand the robot lay here, bleeding into a slowly growing puddle of green goo, until it was hoisted up and hauled away.

Ouch.

Incidentally, Boston Dynamics brought along an eighth fully functional ATLAS, not as a loaner, but as a sacrifice: if necessary, teams could harvest parts from the spare to keep their robots running.

ATLAS Is Awesome. ATLAS Is Awful.

Atlas robot during DARPA Robotics ChallengeMIT team gathers around their fallen ATLAS.Photo: Evan Ackerman/IEEE Spectrum

Speaking of ATLAS, let us say right off the bat that we’re big fans of Boston Dynamics. Those guys are amazing. ATLAS is a beautifully engineered hydraulic-electric humanoid, with powerful legs, capable arms, and impressive sensor suite. On Day 2, Team IHMC and its ATLAS had a flawless run, performing all tasks with dexterity and confidence. They ended up placing second in the competition only because it took them just a little longer (6 minutes) than Team KAIST, the winner, to finish the course. Other teams that used ATLASES (ATLI?) like MIT and WPI-CMU also performed well (seventh and eighth place, respectively), completing most of the tasks, including hard ones like egress. It’s totally possible that an ATLAS team could have finished first at the DARPA Robotics Challenge, and there’s no question in our minds that ATLAS is a cool robot to work with, and watching it compete was thrilling.

But ATLAS is not perfect. It’s big, dangerous, finicky, and, at well over $1 million each, a very pricey robot. Using ATLAS requires insanely strict safety precautions. We were told that the robot’s legs are so strong, it could kick cement blocks into pieces. Sometimes it would move its limbs so violently that it ripped apart its own hydraulic lines. Teams always practiced with ATLAS attached to safety belays, and few tried to teach the robot how to get back on its feet in case of a fall. Its battery didn’t last very long, and it could easily overheat. Fixing it was also tricky, and something the teams often couldn’t do on their own; Boston Dynamics had to come and do the repairs.

So, yeah, ATLAS is a beast. Humongous, hugely expensive, high maintenance. Knowing what we know post-DRC, we think it’s fair to ask if that is the kind of machine that we should be looking at as we build the next generation of disaster robots. We also wonder what will happen to ATLAS now—will DARPA allow any of the teams to keep or buy one for its own use? Would anyone even want one, knowing how costly it is to maintain it? For what it’s worth, we think that if they all end up at the Smithsonian, helping to inspire future generations of robot engineers, that wound’t be too bad.

Right Now, Not Walking Is a Big Advantage

NASA JPL RoboSimian robot during DARPA Robotics ChallengeJPL’s RoboSimian puts its butt-wheels to good use on the drill task.Photo: Evan Ackerman/IEEE Spectrum

Of the top three robots in the DRC Finals, third place went to a robot that rolled on tracks; second place went to a walking biped; and first place went to a biped that had wheels that it could use instead of walking. Three completely different techniques, but after watching two days worth of robots falling over, we’ve been most impressed by robots that at least have the option to avoid bipedal walking.

Hybrid designs seemed to perform the best: KAIST had wheels and legs, CHIMP had tracks and legs, and JPL’s RoboSimian had a quadrupedal design with butt-wheels. In each of these cases, the robot had good mobility without having to worry about falling over all the time.

It’s important to note, however, that in a real disaster area, wheeled mobility may be close to useless, so despite how well the wheeled designs did at the DARPA Robotics Challenge, it shouldn’t minimize the future potential and value of bipedal walking. As IHMC pointed out during a post-competition workshop, bipedal walking lets you move across areas where you only have a footstep-sized safe place to move, and, unless you can fly, no other mobility design does that.

The Course Was Definitely Not Too Hard

DARPA Robotics Challenge courseThe DARPA Robotics Challenge Finals course stands empty at the end of Day 2.Photo: Evan Ackerman/IEEE Spectrum

The consensus from teams seemed to be that the course was definitely not too hard, and honestly, we were expecting something harder. We had sort of figured that DARPA would chain the full versions of all the Trials tasks together into one mammoth disaster course. They did combine itty bitty versions of most of the tasks, which was good, and when Gill Pratt commented that the Finals course was “10 times harder” than the trials course, we’re guessing it was because of the stricter time limit, the egress task, and DARPA’s shenanigans with the wireless communications between teams and robots. 

Considering that many teams did pretty well, was the course too easy? KAIST said that in practice runs, they could complete the whole thing in 28 minutes flat, but they didn’t manage to pull that off in the competition itself. It’s worth noting that quite a few teams finished just under the 60 minute mark, and a few ended up on the stairs and ran out of time, so it’s not like everyone was just ripping right through everything. And being able to complete the entire course just meant that time was a significant factor, which DARPA certainly intended. Being just a few minutes faster made a million dollars worth of difference to one team, and in a real disaster, it could be the difference between saving a life, and being too late.

Robots vs. Door

Want to stop the robot uprising? Just shut your doors.Video: IEEE Spectrum

A recurrent joke at the DRC was that, if we needed to stop the robopocalypse, all we had to do is close our doors. The door task—pulling down a door knob, opening the door, and going through it—proved surprisingly tricky for a number of robots. Some of them seemed physically capable of performing the task; what appeared to be lacking was better perception and control to, among other things, manipulate the knob and avoid bumping into the door frame. At least one robot seemed so terrified of the task it just collapsed backward. 

The Wall Task Was Especially Tricky

Robot with drill during DARPA Robotics ChallengeNIMBRO drilling through the wall.Photo: Evan Ackerman/IEEE Spectrum

Skipping the wall task completely was common; also common was bypassing the wall task, doing the surprise task, and then (maybe) coming back for the wall if there was a comfortable amount of time left. The task involved getting to the drills, seeing the drills, grasping the desired drill (a regular drill or a cut-out tool), turning the drill on, repositioning the robot near the wall, seeing the shape, and then (finally) getting the robot to manage a heavy, moving, vibrating power tool while it chews through drywall.

Generally, if a team decided to attempt a task, they’d either finish it or fall over trying, but several teams were forced to abandon the wall task after they dropped the drill or the drill shut off (it was programmed to operate for 5 minutes; after that, the robots had to turn it back on) and there was not enough time to start all over again. At least two robots, MIT’s ATLAS and JPL’s RoboSimian, even tried punching the wall in a desperate attempt to complete the task. 

Not Enough Surprise in Surprise Tasks

Japanese humanoid robot during DARPA Robotics ChallengeTeam AIST-NEDO’s HRP-2 ponders the plug, the surprise task on Day 2.Photo: Evan Ackerman/IEEE Spectrum

As it turns out, we had a pretty good idea about what the surprise tasks were going to be in advance of the DRC Finals, and so did most of the teams. We hadn’t guessed button, but we had guessed both switch and plug. Of these three, the teams said that plug was by far the hardest. But, these tasks weren’t surprises in the sense that the teams had an “OMG how do we do this?” moment when the task was revealed (that was our hope). Instead, it was just a sort of, “okay, we need to grasp and move in these ways.” The teams even rapidly assembled copies of the switches and plugs at their garages to practice before their official run. It did add a little bit of variability, which is what DARPA was going for, but we think that teams could have handled even more.

During the media briefing after the finals concluded, Gill Pratt explained that DARPA used the surprise tasks to “tune” the difficulty of the DRC, and that DARPA was concerned about the fact that at the rehearsal run no teams had practiced off-belay, so they decided to tune down the surprise tasks pretty heavily.

Lots of Creativity in Driving Techniques

Robot crashes vehicle during DARPA Robotics ChallengeCHIMP getting a little aggressive on the driving task.Photo: Evan Ackerman/IEEE Spectrum

Generally, all of the robots used very similar techniques to complete a given task. Driving was the big exception, and it was fascinating to watch. Teams were allowed five minutes without tools to modify the vehicles for their robots, but even under those constraints there was a lot of creativity on display. The robots used all kinds of mechanisms with levers and cables to turn the steering wheel and press the accelerator; we don’t think any robots bothered to use the brake.

Most robots (and ATLAS in particular) were too bulky to easily sit in the driver’s seat, and even if they had been able to wedge themselves in there, it would have made egress much more challenging. The solution was to get ATLAS to drive while keeping it as far out of the seat as possible, which involved some occasionally bizarre solutions to allow the robots to actuate the controls.

Team TRAC Labs had their ATLAS hold onto what looked like a square piece of wood with a bunch of nails sticking out of one side; the robot would reach behind it and engage the nails into a cog attached to a cable that ran all the way across the vehicle to the steering wheel. When the robot twisted its gripper, it would twist the cable, steering the vehicle. Their ATLAS dropped the block before it exited the vehicle, and I think this makes it the only robot that we saw using a bespoke, disposable tool. 

The ATLAS teams favored platforms that folded up against the side of the vehicle that the robot could kick down to turn one giant step down into two less-giant steps down.

Humanoid getting off car during DARPA Robotics ChallengeLeo slides down its rail-mounted vehicle egress system; the squeeze toy that it used to control the vehicle acceleration is on the dash.Photo: Evan Ackerman/IEEE Spectrum

Leo, Lockheed’s ATLAS, also had a built-in egress assist system, but instead of a drop-down platform, the robot slid out of the vehicle on pivoting metal rails, which allowed it to skip the steps down that robots using platforms had to take. To control the vehicle, Leo actuated a lever to turn the steering wheel with one arm, and to accelerate, it pulled on a doggie squeeze toy (that orange thing in its gripper) attached to a rope. Adorable.

The very best egress technique, though, came from JPL’s RoboSimian. “I thought that the way they did that, seeing this spider-like thing getting out of the car in this dance, it was an elegant and beautiful thing to see,” Gill Pratt said. We couldn’t agree more.

Adaptation Is a Huge Challenge

DRC-HUBO the robot that won the DARPA Robotics ChallengeTeam KAIST’s DRC-HUBO, the DARPA Robotics Challenge winning robot.Photo: Evan Ackerman/IEEE Spectrum

Essentially all of the robots in the DARPA Robotics Challenge could, in principle, complete all of the tasks on the course under ideal conditions. The challenge is dealing with things that are unexpected, and even tiny little changes, or small errors in programming or commands, can lead to catastrophic failures.

For example, we asked IHMC what led to the two falls that they had during their Day 1 run. The fall on the rubble was because the foot placement was too aggressive, and a step forward and down maxed out ATLAS’ ankle, leading to a loss of balance. The fall off the stairs was a software bug that caused the robot to think that it had a full foot on a stair when it only had a half foot on a stair, and when it tried to step up, it lost its balance and fell over. It doesn’t take much.

This is why robots are not ready for real disasters, and won’t be for quite a while, but this is a critical start, and for their complexity the robots that competed in the DRC Finals are more versatile and adaptable than any we’ve seen before.

NIMBRO robot during DARPA Robotics ChallengeNIMBRO demonstrates one of the risks of trying to drive through debrisPhoto: Evan Ackerman/IEEE Spectrum

Also, and this is more of an observation than a conclusion, it’s worth noting that no legged robots tried to deal with the debris, and no wheeled robots tried to deal with the terrain. Teams understandably took the easiest course open to them, but offering options like these meant that robots weren’t challenged in the Finals as much as they were in the Trials. Robotics embraces specialization, but they’ll have to be generalists to be effective in a real disaster area, dealing with both terrain and debris. That’s the only downside of making the DRC Finals so competitive: it created a focus on winning a specific competition as opposed to a focus on making the most capable and versatile robot, since those two things aren’t always the same.

Chimp Is Still Awesome

CMU CHIMP during DARPA Robotics ChallengeSee? Awesome.Photo: Evan Ackerman/IEEE Spectrum

KAIST and IHMC had amazing runs, and they absolutely deserve their first and second place finishes. Our favorite run, though, is still CHIMP’s Day 1 performance, where all kinds of things went wrong (including a fall!) but the robot was able to recover and still score 8 points in under 60 minutes. And to us, that could make them the most successful team to compete in the DARPA Robotics Challenge, at least in terms of versatility and adaptability. Lots of teams did great when nothing went wrong, but CHIMP did great even when things did.

What’s Next

NASA Valkyrie will be used in the Space Robotics ChallengeNASA’s Valkyrie robot will be the star of an upcoming robotics challenge.Photo: Evan Ackerman/IEEE Spectrum

DARPA’s goal with the DRC was not to develop a robotic platform that could be immediately deployed into disaster areas and be helpful. None of the robots that competed last weekend are ready for that. What DARPA is all about is high risk, high reward, long-term technological pushes, and that’s the context that the DRC should be considered in.

Ten years ago, DARPA held another kind of challenge: the Grand Challenge and Urban Challenge, for autonomous vehicles. It was a success, with a handful of robotic cars and trucks completing the courses. Today, we’re just starting to see autonomous vehicle technology that’s anywhere close to being ready for mainstream adoption. DARPA was the genesis of this, but it takes a long, long time to go from DARPA challenge to a technology at a readiness level that we can benefit from. So when we have real disaster robots in five or ten years, we can thank the DARPA Robotics Challenge for being where it all began.

It’s not likely that we’ll see another humanoid challenge of the same magnitude for some time, but there are still a few things to look forward to. Japan will be hosting some sort of robot challenge to coincide with the 2020 Olympics in Tokyo, which should be good. But what we’re really excited about is an announcement from NASA that they’re getting ready to award a handful of Valkyrie robots to university teams in preparation for a robotics challenge intended to explore the possibility of sending humanoid robots into space, and eventually, to Mars.

The DARPA Robotics Challenge is over, but to paraphrase Dr. Gill Pratt at the closing ceremony, “Are we done with the work that needs to be done in robotics? NOOO!

The Conversation (0)