DARPA Robotics Challenge Follow-Up: Competition Videos, Bloopers

Some robots performed well; others not so much

8 min read

DARPA Robotics Challenge Follow-Up: Competition Videos, Bloopers

When DARPA's Virtual Robotics Challenge (VRC) wrapped last Thursday, we got all the results, but not much more. What we wanted was to see some of the actual competition runs themselves, especially the ones that went either very right or very wrong. DARPA released some videos over the weekend giving us a taste of how things went, and we were also treated to an extensive public Q&A session with DARPA program manager Gill Pratt and Nate Koenig of the OSRF.

Let's start with how everything was supposed to go:

Meanwhile, some teams were busy getting, um, creative:

As funny as those outtakes are, some of them were actually successful. I certainly don't mean to pick on the JPL / UCSB / Caltech team here, but they promptly posted a YouTube video showing their VRC competition run and some of their techniques are quite surprising. Have a look (and make sure to watch until the end!):

I have to question how well their "crab walk" technique is actually going to work in either the DARPA Robotics Challenge (DRC) or the "real world" that everyone keeps talking about. As far as I know, ATLAS was not designed to continually scrape along the ground like that, and I can't imagine the ground in the DRC is going to be as forgiving as the ground in the virtual simulator. There's also this quote from the video:

"Our software is easily capable of reliably walking through the bricks using the Boston Dynamics walking controller and our own perception software. We also have a self-righting behavior that allows us to stand up, but we determined that it was not reliable against unfortunately located bricks. Therefore, we chose to crab walk through the bricks as well."

That sort of makes it sound like standing up was tough, so they just went with the crab walk instead because it's easier. It may not be realistic, but it worked efficiently in the sim, so that's what the team went with. There's also the issue of what actions cause "damaging falls" and what actions cause falls that are somehow not damaging. Apparently a standing faceplant on mud does not damage the robot, according to the simulator, but I have trouble believing that the actual ATLAS robot is completely fine with that.

Again, I'm not trying to pick on any specific team here. What I'm getting at is that as sophisticated as the Gazebo simulator is, it's still just a simulator, and it seems like clever VRC teams did their best to exploit that fact in order to achieve the highest scores possible, irrespective of whether or not what they learned and presented in the VRC will actually help them in the real-world competition. The description from the video above spells it out directly:

The term "point space" is newly created, referencing "joint space" but highlighting problem solving that does what it takes to score points with HIGH RELIABILITY. In practice, this involved exploiting the simplified simulator collision geometry (boxes and cylinders) and kinematic singularities, in the DARPA VRC case.

This all leaves me wondering just how legit it is for DARPA to be awarding ATLAS robots in this manner, where high simulator scores do not necessarily correlate with behaviors or techniques that seem, you know, realistic. And if it turns out that crab walking over rough terrain really is the way to go, then ATLAS is not a good platform for that anyway, and some of those other Track A robots are probably going to be more successful. 

Okay, I'll stop being grouchy now! :)



If you're not familiar with Reddit, it's a community website with millions of users who cooperatively vote on content in every subject you can imagine (and lots of subjects that you can't). One of the Reddit sub-communities is a place where nobodies like you and I can ask questions to famous and interesting people who stop by, and last week, DARPA program manager Gill Pratt and OSRF CTO Nate Koenig spent an hour answering questions about the DRC and VRC. I've gone through and excerpted some of the questions and answers and reposted them (in order of popularity) below:

Hi guys! What was the most innovative, most surprising, and/or weirdest solution you saw to one of the virtual challenge tasks?

Nate Koenig, CTO, OSRF
A Fosbury flop into the Polaris Ranger was a very creative solution for car ingress. A more practical solution to car entry used a chest fly press to leverage Atlas into the driver's seat.
Team's were penalized for falling too many times. This lead to innovative walking strategies. Many teams choose to crab-crawl if they fall rather than stand-up. This is a very stable method of traversing uneven and rubble strewn ground.

Two questions, did most of the VRC teams tele-op or hard code the qualification tasks?
Also, will we be able to watch screencasts of the competition? I am particularly interested in seeing how teams fared in the driving competition

Gill Pratt, DARPA
We saw a mixture of tele-op and semi-autonomous operation, both in the qualification and the VRC itself. Most of the better performing teams used semi-autonomous operation.
Watch the DRC website for upcoming video highlights of the VRC.

How optimistic is DARPA towards the possible success and implications of the DRC (Darpa Robotics Challenge)?
If it's not successful, do you think funding will be significantly reduced in the future? In essence, I'm wondering how does DARPA deal with potential failures in light of tackling DARPA-hard problems.

Gill Pratt, DARPA
We are highly optimistic. The VRC teams have shown tremendous ingenuity so far, and we expect this to carry through to the DRC trials. We do expect to see failure as well, but this is part of DARPA's mission to explore high risk, high reward breakthrough technologies. We structured the challenge over three events so contestants would have time to improve their systems.
While the robots being developed in the DRC program will not be hardened to be immediately useful for disaster response, we believe the DRC will agressively push the software and hardware of disaster response robotics forward. We believe that the example of the DARPA Grand and Urban challenges that developed self-driving cars is a useful guide, with technology developed during those challenges now finding its way into driver assist devices and prototype self-driving vehicles.

What do you feel are the hardest issues in simulating a real robot?
What steps have you taken, and will you take in order to make the simulated robot be as realistic to the actual robot as possible?
With the VRC being in simulation, are you concerned about any of the competitors approaches mapping onto real robots?
Did any teams have approaches that would CLEARLY not work on a real robot, but was successful due to the simulated nature of the challenge?

Nate Koenig, CTO, OSRF
One of the primary challenges in simulation is balancing performance with accuracy. A robot with many degrees of freedom that also produces many contacts when manipulating objects introduces numerous constraints on simulation.
We wanted simulation to run as close to real time as possible, while still emulating the physical properties of a real robot. We integrated solutions simulate low-latency, high frequency control loops seen in physical hardware, and improved the speed and accuracy of simulation through software engineering techniques and integrating more realistic friction, contact, and noise models.
Gazebo has maintained an simple and easy transition from simulation to reality. Our focus has been on simulation fidelity, as experienced by the robot. While some changes will be necessary, a majority of the code and algorithms will be transferable to the physical Atlas.
Some of the locomotion strategies used would be inadvisable with the real Atlas. For example, butt scrooching and crab-walking would place a lot of stress on the hands. A few teams also crawled through the mud pit. This would have unfortunate effects in the real world.

What happens to the ATLAS robots after the DRC? Will teams be able to keep them and teach them to fetch beer and play soccer and not take over the world and stuff?

Gill Pratt, DARPA
They won't take over the world, but they will return to DARPA for potential use in future programs.

I've seen DARPA putting out a lot of open programs and "challenges" recently. DARPA's put out the Spectrum Challenge, the Armor Challenge, Cyber Fast Track, and DARPA FANG.
Do you think there is a fundamental shift at DARPA or within government R&D community in general to rely as much as possible on the contributions and innovations of external project teams? Furthermore, do you believe that crowd-sourced innovation will eventually be a major component of DoD innovation, as opposed to the typical goverment/government-funded contractor research model (or can the two coexist)?

Gill Pratt, DARPA
For DRC we're using a blended approach of funded and non-funded participation. In recent years DARPA has explored a variety of Challenges as a means of increasing the pool of participation and innovative ideas. We will continue to investigate opportunities where it makes sense to use Challenges to gain forward momentum in specific areas of research.

Hey Nate, I biked to work and forgot to bring pants, got any I can borrow?

Nate Koenig, CTO, OSRF
I have shorts under my desk. They are even fresh!

What are the plans for the Simulator after this stage of the competition?

Gill Pratt, DARPA
DARPA will continue to fund development of the simulator through the DRC finals, and we expect DRC contestants will use the simulator to help develop software for their real robots. In addition, we are working with OSRF to make the simulator available for use by anyone in the robotics field, including companies, educational instutions, and individuals.

Nate Koenig, CTO, OSRF
We have plans to develop many new and interesting features such as new sensors, more robots, scripting interfaces to facilitate dynamic and controllable environments, fluid dynamics, and much much more.
Gazebo is completely open-source. We like to think of ourselves as guiding an active community of developers, users, and contributors. If there are any new and interesting features you'd like to see in Gazebo, let us know!

Any plans on integrating Gazebo with the FIRST Robotics Competition? Pretty please :)

Nate Koenig, CTO, OSRF
Yes. We are actively working on this.

Thank you so much for doing this! I've been following the DRC for a while and am excited to see the outcomes from it. What's the next big project or step after the DRC? Do you have anything lined up in the pipelines that people can start thinking about?

Gill Pratt, DARPA
Ideally, we want to see the DRC go international, moving from the United States to other parts of the world, perhaps on a rotating basis. Resilience to natural and man-made disasters is a universal human need, and we believe the whole world will want to be involved in its solution.
At DARPA, a number of programs (e.g. Revolutionizing Prosthetics, Autonomous Robotic Manipulation, Maximum Mobility and Manipulation, and the DARPA Robotics Challenge) are producing results that we believe will move technology significantly forward in humanitarian assistance, dsiaster response, and civilian applications, with the promise of significantly improving the human condition.

Given the number of funded teams that didn't make the cut and the number of unfunded teams that achieved impressive results, will DARPA consider going back to a challenge model that does not provide any funding for the initial phase and only starts funding teams upon winning a preliminary competition?

Gill Pratt, DARPA
The DRC used a diversity of methods for teams to enter the competition. We provided intiial funding, most significantly for hardware, because robots capable of doing the DRC tasks are not presently available at low cost. At the same time, we realized that some teams would best demonstrate their competence by writing software instead of writing proposals, so we allowed such teams to compete as well.

If ATLAS falls on its back, can it stand up?

Gill Pratt, DARPA

What happens to the relatively large number of track b teams who didn't qualify? Can they find a hardware partner to participate in the DRC?
Lastly, on a scale of "what" to "wtf", how disappointed are you at the track b teams?

Gill Pratt, DARPA
We are quite impressed with the performance of some of the track B teams, several members of which will go to the DRC Trials. Members of teams who did not qualify may merge with other teams who did, or with track A teams. There is a lot of talent out there beyond the limited number of teams we could select for continued funding, and we expect several of the funded teams to invite other individuals to join them.

What real world products do you think we can see from this in the next 10 years ?

Gill Pratt, DARPA
Aside from robots for disaster response, we believe that task level supervised autonomy, improvements in energy efficiency, and improvements in low-cost manufacturing will enable robots to help improve protection and productiv

How do you know that the ATLAS model in the simulator matches the real robot? If the match is not good, control code developed in the simulator will not work well in the real robot and that would be really bad. So what tests did you do with the actual ATLAS to avoid this problem?

Nathan Koenig, CTO, OSRF
OSRF worked closely with BDI to develop an accurate simulation of Atlas.
In the very near future OSRF will collaborate with an external agency to validate simulation.

[ Reddit AMA ]

[ OSRF ]

[ DRC ]

The Conversation (0)