In February of 2011, NASA launched Robonaut 2 to the International Space Station. It was a huge achievement for the robotics team at NASA’s Johnson Space Center in Houston. There had been other robots in space, but Robonaut was the first advanced humanoid to ever go on a mission beyond Earth. On board the ISS, the robot was intended to eventually work side by side with astronauts, performing some of the dull and repetitive tasks that take up a significant amount of time that the humans on the station could instead be spending on science and discovery.
For a while, things went well. The robot was unboxed from its protective foam packaging, and set up in the Destiny laboratory module. It was powered up for the first time in August of 2011, and by 2012 it was flipping practice switches and cleaning practice handrails while being teleoperated from the ground. Once a month or so, astronauts set Robonaut up and it would do research tasks for several hours at a time, working towards making a transition from an experimental project to a useful helper in the caretaking of human spacecraft. Robonaut even got its own Twitter account: “Check me out. I’m in space!”
But in 2014, NASA decided to go forward with a complex and risky upgrade. The Robonaut sent to the ISS was a torso with a pair of arms and head, and now NASA wanted to add a pair of legs. The idea was to give Robonaut more mobility inside the station. The elongated, wiggly legs would work almost as a secondary pair of arms, allowing the robot to move itself around the station by holding on to hand rails, significantly adding to its capability.
But the upgrade didn’t go according to plan, and instead created a persistent problem that astronauts have been unable to fix. For the last several years, Robonaut has been almost entirely disabled, and publicly available ISS status reports show that the last time the robot completed a full research task was December of 2013. This week, NASA announced that it is bringing Robonaut back to Earth to be fixed.
The decision to give Robonaut mobility had always been part of NASA’s long-term plan for the robot. NASA originally intended for there to be three distinct phases of Robonaut operations. The first phase involved stationary operations, and was completed successfully by the end of 2013 using the Robonaut torso. Phase II was “IVA Mobility,” where IVA stands for “intravehicular.” For this phase, Robonaut needed a way to move around the inside of the ISS, which is where the legs came in. The legs would also be key for Phase III, “EVA Mobility,” which would involve working in vacuum outside of the station, although this scenario would have required even more major hardware upgrades to the robot.
“The space station program was interested in expanding our reach,” Julia Badger, Robonaut Project Manager at NASA’s Johnson Space Center, told IEEE Spectrum this week. “If we were able to move around, we could do logistics management, we could possibly clean filters, or do repairs. Moving around was the next level of capabilities that we’d be able to develop and test.”
But adding new legs to the ISS Robonaut would be much more involved than simply shipping them up to the station and bolting them onto the Robonaut torso. The leg installation would require upgrading a significant amount of Robonaut’s core hardware, including new computers and new wiring to interface the legs to the robot’s main processor, not to mention a complex mechanical assembly process. Further complicating things was the fact that Robonaut was originally developed as a research robot for use on Earth. “Robonaut was not designed at all to be serviceable by astronauts,” Badger said. “It was designed to be a lab robot, and we jumped at the opportunity to send it to space.”
Even for NASA, the leg upgrade was ambitious. Asked if the team was confident it would work, Badger said: “I definitely don’t think we were confident. The ISS basically is about trying new things, so we learned a lot from developing those procedures and having the astronauts run through that complex surgery.” She added that the crew members “did an outstanding job” and that the problem with the robot didn’t involve what the astronauts did, but derived from the complexities of the hardware.
NASA expected that the leg upgrade, which took the Robonaut team 14 hours to do on Earth, would take the astronauts about 20 hours to complete. It ended up taking about 40. The work began on July 16, 2014, and was completed and called successful on August 28.
Almost immediately the Robonaut team at JSC realized something was wrong. On August 29, the robot was partially powered up, but they weren’t able to see any telemetry on the ground. Not long after, a loose wire was fixed and the robot passed intermediary checkouts, but during the next operation on December 17 (the first time motor power was applied to Robonaut after the upgrade), the legs wouldn’t move.
Between January and August 2015, astronauts and ground teams recorded a series of erratic behaviors, including sensor failures, communication failures, and repeated processor lockups. By September, continued troubleshooting suggested that the issue was with the power supply to Robonaut’s processors. “We would start losing power to our computers within our operational window, and it got more and more severe as time went on,” Badger told us. “A power cycle would in general bring it back, just for a little while. The problem was that since it was intermittent, sometimes we’d be able to turn it on and sometimes it would just fail right away as it degraded, we weren’t necessarily able to trust the data—it was very confusing.”
David Wettergreen, a roboticist at Carnegie Mellon University, in Pittsburgh, who specializes in autonomous robots for planetary exploration, explained that the complexity of robotics systems makes error detection and prevention a massive task. “For simpler systems it is reasonable to assume that everything will work, and when it does not, then it can be debugged and repaired, like your car,” he said. “But if a robot won’t be accessible, like an autonomous underwater vehicle, or if safety is paramount, like a self-driving car, then either all of the failure modes need to be understood and mitigated in advance or the robot needs to be able to diagnose and repair itself.” To make things even more challenging, he added, techniques like fault tolerance and fault recovery are areas of substantial active research.
“Robonaut must be both robust and safe,” he said, “so the systems engineering required is at the cutting-edge of technology.”
The ISS daily summary reports provide a glimpse into how much both the astronauts on the station and the Robonaut team on the ground were struggling to understand what was going wrong with the robot. By 2016, it seems as though Robonaut was being taken apart and analyzed almost piece by piece.
[Astronaut Tim] Kopra set up a camcorder to capture standard definition video downlinks of Robonaut operations in the US Lab. He used the oscilloscope, multimeter and current probe to help guide ground teams with this troubleshooting by disengaging the controller card and successfully powering Robonaut without the controller card. The data collected from today’s activity will be analyzed by the ground to determine the cause of the fault in the power supply.
—Excerpt from ISS Daily Summary Report, 3/23/16
Two controller cards that the JSC team suspected could be the culprits were even shipped back to Earth for testing. But once inspected at the lab, the cards checked out okay, and the search for a fix continued, with the team checking and double-checking everything including the mass of cables inside the robot:
The crew performed troubleshooting on the Robonaut chassis. Previous troubleshooting suggested an issue with two card[s] that were returned and tested on the ground. The cards brought down were exonerated and during today’s activities teams received a good response from the Robonaut during first power up. Indicator lights were as expected for a working Robonaut. Various power cycles were performed while trying to narrow down intermittent fault statuses. Eventually the team exonerated several of the cables. The cards were not removed from the CPCI [Compact Peripheral Component Interconnect] chassis and remain installed and all cables were left connected. Teams plan to analyze results to develop any forward actions.
—Excerpt from ISS Daily Summary Report, 2/1/17
It may seem surprising that NASA’s robotics experts weren’t able to diagnose and fix the problem more quickly, since space hardware is painstakingly designed, built, and tested before going into space. But the ISS Robonaut that the astronauts were working on is slightly different from the other five Robonauts at JSC. The space station Robonaut is an R2-B model, while the robots on Earth are R2-Cs, a later version that is “a fairly significant iteration of design,” according to Badger. In the end, she said these differences finally provided the answer, years after the problem first appeared.
“It took us a really long time to figure it out, but what ended up being the case was that the [ISS] robot was missing its ground path from its computer chassis to the ground,” she said. “The current was finding another path through, and it was slowly degrading the robot. Because that robot on orbit was different from the ones on the ground, it was kind of like a mistake that that ground path wasn’t there, and it was very difficult to find.”
What this meant is that electrical current wasn’t flowing the way it should through Robonaut’s body. It’s possible that some of its circuits and processors were getting no power at all while other components were essentially getting fried by too much current. The fault was slowly killing Robonaut.
Wettergreen, from CMU, said he’s encountered similar scenarios. “Errors in properly grounding circuits can lead to really strange symptoms that initially appear unrelated to the root cause—seemingly anything can happen with the electronics,” he said. “It can take a long time to debug because the problem is not easily reproducible and may not occur often, or even in the same way every time.”
Last August, astronauts tried to install a grounding jumper, but that didn’t solve things. The place where they wanted to add the jumper, Badger explained, was covered in a special sealant used to protect sensitive components against shorts caused by foreign objects that can be floating around the ISS. The astronauts had to try to chip that away, and “there was a lot of difficulty,” Badger said. “There was some doubt left on whether that was actually an effective connection or not. We’re also quite confident that there’s likely other damage that was caused by this missing wire.”
It’s this other damage that is the most concerning. Even if the original grounding issue has been solved, and the Robonaut team isn’t sure it has been, Robonaut is almost certainly suffering from significant degradation caused by current running unchecked through its computer chassis. There were some additional things the team could have tried, but when NASA asked if they wanted to use some free cargo space on the return flight of an upcoming resupply mission to bring Robonaut home, the team said yes. Depending on how much damage they find, the plan is to either repair the ISS Robonaut and send it back up to the station, or swap it for one of the R2-C units at JSC and send that one up instead.
Badger wasn’t able to provide a timeframe for when Robonaut may be back on the ISS—there’s no guaranteed mission, and they’ll have to find space, which could be a challenge since Robonaut is a relatively large payload. This means that they might not even be able to have the legs attached when they send it up for the second time, although Badger promises if that’s the case, “it will be much, much simpler to reintegrate.”
Last Friday, Robonaut was packed up to get it ready to come back to Earth. As the team in Houston prepares to get their robot back in shape, Badger reminds us that it’s important to keep the overall Robonaut mission in context.
“The whole point of the ISS is to be able to try different things out. I think [Robonaut] has given us a lot of knowledge on what the requirements for humanoid robots in space will be in the future,” she said. “We’re bringing it home, repairing it, and in the near future after that, we’re hoping to fly it up there again to proceed with our original goals of advancing new technology.”
[ NASA Robonaut ]
Updated 2/16/18 4:50 p.m.: Added comments by CMU roboticist David Wettergreen.
An abridged version of this post appears in the April 2018 print issue as “Robonaut Returns for Repairs.”