ISS Astronauts Operating Remote Robots Show Future of Planetary Exploration

If we want to send humans to Mars, we'll need astronaut-robot collaboration systems like this

3 min read
ISS astronauts control Rollin' Justin robot on a simulated Mars environment
ISS astronauts Jack Fischer and Paolo Nespoli [on screen] operate the humanoid robot Rollin' Justin as part of an experiment to develop control interfaces and telerobotic systems for astronaut-robot collaboration.
Photo: DLR

In late August, an astronaut on board the International Space Station remotely operated a humanoid robot to inspect and repair a solar farm on Mars—or at least a simulated Mars environment, which in this case is a room with rust-colored floors, walls, and curtains at the German Aerospace Center, or DLR, in Oberpfaffenhofen, near Munich.

DRL Rollin' Justin humanoid robot Rollin’ Justin at DLR’s simulated Martian solar farm. Photo: DLR

European Space Agency astronaut Paolo Nespoli commanded the humanoid, called Rollin’ Justin, as the robot performed a series of navigation, maintenance, and repair tasks. Instead of relying on direct teleoperation, Nespoli used a tablet computer to issue high-level commands to the robot. In one task, he used the tablet to position the robot and have it take pictures from different angles. Another command instructed Justin to grasp a cable and connect it to a data port.

Roboticists call this approach “supervised autonomy,” and it offers a number of advantages over either full autonomy (in which the robot attempts to do everything on its own) and direct teleoperation (in which the astronaut needs to control every movement of the robot). Supervised autonomy is a robust way of handling unexpected errors and limitations like communication delays. The astronaut acts a supervisor of the robot, and if the robot gets stuck, the human can help it complete the task.

“This concept relies on the robot’s local intelligence to reason and plan the commanded tasks,” says Dr. Neal Y. Lii, the experiment’s principal investigator at DLR’s Robotics and Mechatronics Center. He explains that this allows the operator to carry out tasks without the cognitive strain and pressure of an immersive telepresence system with haptic feedback and visual servo control. “The robot’s intelligence always keeps the robot in a safe state so that it can wait for the feedback and command from the astronaut.”

The experiment, called SUPVIS Justin, was led by DLR in partnership with ESA. It is part of a broader program, the Multi-Purpose End-To-End Robotic Operation Network, or METERON project. Initiated by ESA with DLR, NASA, and Roscosmos, METERON consists of a series of space telerobotics experiments. The goal is to develop advanced human-robot collaboration capabilities to help with future planetary exploration missions.

The idea is that, if humans want to go to Mars, they’ll need to build and maintain habitats and other infrastructure on the surface of the planet, and robots could be a huge help. Before landing, astronauts would remain in orbit and send robots to the surface to work on the needed infrastructure. ESA, and NASA, too, appear to believe that this is the only realistic option to bring humans to Mars. Hence the mock Martian solar farm at DLR.

ISS astronauts Jack Fischer and Paolo Nespoli (on screen) operate the humanoid robot Rollin' Justin as part of an experiment to develop control interfaces and telerobotic systems for astronaut-robot collaboration. Astronauts Jack Fischer (left) and Paolo Nespoli use a tablet PC to operate Rollin' Justin from the ISS. Image: DLR

In the experiment in August, Italian astronaut Nespoli performed two sequences of tasks, or protocols. In the first, he used the robot to perform a solar panel unit inspection and reboot mission. In the second protocol, he performed a system software upgrade mission. 

Nespoli went through the two protocols so efficiently that, with some time left, he offered a surprise to the DLR team on the ground: He invited his American colleagues Jack Fischer and Randy Bresnik to try the experiment as well. Although they had never trained to use the tablet interface, the two NASA astronauts, with assistant from Nespoli and the team on the ground, were able to complete some of the tasks without difficulty.

DLR ground control team during SUPVIS Justin experiment Ground control crew (left to right): co-investigator Thomas Krüger (ESA), co-investigator Daniel Leidner (DLR), co-investigator Peter Birkenkampf (DLR), and principal investigator Neal Y. Lii (DLR). Photo: DLR

Dr. Lii says two more SUPVIS Justin sessions are scheduled. For the next experiment, his team will make improvements to the tablet’s user interface as well as the robot’s functionalities based on the experience of the ISS crew. “At the same time, we will increase the task complexity and difficulty with each successive experiment session to get a view into the performance envelope we can expect from the astronauts,” he says. In the future, they also plan to test telepresence systems that offer a more immersive experience, because such systems might be necessary during certain missions.

“This allows the user to move between using the robot as a coworker  in supervised autonomy form, or as a haptically coupled physical extension on location,” Dr. Lii says. “Ultimately, our hope is to assemble a benchmark/guideline/playbook for designing intuitive space robotic teleoperation systems.”

The Conversation (0)

How the U.S. Army Is Turning Robots Into Team Players

Engineers battle the limits of deep learning for battlefield bots

11 min read
Robot with threads near a fallen branch

RoMan, the Army Research Laboratory's robotic manipulator, considers the best way to grasp and move a tree branch at the Adelphi Laboratory Center, in Maryland.

Evan Ackerman

“I should probably not be standing this close," I think to myself, as the robot slowly approaches a large tree branch on the floor in front of me. It's not the size of the branch that makes me nervous—it's that the robot is operating autonomously, and that while I know what it's supposed to do, I'm not entirely sure what it will do. If everything works the way the roboticists at the U.S. Army Research Laboratory (ARL) in Adelphi, Md., expect, the robot will identify the branch, grasp it, and drag it out of the way. These folks know what they're doing, but I've spent enough time around robots that I take a small step backwards anyway.

This article is part of our special report on AI, “The Great AI Reckoning.”

The robot, named RoMan, for Robotic Manipulator, is about the size of a large lawn mower, with a tracked base that helps it handle most kinds of terrain. At the front, it has a squat torso equipped with cameras and depth sensors, as well as a pair of arms that were harvested from a prototype disaster-response robot originally developed at NASA's Jet Propulsion Laboratory for a DARPA robotics competition. RoMan's job today is roadway clearing, a multistep task that ARL wants the robot to complete as autonomously as possible. Instead of instructing the robot to grasp specific objects in specific ways and move them to specific places, the operators tell RoMan to "go clear a path." It's then up to the robot to make all the decisions necessary to achieve that objective.

Keep Reading ↓ Show less