The December 2022 issue of IEEE Spectrum is here!

Close bar

Point-and-Click Method Makes Robot Grasping Control Less Tedious

Teleoperating complex robots is really hard, and Georgia Tech is working to fix that

3 min read
Georgia Tech researchers developed a novel point-and-click robot grasping system
Image: Georgia Tech RAIL Lab via YouTube

Until all robots everywhere are autonomous all the time, humans are going to have to take over once in awhile. This is going to happen more frequently as robots that are almost but not quite fully autonomous get deployed in residential, commercial, and industrial environments. When these robots get stuck on a task (and they definitely will get stuck), a human operator could hop in via telepresence to help them out. One problem, though, is that right now this teleoperation process is awfully tedious.

For most grasping tasks, when a robot needs help, it means that a human needs to manually position every single degree of freedom of the gripper while squinting at some low-resolution 3D point cloud. Georgia Tech researchers are working on making the process significantly less painful. Their approach involves getting rid of all of that manual positioning and using a friendly, interactive interface that takes care of everything with just one or two clicks.

Between the full manual and point-and-click grasping approaches shown in the above video, the researchers, from Georgia Tech’s Robot Autonomy and Interactive Learning (RAIL) lab, led by Professor Sonia Chernova, also implemented a middle ground “constrained positioning” method, which intelligently limits the amount of degrees of freedom that a user needs to position. Instead, the user needs to select only a grasp point, approach angle, and grasp depth. If you put these approaches together, you get a spectrum of options for teleoperated grasping, ranging from full 6-DoF manual control to 3-DoF constrained positioning grasping to single-click mostly automated grasping. You can see all of these methods in action here.

Georgia Tech's point-and-click teleoperation system offers a spectrum of options for robot grasping, from full 6-DoF manual control to 3-DoF constrained positioning grasping to single-click mostly automated grasping.

As the autonomy of these grasping approaches improves, the system makes increasing use of scene information, although it’s not necessarily doing scene or object recognition. In other words, it needs basic depth data in order to help you out with grasping, but it doesn’t need to understand what it’s looking at, making it easy to scale up and deploy to new environments without training. The three approaches also use only a mouse and keyboard, and are accessed through a friendly looking web interface, which helps users to get comfortable with the system. Comfort is an important point, because ideally, you want non-expert users to be able to control your robots without getting frustrated at the task, the robot, and (ultimately) at you, the robot owner.

A study of non-expert users showed that the point-and-click interface was the most effective, helping users to “complete a greater number of tasks more quickly, complete tasks more consistently, and make fewer mistakes,” as the RAIL lab researchers explain: 

While point-and-click had the clearest advantages over the other interfaces, constrained positioning does have a significant advantage over free positioning in reducing the number of errors made by users. The most frequent type of error was missing a grasp, in which the gripper failed to make contact with any part of the environment, object or otherwise. Both constrained positioning and point-and-click require the user to focus their end-effector positioning around a point cloud by clicking on a point to initiate the interaction. Constraining the interaction to a physical surface significantly reduces the number of missed grasps, resulting in more efficient use of the arm.

“A Comparison of Remote Robot Teleoperation Interfaces for General Object Manipulation,” by David Kent, Carl Saldanha, and Sonia Chernova from Georgia Tech, was presented at HRI 2017 in Vienna, Austria.

The Conversation (0)

The Bionic-Hand Arms Race

The prosthetics industry is too focused on high-tech limbs that are complicated, costly, and often impractical

12 min read
A photograph of a young woman with brown eyes and neck length hair dyed rose gold sits at a white table. In one hand she holds a carbon fiber robotic arm and hand. Her other arm ends near her elbow. Her short sleeve shirt has a pattern on it of illustrated hands.

The author, Britt Young, holding her Ottobock bebionic bionic arm.

Gabriela Hasbun. Makeup: Maria Nguyen for MAC cosmetics; Hair: Joan Laqui for Living Proof

In Jules Verne’s 1865 novel From the Earth to the Moon, members of the fictitious Baltimore Gun Club, all disabled Civil War veterans, restlessly search for a new enemy to conquer. They had spent the war innovating new, deadlier weaponry. By the war’s end, with “not quite one arm between four persons, and exactly two legs between six,” these self-taught amputee-weaponsmiths decide to repurpose their skills toward a new projectile: a rocket ship.

The story of the Baltimore Gun Club propelling themselves to the moon is about the extraordinary masculine power of the veteran, who doesn’t simply “overcome” his disability; he derives power and ambition from it. Their “crutches, wooden legs, artificial arms, steel hooks, caoutchouc [rubber] jaws, silver craniums [and] platinum noses” don’t play leading roles in their personalities—they are merely tools on their bodies. These piecemeal men are unlikely crusaders of invention with an even more unlikely mission. And yet who better to design the next great leap in technology than men remade by technology themselves?

Keep Reading ↓Show less