Robots Have a Hard Time Grasping These "Adversarial Objects"

To make robot grasping more robust, researchers are designing objects that are as difficult as possible for robots to manipulate

UC Berkeley researchers are designing objects that are as difficult as possible for robots to grasp.
Photo: UC Berkeley
Inspired by the concept of adversarial images, UC Berkeley researchers are exploring physical adversarial objects carefully designed to be tricky for conventional robot grippers to pick up.

There’s been a bunch of research recently into adversarial images, which are images of things that have been modified to be particularly difficult for computer vision algorithms to accurately identify. The idea is that these kinds of images can be used to help design more robust computer vision algorithms, because their “adversarial” nature is sort of a deliberate worst-case scenario—if your algorithm can handle adversarial images, then it can probably handle most other things.

Researchers at UC Berkeley’s Laboratory for Automation Science and Engineering (AUTOLAB), directed by Professor Ken Goldberg, have been extending this concept to robot grasping, with physical adversarial objects carefully designed to be tricky for conventional robot grippers to pick up. All it takes is a slight tweak to straightforward three-dimensional shapes, and a standard two-finger will have all kinds of trouble finding a solid grasp.

The key to these adversarial objects is that they look easy to grasp, but at least for a two-finger (parallel-jaw) gripper, they’re not. The difference between what the objects look like and what their actual geometries are is subtle: In one of the examples, you can see a cube with some shallow pyramids on three of the six sides—the smallest pyramid has a slope of just 10 degrees. The side opposite each pyramid is a regular, flat face, and the result is that there are no directly opposing faces on the cube. This causes problems for two-finger grippers, which work by pinching things, and if you’re trying to pinch against an angled surface, the force you exert will tend to cause the object to twist, often leading to a failed grasp. 

Grasp planners often look for smooth opposing surfaces that are “pinchable,” and because the difference between the adversarial cube and a true cube is small enough that it wouldn’t necessarily be picked up by a 3D sensor, most robotic systems would be like, “Oh look, a cube, that’s easy!” And then they’d likely fail, with the cube twisting and slipping out of its grasp.

As the complexity of a shape increases, it gets harder to develop an adversarial version. With a cuboctahedron (a polygon that has eight triangular faces and six square faces), the researchers randomly perturbed the vertices of the shape (in simulation) until they ended up with one that had no directly opposing surfaces. For even more complex shapes, like intersecting cylinders, adversarial examples were generated with a deep learning algorithm. 

In some preliminary real-world testing, a parallel jaw gripper with point contact fingers tried to pick up some of these adversarial objects. In each case, the computed grasp (based on a Dex-Net policy run on the object in simulation) was predicted to succeed 100 percent of the time, but the actual success rates on the adversarial cubes and cuboctahedrons was just 13 percent.

The researchers say that they plan to test these objects with different gripper types, as well as suction grasps, to see if they can come up with the most adversarial adversarial objects. And remember, the point of all this isn’t just to frustrate your poor grasping algorithm—it’s to develop tools that will help make robot grasping robust enough to (hopefully) reliably work in the real world.

Adversarial Grasp Objects,” by David Wang, David Tseng, Pusong Li, Yiding Jiang, Jeffrey Mahler, and Ken Goldberg from UC Berkeley will be presented at the IEEE International Conference on Automation Science and Engineering (CASE) next month.

Robotics News

Biweekly newsletter on advances and news in robotics, automation, control systems, interviews with leading roboticists, and more.

About the Automaton blog

IEEE Spectrum’s award-winning robotics blog, featuring news, articles, and videos on robots, humanoids, automation, artificial intelligence, and more.