The December 2022 issue of IEEE Spectrum is here!

Close bar

Robots Have a Hard Time Grasping These "Adversarial Objects"

To make robot grasping more robust, researchers are designing objects that are as difficult as possible for robots to manipulate

4 min read
UC Berkeley researchers are designing objects that are as difficult as possible for robots to grasp.
Inspired by the concept of adversarial images, UC Berkeley researchers are exploring physical adversarial objects carefully designed to be tricky for conventional robot grippers to pick up.
Photo: UC Berkeley

There’s been a bunch of research recently into adversarial images, which are images of things that have been modifiedto be particularly difficult for computer vision algorithms to accurately identify. The idea is that these kinds of images can be used to help design more robust computer vision algorithms, because their “adversarial” nature is sort of a deliberate worst-case scenario—if your algorithm can handle adversarial images, then it can probably handle most other things.

Researchers at UC Berkeley’s Laboratory for Automation Science and Engineering (AUTOLAB), directed by Professor Ken Goldberg, have been extending this concept to robot grasping, with physical adversarial objects carefully designed to be tricky for conventional robot grippers to pick up. All it takes is a slight tweak to straightforward three-dimensional shapes, and a standard two-finger will have all kinds of trouble finding a solid grasp.

adversarial objects for robotsStarting from a cube, the adversarial object evolves to become increasingly more difficult for a two-finger gripper to pick up—when it tries to pinch against an angled surface, the object twists and slips off.Image: UC Berkeley

The key to these adversarial objects is that they look easy to grasp, but at least for a two-finger (parallel-jaw) gripper, they’re not. The difference between what the objects look like and what their actual geometries are is subtle: In one of the examples, you can see a cube with some shallow pyramids on three of the six sides—the smallest pyramid has a slope of just 10 degrees. The side opposite each pyramid is a regular, flat face, and the result is that there are no directly opposing faces on the cube. This causes problems for two-finger grippers, which work by pinching things, and if you’re trying to pinch against an angled surface, the force you exert will tend to cause the object to twist, often leading to a failed grasp. 

Robot fails to grasp adversarial objectA parallel jaw gripper with point contact fingers successfully grasps a regular cube (left) and fails to grasp an adversarial cube designed by the researchers (right).Image: UC Berkeley

Grasp planners often look for smooth opposing surfaces that are “pinchable,” and because the difference between the adversarial cube and a true cube is small enough that it wouldn’t necessarily be picked up by a 3D sensor, most robotic systems would be like, “Oh look, a cube, that’s easy!” And then they’d likely fail, with the cube twisting and slipping out of its grasp.

Researcher uses metal thimbles to simulate a robot gripperThe adversarial shapes work on humans, too—wearing thimbles to emulate the cold unfeeling steel of a robot and using two fingers in a pinching grasp, the researchers were able to verify that the objects were hard to pick up.Photo: UC Berkeley

As the complexity of a shape increases, it gets harder to develop an adversarial version. With a cuboctahedron (a polygon that has eight triangular faces and six square faces), the researchers randomly perturbed the vertices of the shape (in simulation) until they ended up with one that had no directly opposing surfaces. For even more complex shapes, like intersecting cylinders, adversarial examples were generated with a deep learning algorithm. 

The point of all this isn't just to frustrate your poor grasping algorithm—it's to develop tools that will help make robot grasping robust enough to (hopefully) reliably work in the real world.

Adversarial objects for robotsFor complex shapes, like intersecting cylinders, the researchers used deep learning algorithms to generate adversarial designs.Image: UC Berkeley

In some preliminary real-world testing, a parallel jaw gripper with point contact fingers tried to pick up some of these adversarial objects. In each case, the computed grasp (based on a Dex-Net policy run on the object in simulation) was predicted to succeed 100 percent of the time, but the actual success rates on the adversarial cubes and cuboctahedrons was just 13 percent.

The researchers say that they plan to test these objects with different gripper types, as well as suction grasps, to see if they can come up with the most adversarial adversarial objects. And remember, the point of all this isn’t just to frustrate your poor grasping algorithm—it’s to develop tools that will help make robot grasping robust enough to (hopefully) reliably work in the real world.

Adversarial Grasp Objects,” by David Wang, David Tseng, Pusong Li, Yiding Jiang, Jeffrey Mahler, and Ken Goldberg from UC Berkeley will be presented at the IEEE International Conference on Automation Science and Engineering (CASE) next month.

The Conversation (0)

The Bionic-Hand Arms Race

The prosthetics industry is too focused on high-tech limbs that are complicated, costly, and often impractical

12 min read
A photograph of a young woman with brown eyes and neck length hair dyed rose gold sits at a white table. In one hand she holds a carbon fiber robotic arm and hand. Her other arm ends near her elbow. Her short sleeve shirt has a pattern on it of illustrated hands.

The author, Britt Young, holding her Ottobock bebionic bionic arm.

Gabriela Hasbun. Makeup: Maria Nguyen for MAC cosmetics; Hair: Joan Laqui for Living Proof

In Jules Verne’s 1865 novel From the Earth to the Moon, members of the fictitious Baltimore Gun Club, all disabled Civil War veterans, restlessly search for a new enemy to conquer. They had spent the war innovating new, deadlier weaponry. By the war’s end, with “not quite one arm between four persons, and exactly two legs between six,” these self-taught amputee-weaponsmiths decide to repurpose their skills toward a new projectile: a rocket ship.

The story of the Baltimore Gun Club propelling themselves to the moon is about the extraordinary masculine power of the veteran, who doesn’t simply “overcome” his disability; he derives power and ambition from it. Their “crutches, wooden legs, artificial arms, steel hooks, caoutchouc [rubber] jaws, silver craniums [and] platinum noses” don’t play leading roles in their personalities—they are merely tools on their bodies. These piecemeal men are unlikely crusaders of invention with an even more unlikely mission. And yet who better to design the next great leap in technology than men remade by technology themselves?

Keep Reading ↓Show less