Adversarial Objects Train Robots To Improve Ability To Grasp

Activities humans take for granted prove to be harder than they look for robots. So researchers are designing methods to train them. JL

Evan Ackerman reports in IEEE:

Adversarial images are images that have been modified to be difficult for computer vision algorithms to identify. These images can be used to design more robust computer vision algorithms, because their “adversarial” nature is a deliberate worst-case scenario—if your algorithm can handle adversarial images, then it can handle most things. Researchers have been extending this concept to robot grasping, with physical adversarial objects designed to be tricky for conventional robot grippers to pick up. The point is to develop tools that will make robot grasping robust enough to work in the real world.
There’s been a bunch of research recently into adversarial images, which are images of things that have been modified to be particularly difficult for computer vision algorithms to accurately identify. The idea is that these kinds of images can be used to help design more robust computer vision algorithms, because their “adversarial” nature is sort of a deliberate worst-case scenario—if your algorithm can handle adversarial images, then it can probably handle most other things.
Researchers at UC Berkeley have been extending this concept to robot grasping, with physical adversarial objects carefully designed to be tricky for conventional robot grippers to pick up. All it takes is a slight tweak to straightforward three-dimensional shapes, and a standard two-finger will have all kinds of trouble finding a solid grasp.

Image: UC Berkeley

Starting from a cube, the adversarial object evolves to become increasingly more difficult for a two-finger gripper to pick up—when it tries to pinch against an angled surface, the object twists and slips off.

The key to these adversarial objects is that they look easy to grasp, but at least for a two-finger (parallel-jaw) gripper, they’re not. The difference between what the objects look like and what their actual geometries are is subtle: In one of the examples, you can see a cube with some shallow pyramids on three of the six sides—the smallest pyramid has a slope of just 10 degrees. The side opposite each pyramid is a regular, flat face, and the result is that there are no directly opposing faces on the cube. This causes problems for two-finger grippers, which work by pinching things, and if you’re trying to pinch against an angled surface, the force you exert will tend to cause the object to twist, often leading to a failed grasp.

The point of all this isn’t just to frustrate your poor grasping algorithm—it’s to develop tools that will help make robot grasping robust enough to (hopefully) reliably work in the real world.

Grasp planners often look for smooth opposing surfaces that are “pinchable,” and because the difference between the adversarial cube and a true cube is small enough that it wouldn’t necessarily be picked up by a 3D sensor, most robotic systems would be like, “Oh look, a cube, that’s easy!” And then they’d likely fail, with the cube twisting and slipping out of its grasp.

Photo: UC Berkeley

The adversarial shapes work on humans, too—wearing thimbles to emulate the cold unfeeling steel of a robot and using two fingers in a pinching grasp, the researchers were able to verify that the objects were hard to pick up.

As the complexity of a shape increases, it gets harder to develop an adversarial version. With a cuboctahedron (a polygon that has eight triangular faces and six square faces), the researchers randomly perturbed the vertices of the shape (in simulation) until they ended up with one that had no directly opposing surfaces. For even more complex shapes, like intersecting cylinders, adversarial examples were generated with a deep learning algorithm.

Image: UC Berkeley

For complex shapes, like intersecting cylinders, the researchers used deep learning algorithms to generate adversarial designs. 

In some preliminary real-world testing, a parallel jaw gripper with point contact fingers tried to pick up some of these adversarial objects. In each case, the computed grasp (based on a Dex-Net policy run on the object in simulation) was predicted to succeed 100 percent of the time, but the actual success rates on the adversarial cubes and cuboctahedrons was just 13 percent. The researchers say that they plan to test these objects with different gripper types, as well as suction grasps, to see if they can come up with the most adversarial adversarial objects. And remember, the point of all this isn’t just to frustrate your poor grasping algorithm—it’s to develop tools that will help make robot grasping robust enough to (hopefully) reliably work in the real world.