The December 2022 issue of IEEE Spectrum is here!

Close bar

Clever Cornell Robot Finds your Lost Keyboard

Cornell has taught a robot to categorize scenes and locate objects based on their relationships to other objects, which is way cooler than I just made it sound

2 min read
Clever Cornell Robot Finds your Lost Keyboard

Yesterday we posted a video showing how robots can use machine learning to adapt to objects and situations that they've never seen before. The same research group at Cornell has also been working on clever ways to allow robots to efficiently interpret scenes and identify objects, which is one of those things that robots are going to have to be good at before they can really become helpful in our homes.

Humans have the ability to look at a scene and immediately pick out important elements while ignoring everything else, because we have brains that are awesome like that. Robots, in general, don't really work that way. They have to examine each and every pixel of what they're looking at and decide whether or not it's something they're interested in. So if you ask a robot go to find you (say) a computer keyboard, it's got to enter a room and methodically search every pixel-equivalent area until it finds what it's looking for. While the robot is likely to find its target in this manner, it'll take roughly forever to do so.

What the Cornell research group has been teaching robots to do is to be able to rapidly break down a scene into general categories, and then recognize how some categories are related to others. In a general sense, this is the same thing that we humans do. For example, if you enter a room and want to locate a computer keyboard, you first (subconsciously) identify the places that you don't need to pay attention to, like the walls and ceiling and probably the floor. Next, if you don't immediately see the keyboard, you might take a closer look at the places where keyboards like to hang out, namely in close proximity to a computer. You see a computer, you get closer, and hey, there's the keyboard. Nicely done.

Now, watch a robot do the exact same thing: 

The key to getting this to work goes back to the machine learning and adaptability system that Cornell has been working on. First, they showed a robot a series of about 50 different scenes, with most objects in the scene labeled. The robot read the labels and then remembered characteristics of the associated objects (like color, texture, and what other objects are nearby), such that it was then able to use those characteristics categorize entirely new objects in environments it's never experienced.

Armed with this knowledge and way of thinking (if you want to call it that), the robot was then able to enter a room it had never been in before in search of a keyboard, and "know" that since it spotted a monitor, and keyboards are often found near monitors, it should go check out the monitor. And of course, the keyboard is right there.

This ability to follow chains of reasoning to identify, categorize, and locate objects is obviously still being developed, but again, giving robots the ability to understand context and use it to adapt to new things brings them that much closer to being the machines that take over all of our household chores and leave us with nothing better to do than lie around and play video games all day. Because that's what we all secretly want, right? ...Right?

[ Cornell ]

The Conversation (0)

The Bionic-Hand Arms Race

The prosthetics industry is too focused on high-tech limbs that are complicated, costly, and often impractical

12 min read
A photograph of a young woman with brown eyes and neck length hair dyed rose gold sits at a white table. In one hand she holds a carbon fiber robotic arm and hand. Her other arm ends near her elbow. Her short sleeve shirt has a pattern on it of illustrated hands.

The author, Britt Young, holding her Ottobock bebionic bionic arm.

Gabriela Hasbun. Makeup: Maria Nguyen for MAC cosmetics; Hair: Joan Laqui for Living Proof

In Jules Verne’s 1865 novel From the Earth to the Moon, members of the fictitious Baltimore Gun Club, all disabled Civil War veterans, restlessly search for a new enemy to conquer. They had spent the war innovating new, deadlier weaponry. By the war’s end, with “not quite one arm between four persons, and exactly two legs between six,” these self-taught amputee-weaponsmiths decide to repurpose their skills toward a new projectile: a rocket ship.

The story of the Baltimore Gun Club propelling themselves to the moon is about the extraordinary masculine power of the veteran, who doesn’t simply “overcome” his disability; he derives power and ambition from it. Their “crutches, wooden legs, artificial arms, steel hooks, caoutchouc [rubber] jaws, silver craniums [and] platinum noses” don’t play leading roles in their personalities—they are merely tools on their bodies. These piecemeal men are unlikely crusaders of invention with an even more unlikely mission. And yet who better to design the next great leap in technology than men remade by technology themselves?

Keep Reading ↓Show less