# Modeling Uncertainty Helps MIT's Drone Zip Around Obstacles

## This drone keeps track of what it doesn't know to quickly plan aggressive maneuvers

Catch that drone!
Photo: Jonathan How/MIT CSAIL

It’s not too hard to make a drone that can fly very fast, and it’s not too hard to make a drone that can avoid obstacles. Making a drone that can do both at once is much more difficult, but it’s necessary in order for them to be real-world useful.

At MIT CSAIL, Pete Florence (in Russ Tedrake’s lab) has developed a new motion planning framework called NanoMap, which uses a sequence of 3D snapshots to allow fast-moving (10 m/s) drones to safely navigate around obstacles even if they’re not entirely sure where they are.

Here’s a video of MIT’s drone in action. Don’t worry if you don’t catch all the details, as we’ll take a crack at explaining what’s going on afterwards:

I don’t mind telling you, this is one of those papers which reminds me that I have a degree in geology rather than robotics. Oof. So, let’s start with the key idea of NanoMap, which the paper helpfully makes explicit right there in the abstract:

The key idea of NanoMap is to store a history of noisy relative pose transforms and search over a corresponding set of depth sensor measurements for the minimum-uncertainty view of a queried point in space.

If that doesn’t completely make sense, I’m right there with you, but this figure should help a bit with the basic idea:

This figure shows how NanoMap evaluates motion plans (blue line), given a series of depth sensor measurements over time (gray triangles). For each sample point (red dot), the history of measurements is searched until a view is found (orange triangle) that contains the sample point.Image: MIT CSAIL

As the drone moves forward, it takes a continuous sequence of depth sensor snapshots (at something like 30 Hz, depending on the sensor), represented by the gray triangles* above. See that little curvy blue line? Let’s say that’s the trajectory that you want the drone to fly along next. To get to the first point in that trajectory (the red dot in the second diagram from left), the drone has a good enough view of what’s going on from right where it is. But to plan farther ahead, the drone needs information about areas outside of the current field of view of its depth sensor. NanoMap then starts looking backwards through its collection of snapshots, until it finds one that shows the area it needs to plan into. If it can’t find a good snapshot, then it’ll have to slow down and look around a bit, but if it does find one, it has the information it needs to move much more aggressively, since it already knows where obstacles might be.

[shortcode ieee-pullquote quote=""If NanoMap wasn't modeling uncertainty and the drone drifted just 5 percent away from where it was expected to be, the drone would crash more than once every four flights. Meanwhile, when it accounted for uncertainty, the crash rate reduced to 2 percent."" float="right" expand=1]

The problem with this technique is that the farther back the drone has to look to find the right snapshot, the less certain it’s going to be about where exactly it was (relative to where it is now) when that snapshot was taken, and consequently where it needs to move to avoid any obstacles that the snapshot contains. This stems from the fact that autonomous robots tend to be lousy at independently keeping track of their own position. Because IMUs aren’t perfect, and and cameras and lidar aren’t perfect, a robot that’s estimating where it is relative to where it’s been will become increasingly less accurate.

Many robots that localize themselves on maps that they build (via simultaneous localization and mapping, or SLAM) use what’s called loop closure to try and compensate for this. They wander around, building up a map that gradually gets less accurate as the robot’s sensors drift and it becomes less certain about its location. After the robot has wandered for a while, it loops back around, and once it recognizes where it’s started, it can compare the starting point it remembered with what it sees now and close the loop, calibrating (to some extent) the rest of the map. Here’s a video of this in practice—note how the map snaps into something much more right-angle-y (and accurate) as soon as the robot recognizes that it’s been in that big oval room before:

This is the conventional way of doing autonomous navigation: Compensating for uncertainty through multiple views of the same landmarks—spaces or objects—that can be calibrated against one another, ending up with a nice big accurate map. NanoMap does away with this entirely. Instead, it just uses that sequence of snapshots and compensates for uncertainty by modeling what the uncertainty actually is for each snapshot, looking for the snapshot with the lowest uncertainty, and then incorporating the uncertainty into the motion plan. The higher the uncertainty, the farther away the drone will fly from obstacles, since the space in which each obstacle could potentially be gets bigger.

With drones, most of the uncertainty comes from the accuracy of measuring acceleration with an IMU. Not getting an acceleration measurement quite right (which you never really do) leads to inaccuracies in estimations of velocity and position that get worse over time. This is called drift, and the more extreme the maneuvers that your drone is attempting, the more severe the drift, meaning that drones that move faster and more dynamically find it much more challenging to accurately estimate where they are, not surprisingly.

The MIT drone used an Intel RealSense r200 (for outdoor environments) and an Asus Xtion (for indoor environments) as the depth camera sensor. A Hokuyo 2D lidar sensor aided with obstacle detection but was used in a memoryless fashion.Photo: Jonathan How/MIT CSAIL

In testing, the researchers found that their uncertainty modeling really started to pay off when drift got much worse than 20 cm/s or so. Up to about 75 cm/s of drift, planning with NanoMap and incorporating uncertainty was able to keep the drone from crashing 97-98 percent of the time. With drift well over 1 m/s, the drone was only safe 10 percent of the time, but that was three times more robust than testing without uncertainty modeling. The press release sums it up overall:

If NanoMap wasn’t modeling uncertainty and the drone drifted just 5 percent away from where it was expected to be, the drone would crash more than once every four flights. Meanwhile, when it accounted for uncertainty, the crash rate reduced to 2 percent.

So that’s pretty good, right? MIT has already shown how well this works through flight experiments conducted as part of DARPA’s FLA program, and NanoMap is open source and available for you to play with on Github.

#### * Technically, these are called “frustums,” or pyramids with the tops chopped off. Frustum is an awesome word that can refer to other decapitated shapes as well, like lampshades, which are frustum-ized cones.

The Conversation (0)

## The Bionic-Hand Arms Race

### The prosthetics industry is too focused on high-tech limbs that are complicated, costly, and often impractical

Horizontal

The author, Britt Young, holding her Ottobock bebionic bionic arm.

Gabriela Hasbun. Makeup: Maria Nguyen for MAC cosmetics; Hair: Joan Laqui for Living Proof
DarkGray

In Jules Verne’s 1865 novel From the Earth to the Moon, members of the fictitious Baltimore Gun Club, all disabled Civil War veterans, restlessly search for a new enemy to conquer. They had spent the war innovating new, deadlier weaponry. By the war’s end, with “not quite one arm between four persons, and exactly two legs between six,” these self-taught amputee-weaponsmiths decide to repurpose their skills toward a new projectile: a rocket ship.

The story of the Baltimore Gun Club propelling themselves to the moon is about the extraordinary masculine power of the veteran, who doesn’t simply “overcome” his disability; he derives power and ambition from it. Their “crutches, wooden legs, artificial arms, steel hooks, caoutchouc [rubber] jaws, silver craniums [and] platinum noses” don’t play leading roles in their personalities—they are merely tools on their bodies. These piecemeal men are unlikely crusaders of invention with an even more unlikely mission. And yet who better to design the next great leap in technology than men remade by technology themselves?

As Verne understood, the U.S. Civil War (during which 60,000 amputations were performed) inaugurated the modern prosthetics era in the United States, thanks to federal funding and a wave of design patents filed by entrepreneurial prosthetists. The two World Wars solidified the for-profit prosthetics industry in both the United States and Western Europe, and the ongoing War on Terror helped catapult it into a US \$6 billion dollar industry across the globe. This recent investment is not, however, a result of a disproportionately large number of amputations in military conflict: Around 1,500 U.S. soldiers and 300 British soldiers lost limbs in Iraq and Afghanistan. Limb loss in the general population dwarfs those figures. In the United States alone, more than 2 million people live with limb loss, with 185,000 people receiving amputations every year. A much smaller subset—between 1,500 to 4,500 children each year—are born with limb differences or absences, myself included.

Today, the people who design prostheses tend to be well-intentioned engineers rather than amputees themselves. The fleshy stumps of the world act as repositories for these designers’ dreams of a high-tech, superhuman future. I know this because throughout my life I have been fitted with some of the most cutting-edge prosthetic devices on the market. After being born missing my left forearm, I was one of the first cohorts of infants in the United States to be fitted with a myoelectric prosthetic hand, an electronic device controlled by the wearer’s muscles tensing against sensors inside the prosthetic socket. Since then, I have donned a variety of prosthetic hands, each of them striving toward perfect fidelity of the human hand—sometimes at a cost of aesthetics, sometimes a cost of functionality, but always designed to mimic and replace what was missing.

In my lifetime, myoelectric hands have evolved from clawlike constructs to multigrip, programmable, anatomically accurate facsimiles of the human hand, most costing tens of thousands of dollars. Reporters can’t get enough of these sophisticated, multigrasping “bionic” hands with lifelike silicone skins and organic movements, the unspoken promise being that disability will soon vanish and any lost limb or organ will be replaced with an equally capable replica. Prosthetic-hand innovation is treated like a high-stakes competition to see what is technologically possible. Tyler Hayes, CEO of the prosthetics startup Atom Limbs, put it this way in a WeFunder video that helped raise \$7.2 million from investors: “Every moonshot in history has started with a fair amount of crazy in it, from electricity to space travel, and Atom Limbs is no different.”

We are caught in a bionic-hand arms race. But are we making real progress? It’s time to ask who prostheses are really for, and what we hope they will actually accomplish. Each new multigrasping bionic hand tends to be more sophisticated but also more expensive than the last and less likely to be covered (even in part) by insurance. And as recent research concludes, much simpler and far less expensive prosthetic devices can perform many tasks equally well, and the fancy bionic hands, despite all of their electronic options, are rarely used for grasping.

Activity arms, such as this one manufactured by prosthetics firm Arm Dynamics, are less expensive and more durable than bionic prostheses. The attachment from prosthetic-device company Texas Assistive Devices rated for very heavy weights, allowing the author to perform exercises that would be risky or impossible with her much more expensive bebionic arm.Gabriela Hasbun; Makeup: Maria Nguyen for MAC cosmetics; Hair: Joan Laqui for Living Proof

## Function or Form

In recent decades, the overwhelming focus of research into and development of new artificial hands has been on perfecting different types of grasps. Many of the most expensive hands on the market differentiate themselves by the number and variety of selectable prehensile grips. My own media darling of a hand, the bebionic from Ottobock, which I received in 2018, has a fist-shaped power grip, pinching grips, and one very specific mode with thumb on top of index finger for politely handing over a credit card. My 21st-century myoelectric hand seemed remarkable—until I tried using it for some routine tasks, where it proved to be more cumbersome and time consuming than if I had simply left it on the couch. I couldn’t use it to pull a door shut, for example, a task I can do with my stump. And without the extremely expensive addition of a powered wrist, I couldn’t pour oatmeal from a pot into a bowl. Performing tasks the cool bionic way, even though it mimicked having two hands, wasn’t obviously better than doing things my way, sometimes with the help of my legs and feet.

When I first spoke with Ad Spiers, lecturer in robotics and machine learning at Imperial College London, it was late at night in his office, but he was still animated about robotic hands—the current focus of his research. Spiers says the anthropomorphic robotic hand is inescapable, from the reality of today’s prosthetics to the fantasy of sci-fi and anime. “In one of my first lectures here, I showed clips of movies and cartoons and how cool filmmakers make robot hands look,” Spiers says. “In the anime Gundam, there are so many close-ups of gigantic robot hands grabbing things like massive guns. But why does it need to be a human hand? Why doesn’t the robot just have a gun for a hand?”

It’s time to ask who prostheses are really for, and what we hope they will actually accomplish.

Spiers believes that prosthetic developers are too caught up in form over function. But he has talked to enough of them to know they don’t share his point of view: “I get the feeling that people love the idea of humans being great, and that hands are what make humans quite unique.” Nearly every university robotics department Spiers visits has an anthropomorphic robot hand in development. “This is what the future looks like,” he says, and he sounds a little exasperated. “But there are often better ways.”

The vast majority of people who use a prosthetic limb are unilateral amputees—people with amputations that affect only one side of the body—and they virtually always use their dominant “fleshy” hand for delicate tasks such as picking up a cup. Both unilateral and bilateral amputees also get help from their torsos, their feet, and other objects in their environment; rarely are tasks performed by a prosthesis alone. And yet, the common clinical evaluations to determine the success of a prosthetic are based on using only the prosthetic, without the help of other body parts. Such evaluations seem designed to demonstrate what the prosthetic hand can do rather than to determine how useful it actually is in the daily life of its user. Disabled people are still not the arbiters of prosthetic standards; we are still not at the heart of design.

The Hosmer Hook [left], originally designed in 1920, is the terminal device on a body-powered design that is still used today. A hammer attachment [right] may be more effective than a gripping attachment when hammering nails into wood.Left: John Prieto/The Denver Post/Getty Images; Right: Hulton-Deutsch Collection/Corbis/Getty Images

## Prosthetics in the Real World

To find out how prosthetic users live with their devices, Spiers led a study that used cameras worn on participants’ heads to record the daily actions of eight people with unilateral amputations or congenital limb differences. The study, published last year in IEEE Transactions on Medical Robotics and Bionics, included several varieties of myoelectric hands as well as body-powered systems, which use movements of the shoulder, chest, and upper arm transferred through a cable to mechanically operate a gripper at the end of a prosthesis. The research was conducted while Spiers was a research scientist at Yale University’s GRAB Lab, headed by Aaron Dollar. In addition to Dollar, he worked closely with grad student Jillian Cochran, who coauthored the study.

Watching raw footage from the study, I felt both sadness and camaraderie with the anonymous prosthesis users. The clips show the clumsiness, miscalculations, and accidental drops that are familiar to even very experienced prosthetic-hand users. Often, the prosthesis simply helps brace an object against the body to be handled by the other hand. Also apparent was how much time people spent preparing their myoelectric prostheses to carry out a task—it frequently took several extra seconds to manually or electronically rotate the wrists of their devices, line up the object to grab it just right, and work out the grip approach.The participant who hung a bottle of disinfectant spray on their “hook” hand while wiping down a kitchen counter seemed to be the one who had it all figured out.

In the study, prosthetic devices were used on average for only 19 percent of all recorded manipulations. In general, prostheses were employed in mostly nonprehensile actions, with the other, “intact” hand doing most of the grasping. The study highlighted big differences in usage between those with nonelectric, body-powered prosthetics and those with myoelectric prosthetics. For body-powered prosthetic users whose amputation was below the elbow, nearly 80 percent of prosthesis usage was nongrasping movement—pushing, pressing, pulling, hanging, and stabilizing. For myoelectric users, the device was used for grasping just 40 percent of the time.

More tellingly, body-powered users with nonelectric grippers or split hooks spent significantly less time performing tasks than did users with more complex prosthetic devices. Spiers and his team noted the fluidity and speed with which the former went about doing tasks in their homes. They were able to use their artificial hands almost instantaneously and even experience direct haptic feedback through the cable that drives such systems. The research also revealed little difference in use between myoelectric single-grasp devices and fancier myoelectric multiarticulated, multigrasp hands—except that users tended to avoid hanging objects from their multigrasp hands, seemingly out of fear of breaking them.

“We got the feeling that people with multigrasp myoelectric hands were quite tentative about their use,” says Spiers. It’s no wonder, since most myoelectric hands are priced over \$20,000, are rarely approved by insurance, require frequent professional support to change grip patterns and other settings, and have costly and protracted repair processes. As prosthetic technologies become more complex and proprietary, the long-term serviceability is an increasing concern. Ideally, the device should be easily fixable by the user. And yet some prosthetic startups are pitching a subscription model, in which users continue to pay for access to repairs and support.

Despite the conclusions of his study, Spiers says the vast majority of prosthetics R&D remains focused on refining the grasping modes of expensive, high-tech bionic hands. Even beyond prosthetics, he says, manipulation studies in nonhuman primate research and robotics are overwhelmingly concerned with grasping: “Anything that isn’t grasping is just thrown away.”

TRS makes a wide variety of body-powered prosthetic attachments for different hobbies and sports. Each attachment is specialized for a particular task, and they can be easily swapped for a variety of activities. Fillauer TRS

## Grasping at History

If we’ve decided that what makes us human is our hands, and what makes the hand unique is its ability to grasp, then the only prosthetic blueprint we have is the one attached to most people’s wrists. Yet the pursuit of the ultimate five-digit grasp isn’t necessarily the logical next step. In fact, history suggests that people haven’t always been fixated on perfectly re-creating the human hand.

As recounted in the 2001 essay collection Writing on Hands: Memory and Knowledge in Early Modern Europe, ideas about the hand evolved over the centuries. “The soul is like the hand; for the hand is the instrument of instruments,” Aristotle wrote in De Anima. He reasoned that humanity was deliberately endowed with the agile and prehensile hand because only our uniquely intelligent brains could make use of it—not as a mere utensil but a tool for apprehensio, or “grasping,” the world, literally and figuratively.

More than 1,000 years later, Aristotle’s ideas resonated with artists and thinkers of the Renaissance. For Leonardo da Vinci, the hand was the brain’s mediator with the world, and he went to exceptional lengths in his dissections and illustrations of the human hand to understand its principal components. His meticulous studies of the tendons and muscles of the forearm and hand led him to conclude that “although human ingenuity makes various inventions…it will never discover inventions more beautiful, more fitting or more direct than nature, because in her inventions nothing is lacking and nothing is superfluous.”

Da Vinci’s illustrations precipitated a wave of interest in human anatomy. Yet for all of the studious rendering of the human hand by European masters, the hand was regarded more as an inspiration than as an object to be replicated by mere mortals. In fact, it was widely accepted that the intricacies of the human hand evidenced divine design. No machine, declared the Christian philosopher William Paley, is “more artificial, or more evidently so” than the flexors of the hand, suggesting deliberate design by God.

Performing tasks the cool bionic way, even though it mimicked having two hands, wasn’t obviously better than doing things my way, sometimes with the help of my legs and feet.

By the mid-1700s, with the Industrial Revolution in the global north, a more mechanistic view of the world began to emerge, and the line between living things and machines began to blur. In her 2003 article “ Eighteenth-Century Wetware,” Jessica Riskin, professor of history at Stanford University, writes, “The period between the 1730s and the 1790s was one of simulation, in which mechanicians tried earnestly to collapse the gap between animate and artificial machinery.” This period saw significant changes in the design of prosthetic limbs. While mechanical prostheses of the 16th century were weighed down with iron and springs, a 1732 body-powered prosthesis used a pulley system to flex a hand made of lightweight copper. By the late 18th century, metal was being replaced with leather, parchment, and cork—softer materials that mimicked the stuff of life.

The techno-optimism of the early 20th century brought about another change in prosthetic design, says Wolf Schweitzer, a forensic pathologist at the Zurich Institute of Forensic Medicine and an amputee. He owns a wide variety of contemporary prosthetic arms and has the necessary experience to test them. He notes that anatomically correct prosthetic hands have been carved and forged for the better part of 2,000 years. And yet, he says, the 20th century’s body-powered split hook is “more modern,” its design more willing to break the mold of the human hand.

“The body powered arm—in terms of its symbolism—(still) expresses the man-machine symbolism of an industrial society of the 1920s,” writes Schweitzer in his prosthetic arm blog, “when man was to function as clockwork cogwheel on production lines or in agriculture.” In the original 1920s design of the Hosmer Hook, a loop inside the hook was placed just for tying shoes and another just for holding cigarettes. Those designs, Ad Spiers told me, were “incredibly functional, function over form. All pieces served a specific purpose.”

Schweitzer believes that as the need for manual labor decreased over the 20th century, prostheses that were high-functioning but not naturalistic were eclipsed by a new high-tech vision of the future: “bionic” hands. In 2006, the U.S. Defense Advanced Research Projects Agency launched Revolutionizing Prosthetics, a research initiative to develop the next generation of prosthetic arms with “near-natural” control. The \$100 million program produced two multi-articulating prosthetic arms (one for research and another that costs over \$50,000). More importantly, it influenced the creation of other similar prosthetics, establishing the bionic hand—as the military imagined it—as the holy grail in prosthetics. Today, the multigrasp bionic hand is hegemonic, a symbol of cyborg wholeness.

And yet some prosthetic developers are pursuing a different vision. TRS, based in Boulder, Colo., is one of the few manufacturers of activity-specific prosthetic attachments, which are often more durable and more financially accessible than robotic prosthetics. These plastic and silicone attachments, which include a squishy mushroom-shaped device for push-ups, a ratcheting clamp for lifting heavy weights, and a concave fin for swimming, have helped me experience the greatest functionality I have ever gotten out of a prosthetic arm.

Such low-tech activity prostheses and body-powered prostheses perform astonishingly well, for a tiny fraction of the cost of bionic hands. They don’t look or act like human hands, and they function all the better for it. According to Schweitzer, body-powered prostheses are regularly dismissed by engineers as “arcane” or derisively called “Captain Hook.” Future bionic shoulders and elbows may make a huge difference in the lives of people missing a limb up to their shoulder, assuming those devices can be made robust and affordable. But for Schweitzer and a large percentage of users dissatisfied with their myoelectric prosthesis, the prosthetic industry has yet to provide anything fundamentally better or cheaper than body-powered prostheses.

## The Breakthroughs We Want

Bionic hands seek to make disabled people “whole,” to have us participate in a world that is culturally two-handed. But it’s more important that we get to live the lives we want, with access to the tools we need, than it is to make us look like everyone else. While many limb-different people have used bionic hands to interact with the world and express themselves, the centuries-long effort to perfect the bionic hand rarely centers on our lived experiences and what we want to do in our lives.

We’ve been promised a breakthrough in prosthetic technology for the better part of 100 years now. I’m reminded of the scientific excitement around lab-grown meat, which seems simultaneously like an explosive shift and a sign of intellectual capitulation, in which political and cultural change is passed over in favor of a technological fix. With the cast of characters in the world of prosthetics—doctors, insurance companies, engineers, prosthetists, and the military—playing the same roles they have for decades, it’s nearly impossible to produce something truly revolutionary.

In the meantime, this metaphorical race to the moon is a mission that has forgotten its original concern: helping disabled people acquire and use the tools they want. There are inexpensive, accessible, low-tech prosthetics that are available right now and that need investments in innovation to further bring down costs and improve functionality. And in the United States at least, there is a broken insurance system that needs fixing. Releasing ourselves from the bionic-hand arms race can open up the possibilities of more functional designs that are more useful and affordable, and might help us bring our prosthetic aspirations back down to earth.