The December 2022 issue of IEEE Spectrum is here!

Close bar

The Buddy System: Human-Computer Teams

The highest accuracy in face recognition comes through the partnering of a human expert with face recognition software

3 min read
: A new study from the University of Maryland and colleagues indicates that combining the expertise of trained specialists called forensic face examiners with state-of-the-art face recognition software gives the best accuracy.

University of Maryland

Is that him? Is she the one?

A prized attribute among law enforcement specialists, the expert ability to visually identify human faces can inform forensic investigations and help maintain safe border crossings, airports, and public spaces around the world.

The field of forensic facial recognition depends on highly refined traits such as visual acuity, cognitive discrimination, memory recall, and elimination of bias. Humans, as well as computers running machine learning (ML) algorithms, possess these abilities. And it is the combination of the two—a human facial recognition expert teamed with a computer running ML analyses of facial image data—that provides the most accurate facial identification, according to a recent 2018 study in which Rama Chellappa, Distinguished University Professor and Minta Martin Professor of Engineering, and his team collaborated with researchers at the National Institute of Standards and Technology and the University of Texas at Dallas.

Chellappa, who holds appointments in UMD’s Departments of Electrical and Computer Engineering and Computer Science and Institute for Advanced Computer Studies, is not surprised by the study results. “For this facial recognition task, like a lot of tasks in the future, humans will need a computer ML buddy to do it really well,” he says.

Pairing human expertise at decoding subtle cues such as emotional signals, context, and remembered experience with ML’s blazing computational power leverages the strengths of each. It also compensates for deficiencies. “Machines can mess up,” Chellappa says. “A dog can be mistaken for a traffic sign by a machine if the algorithm is given poor-quality data. Human judgment and experience can easily spot that kind of error and correct it.”

University of Maryland

In the early days of computer vision research, Chellappa’s labs followed the common research approach of mathematically connecting inputs of data to obtain outputs that enabled them to explore the underlying physics of image formation and geometry of objects. Mapping the contours and appearances of a face, for example, was a common task.

All this changed in 2012. That’s the year the technique for layering input–output data emerged. This led to the ML approach known as deep learning, supported by an algorithmic configuration referred to as neural networks.

Deep learning affords greater ML accuracy than ever before by overcoming the constraints of limited amounts of data. The layering of data makes more data points available for analysis and powers more robust, accurate analyses of patterns and predictions. Around the same time, affordable advances in computing and graphics processing power also became available.

Combined, the confluence of the two—development of neural network deep learning and faster machine processing—led to the 2012 breakthrough that now powers technologies central to computer vision and facial recognition. And so much more, from driverless cars to voice recognition and chatbots.

“I call it the earthquake of 2012,” Chellappa says. “It completely turned our world upside down.”

Though deep learning contributes enormously to improved accuracy in facial recognition, Chellappa believes in the conclusion suggested by the 2018 study: the future is still human. Optimal accuracy in facial recognition and other ML-mediated tasks will likely come from humans and machines teaming up.

For example, he sees enormous potential to apply deep learning and the buddy system of human–computer teams to issues in medicine and human health. In a cardiac exam, the ML-enabled computer could make diagnostic suggestions from patterns in the patient’s heart data and help predict patients at risk for a cardiac event. The human medical professional would analyze the machine’s report and make the final diagnostic decision, and design the plan for how and when to treat.

Says Chellappa: “ML is remarkably powerful in identifying patterns and making predictions, especially when there is a lot of high-quality data, so I’m very excited about this deep learning potential in medicine to improve care. It’s an exciting time to be in computer vision and machine learning.”

The Conversation (0)

The Bionic-Hand Arms Race

The prosthetics industry is too focused on high-tech limbs that are complicated, costly, and often impractical

12 min read
A photograph of a young woman with brown eyes and neck length hair dyed rose gold sits at a white table. In one hand she holds a carbon fiber robotic arm and hand. Her other arm ends near her elbow. Her short sleeve shirt has a pattern on it of illustrated hands.

The author, Britt Young, holding her Ottobock bebionic bionic arm.

Gabriela Hasbun. Makeup: Maria Nguyen for MAC cosmetics; Hair: Joan Laqui for Living Proof

In Jules Verne’s 1865 novel From the Earth to the Moon, members of the fictitious Baltimore Gun Club, all disabled Civil War veterans, restlessly search for a new enemy to conquer. They had spent the war innovating new, deadlier weaponry. By the war’s end, with “not quite one arm between four persons, and exactly two legs between six,” these self-taught amputee-weaponsmiths decide to repurpose their skills toward a new projectile: a rocket ship.

The story of the Baltimore Gun Club propelling themselves to the moon is about the extraordinary masculine power of the veteran, who doesn’t simply “overcome” his disability; he derives power and ambition from it. Their “crutches, wooden legs, artificial arms, steel hooks, caoutchouc [rubber] jaws, silver craniums [and] platinum noses” don’t play leading roles in their personalities—they are merely tools on their bodies. These piecemeal men are unlikely crusaders of invention with an even more unlikely mission. And yet who better to design the next great leap in technology than men remade by technology themselves?

Keep Reading ↓Show less