A Japanese roboticist is building androids to understand humans—starting with himself
Photo: Makoto Ishida
Hiroshi Ishiguro, a roboticist at Osaka University, in Japan, has, as you might expect, built many robots. But his latest aren’t run-of-the-mill automatons. Ishiguro’s recent creations look like normal people. One is an android version of a middle-aged family man—himself.
Ishiguro constructed his mechanical doppelgänger using silicone rubber, pneumatic actuators, powerful electronics, and hair from his own scalp. The robot, like the original, has a thin frame, a large head, furrowed brows, and piercing eyes that, as one observer put it, “seem on the verge of emitting laser beams.” The android is fixed in a sitting posture, so it can’t walk out of the lab and go fetch groceries. But it does a fine job of what it’s intended to do: mimic a person.
Ishiguro controls this robot remotely, through his computer, using a microphone to capture his voice and a camera to track his face and head movements. When Ishiguro speaks, the android reproduces his intonations; when Ishiguro tilts his head, the android follows suit. The mechanical Ishiguro also blinks, twitches, and appears to be breathing, although some human behaviors are deliberately suppressed. In particular, when Ishiguro lights up a cigarette, the android abstains.
It’s the perfect tool for Ishiguro’s field of research: human-robot interaction, which is as much a study of people as it is of robots. “My research question is to know what is a human,” he tells me between spoonfuls of black sesame ice cream at an Osaka diner. “I use very humanlike robots as test beds for my hypotheses”—hypotheses about human nature, intelligence, and behavior.
Robots, Ishiguro and others say, are poised to move from factories into daily life. The hope is that robots will one day help people with a multitude of tasks—they’ll do household chores, care for the elderly, assist with physical therapy, monitor the sick at hospitals, teach classes, serve cappuccinos at Starbucks, you name it. But to be accepted in these roles, robots may have to behave less like machines and more like us.
Researchers have, of course, long been interested in making robots look and act more like human beings. Among the most notable efforts in this regard are Waseda University’s Wabot, MIT’s Cog, NASA’s Robonaut, Sarcos’s Sarcoman, the Toyota partner robots, Japan’s METI HRP series, Sony’s Qrio, and perhaps most famous of all, Honda’s Asimo.
These robots are all mechanical looking, Ishiguro says, but our brains are wired to relate to other humans—we’re optimized for human-human, not human-Asimo, interaction. That’s why he builds robots that look like people, as part of his work at the Advanced Telecommunications Research Institute International, known as ATR, where he’s a visiting group leader. To describe an android copy of a particular individual, he coined the term “geminoid,” after geminus, which is Latin for twin. He named his mechanical brother Geminoid HI-1.
By building humanlike robots Ishiguro hopes to decipher what the Japanese call sonzaikan—the feeling of being in the presence of a human being. Where does the sense of humanness come from? And can you convey those qualities with a robot?
The idea of connecting a person’s brain so intimately with a remotely controlled body seems straight out of science fiction. In The Matrix, humans control virtual selves. In Avatar, the controlled bodies are alien-human hybrids. In the recent Bruce Willis movie Surrogates, people control robot proxies sent into the world in their places. Attentive viewers will notice that Ishiguro and the Geminoid have cameo roles, appearing in a TV news report on the rapid progress of “robotic surrogacy.”
Ishiguro’s surrogate doesn’t have sensing and actuation capabilities as sophisticated as those in the movie. But even this relatively simple android is giving Ishiguro great insight into how our brains work when we come face to face with a machine that looks like a person. He’s also investigating, with assistance from cognitive scientists, how the operator’s brain behaves. Teleoperating the android can be so immersive that strange things happen. Simply touching the android is enough to trigger a physical sensation in him, Ishiguro says, almost as though he were inhabiting the robot’s body.
“Like all Dr. Frankensteins of literature, he’s raising some deep, powerful questions about our humanity and our creations, and it’s scary, but it’s also important that we confront these questions, and he’s doing that not in the realm of fiction but in the laboratory,” says IEEE Fellow Ken Goldberg, a robotics professor at the University of California, Berkeley. “We’re going to learn something about machines—but even more about ourselves.”
Hiroshi Ishiguro stomps on the accelerator. The black Mazda RX-8 roars onto the highway, the heavy-metal Scorpions blasting from the speakers. We’re driving to Osaka University’s Toyonaka Campus. Ishiguro is wearing aviator sunglasses, black polyester pants, a black vest on top of a black shirt, along with a black belt, socks, and shoes.
“Give me question,” he says, his eyes fixed on the road.
I ask whether he always dresses in black.
“Why do you change your clothes?” he says. “Do you change your name? So why do you change your clothes? Name is identity. Face is identity. But the majority of your [appearance] comes from your clothes. You should not change your clothes. Do you agree?”
I meekly suggest that all-black attire might get a bit hot in the summer.
“We have air conditioners,” he says. “Next question.”
For an academic, Ishiguro lives a frantic existence. He works at four labs, oversees some 50 students, is a cofounder of a robotics start-up, and constantly travels to conferences around the world. No wonder he needs a copy of himself.
Ishiguro, who is 46, started building robots more than a decade ago, after abandoning the idea of becoming an oil painter. One of the first robots he helped put together, Robovie, looked like a trash can with arms. Another robot he worked with, Wakamaru, developed by Mitsubishi Heavy Industries, resembled an overgrown insect. People had mixed reactions to those robots, and that got him thinking about how crucial appearance is to communication. And yet, he realized, the connection hadn’t been properly investigated.
Indeed, consider that the world population of industrial and service robots is about 8.5 million, according to some estimates. That includes manufacturing robots, palletizing robots, surgical robots, bomb-disposal robots, milking robots, meat-handling robots, underwater and aerial robots, some 4 million home vacuum cleaners, and one that cleans the Louvre’s glass pyramid. That’s to say nothing about the vast assortment of robot dogs, dinosaurs, seals, and other robot toys. It’s a fine bunch of helpful or at least entertaining hardware. Yet none of it matches the robots that science fiction promised the future would bring—Astro Boy, C-3PO, Data, Rosie, David, Number Six. Today’s robots are closer to other appliances in appearance. And even the best of them aren’t very good company.
Many roboticists believe this is going to change. They have visions of robots that will assist us, talk to us, even care about us. Social robots are coming to life in labs at MIT, Carnegie Mellon, Georgia Tech, and elsewhere. Though nobody knows yet what types of robots will become major applications (roboticists avoid the term “killer app,” understandably), they may soon be among us.
But do they have to look like us?
Behind the Curtain: Ishiguro teleoperates his android using a computer that captures his voice and facial expressions. Photo: ATR Intelligent Robotics and Communication Laboratories
Some roboticists I spoke to question whether humanlike androids are a good idea, or even necessary. One researcher, who asked not to be named because he collaborates with institutions with which Ishiguro is affiliated, says the Geminoid is “not very convincing—and a bit creepy.” In some demonstrations, he notes, the android barely moves its eyes and lips, appearing catatonic. On YouTube, many commenters call videos of the Geminoid “scary” and “freaky.” The tech blog Gizmodo included the android in its list of “10 Creepy Machines From Robot Hell.”
Ishiguro is unfazed. He acknowledges that in some situations you don’t need or want your robot to resemble a person. You don’t need an android to vacuum your house if a saucer on wheels can do a better job. And a robotic assistant that helps an elderly person out of the tub probably shouldn’t sport a human face.
He’s also well aware that, although people might better connect with a robot when it resembles another human being, when it gets the nuances wrong it may seem more like a zombie or an animated corpse. The Japanese roboticist Masahiro Mori called this descent into creepiness, as lifelike appearance is approached but not attained, the “uncanny valley.”
Ishiguro says the possibility that his creations might result in revulsion won’t stop him from “trying to build the robots of the future as I imagine them.” He is convinced that human-looking robots are a natural interface for humans to interact with and that the uncanny valley idea may be too simplistic to explain people’s reactions to robots. We may simply come to accept lifelike androids as we’re exposed to more of them and, in the future, rely on them for our care and other needs.
Or as he puts it in Phie Ambo’s 2007 documentary Mechanical Love, “If we’re short on people, why not make some?”
We’re standing outside Osaka University’s Intelligent Robotics Laboratory, which Ishiguro directs. It’s a Sunday, the lab is empty, and we’re locked out. Ishiguro can’t recall the code of the entry lock and is on the phone with a colleague. When the door finally opens, robotic creatures great and small are everywhere—and they appear to be staring right at me. I ask about a diminutive android standing in a corner.
“That is my daughter’s copy,” Ishiguro says. He explains that the robot, built in 2001 and called Repliee R1, was based on his then 4-year-old daughter. It has nine DC motors in its head, prosthetic eyeballs, and silicone skin. Ishiguro says that the eyelids looked unnatural and that only the mouth moved, so the robot’s facial expressions couldn’t change much. When the neck turned, a lack of stiffness in the body would cause the robot to shake. People told him that it gave them a strange, eerie feeling. That includes the robot’s master template—Ishiguro’s daughter.
“She was scared very much,” he says, “and almost cried.” It took several meetings until she warmed up to her double.
Surely Ishiguro must have been staring into the abyss of the uncanny valley. His response was to attempt a flying leap over the chasm—to build a markedly better android. This time he modeled it after an adult. Ishiguro shows me the result.
“This is Repliee Q2,” he tells me, almost as if introducing a friend. The android has long golden-brown hair, glossy lips, perfect skin, and 42 pneumatic actuators embedded in her petite body. Perched on a stool, she—er, the android—can talk about itself and sing Disney tunes. Ishiguro built it with help from the robotics firm Kokoro, which has considerable experience with animatronics. Kokoro engineers and Ishiguro’s team worked particularly hard to duplicate a woman’s facial expressions, blinking, and hand movements. And they wrote software to simulate involuntary movements like breathing and subtle head twitches.
“All humans have these,” Ishiguro says, “unless they’re dead.”
For the 2005 World Exposition in Aichi, Ishiguro replaced the face, copying that of a famous Japanese newscaster, Ayako Fujii. Visitors lined up to talk to the robotic impersonator.
“An elderly gentleman came over and asked, ’Where’s the android?’ and he was standing right next to it,” says Karl MacDorman, a former Ishiguro collaborator and now director of the Android Science Center at Indiana University–Purdue University, in Indianapolis. “Androids can press our Darwinian buttons—they are perfect tools to study how our brains work.”
Despite its success in many regards, this android revealed a serious problem. The techniques of artificial intelligence remain too primitive to reproduce everyday human behavior, like the ability to sustain a conversation. So the robot’s lifelike appearance proved deceiving: People expected more than it could deliver.
After some careful reflection on the problem, Ishiguro decided to shift his focus from autonomous androids to teleoperated ones. For that, he figured he’d start with a copy of a real person, someone he could trust and who’d be willing to become a guinea pig in long, tedious experiments. He’d reproduce himself.
Building on what he had done with Repliee Q2, Ishiguro added even more actuators to the Geminoid: 13 in the face, 15 in the torso, and 22 in the arms and legs. His team used a cast to make a perfect copy of his physique. They improved the body-control software to generate even finer motions and developed a computer vision system to synchronize the Geminoid’s lips to the operator’s.
So how did people react?
In the three years since completing the Geminoid, Ishiguro has tested the android using dozens of volunteers, colleagues, and students, as well as his wife and daughter. In controlled experiments, he observed that people may feel uneasy at first, but they quickly adapt, and most are capable of having a natural conversation with his replica. No one ran away from the android screaming in horror. He also showed that the Geminoid could act as a good salesman, that children were eager to play games with it, and that pet owners were particularly skilled at detecting its nonverbal cues.
His main conclusion is that in many situations people respond to the Geminoid in much the same way they do to other individuals. What’s more, people who know him, Ishiguro says, could experience his personality and authority: The android, though not perfect, was able to convey sonzaikan—his presence.
“To look back at the trajectory of androids in his lab, the progress is striking,” says Andrea Thomaz, a robotics professor at Georgia Tech who is familiar with Ishiguro’s work. The exciting part about his and others’ work on androids, she adds, “is that it forces us to more fully understand human behavior at every level.”
For Ishiguro, having a robot clone has practical uses as well—such as being in two places at once. Most of the time the Geminoid resides at ATR, in Kyoto, an hour’s drive from Osaka, so Ishiguro uses the robot to attend meetings there.
“Very convenient,” he says with a smile. “The problem is I can’t get salary from ATR if I use the Geminoid. Therefore they still make me go there.”
It’s late at night, and Ishiguro sits in his Osaka University lab. He’s teleoperating the Geminoid, which is lounging 9000 kilometers away at a café in Linz, Austria. The android is there as part of the Ars Electronica festival. Ishiguro is chatting with a group of kids who’d taken notice of the robot. Then a boy pokes the android’s face.
“Yahh, don’t touch to my face,” Ishiguro yells. The kids laugh.
But what just happened was quite profound. Ishiguro says that when the kid poked the robot’s face, he felt a tingle on his cheek—even though nothing was attached to his face. For an instant, Ishiguro’s brain mistook his body for another—in Austria.
Ishiguro has experienced this many times in different settings, all with the same result: He feels as though the touch were real. He believes the phenomenon stems in part from observing the synchronization of the android’s head and lips with his own. When Ishiguro uses the Geminoid, he sees video from external cameras that show the robot’s field of view and also its face. Because he is watching the robot’s lips move as he speaks and seeing its head move when he turns his own neck, Ishiguro’s brain starts to treat the robot as an extension of his own body.
Even more surprising, Ishiguro says that when other people teleoperate the Geminoid, after a while they, too, may experience the “phantom poke.” This means that an android doesn’t even have to look like you for you to think of its body as your own!
Peter W. Halligan, a psychology professor at Cardiff University, in Wales, who is not involved in the Geminoid project, says the phenomenon “sounds fascinating.” He notes that a possible explanation is that humans have mirror neurons that fire both when a subject is touched and when the same subject observes another being touched, in this case on the face. In normal situations, the sensation is suppressed. Is the act of teleoperating a robot tricking the brain?
Indeed, the brain fills in many gaps in the signals our sensing organs send to it, and our sense of embodiment is more malleable than most people think. Recently, Swedish researchers showed that when presented with certain visual cues, subjects wearing head-mounted displays could experience other people’s bodies—or a mannequin’s—as their own.
Ishiguro is collaborating with cognitive scientists to investigate the phenomenon. He’s planning to use brain scans to find out what happens inside the operator’s head when the Geminoid gets poked. He believes his results will show humans can easily adapt to robotic bodies.
Whether each of us will have a geminoid in the future remains to be seen. But lifelike androids are already helping Ishiguro and others explore some of the pressing questions in human-robot interaction. What do people expect from robots? What social behaviors should they exhibit? And how do we get their look right? In the coming years, researchers will have to answer these questions to come up with design principles for building the next generation of social robots.
Ishiguro, for his part, believes that robotic telepresence will become a major communication medium over the next decade. Eventually, he speculates, humanlike robots will become truly integrated into society—not just for factory automation or as labor-saving devices but as replacements for someone’s physical presence. A future where androids may become so advanced that we’re unable to distinguish them from ourselves doesn’t frighten Ishiguro.
“Humankind is always trying to replace human abilities with machines. That’s our history,” he says. “I’m doing the same thing. Nothing special.”
This article originally appeared in print as “The Man Who Made a Copy of Himself.”
To Probe Further
For more photos of Ishiguro’s robots, go to “The Amazing Androids of Hiroshi Ishiguro.”