The Heaphy Project: Crowdsourced Robot Servants and the Willow Garage Spin-off That Never Was

Image: Tim Field/Heaphy Project
A Willow Garage project to train remote workers to teleoperate robot servants looked promising. So why was it abandoned?

Willow Garage, the now-disbanded Silicon Valley robotics incubator, unleashed some of the most impactful robot technologies of the past half-decade. Today, this formidable legacy lives on, in great part, through a host of Willow spin-offs that remain at the forefront of robotics. This article is not about them.

It’s about a Willow Garage project that was, at one point, on the verge of being spun out of the incubator, but is now all but forgotten. The project, called Heaphy, recognized that robots were still too limited in what they could do autonomously, and that instead of improving artificial intelligence, an alternative would be relying on more human intelligence.

Heaphy aimed to recruit people online and train them to operate Willow’s powerful PR2 robots to carry out everyday tasks like serving drinks and collecting trash. The remote operators would be paid for their work and could be anywhere in the world, using only a web browser to control the robots. Thus the idea of crowdsourced telerobotic labor was born.

To be sure, the notion of a stranger halfway around the world commanding a robot servant and doing your chores for you sounds like something straight out of a sci-fi movie. But as it happened, Heaphy demonstrated that that concept could potentially work in the real world.

Video: Tim Field/Heaphy Project
A 2012 video gives an overview of the Heaphy project, showing how remote operators could use a web browser to take control of a PR2 robot and navigate around Willow Garage’s office, grab a bottle from a refrigerator, or unload clothes from a dryer.

“We used the robot to fetch snacks, play tic-tac-toe, open beer bottles, and make microwave popcorn,” Tim Field, the Willow engineer behind the system, tells IEEE Spectrum. He named the project “Heaphy” after a favorite walking track in his native New Zealand.

Though Heaphy has been publicly discussed before, earlier reports haven’t described Willow’s plan to commercialize the technology—and how the project ended up abandoned instead.

The Heaphy project started in 2009. Field says he got the idea from Scott Hassan, the founder of Willow Garage and now CEO of Suitable Technologies. That year, Hassan—who had long wanted a web-based system to control the PR2—showed Field new 3D graphics applications Google was developing. Field was impressed that these applications ran in a browser window, and proposed building a web teleoperation system like the one Hassan wanted, capable of connecting remote human users to Willow’s robots. Hassan and Steve Cousins, Willow’s CEO, liked the idea, and it’s easy to see why.

For all the technological advances in recent years, even the most sophisticated robots struggle to work on their own. Robot vision and manipulation are getting better, but they remain too crude for most practical applications. Robots still have trouble identifying and discriminating between objects, manipulating items in real environments, and making commonsense decisions about what to do with them. (Robot navigation, on the other hand, is improving at a faster pace, and Cousins is now exploring that with his own start-up, Savioke.)

Field believed that robots were “years or decades away” from the sort of dexterous tasks that minimum wage workers are capable of doing. He thought that a more sensible approach—at least until roboticists could overcome the limitations of machine learning and AI—was simply to put humans in the loop.

To find remote workers eager to participate in the project, Field turned to Amazon’s Mechanical Turk, the crowdsourcing service where online users receive small cash rewards to perform tasks such as data entry and audio transcription. Mastering an advanced robot worth nearly half a million dollars was not your typical Mechanical Turk job.

“It’s really an interface problem,” says another former Willow engineer. (He asked not to be identified because he doesn’t have permission from his current employer to speak to the media.) “You’ve got a person who is much more capable than a robot. The trick is how to get that intelligence into the robot.”

Commercial telerobotic systems—the kind used in nuclear power plants and underwater industrial operations—typically feature high-end joysticks and custom controllers. That was not an option for Heaphy, which had to rely on the computers owned by “Turkers,” as the Mechanical Turk workers are known.

Working with two colleagues, Field started to build a web-based interface that could give Turkers complete control of a PR2. Using only their keyboard and mouse, the workers would be able to operate the robot’s mobile base, dual arms, cameras and other sensors.

A critical problem the project had to solve was network latency. Getting data from the robot to a Turker, over the Internet, and back again resulted in lag, and lag makes teleoperation very difficult. Imagine you’re driving a car, but every time you turn the steering wheel, it takes a couple of seconds for the car to respond; it’s extremely disorienting, and you’re probably going to crash.

To reduce latency, Field and his colleagues painstakingly optimized their hardware and software. They also made a key improvement to the user interface: when you’re operating the robot’s gripper, you see a virtual gripper superimposed on the video feed [shown in the image below]. The virtual gripper respond to your commands with no delay, so you don’t get disoriented due to the latency effect. A moment later you see the actual hand executing the motion you just did with the virtual one.

“Moving the arm was very difficult for people,” remembers the former Willow engineer. “You had to have fine motor skills with the mouse. People might spend five hours doing that one task.”

Once Turkers had completed a training program based on a realistic 3D simulator and split into levels of increasing complexity, they were allowed to control live robots. That, in turn, led to a range of new challenges.

“The PR2 is a 200-kilogram machine that can travel at a meter a second,” says Field, adding that he once drove the robot out of Willow Garage’s Menlo Park office, down the road, and back in through another entrance. Letting a random Turker do something like that was definitely not an option, so the engineers started the Turkers on a robot locked in an empty room. Outside, a sign on the door read, “Don’t come in here!”

Eventually, a group of Turkers learned to control the PR2 well enough that they were allowed to go out of the locked room and started doing some cleaning tasks around the office. The best operator was a young gamer living in Illinois. Whenever Willow wanted to demonstrate Heaphy, Field would make sure he was controlling the robot. 

By mid 2011, Heaphy had evolved into a complete system integrating a web user interface, a cloud-based backend, and the PR2 robots. And how well did it work?

“It was stunning to see the difference Heaphy made to how the PR2 could perform,” Cousins says. “A skilled operator could really make the PR2 do a lot of stuff.” But at the same time he notes that “it still took a lot longer to do a task ‘through’ the robot than a person could do it using his own body.”

A European roboticist who saw a Heaphy demo during a visit to Willow says it was impressive to see a teleoperated PR2 doing tasks he knew the robot alone couldn’t possibly do. “It’s cool to see that the key problem for getting a robot to do my laundry and dishes is really software,” he says.

But Heaphy also had its share of critics inside Willow. While none would comment on the record for this story, privately they say Heaphy was a small and isolated project that contrasted with the other efforts taking place at Willow. Part of the problem was that Heaphy was a proprietary system, and its code was stored in a restricted repository. Although the Heaphy engineers developed most of their code from scratch, they also relied on software developed by other Willow employees as part of the Robot Operating System (ROS), which is notably an open-source project

Field acknowledges that “work from several ROS engineers was invaluable” to his project. He kept it closed “to maintain focus” and protect its commercial prospects. And he notes that researchers involved with the PR2 program, both internal and external to Willow, were collaborating on their own alternative open-source remote manipulation systems, in particular PR2 Remote Lab and Interactive Manipulation.

In October 2011, Cousins sent a memo to the staff, saying that “creating spin-outs and commercial impact is part of our core mission.” Earlier that year, an internal telepresence robot project became Willow’s first spin-off, Suitable Technologies, which now sells the Beam remote presence system. Heaphy, he announced in the memo, had emerged as the next spin-off candidate. “Heaphy has been kept proprietary in order to prepare it for commercialization and is now ready to move to the next phase.”

Over the next several months, Field worked with Cousins and Bob Bauer, director of commercialization, developing a business plan for the start-up, meeting with potential customers, and taking steps to protect its intellectual property. They envisioned a variety of potential applications for crowdsourced robots, including carrying out tasks at factories, assisting individuals with disabilities and the elderly, and acting as security guards at remote facilities.

But as enamored as he was by the technology, Field still had reservations about Heaphy. He knew that the security and safety of the system had to be improved further to make sure a rogue operator couldn’t run amok with a PR2. And then there were other concerns, such as privacy, and how to prevent remote users from spying on people or things they weren’t supposed to.

Most significant, Field was having second thoughts about Heaphy’s commercial possibilities. Generally, the plan was paying around US $15 an hour for actual cleaning work. But considering the costs of the robot itself, plus training, installation, support, and everything else—he didn’t see how they would make any money. “I had a soul search and thought, realistically, that we were five or 10 years away from this being something commercially viable,” he recalls thinking. 

Video: Tim Field/Heaphy Project
In one demo, Turkers had to use a PR2 to set a table with plates, bowls, and chopsticks in a mock sushi restaurant. They also had to pick up dishes from a rotating tray that simulated the conveyor belts seen in some Japanese restaurants.

Field decided to leave Willow Garage to pursue other projects, and went on to found a start-up dedicated to 3D image capture and visualization. He doesn’t know what, if anything, Willow did with the code and data created as part of Heaphy. “If no one took it over, it’s probably gone,” he says. In the end, it appears that Heaphy simply languished until Willow ceased developing new systems in 2013.

The other Willow engineer says that although Heaphy was abandoned, the idea of telerobotic labor remains very much alive. He points to new projects such as a teleoperated PR2 being used at Oregon State University to safely change bed linens, an application that could find use in hospitals treating patients with highly infectious diseases.

“There has been a paradigm shift in the past few years,” he says. “People have realized that we will not solve AI. It’s just not going to happen.”

Not everyone will agree with that. But what is certain is Heaphy, like many other technologies, was a little ahead of its time. It didn’t go on to become as influential as other Willow Garage projects, but it showed us a possibility we might want to embrace: that before we can have fully autonomous robots capable of helping us, we’ll have to bring in humans to close the gap. 



IEEE Spectrum's award-winning robotics blog, featuring news, articles, and videos on robots, humanoids, drones, automation, artificial intelligence, and more.
Contact us:

Erico Guizzo
Senior Writer
Evan Ackerman
Jason Falconer
Angelica Lim

Newsletter Sign Up

Sign up for the Automaton newsletter and get biweekly updates about robotics, automation, and AI, all delivered directly to your inbox.