Telepresence: A Manifesto

Photo: Dan McCoy/Rainbow

Mind and Machine: Marvin Minsky holds a 14-jointed, three-elbowed, computer-controlled, hydraulically muscled mechanical arm that he built at his MIT lab in the 1970s. This photo appeared in the article "Telepresence," republished here, in the June 1980 issue of Omni.

Over 30 years ago, MIT professor and artificial intelligence pioneer Marvin Minsky laid out an ambitious plan calling for the development of advanced teleoperated robotic systems that would usher in a "remote-controlled economy." He coined the term "telepresence" to describe these systems, which in his futuristic vision would transform work, manufacturing, energy production, medicine, and many other facets of modern life. His plan appeared as an essay in the June 1980 issue of the influential—and now defunct—science and science fiction magazine Omni.

Today, despite the advances in computers and robotics, Minsky's manifesto remains as current and compelling as ever, a powerful call for a technology that could bring about huge societal benefits. IEEE Spectrum reproduces here his essay in full.

You don a comfortable jacket lined with sensors and musclelike motors. Each motion of your arm, hand, and fingers is reproduced at another place by mobile, mechanical hands. Light, dexterous, and strong, these hands have their own sensors through which you see and feel what is happening. Using this instrument, you can "work" in another room, in another city, in another country, or on another planet. Your remote presence possesses the strength of a giant or the delicacy of a surgeon. Heat or pain is translated into informative but tolerable sensation. Your dangerous job becomes safe and pleasant.

The crude robotic machines of today can do little of this. By building new kinds of versatile, remote controlled mechanical hands, however, we might solve critical problems of energy, health, productivity, and environmental quality, and we would create new industries. It might take 10 to 20 years and might cost $1 billion—less than the cost of a single urban tunnel or nuclear power reactor or the development of a new model of automobile.

To convey the idea of these remote control tools, scientists often use the words "teleoperators" or "telefactors." I prefer to call them "telepresences," a name suggested by my futurist friend Pat Gunkel. Telepresence emphasizes the importance of high quality sensory feedback and suggests future instruments that will feel and work so much like our own hands that we won't notice any significant difference.

Telepresence is not science fiction. We could have a remote controlled economy by the twenty first century if we start planning right now. The technical scope of such a project would be no greater than that of designing a new military aircraft.

A genuine telepresence system requires new ways to sense the various motions of a person's hands. This means new motors, sensors, and lightweight actuators. Prototypes will be complex, but as designs mature, much of that complexity will move from hardware to easily copied computer software. The first ten years of telepresence research will see the development of basic instruments: geometry, mechanics, sensors, effectors, and control theory and its human interface. During the second decade we will work to make the instruments rugged, reliable, and natural.

Photo: AFP/Getty Images

Nuclear Scare: In March 1979, a nuclear core at the Three Mile Island power plant in Dauphin County, Pa., suffered a partial meltdown, leaking radioactive gases and water in one of the most significant nuclear accidents ever to occur in the United States. The author contends that telepresence robots should be part of nuclear reactor design to help make repairs and, in case of accidents, to handle damage.

Three Mile Island really needed telepresence. I am appalled by the nuclear industry's inability to deal with the unexpected. We all saw the absurd inflexibility of present day technology in handling the damage and making repairs to that reactor. Technicians are still waiting to conduct a thorough inspection of the damaged plant—and to absorb a year's allowable dose of radiation in just a few minutes. The cost of repair and the energy losses will be $1 billion; telepresence might have cut this expense to a few million dollars.

The big problem today is that nuclear plants are not designed for telepresence. Why? The technology is still too primitive. Furthermore, the plants aren't even designed to accommodate the installation of advanced telepresence when it becomes available. A vicious circle!

Perhaps you have seen the current style of remote control arms used at nuclear facilities. They are little better than pliers, unable to do many things you can do with your own hands. Anyone can buy a simple remote manipulator off the shelf. It usually consists of an input unit for the operator to control and of an output device that does the work. Typically, the input is a handle attached to a jointed armlike linkage. When you squeeze the handle, a gripper closes at the output. But no such device demonstrates true telepresence. The remote gripper may well imitate the motion of your hand, but the remote arm does not follow your arm's curve, and so you cannot always reach around obstacles. The dynamics are unnatural, and the designs skimp on many shoulder, elbow, and wrist motions. The hands have unnatural wrists. The conventional grippers can pinch or clamp but can't twist, shear, roll, or bend. They can't use ordinary scissors. Instead, someone has to remove the hand and replace it with a special tool for that particular task.

If people had a bit more engineering courage and tried to make these hands more like human hands, modeled on the physiology of the palm and fingers, we could make nuclear reactor plants and other hazardous facilities much safer.

My first vision of a remote controlled economy came from Robert A. Heinlein's prophetic 1942 short story, "Waldo." I suppose Heinlein had heard about myasthenia gravis, a disease causing profound muscle weakness. His hero, Waldo, a wealthy young man, is afflicted with it. So Waldo constructs a satellite and invents telepresence devices; he can lie there in zero gravity and operate his inventions effortlessly. Waldo creates dozens of mechanical hands, some merely monkey fists in size, some micrometers in span; he rigs others so huge that each "hand" spreads six meters from little fingertip to thumb tip. The hands imitate everything he did; he spends all his time out in space operating factories on Earth. Almost 40 years after he wrote "Waldo," Heinlein had many suggestions for this article.

Developing telepresence will involve hard scientific and engineering problems, but I believe we should go ahead. Present devices are so clumsy, they are used only when nothing else works. Once improved, however, telepresence will bring us:

Photo: John Hoagland/Liaison/Getty Images

Water and Fire: In June 1979, an explosion at the Ixtoc I offshore drilling rig, operated by Mexico's Pemex, dumped more than 3 million barrels of crude oil into the Gulf of Mexico. This year's BP spill is estimated to have spewed some 4 million barrels. Telepresence technology, the author argues, could help mitigate this type of accident.

- Safe and efficient nuclear-power generation, waste processing, and land and sea mining. Last year's Gulf of Mexico undersea oil blowout is the kind of accident that I'm convinced telepresence technology could have helped to mitigate.

- Advances in fabrication, assembly, inspection, and maintenance systems. With telepresence one can as easily work from a thousand miles away as from a few feet. Manual labor could easily be done without leaving your home. People could form "work clubs." One region of the world could export the specialized skills it has. Anywhere. A laborer in Botswana or India could market his or her abilities in Japan or Antarctica.

- The elimination of many chemical and physical health hazards and creation of new medical and surgical techniques. If we miniaturize telepresence for use in microsurgery, for example, surgeons could repair or replace many little blood vessels in the brain. Other organs beyond the reach of scalpel and forceps could similarly be repaired or substituted.

- A reduction of transportation costs and of energy and commuting time, enabling one person to do different jobs in different places. Mass transportation could be replaced by ubiquitous taxis, remotely controlled by teleoperation. Telepresence devices could fix sewers, electrical conduits, and water mains from within. Teleoperation will do away with all hazardous and unpleasant tasks.

- The construction and operation of low-cost space stations. Telepresence might prove invaluable for solar power satellite construction—for amassing materials in space and supplies for the human work force. Telepresence would be able to assemble various orbital structures.

There are many places here on Earth more dangerous to humans than outer space is. Mines, for example. In a remote controlled mining operation, there are no people to be hurt. A fire in a mineshaft or a collapse would elicit no more response than: "Well, it is very sad. We've lost six robots."

Remote-controlled mining can exploit buried resources efficiently and humanely. No one is exposed to the dangers of explosions or of breathing in coal dust. No more black-lung disease. We will mine the meter-thin deposits of anthracite coal now lodged in formations we cannot reach. Underground combustion and gasification schemes, which are presently unfeasible because they cannot be controlled, may then be feasible.

The biggest challenge to developing telepresence is achieving that sense of "being there." Can telepresence be a true substitute for the real thing? Will we be able to couple our artificial devices naturally and comfortably to work together with the sensory mechanisms of human organisms?

When any job becomes too large, small, heavy, or light for human hands, it becomes difficult to distinguish the inertia and elasticity of the instrument from what it's working on. Telepresence will be able to adjust and compensate for such problems, thus making the job easier. For instance, a remote miner could dig a narrow seam without himself having to stoop or crawl. Machines will incorporate new theories of human sensory pattern perception and feedback control to "reflect" accurately to the user the modified remote sensations.

We have talked of mining, but no matter how much coal we mine, we are, like it or not, becoming dependent on nuclear power. Even if it were to be banned in the United States, we cannot prevent its proliferation abroad. The nuclear designers try to anticipate and avoid all modes of failure. But all reactors have the potential to break down: High temperatures weaken structural materials; generators apply high pressures to those weakened structures; and radiation damage makes inspection difficult, while aggravating structural damage and corrosion.

These problems compel designers to choose between two extremes. One is to build each part with monumental toughness—to minimize human exposure—and hope this system never fails; this is today's designers' favorite approach. But in the end breakdown and failure occur anyway, requiring man's intervention. Even a minimal failure shuts down a reactor for months.

I think the better extreme is to build modular systems that permit periodic inspection, maintenance, and repair. Telepresence would prevent crises before they could arise.

If no one were in the buildings, no one would be exposed to radiation. Then we could all stop quarreling about "tolerable" and "threshold" doses. If nothing enters or leaves the reactor except by way of telepresence machines, no one can steal anything. Computers—or skeptical people—can monitor for unusual activities over viewing channels. This allows few opportunities for sabotage, and it makes it easier to combine power generation, fuel processing, and waste management.

We can employ telepresence in any environment alien to humans. Most of the earth, for example, is ocean; "moonwalks" on the ocean floor at two miles' depths are technically more difficult to execute than moonwalks on the moon or Mars. Remotely operated seafloor "construction crews" could bypass the prohibitive hazards of manned exploration, avoiding the risk of weather troubled ships and treacherous towers in mining on the continental shelf. The U.S. Office of Naval Research has some remote controlled deep sea exploration projects, and eventually such systems will explore for and extract deep sea petroleum and minerals. Eventually entire undersea industrial plants could be so controlled from the surface.

There are already some undersea manipulators. The Alvin submersible, at Woods Hole, Massachusetts, is wonderful, but its manipulator is used mainly for picking up samples. You couldn't tie a knot with it. I'd like to see one that can do anything that fingers can do.

Photo: NASA

Space Explorers: A model of NASA's Viking lander sits on a mock Mars landscape. In 1976, NASA successfully sent two Viking landers to the Red Planet. The mission provided a wealth of scientific data and stunning photographs—and fomented a debate on whether space exploration should rely on humans or robots.

In space the amazing success of Vikings 1 and 2 shows how much can be done with remote control—even with daylong transmission delays. Yet the Viking spacecraft had pathetic limitations. There was no way to reconfigure the equipment to make use of what was learned; a week of breathless planning was required just to get Viking 2 to turn a stone over.

I think the best way to explore the planets is to have people in orbiting spacecraft operate telepresences that maneuver on the surface. A Mars Rover with good telepresence manipulators could make extensive excavations, then reconfigure scientific equipment to exploit what has been discovered.

Think how much more we could have learned with a permanent vehicle on the moon. The Earth Moon speed of light delay is short enough for slow but productive remote control. With a lunar telepresence vehicle making short traverses of one kilometer per day, we could have surveyed a substantial area of the lunar surface in the ten years that have slipped by since we landed there,

Among the most exciting prospects for solving our energy woes is to build a ring of solar power satellites in orbit around Earth. Safe, free solar energy could then be collected and beamed back to receivers located near our cities. The main problem is the cost: We must put sufficiently large structures in space to gather enough sunlight, since each station requires thousands of acres of reflectors and collectors. And then there's the cost of sending people into space to build them.

Telepresence could save billions of dollars by employing remote controlled hands stationed in orbit and controlled by technicians on Earth and on the moon. Most satellite construction could be done by people working in their own homes and offices.

To circumvent the cost of lifting satellite payloads against Earth's powerful gravity, scientists have been devising ways to manufacture and launch materials directly from the moon or from the asteroids. Building such lunar facilities, however, would be impossibly expensive if carried out entirely by men in space suits on the moon. Instead, why not use cheap, Earth based labor via telepresence to build moon factories? Imagine having to go no farther than your study to operate a crane on Mare Imbrium. We need only send telepresence machines on inexpensive one way trips.

The scenario includes sending 20 real men and women to the moon. It's not very difficult. Saturn V's have the potential to send up a crew of scientists and engineers with many months' supplies. (If only they didn't have to bring them back!) So we ferry up a return vehicle for use in emergencies. Then we send up more permanent housing and, finally, the superflexible telepresence equipment needed to construct the first lunar installations. The people are there to supervise the work and to fix the equipment when necessary.

One major obstacle to all this has been NASA's legislated inability to deal with such far reaching concepts as telepresence. The U.S. space program is entirely mission oriented. NASA never gets appropriations to make better manipulators or navigational devices, as things in themselves. Even so, scientists at Ames Research Center, in California, managed to develop a startlingly nice telepresence, a remote-controlled space suit. It looks like a real space suit; you put your arm into the master suit and the slave suit moves just like your arm. It's an extremely good arm, a perfect imitation. Your arm feels natural in it. But it doesn't have any hand.

The space shuttle, too, has an arm. It is very long, and it takes about half a minute to complete any motion. But there isn't any reason to hurry. In zero gravity nothing weighs anything; so one can use a 100-pound, long, slender pipe to move a 10-ton load very slowly. (There's a simulator for this at Marshall Space Flight Center, in Alabama. It is a model of the fuel tank of the space shuttle—a helium balloon as big as a house. Sitting in its hangar, it weighs nothing. When you press on it, no movement occurs for about 30 seconds, but then it begins to move. If you grab on to it, it pulls you right up, too, and because it has no weight, you can lift it with your hand, but it has a mass of half a ton!)

While the shuttle arms are merely glorified construction cranes, they are the beginnings of giant teleoperators. At the other end of the size spectrum, biologists have long used micromanipulators, tiny teleoperators. But none of them have any sensors. If we were to miniaturize telepresences for surgery, we could develop touch reflecting microhands on slender probes that reach through the vessels' narrowest passages. Further in the future a surgeon could direct a semi intelligent procedure, including several simultaneous microtelepresences, to make smaller repairs swiftly.

The first crude remote controlled mechanical hands were built around 1947 at Argonne National Laboratories, in Illinois, for handling dangerous chemicals. In 1954 the late Ray Goertz, a scientist at Argonne, developed hands in which electric motors "reflected" some of the forces back so that the operator could feel something of what was happening, at least resistance and pressure, if not textures. Paradoxically, the very first telepresences could relay sense of feel better than later electric models could, because they used rigidly linked cables and pulleys. Later electric motors were stronger and could work at greater distances, but they lost that sense of feel one got through the cables. More advanced models measured forces at the output and used additional motors to reflect those forces back to the user's hands. When the remote claw hit something, the input became harder to push. This helped, but the force reflection was still inadequate for performing delicate work.

Early pioneers like Goertz had the fantasy of building better robots of various kinds, and then people got interested in my field, artificial intelligence—getting robots to do smart things. And we did get them to do simple mechanical things, some factory work, like assembling a motor. But they were always handicapped by those terrible claw hands.

Photos: Schenectady Museum Archives/GE

Iron Man: Ralph Mosher, an engineer working for General Electric in the 1950s, developed a robotic exoskeleton called Hardiman. The mechanical suit, consisting of powered arms and legs, could give him superhuman strength. Mosher subsequently made a simpler version that permitted him to sit in his chair and pick up refrigerators.

To create true telepresences, we must supply more natural sensory channels—touch, pressure, textures, vibration. We must learn which sensory defects are most tolerable. In 1958 Ralph Mosher, an engineer working for General Electric, developed a telepresence—called Hardiman—that had good dexterity and compensation. It had only two fingers, but those fingers each had three joints so that they could wrap around any object. Hardiman could lift hundreds of pounds; it transformed you into a superperson. But it was never put to any practical use. Mosher subsequently made a simpler version that permitted him to sit in his chair and pick up refrigerators.

Another big manipulator was designed and built in the late 1950s as part of a project to build nuclear airplanes. But Congress finally decided the plane, designed to stay aloft for a year without landing, wouldn't be safe.

Although little work has been done since the '50s, there now exist a few more versatile experimental manipulators. Electrotechnical Laboratory, in Tokyo, has made a three fingered, 12 jointed hand that can roll a baton. But that's about all it can do. A group at Stanford University invented a long, snakelike tentacle that can wrap around objects. I once built a 14 joint, multi elbowed arm that can easily reach around things in its way. But no project has the resources to bring such systems to practical perfection.

I think we should make telepresences that compare well with the human hand: a five fingered device capable of imitating natural motions. It should be mobile. We might then adapt designs and concepts from the arm to make legs, yielding a system able to work wherever people can, not only on carefully prepared floors.

To control such an instrument, we will also want a light, well articulated sleeve that includes effectors to reflect the sensations. This will require advanced materials and new muscle imitating devices; for visual feedback, we'd need slender fiber optic probes articulated to emulate the operator's head and eye motions. We probably would want to have an eye of some sort on the fingers.

A Philco engineer named Steve Moulton made a nice telepresence eye. He mounted a TV camera atop a building and wore a helmet so that when he moved his head, the camera eye on top of the building moved, and so did a viewing screen attached to the helmet.

Wearing this helmet, you have the feeling of being on top of the building and looking around Philadelphia. If you "lean over," it's kind of creepy. But the most sensational thing Moulton did was to put a two to one ratio on the neck so that when you turn your head 30 degrees, the mounted eye turns 60 degrees; you feel as if you had a rubber neck, as if you could turn your "head" completely around!

Why did telepresence stop evolving 20 years ago? One reason is that research funds declined while costs escalated. By 1960, no laboratory could afford to take another step. But a more fundamental reason for this stagnation is that engineers are far too clever at solving immediate problems. This has led to endless repetition of the same scenario: An application needs a better manipulator, for example, to join two pipes together; an existing mechanical hand would help, but only if it had another joint in one of its fingers. One must add another control channel, design a new sensor and input feedback device, modify a microcomputer program, retrain the operators, and reengineer the older tools. All this puts enormous strains on a company's budget. Ultimately pipe fittings are redesigned so that the old, clumsy hand can manage.

A good production engineer can solve almost any specific problem by using a special jig, fabricating a new part, developing a special tool for the hand, or replacing the hand with a special tool. Each problem gets solved, to be sure, but the overall technology becomes antiquated and goes unnoticed until an accident such as the one at Three Mile Island or the Mexican oil spill occurs and we find out there is no way at all to turn a valve from afar, or to replace it.

Several major companies have been involved in telepresence research from time to time—AMF Hughes, General Mills, IBM, and others—though none of them have reached "critical mass." Many smaller firms possess more precious skills—Unimation, Central Research Laboratories, Programmed and Remote, and others. The Defense Advanced Research Projects Agency, working with the army, once supported work on powered armored suits, like those in Heinlein's Starship Troopers, but the work was abandoned. University workers have had many good ideas, designs, and prototypes, but they could never afford to engineer complete systems. There are important projects at Carnegie, MIT, Jet Propulsion Laboratory, Stanford, and other university labs.

Part of the problem has been that telepresence has never been anybody's baby; such a project demands centralization. It requires imaginative specialists in sensors, effectors, control theory, artificial intelligence, software, engineering, psychology, and first class facilities for mechanical and electrical engineering and materials science. It will need strong resources for interactive computation and for real time physical simulation.

It would be difficult to assemble such an organization in today's peacetime atmosphere. What we need is a modern league of working centers connected by a computer network. Such a network would contain a central data bank somewhere in the United States, combining administrative and engineering resources that need centralization. Perhaps there could be computer centers close to universities or industrial locations, working together through communications networks such as ARPANET (the computer conferencing network of the artificial intelligence community).

I can't imagine anyone doubting that telepresence is possible. It's a matter of solving many problems that are hard, but not impossible. In the mechanical area the same things have been done repeatedly; engineers spend their time arguing about what kind of wrist is better. Some think the wrist should go round and round, spinning forever, especially on a robot that's using a screwdriver. That's all right for the industrial robots, but there isn't much point in that for a telepresence, because you can't spin your own wrist around to control it.

Some research has been done in the psychology of spatial perception, in terms of feedback controls and the interrelationship between electronics and the human nervous system. There is a device, for example, developed by J. C. Bliss and J. G. Linvill, at Stanford, that can translate print into "feel," which enables the blind to read conventional printed matter. It's a gadget that fits conveniently on your fingertip and has a lot of miniature photocells to sense light and a lot of little vibrators that allow the finger to sense remotely the fine shape of the letters. In my own laboratory, graduate student Danny Hillis recently fabricated a thin, skinlike material that can "feel'' and transmit small tactile surface features.

Someone could develop similar devices for telepresence—vibrating patterns, for example, that would convey the sensations of "hot" or "cold." However, very little is known about tactile sensations. It seems quite ironic to me that we already have a device that can translate print into feel, but that we have nothing that can translate feel into feel.

Eventually telepresence will improve and save old jobs and create new ones. Later, as we learn more about robotics, many human telepresence operators will be able to turn their tasks over to the robots and become "supervisors." In the long run, since each step toward telepresence is a step toward robots, telepresence sensors and output devices could be controlled by computers rather than by people. This becomes inevitable as we learn more about artificial intelligence.

Computers equipped with artificial hands and eyes have actually grasped and moved objects in accord with verbal commands. A complicated precision bearing was so assembled at MIT; an entire pump was put together at Stanford; a toy automobile was constructed in Edinburgh, Scotland. Similar work has been done at SRI International. These laboratory programs are too undependable for practical use because, although we can do many things with computers, we cannot get them to do many things any child can do. Someday our machines could do all our work for us, but that is a long way off, and it would need another whole article to begin to explain the problems.

If teleoperator technology promises wealth and freedom beyond dreams, is there a dark side? People who issue manifestos should think about such matters. The solution may be to grant those who want to live in the "old ways" their chance, while those who want new gifts should also have theirs. I think the gifts promise better, richer, and longer lives. Might telepresence, though, have a special tendency to make workers feel alienated? Perhaps, yes, even with superb technology. Many jobs will become intensely more interesting and more creative; many worlds will be expanded.

If each step toward telepresence were also a step toward the economic pain and psychic grief of unemployment, one might consider working against it. Yet a generation of reforms is already eliminating many of the unsafe jobs that telepresence could preserve. Telepresence offers a freer market of men's and women's skills, rendering each worker less vulnerable to the moods and fortunes of one employer.

Finally, in a strange sense, the question of "technological unemployment" may become moot. Many young people today consider it demeaning to be bound to any single employer, occupation, or even culture. Perhaps many of us sense—at least on some level—that little of what we do really has to be done. Our attitudes about work, about changing the quality of it, depend as much on our own dispositions and our alternatives as on the jobs themselves. In effect, most of us already feel technologically unemployed.

Postscript: from the London Telegraph Foreign Service

PARIS—The French government has authorized Electricité de France, the French central electrical generating board, to go ahead with loading two new nuclear power stations with enriched uranium fuel. The approval came despite the detection [in 1978] of cracks several millimeters wide in the tube end-plates of the steam generators and in the tubes connecting them to the reactors.

The existence of the cracks first was disclosed…by nuclear engineers who pointed out that once the reactors go into operation, repairs would be impossible for lack of appropriate robotic equipment. François Kosciusko Morizet, government director of industrial quality and security, countered that the defects were very carefully examined and were found to be superficial. In the very worst of hypotheses, the cracks would not give trouble for five to six years, by which time repairs would be easy, he said, since from 1981 on, France would have robots, able to repair such defects automatically. [Italics mine. —M.M.]

Since France is walking the tightrope of possible power shortages and Electricité de France has warned that possible power cutbacks may be necessary …, the government has decided that the economic risk of holding up the electricity generation timetable by repairing the reactor cracks now is greater than the danger of serious accidents later.

My work in artificial intelligence is carried on in a world as much fiction as science. This essay used specific suggestions from Isaac Asimov and Robert A. Heinlein as well as from Carl Sagan, Brian O'Leary, Edward Purcell, and many others.

For more technical details I recommend Remotely Manned Systems, edited by Ewald Heer (Caltech, 1973), and Human Factors Applications in Teleoperator Design and Operations, by E. G. Johnsen and W. R. Corliss (Wiley, 1971). For a discussion of intelligent machines, see M. Minsky, "Computer Science and the Representation of Knowledge," in The Computer Age: A Twentieth Century View, edited by M. Dertouzos and J. Moses (MIT 1979).

In memoriam, 2001: This essay was published in Omni because Kathy Keeton Guccione, its founder, instantly grasped these ideas and suggested that Omni could reach more readers than would any technical journal. She was a powerhouse of support for space research and other far-out sciences—and a friend that a legion of writers will miss.

This article originally appeared in Omni magazine, June 1980 issue, pages 44-52, and is reproduced here with permission from Marvin Minsky.