Of Modes and Men

Cut-and-paste, the one-button mouse, WYSIWIG desktop publishing--these are just a few of the user interface innovations pioneered by Larry Tesler

Like Woody Allen's 1983 movie character, Zelig, who appears at every significant historical event of his era, [see sidebar, "Larry Tesler"] has had a hand in major events making computer history during the past 30 years. When the first document-formatting software was developed at Stanford University in 1971, Tesler was coding it. When a secretary first cut and pasted some text on a computer screen at Xerox Corp.'s Palo Alto Research Center (PARC) in 1973, Tesler was looking over her shoulder. When the first portable computer was turned on in an airport waiting area (and on an airplane), Tesler had his fingers on the keyboard. When Steve Jobs went to PARC in 1979 to see the legendary demo that is purported to have set the stage for a revolution in computing, Tesler had his hand on the mouse.

And when Apple Computer Inc.'s infamous Newton handheld computer failed spectacularly in the early 1990s, taking millions of dollars of investment and a few careers down with it, Tesler was there, too. Hey, nobody gets it right 100 percent of the time.

So why haven't you heard of him? [see sidebar, "Larry's Lexicon"].

"Larry generated a lot of the basic ideas for the work we were doing," says Douglas Fairbairn, a former colleague at PARC. "But he doesn't have a big ego, so his name didn't get attached to things. He wasn't the one guy who did one big thing you'll remember him for; he was a collaborator on many things."

This past May, Tesler joined Yahoo Inc., in Sunnyvale, Calif., as vice president of user experience and design. At Yahoo, Tesler has perhaps his biggest opportunity yet to shape the way hundreds of millions of people interact with technology as they search the Web, watch and listen to media, share things they've created or found, correspond with family and friends, and even find love through the Internet.

When Tesler was a child, his parents couldn't quite figure him out, so they had him tested by a career counselor. The counselor told them that the results were unusual--Tesler registered a strange combination of sensitivity to people and fascination with math. The best career choice the counselor could suggest was working as an architect or maybe becoming a certified public accountant.

The field of computer usability did not yet exist. At the time, people had to learn arcane codes to communicate with computers, typically via punch cards and printouts. The idea of making a computer easy for the average person to use was still a new one when Tesler came along.

"I was born to do it and feel very lucky to have been in the right place at the right time," he says.

It's a recurring theme in his career, starting with Stanford University, in California, in the early 1960s. Computer time was available free of charge to anybody who wanted it, and Tesler, a math major, wanted it very much. He wrote a few little programs for his own amusement, and he programmed enhancements to the software that created graphic patterns for the 3465-seat card section at Stanford Stadium.

Soon he got a paid job in Stanford's computer laboratory developing software that would make strings of computer code easier for humans to read, by formatting the text into either fixed-width columns or free-form lines. It wasn't a new idea, he says, just a nicer implementation of an old idea.

But this was the first bit of computer code Tesler wrote that was meant to be used by large numbers of people. And he was hooked, by programming in general and by text formatting in particular. Text formatting--the ability to display or print out type in different sizes and with different fonts, spacing, and so on--doesn't generally get much attention in treatises on how the computer has transformed civilization. But it is actually one of the most common computer functions and the foundation of such programs as Microsoft Word. After college in the early 1960s, Tesler spent a few years working as a computer consultant. As one of only four listed in the Palo Alto, Calif., area phone book, he got a lot of calls. He wrote programs that scheduled classroom use for the San Jose school district and that simulated the distribution of nuclear fallout for the Stanford Research Institute (now called SRI International).

When the Silicon Valley recession of the late 1960s dried up consulting contracts, he managed to find a job at the Stanford Artificial Intelligence Laboratory, where he considered how computers might be made to understand natural human language and even to express simulated human thoughts and feelings.

"This work was technically fascinating," Tesler recalls, "but one day I woke up and realized that this stuff wouldn't be practical for decades."

And he had bigger woes, too: by 1970 his brief marriage to his college sweetheart had broken up. He packed his 5-year-old daughter and her favorite toys into his aging Dodge Dart and moved to Oregon, joining a wave of Vietnam-era dropouts. There, he and a group of friends bought land and began building their own houses. Eventually, he had a nice house but no savings. That was when he noticed that there was only one computer within a 60-kilometer radius, at a bank.

The bank wouldn't hire him, so Tesler called his former colleagues at the Stanford AI laboratory for help. He hadn't been in touch with them for six months.

"Did you know that Xerox started a computer research center in Palo Alto?" one asked Tesler. "About a week after you left for Oregon, Alan Kay, who planned to go to Xerox himself, came here looking for you." Kay, now a senior fellow with Hewlett-Packard Co., in Palo Alto, recalls that he even went to Oregon to try to track Tesler down. Kay had worked briefly at the Stanford AI lab and had considered Tesler "one of the most interesting intellectuals and one of the best programmers there."

Tesler immediately contacted Robert Taylor, director of the computer science laboratory at PARC, asking for a part-time job, so he could continue to live in Oregon. But PARC would consider Tesler only for full-time positions; Tesler wasn't interested. A month later, still jobless, he reconsidered, but by then Xerox had instituted a hiring freeze. The opportunity was gone.

Tesler went back to the Stanford AI lab, hoping he could find something to do there that was more "real" than advanced research. The lab manager, Les Earnest, was interested in computer typesetting, a nascent technology at the time, and assigned Tesler to develop a "document compiler," something that could format text and generate footnotes, tables of contents, indexes, and bibliographies.

So in 1971 Tesler wrote a piece of software that would allow users to tuck formatting commands inside normal text as they typed, like inserting a parenthetical expression. When the computer displayed or printed text coded in this manner, it would adjust the text as instructed--boldface or italic, for example--and hide the instruction. This software, called Pub, was one of the first, if not the first, of what would become a "markup language with embedded tags and scripting." The best-known such language today is Dynamic HTML, used in Web design. Pub was distributed through the ARPANet, a precursor to the Internet, and widely used in universities.

Tesler continued to check in with PARC, and at the end of 1971, Xerox offered him a job in the On-Line Office System Group, then building an ambitious distributed operating system for office applications. Tesler was interested in personal computers, so the project didn't thrill him. And he thought the offered pay was too low. So he turned PARC down. Again.

More than a year went by. It was early 1973, and PARC was in the thick of developing what would be the first personal computer, the Alto. It made Tesler yet another job offer, and this time he didn't refuse. He would split his time between the Office System Group and Alan Kay's Learning Research Group.

One goal of the Learning Research Group was to develop Smalltalk, the first dynamic object-oriented programming language--the first language, that is, designed to create programming entities that had characteristics and attributes that could be passed on to other entities. A generic "job" entity, for example, might have such attributes as a salary, a department, and health benefits. Another entity, such as "manager," could be created, automatically inheriting those attributes and gaining additional ones, such as supervisory duties or security clearances. Smalltalk back in the 1970s contained almost all of the concepts in today's enormously popular Java language.

As one of Tesler's first tasks at PARC, he and a co-worker wrote a paper on the future of interactive computing, which for the first time talked about cut-and-paste as a way of moving blocks of text, images, and the like. It also described representing documents and other office objects stored on the computer as tiny images--icons--instead of as a list of names [see photo, ].

Today, icons are everywhere--as folders, documents, and the trash can or recycle bin for deleted files. The radicalness of the concepts is apparent only when you understand that, in those days, opening up a document required knowing--and typing in--its exact name and document type.

Before Tesler arrived, the Office System Group had already formulated some ideas about how they expected users to interact with computers. Their concept was based on the three-button mouse and depended heavily on the use of modes. In a mode-based system, you first tell the computer what type of thing you are going to do, and then you do it. So if you want to insert text in a document, you put the computer in insert mode, select your insert point, type in the text, and then exit the insert mode. If you want to delete, you put the computer in delete mode, select the text to be removed, execute the deletion, and exit the mode. It could take weeks or months to become proficient in the use of the system, but at the time, practitioners believed nothing much simpler than that could offer as much power to the user.

"I was aghast," Tesler says. He reasoned that ordinary users wouldn't invest more than a few minutes in learning a user interface and that they would lose track of modes, risking deleting instead of inserting, for example, with potentially disastrous results. But Tesler found few allies among his colleagues, who, incredibly to Tesler, felt that a moded interface was intuitive.

Meanwhile, PARC hired Sylvia Adams, now Sylvia Amundsen, a secretary whose previous high-tech experience had been limited to the IBM Selectric typewriter. When she arrived for her first day of work, Tesler recalls, "I grabbed her. I didn't want her to get contaminated by some word processor that we were using."

He sat her in front of a blank computer screen and gave her a printed page of text that he had marked up with corrections. "See this text?" he asked. "Pretend it's on the screen. And see these proofreading marks? Your job is to make those changes on the screen. How would you do it?"

"Well," she said, "I have to insert something there, so I would point there, and then I would type what I wanted. And to delete this, I would draw through it." Tesler took notes as Adams invented, in effect, the modeless user interface for text editing.

Then Tesler had her use the moded system in use at PARC. She hated it; she got stuck frequently in the wrong mode and had trouble getting out of it. She proved his theory that it was a bad system for a nontechnical beginner.

The test was enough to convince Tesler's boss, Bill English, to allow Tesler to work on an alternative to the moded interface. As he developed it, he continued his user experiments, regularly grabbing "civilians" from the building lobby--delivery people, friends picking up employees for lunch, anybody without computer experience. Tesler sat them down in front of a computer screen now running prototype software and asked them to edit text.

"They were such novices," Tesler recalls. "With one couple, the first thing they said when looking at the crisp text on the screen was, 'You get really good TV reception in here.'"

After writing a simple text editor called Mini Mouse, he went on to develop Gypsy, a modeless text-processing system that was the first to use many now-familiar elements: the cut-and-paste function to move text, a fill-in form to enter search terms, selection of text by holding down a mouse button and dragging the cursor through it, bold and italic type styles, and what-you-see-is-what-you-get printing. Tesler coined that phrase and other now-common terms [see http://www.spectrum.ieee.org for more]. Gypsy also first implemented click-to-open files; previously, the only way to open a file was to type its name and the Open command. Gypsy was never commercialized, but it was used for years to edit manuscripts at Ginn and Co., a textbook publisher then owned by Xerox. Tesler went on to develop other pathbreaking text-formatting software, but none of it made it to market.

In the mid-1970S, the Holy Grail of Kay's Learning Research Group at PARC was the Dynabook. A name dreamed up by Kay, Dynabook was to be a portable computer smaller than today's laptops. It would do just about everything anyone could then imagine a computer doing, including storing everything you needed on a daily basis and communicating with other computers when necessary. It was a pretty audacious idea. At the time, PARC's Alto, one of the smallest stand-alone computers then in existence, was about the size and weight of a hotel minibar refrigerator. The Dynabook was clearly way off in the future.

 

"I was born to do it and feel very lucky to have been in the right place at the right time"

 

Kay regarded the 1973 Alto as an interim Dynabook that did the right things but was too big to carry around. In 1978 Kay and his colleague, Adele Goldberg, felt it was finally possible to build the next computer that would lead to Dynabook: Notetaker. It would be portable, use floppy disks (then a brand-new storage medium), and run on batteries. It would have only one function--it would allow students to take notes. PARC's Fairbairn set out to design the hardware and enlisted Tesler to assist.

Tesler didn't know a whole lot about hardware; his entire hardware engineering experience consisted of once building a joystick. Fairbairn, a hardware pro, said, "Hey, it's not hard," and handed him a manual for the just-introduced Intel 8086, which was to become the first commercially successful 16-bit microprocessor.

Tesler read the manual and designed the circuitry for the processor board while Fairbairn worked on the many other boards, interfaces, and specialized hardware needed. Within a week of the release of sample chips, the processor board was wrapped. And it didn't work. The fault was neither Tesler's nor his prototype software's. The Intel documentation had numbered the pins on a peripheral chip backward, and as the first developer to build a prototype, Tesler was the one who found the bug.

Because the PARC researchers' vision for the Dynabook included networking, the Notetaker was to include an Ethernet card. Existing ones contained more than 100 chips. But the size constraints for a portable machine meant the Notetaker's Ethernet card could hold 24 chips, tops. Tesler decided to move a number of functions to software, ending up needing only 20 chips. In effect, he had come up with the first software implementation of the Ethernet protocol.

The Notetaker worked; 10 prototypes were built, and Tesler and Fairbairn went on the road. They crisscrossed the country, visiting Xerox executives in several locations, trying to convince someone--anyone--that Notetaker should be a product. Some executives seemed excited but took no action beyond referring the duo to others. This went on for months.

One night in 1979, in the midst of this road show, Tesler and Fairbairn were sitting in the gate area at San Francisco International Airport, waiting to board their red-eye flight to New York City. Because the Notetaker was a secret project, they had never turned the 16-kilogram computer on outside of a Xerox building. But the gate area was nearly empty. So they opened up Notetaker, powered it up from its built-in battery, ran its Smalltalk program, and quickly turned it off again [see photo, ].

"That was the first time a portable computer was ever used in an airport," Tesler says. "We think."

The duo couldn't resist, Fairbairn says, turning the computer on again, during that flight. History was made once again. The flight attendants didn't even notice.

By the Fall of 1979, it had become clear that Xerox executives just weren't buying the Notetaker. They had their hands full competing with an onslaught of low-cost copiers from Asia. Meanwhile, in December 1979, Steve Jobs and his colleagues from Apple Computer Inc., in Cupertino, Calif., took their now-famous tour of PARC. Tesler was on hand to demonstrate the Alto's user interface.

"The questions the Apple people were asking totally blew me away," Tesler says. "They were the kind of questions Xerox executives should have been asking but didn't. They asked: 'Why don't the windows refresh automatically? Why did you do the menus this way?'"

"And the question I remember most was from Steve Jobs. He said, 'You guys are sitting on a gold mine here. Why aren't you making this a product?'"

After that meeting, Tesler decided that his next job would be at Apple. He would stay at PARC long enough to get them to accept the Notetaker, finish the page layout system still in development, and give a scheduled talk at the 1980 Association for Computing Machinery Siggraph conference, the big annual meeting for the computer graphics community.

The Siggraph talk was key, Tesler felt, because he had for the first time obtained permission from Xerox to publicly demonstrate some of the word-processing and text-formatting programs developed at PARC. He believed this demonstration would take trade secret status away from PARC's pioneering concepts, thereby allowing him to work on similar technology at Apple without breaking his confidentiality agreement.

He wasn't the only researcher talking about leaving PARC in the early 1980s. Kay recalls that life at PARC was getting to be more and more of a strain.

Tesler started working for Apple shortly after his Siggraph speech. Bruce Daniels, the software manager on the Lisa personal computer project, hired him.

"There was no question that we would hire him," recalls Daniels, now a program manager at Sun Microsystems Inc., in Menlo Park, Calif. "We didn't need to interview him. We had seen this vision at PARC, we loved it, but how could we build it? He knew everything; he was this font of wisdom, he could answer any question, he was passionate--it was the proverbial match made in heaven."

Lisa, released in 1983, was technologically dazzling. Coming just a year after Xerox's successor to the Alto, the Star office workstation, Lisa was the first commercially available personal computer that, like the networked office Star, used windows, icons, and a mouse--all the elements of the graphical user interface that we now take for granted. Ultimately, though, the US $8000 system was commercially unsuccessful. Apple stopped manufacturing the Lisa after only two years; just 30 000 had been sold.

In the Lisa group, Tesler contributed to "absolutely everything," Daniels says [see photo, ]. Tesler, more modestly, says that Bill Atkinson and Jef Raskin were well into the Lisa interface design when he got there, but he feels he had some influence. He talked them into using cut-and-paste, he recalls, and wrote a memo that argued persuasively for the one-button mouse, a concept that Tesler had advocated at PARC and that Raskin had been promoting at Apple.

"I had asked him to suggest how many buttons should go on the mouse," Daniels recalls. "I made the decision based on his comments, and the Apple mouse still has just one button."

Tesler brought his idea of click-and-drag for selecting things over from PARC as well, along with the pop-up menus that had been used in Smalltalk. He designed the software widget that makes windows divisible (as is commonplace now in the spreadsheet program Microsoft Excel). And when Atkinson was working on the menu bar, trying to attach it to windows in such a way that it wouldn't obscure text or get too small, Tesler pointed to the top of the screen. "I said, 'It should be at the top,'" he recalls. "It should be there, always in the same place." Atkinson implemented the menu bar that night, and by dawn he had come up with menus that dropped down as you dragged the mouse and shortcut keys.

"He invented the whole thing," Tesler says, "but I gave him the inspiration."

Meanwhile, the Macintosh project, for a cheaper computer than the Lisa, geared up. Tesler was not officially part of it; he continued to develop software for the Lisa group. But he spent one day a week in the Mac building critiquing the user interface under development. Convinced by tests with ordinary users, the Mac designers ultimately decided to base the Mac interface on Lisa, with a menu bar at the top and folders and documents on the screen, instead of the metaphor then being tried--a large picture of a floppy disk taking up most of the screen, with icons of documents scattered on top of it.

In 1986 Tesler was named vice president of Apple's newly formed Advanced Technology Group, the group that contributed to the design of the next-generation product, the Macintosh II. At one point, researchers hooked 24 Mac IIs together to render an animated short, called Pencil Test, which was shown at the 1988 Siggraph conference. Movie director John Lasseter, more recently of Pixar fame, advised on the project; at the time, he was dating a woman in the group (whom he later married).

Struggles in doing Pencil Test, particularly in synchronizing sound and video, led to another small research group, the Time Lords. The software that the Time Lords developed was called QuickTime, a form of which still ships with every Macintosh computer and is also widely used on Windows computers.

With Tesler at the helm, the Advanced Technology Group grew to some 200 people. One of the many projects under development was a full-function handheld Mac. A prototype was up and running when Tesler made what he recalls as one of the worst decisions of his career: he killed the project. Instead, he devoted himself to the development of the Newton, a handheld computer that was to be based on a new technology incompatible with the Macintosh line of computers.

The year was 1990. Tesler had just taken over management of the Newton project, which had been under way for several months. Apple president Jean-Louis Gassee, who had championed the project, had just quit the company; the Newton project leader left a week later.

Competition between the handheld Mac group and the Newton group had developed; it was obvious that one had to go. Tesler had to choose which. The handheld Mac worked; the Newton was still just research. However, Newton proponents convinced him that the handheld Mac was a dead end, because it ran an old version of the Mac operating system and wasn't going to be powerful enough to run the new version when it came out.

The handheld Mac was relegated to history, and all bets were on the Newton.

"I did a few things right at the beginning of that project," Tesler recalls, "and then I made an unbelievable series of mistakes."

When Tesler joined the Newton group, the computer was envisioned as a 20- by 28-centimeter machine, weighing about 3 kilograms and selling for over $7000. He insisted developers get the price down to $1000, and he worked out an arrangement to obtain processors designed by Acorn Computer Ltd., in Cambridge, England. He even convinced Apple to invest in a newly created company, Advanced RISC Machines Ltd., also in Cambridge, that would produce them.

That, Tesler recalled, was one of his good decisions, as was taking the nascent spread-spectrum wireless communications technology being developed for the Newton and moving it into the Advanced Technology Group. That project team eventually joined with development teams in other companies and created the Wi-Fi wireless local-area network standard.

Then the mistakes began. For one, Tesler killed an internal handwriting recognition project, but he didn't give up on handwriting recognition altogether. Instead, he licensed technology from ParaGraph, a company founded in Moscow. It didn't work on Newton's relatively slow processor.

Looking back, he says he should have realized that while the Newton technology had great potential, it was too immature for a product that needed to get out the door. Eventually, the Newton handheld computer was released.

"And it was a disaster," Tesler recalls. He left the group just before the product was shipped, because at that point he was against releasing the product at all. He stayed at Apple for four more years as chief scientist. "It was a consolation prize," he says. He proposed a number of projects; none got funded. Eventually, he took on the job of managing network products, a profitable business but one without the risk-taking innovation that excited him. His final task before quitting in 1997 was to shut down the Advanced Technology Group; the company felt it could no longer afford basic research.

"I did a few things right at the beginning... and then I made an unbelievable series of mistakes"

While closing the group, Tesler became enamored with one project, a programming language called Cocoa. Like many of his technical loves, Cocoa embraced object-oriented programming. But unusually for such a language, Cocoa was to be used to allow schoolchildren, with no previous programming experience, to easily create their own simulations and video games. With Apple's blessing, Tesler spun Cocoa off into a company, Stagecast Software Inc., now in Burlingame, Calif.

The software was renamed Stagecast Creator. It was aimed to the then-hot creativity and education market. It is still available and today is a standard tool in computer camps.

The group moved into offices in Palo Alto in 1997. Tesler met with potential investors, developed marketing strategies, reviewed code under development, and even bought the office trash cans.

Unfortunately for Stagecast, the product was released in 1999, just as the education software market went bust. By mid-2000, it was clear that Stagecast wasn't going to be able to support its 15 employees, much less make a fortune for its investors. Even though the product was loved by its users and won countless awards, the market was just too small.

"Millions of dollars went down the drain," says Tesler, who never drew a salary from Stagecast, "including a lot of my personal money. A friend had told me that if I wasn't willing to invest half my assets in an idea, I didn't believe in it enough. And the more my investors and I sank into it, the more I felt that some day we'd figure out how to turn the corner. But after the market crash, what had been a fraction of my wealth suddenly became almost all of it."

Still, he had no plans to leave, thinking he would stay with the company, maintain the product, and eventually find a buyer.

His wife finally intervened. "She told me, 'Stop dreaming, it's over,'" he recalls. "And she got me to face the fact that I was just going to have to walk away."

So he left. Two employees remained and, to date, make enough from keeping the product on the market to support themselves modestly. Meanwhile, a recruiter put Tesler in touch with Amazon.com Inc.

At first, he was hesitant. He perceived Amazon as a retail company, and he didn't know anything about retail. Besides, the job was at the company's home office in Seattle, about 1300 km north of his home in Portola Valley, Calif. But, at the urging of a former Apple co-worker who was at Amazon, Tesler went to Seattle to meet with several people, and he was hooked.

"When I got there, it was very clear that Amazon is a technology company that happens to be a retailer," he says. And while they couldn't do much more than hint about it, Tesler felt that Amazon was getting ready to do something interesting in user interface design.

He joined Amazon in October 2001 as an engineering vice president and spent his first few months getting his bearings. He then took on the position of vice president of shopping experience, focusing on the usability of Amazon's Web site. He contributed to the recent launch of the tool that allows customers to search within books, as well as other user interface developments that are still confidential.

He loved it, particularly, he says, "going home and realizing that tens of millions of people are going to be using what I just worked on."

It's a recurring theme with him: the sheer delight of knowing that people use his work. His happiest memories from the Stagecast years were of trips to classrooms where kids were using his software. He still gets a little thrill when friends mention how they found what they needed at Amazon.

This past spring Tesler left Amazon. His problem wasn't with the company but rather with having a job in Seattle and a wife and roots in Silicon Valley. In May he joined Yahoo. Will he have his hand in another interface revolution there?

Changing the way people do things is never easy. Says Daniels: "You have to have the ability to guess years ahead and the self-assurance that you are right."

Tesler, Daniels thinks, can do that. And so does Yahoo. Yahoo expects Tesler to spark innovations all over the company.

"There's never been a more exciting time for user interaction innovators," Tesler says. "What we create today may not be as seminal as what we created 30 years ago. But the challenges are greater, and the solutions will have immediate impact on a lot of people."

Besides, he says, "It should be fun."

Advertisement
Advertisement