Robots, Incorporated

Microsoft’s best and brightest are quietly trying to bring robotics into the mainstream

14 min read
Robots, Incorporated
Photo: Brian Smale

Software pundits and tech analysts can be forgiven for overlooking Microsoft’s new robotics group. Compared with the company’s billion-dollar businesses—Windows, MSN, Xbox, and more—robotics is nonexistent. Microsoft is giving the group’s software away for free for noncommercial use. In other ways, robotics is merely minuscule. And indeed, the company is hardly betting the farm on it, having devoted only 11 of its 76 000 employees to creating Robotics Studio 1.0.

Yet this tiny group of elite software engineers, housed in a small set of open offices known as the “Broom Closet,” handpicked by a 26-year company veteran who has the ear of Bill Gates, and tucked into a tiny corner of the company’s research budget, has put together a set of tools that may bring robot manufacturers under one roof, the way Windows did for most PC makers. Indeed, future versions may someday find their way into more machines than Windows did—and be just as lucrative. Microsoft’s eventual plan is to charge users US $399 to license up to 200 copies of the software components that go into a commercial robot.

Right now, the robotics world is rife with devices that don’t easily work together or with standard programming tools. Take the Create, a generalized, programmable version of the popular Roomba vacuum-cleaning robot. The Roomba’s maker, iRobot Corp. of Burlington, Mass., stripped out the vacuuming-specific parts and put in a cargo bay, a serial cable, 32 different sensors, and a 25-pin expansion port. At $130, it’s a budding roboticist’s dream. But to program it, you have to write in C or C++. If you want to add a webcam or a robotic arm from another manufacturer, you have to write more code—first for the accessory and then for integrating it into the robot. If you later swap out the new unit for a better one from a different vendor, you have to invent that wheel all over again.

Good robotics programming is far harder than writing a typical application for personal computers. Each component is expected to act autonomously and react to complicated events in the world of a kind that a printer or mouse never has to deal with.

Robotics Studio, released in December, aims to handle much of that complexity for robot programmers. It isn’t an operating system. But manufacturers will use it to write software for their robotic components much as a maker of a device that hooks up to a PC does, whether it’s a printer, an LCD display, or a data-acquisition sensor. Once such a service is written—telling, for example, a robotic arm to move up or down, grip or release, rotate n degrees, and so on—the action can be done with a single instruction. And when you substitute a new arm, the same commands work in the same way, so a minimum of reprogramming is needed. Microsoft’s software, in other words, will do what MS-DOS and then Windows did: nurture an ecosystem in which new devices spawn new programs for more and more end users who in turn inspire yet more innovation—the same virtuous cycle that brought explosive growth to the cottage PC industry 25 years ago.

Whether that cycle will develop remains to be seen, but there are signs it may have already begun. The tool kit has been downloaded more than 100 000 times since its December release. An enhanced version, previewed in April, will be used this fall in computer-science and engineering classes at Georgia Tech, Carnegie Mellon, and other schools. And it’s already being tested by a variety of manufacturers, from makers of the tiny iRobot units to Kuka Robot Group, in Augsburg, Germany, which in May released the first robot able to lift 1000 kilograms. Even though the software is free for many, managers at Microsoft say they’re confident that once it’s in millions of machines, moneymaking businesses will emerge. The company’s free media player, for example, was the seed from which its Internet-based television software—with customers such as AT&T and Verizon Communications—sprouted.

Today’s $11 billion robot sector—mostly industrial robots—will double by 2010, according to estimates by the Japan Robot Association, and it should exceed $66 billion by 2025. Most of the growth will be in nonindustrial applications—especially, analysts say, in areas such as toys, transportation, and health and senior care. Imagine a robot helping a recovering heart-attack patient get some exercise by walking her down a hospital corridor, carrying her intravenous medicine bag, monitoring her heartbeat and other vital signs, and supporting her weight if she weakens.

The International Federation of Robotics predicts that 5.6 million robots for domestic, entertainment, and leisure applications will be sold from 2006 to 2009, and right now the field is wide open. Microsoft’s competitors include Player, an open-source project partially funded by the U.S. National Science Foundation, DARPA, and various artificial intelligence labs; Gostai, in Paris, a small maker of open-source robotics software; and Evolution Robotics, based in Pasadena, Calif., and Tokyo. None has anything near the vast resources of a Microsoft. It’s no wonder the 10 people who wrote Robotics Studio 1.0 say they believe they’re the pioneers of the next big thing, not just for their company, but for the world.

At Microsoft, even the lowliest new programmers have their own offices, and buildings on the company’s sprawling Redmond, Wash., campus consist largely of corridor after corridor of individual offices, each with a large window equipped with blinds for privacy. Except, that is, for one particular corner of the third floor in Building 113. There you’ll find a large open space—the Broom Closet—with a couch, easy chairs, a coffee table, a giant LCD television, and more robots, robot accessories, and robotic toys than you thought existed.

There are iRobot’s Roomba and Create robots, of course. Attached to another robot are a radio and antenna, small stereo speakers, and some kind of sensor attachment that looks for all the world like a small coffeemaker. There’s also the Traxster, a robot made by Summerour Robotics Corp. of Atlanta, which has wheels that run in tracks, as on a tank, and optionally includes vision sensors connected to an articulated neck. There are joysticks, keyboards, and remotes of various kinds; a low, circular, wheeled robot that looks like a cross between the Roomba and a blue ladybug; several Lego Mindstorms robots; a black-and-white spaceman robot that looks handmade; and a green-and-purple stuffed dinosaur. When I ask whether the dinosaur is a robot, Ioana Butoi, a 26-year-old Romanian software engineer on whose desk it sits, answers shyly, “No. But it could be.”

Setting up shop in what was assumed to be a utility closet was the idea of Tandy Trower, manager of the group and its sole office dweller. “I wanted a small group that spent time together,” he says. “Good things happen in small groups of people who talk to one another a lot.”

In late 2005, Trower cherry-picked its members from every area, including two engineers from the first team to work on the Xbox, another project in which Microsoft tried to do something completely different. Today, the Xbox is the heart of the company’s $4 billion entertainment division.

The idea for the robotics group came from several different sources. The first was Craig Mundie, the company’s chief research and strategy officer. Back in 2000, he took a broad look at trends in computing and the Internet. What he foresaw was a “sea-change of increasing complexity. There would be processors everywhere,” he says. Computation would be distributed across different processors in a single chip, a single device, or across a network, local or otherwise. Processors would be loosely coupled; that is, they would come and go at will. And computing was moving to a services-based model, meaning that software would increasingly be written for a network cloud—a company’s network or the Internet itself—instead of individual computers.

That’s just the opposite of what happens on a PC, where a single processor is in charge; peripherals ask to be connected and disconnected and in between have to request some of the processor’s valuable attention. We’re all familiar with a print job or a Web browser timing out because the printer or a remote Web site doesn’t respond quickly enough. Now imagine the timing and attention problems a robot will have—the feet want more information from the eyes before deciding where to step, while the eyes can provide that information only when the next step is taken. Or there might arise two unrelated but equally critical tasks, such as walking beside a hospital patient and simultaneously regulating the flow of her intravenous medications.

In programmer-speak, each of those tasks is a thread. A conventional program can run only a single thread to completion or put it on hold while another thread runs. In a computer with multiple processors, or a single processor with multiple cores, more than one thread can run simultaneously, each taking in a stream of data from a set of sensors and responding to the data in some way. But there’s still the problem of coordinating the two threads and the responses. A thread managing the hospital patient’s heart rate might tell the thread managing the IV drip to stop one of the medications from flowing.

To work on such problems, Mundie put together a team of researchers, known as the Advanced Technology Incubation Group. They came up with something called a concurrency and coordination runtime (CCR). The CCR hides the complexity of managing multiple threads simultaneously by letting programmers create a software object called a dispatcher, which can manage multiple threads (typically one for every processor in the computer) and assign scheduling priorities for each one. The CCR even lets a programmer create multiple dispatchers, which are managed through a class of objects called arbiters. Other tools in the CCR let threads share data or claim it exclusively, pass data from one thread to another, and let one thread command another to do something.

In Mundie’s picture of computing’s future , processing and information aren’t just distributed among the components of a system; they’re strewn throughout its environment. Consider the Roomba, iRobot’s lowly $119 vacuum-cleaning robot. Today, it employs several strategies to navigate a room in ever-widening circles as it cleans. But most things in the room haven’t moved since it vacuumed yesterday, and some things—such as the walls—don’t move at all. In the distributed, services-oriented model, some other computer in the household could be a repository of information about the location of walls and furniture and electrical outlets. Any robot moving about the room could draw from that data and update it when, say, a chair gets shifted from one place to another. Access to information about the layout of a house is a service that would be available to every robot. Access to electricity is another.

A small army of Roombas, communicating with one another directly or through a household server, could quickly vacuum an entire house. Eventually, the tables and chairs themselves would be smart enough to report their new locations when they get moved. And new robots coming into the household would quickly acquire whatever information they need, just as servants do when one royal family visits another. In Robotics Studio, such services, whether they reside in another component within the same robot, in a local computer, or across the Internet, show up whenever they’re available. Microsoft calls this DSS, for decentralized software services.

CCR and DSS are the two key technologies developed by the Advanced Technology Incubation Group to have ended up in Robotics Studio. Although the main burden of making sure that a robot doesn’t get bogged down doing one thing, to the exclusion of other vital tasks, belongs to the operating system, and so-called real-time operating systems (RTOSs) have been around for decades now, CCR and DSS ensure that the benefits of an RTOS don’t get lost at higher levels of programming.

The Studio software hides complexity from programmers in some other ways as well. Services in a program can be displayed as simple block icons, and by connecting them with arrows, programmers create relations between them—such as sending data from one to another. When Trower demonstrates this process on the big television screen, it seems almost too easy. (Programming something that could guide a heart patient down a hospital corridor would be considerably more complicated.) The software tool kit also contains tutorials, sample programs, and generic robotics code, as well as a simulation tool that lets you test your program without having to risk sending an expensive robot down a flight of stairs headfirst.

Robotics Studio is written for Windows, but it doesn’t follow that the robot itself is using any form of that operating system. Of course Microsoft would like robot makers to use Windows. But many of the robots in the Broom Closet don’t even use a formal operating system.

When Mundie’s group wrote a base of code for multiprocessor programming, they didn’t have robotics, or anything else, specifically in mind. Mundie simply had assigned several architects to write some general software for concurrent, distributed computing. So he needed someone to figure out what applications could take advantage of the new software. Trower was given that job. He looked for tasks that were highly distributed, requiring a great deal of local autonomy, and used lots of computation in real time. He would eventually find the field of robotics, but not right away.

He first considered biologically inspired technologies, such as neural networks and genetic algorithms, “because they’re inherently built on the idea of distributed processing and concurrency,” he says. Mundie’s ideas were applicable to it, he says, but biotech was ”too immature” for anything that Microsoft could add value to.

After an entire year of searching, Trower was reassigned to Bill Gates’s staff. “I went to work for Bill as an extra set of strategic eyes and ears,” he says. “I kind of stumbled upon people from the robotics community who were knocking on our door.” For example, at a meeting with Microsoft CEO Steve Ballmer, the head of Lego, maker of the robot toy Mindstorms, told Ballmer, “We ought to do something together.” There was also a visit to the Microsoft campus by Red Whittaker, whose Carnegie Mellon team would soon do better than any other in the first DARPA Grand Challenge, a competition for robotic vehicles that required them to undertake an arduous trek through the desert.

Unknown to Trower, Whittaker was far from the only university professor telling Microsoft about the importance of robotics. The company has an entire group, now known as ER&P, for external research and programs, that, among other things, hires hundreds of computer-science students as interns, funds faculty and graduate student research, and brings several hundred professors from around the world to Redmond every summer for a Faculty Summit, honoring five as New Faculty Fellows, and giving them $200 000 each to spend on their research. Through these various relations, Stewart Tansley, program manager in ER&P, got a clear message from computer-science professors: we want to use robots in our classrooms, but it’s too hard. [See photo, "Straight From the Shoulder."]

“The idea of robots, of giving machines the powers of humans, is very powerful in our culture,” Tansley says, “going all the way back to the Greeks. We kept hearing that robotics research was popular but challenging. Students wanted to program robots, but they were spending all their time on fundamental engineering. There was a lot of reinvention of the wheel.”

Microsoft, Tansley says, is the software equivalent of a plumbing company. “The notion of doing the fundamental plumbing for robotics seemed like a good idea.” So in December 2003, he set about using a uniquely Microsoft invention to get the company focused on robotics. With a colleague, he started to write a paper on the topic for Think Week, Gates’s semiannual retreat during which he reads a hundred or more papers from employees at every level of the company. “While writing it,” Tansley recalls, “we came across Tandy Trower.” Trower decided to write his own Think Week paper about robotics.

That winter, Gates would hear about robotics directly. A few months after reading Trower’s and Tansley’s Think Week papers, he visited six schools on a university tour, another of his semiregular brainstorming exercises. By his account, at every stop students and faculty were excited to show him at least one robotics project.

The message to Gates was clear: go anywhere in the world, from Germany to Korea, and there’s an excitement, an anticipation that something is happening with robots. They’re a powerful attractor for students and everyone else. And concurrent, distributed programming on multicore multiprocessors was the new, disruptive technology that was going to take robots out of their largely industrial settings and put them everywhere.

Once Gates decided to involve Microsoft in robotics, the next step was to figure out how. Should Microsoft write an operating system specifically for robots? What other resources existed within the company that could help? Even in late 2003, Microsoft had a set of programming tools for Web services, called .Net; a small, efficient version of Windows, called Windows CE, which today can be found embedded in everything from ATMs and cellphones to gas pumps; language products designed for Web-style programming, such as C#; and Mundie’s codebase, including CCR and DSS. Gates sent Trower on an open-ended mission to figure out just what Microsoft should do, and who within the company should do it. After a five-month study covering everything from Lego’s Mindstorms to the latest industrial robots, Trower reported back to Gates: “I told him I thought there was a business here for Microsoft and that I might want to run it myself.”

Trower’s modest title—general manager, Microsoft Robotics Group—accurately reflects the genial 26-year veteran’s self-effacing ways but hides both his influence at Microsoft and a résumé tailor-made for running such a venture. He arrived in 1981 to manage the company’s BASIC language products, already its biggest business and one that would develop into a division that sold compilers for C, Fortran, and Pascal, as well as BASIC. Next, he managed the first two releases of Windows and, even before that, developed programs for it, including Microsoft’s famous flight simulator. He founded the company’s first usability lab. Eventually, Trower became a minister-without-portfolio reporting directly to Gates.

Besides giving the robotics group its manager, Gates can be credited with finding an ideal niche within Microsoft for it. He told Trower to form a regular business unit with a staff, a budget, and its own quarters—the Broom Closet—in a building that also housed other research groups. Version 1.0 of Robotics Studio would have a release date, just like any commercial product, and was to be updated and maintained like any other release. But Gates also located the group organizationally within Microsoft’s research division, freeing Trower from the need to produce revenue. There would be no product managers telling him who his customers were or what features they needed. In place of market research, Trower was to rely on his five-month study, his years at Microsoft, and his knowledge of what programmers need and users want.

By maneuvering the new group into a gray space between Microsoft’s business and research wings, Gates created a start-up right in the middle of the vast Redmond campus, a skunkworks that had the boss’s blessing. Soon after, Mundie gave Trower the fruits of the Advanced Technology Incubation Group’s research, CCR and DSS, as well as the two programmers who were their principal architects, George Chrysanthakopoulos and Henrik Frystyk Nielsen. By then, Trower had chosen several of the group’s eight other software engineers.

The resulting team is as eclectic as the special-forces crew in the World War II movie The Dirty Dozen. Just as Lee Marvin’s Major Reisman had a mix of sharpshooters, demolition experts, and so on, Trower needed specialists for everything from operating-system-level programming to user interfaces. [See photo, " ." Since the photo was taken, a Korean software engineer, Young Joon Kim, has joined the group, making it an even dozen after all.] No three members were born in the same country or were from the same part of the company.

Chrysanthakopoulos, a wiry, entertaining Athens-born electrical engineer who doesn’t stop talking until someone tells him to (a task other team members don’t shy away from, to everyone’s merriment), wrote much of the CCR and became the robotics team’s technical lead. Prior to joining Mundie’s group, he wrote software for the Xbox, as did another denizen of the Broom Closet, Kyle Johns. Nielsen, a Dane, became the group’s program manager. Along with DSS, he wrote an associated Web-specific protocol, DSSP. Nielsen is an old hand at such things: for his master’s thesis in electrical engineering, he had helped World Wide Web inventor Tim Berners-Lee write the Web’s fundamental protocol, HTTP.

The two youngest team members, Pavel Khijniak, a Russian-born 24-year-old who works on user interfaces, and Butoi, the shy Romanian, came from the Windows group. Two other members were managers elsewhere in the company before joining the robotics team. Joseph Fernando was a development manager in the digital media division. A gentle accent is the only sign of his Sri Lankan origins. Johns, one of the two U.S.-born members, was a lead programmer working on graphics performance tools for the Xbox. “They came here as individual contributors,” Trower says, “because they were very excited and passionate about working in this particular area.”

Indeed, if there’s anything that unifies this motley crew, it’s a love of robots. For example, even before the robotics group was formed, David Lee, who grew up in Utah, worked on an entry for the second DARPA Grand Challenge with his brother and a friend, both of whom are electrical engineers. He was able to write all their vehicle’s software with Microsoft’s .Net programming package, much to his surprise. (He says their entry failed to make the second-round cut, largely because of the vehicle’s poor turning radius.)

When I visited the team in March, several members were working on a sample entrant for a robot competition that the group was sponsoring at an upcoming conference on embedded systems. A strip of duct tape ran down the middle of a sheet of plywood, taking up much of the Broom Closet’s limited floor space. The idea was to test the ability of infrared sensors to read surfaces of various textures and degrees of reflectivity. All the contestants were to use an iRobot Create chassis; an embedded computer made by ICOP Technology of El Monte, Calif.; and, of course, Microsoft Robotics Studio 1.5.

To look at the team in this robotics playground, it seems to be all fun and games. But there’s also an air of seriousness. Trower’s recruits know this is a unique opportunity to help Microsoft change direction, a task often compared to turning an aircraft carrier.

Team members seem to be unaware of how much attention the company’s top management is paying to the Broom Closet. Several mentioned that at their previous positions at Microsoft, they were 10 or 11 rungs below “Bill,” without knowing that they are now only one hop away—via Trower—from Gates, who, though he is no longer CEO or chief strategist, can still make or break a project with a single word.

Are ubiquitous robots, dreamed of for millennia, in our immediate future, or are they still a number of years over the horizon? At least the world Mundie imagined seven years ago is here, with data centers filled with multiprocessor servers and desktops everywhere sporting multicore personal computers for less than $2000. We’re about to see whether the other half of his and Trower’s and Gates’s vision is correct. Will the new processors lead us away from PCs and toward a future filled with robots—robots running Microsoft’s software?

To Probe Further

At https://channel9.msdn.com/wiki/default.aspx/Channel9.MSRoboticsStudio, Microsoft has technical information and a community forum for Robotics Studio developers, as well as a link to the software’s home page.

The Player Project software is available at SourceForge: https://playerstage.sourceforge.net. Gostai, another provider of a universal programming tool for robotics, is at https://www.gostai.com/urbi.html. Evolution Robotics is at https://www.evolution.com.

This article is for IEEE members only. Join IEEE to access our full archive.

Join the world’s largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum’s articles, podcasts, and special reports. Learn more →

If you're already an IEEE member, please sign in to continue reading.

Membership includes:

  • Get unlimited access to IEEE Spectrum content
  • Follow your favorite topics to create a personalized feed of IEEE Spectrum content
  • Save Spectrum articles to read later
  • Network with other technology professionals
  • Establish a professional profile
  • Create a group to share and collaborate on projects
  • Discover IEEE events and activities
  • Join and participate in discussions