Beyond Second Life: Philip Rosedale’s Gutsy Plan for a New Virtual-Reality Empire

High Fidelity’s decentralized architecture aims to power the next generation of virtual-reality worlds


Philip Rosedale, the Willy Wonka of virtual reality, is giving me a tour of his bustling office in San Francisco in August when his blue eyes sparkle with a better idea. “Let’s just go in-world instead,” he suggests. I follow him to a windowless back room. Waiting there for us are a large HDTV screen, a computer, a few cardboard boxes, and two small, black infrared light-emitting beacons that point down from the high corners.

We slip on our beveled, black HTC Vive headsets, and my eyes adjust to the virtual world. Instantly, I’m teleported to a large living room filled with playthings. A dart board hangs on the wall, a jukebox sits in the corner, a craps table stands beside me. Bows and arrows litter the floor. “My God, it’s a mess in here,” he says through my headset.

In the physical world, Rosedale is a graying 48-year-old in Converse sneakers. Here, Rosedale’s avatar is an almond-eyed woman with short dark hair and blue jeans. I follow Rosedale outside by pushing a button on a controller in my hand and feel a woozy disparity between the motion in-world and my actual stasis. “We’re going to fix that,” Rosedale reassures me.

Rosedale relishes the surreal possibilities of life inside VR. He hands me a garden gnome and suggests we play tetherball. He uses a sword to smack the ball in my direction and urges me to whack it back with the gnome. Tossing the sword on the ground, Rosedale shows me how I can “rez”—or create—my own objects to play with by selecting them from a menu. I use my controller to click something called the Floating Space Cantina, and a huge purple gazebo crashes from the sky onto the lawn before me. “Wow, that’s cool,” Rosedale says, marveling at the structure. “I guess someone just made that.”

This is a beta demonstration of High Fidelity, open source software created by Rosedale’s company of the same name that lets you build and deploy your own virtual world. Rosedale calls this “social VR.” Most VR experiences—such as games and films—are designed for a single person. Social VR is all about sharing moments with others. The concept is an evolution of his pioneering virtual world, Second Life, which he created in 2003.

Having seen, and monetized, the first Internet boom, Rosedale wants High Fidelity to do for VR what the World Wide Web did for the Internet: provide a new means through which people can stake out ground online. Instead of surfing to people’s Web pages, you’ll be teleporting to virtual worlds they’re running on their own computer servers. To get from one place to the next, rather than clicking on a hyperlink, people will simply click on a 3D object or “portal.” Coded using that favorite of Web programmers, JavaScript, it will connect them to another’s server, allowing them to open the door of someone’s castle or step into the hut of someone’s tropical island. “What we believe is going to happen is the Internet all over again,” he declares.

This master plan is similar to the one that inspired Second Life, which supported 1.1 million active users per month at its peak. But despite the hype, Second Life never reached mass adoption. Achieving full Internet-scale VR, Rosedale has realized, comes down to the servers.

Second Life had at least 10,000 servers around the country all run by his former company, Linden Lab. As Second Life’s popularity grew, his employees became bogged down with maintenance and capacity issues. The question for High Fidelity became how to break that model and put the servers out in the wild. By distributing High Fidelity online for free, Rosedale wants to foster a do-it-yourself, interconnected community that transforms our virtual lives. “We are closer than probably people think to having an Internet-scale set of servers that present interconnected personal spaces,” he says.

Rosedale has a knack for painting utopian visions of the future. But to reach the masses, High Fidelity will need to develop far beyond quirky gnome-and-gazebo demos. His team must create sophisticated synchronization software that is also easy to use, writing code that lets people transform their laptops into VR servers while also handling the job of instantly coordinating people’s actions across worlds. Ultimately, these systems must be able to scale to handle the millions whom he hopes will join him in his funky alternative universe.

Weaned on science fiction and “Star Trek,” Rosedale had an early fascination with the potential of virtual worlds and tried to build his own head-mounted display in his teens. Studying physics at the University of California, San Diego, he devoted himself to solving the software challenges of VR instead. The video compression technology he coded caught the attention of RealNetworks, an early streaming company in Seattle, which bought his wares and made him its chief technology officer. But it was seeing the simulated future presented in the seminal sci-fi film The Matrix, in 1999, and reading the 1992 science fiction novel Snow Crash (which popularized the use of the term “avatar” to mean an online stand-in), that inspired Rosedale to leave Real, move to San Francisco, and make his own virtual world. “I was obsessed,” he says.

In 2003, his company, Linden Lab, launched Second Life as the Net’s first free-form virtual reality community. People could create their own avatars and online homes. Fueled by media buzz, Second Life attracted individuals, corporations, and governments (even the IEEE invested in a Second Life island). Users spent over US $500 million annually worth of virtual cash (in lindens), and developed their own political systems, newspapers, and in-world design firms. It was like virtual reality’s pioneer town—predating the kind of activity we’d later see with large community games like Minecraft or World of Warcraft.

By 2009, however, Rosedale realized the pressure of running a $100 million company with about 200 employees was distracting him from his lifelong goal of making a truly immersive VR experience, so he resigned as CEO (though he remains a shareholder). He had seen Second Life run up against its natural limits in employee and server capacity and was eager to obliterate those limits. “How do we do the kind of thing that Second Life did so well for a million people with mice and keyboards but scale that up to a billion?” Rosedale recalls asking himself.

Part of the answer, it turns out, was to wait for virtual reality to go mainstream. Serious research into virtual reality dates back to the late 1960s, but for decades the goggles were unwieldy, the graphics too unconvincing, and the latency—the lag between making a movement and getting visual feedback on that movement—was too stomach-churning to reach the masses. But improvements in device design and new software for producing 3D objects and environments have made it a far more compelling technology. As gaming, Hollywood, the military, theme parks, and others race to cash in, Goldman Sachs predicts virtual and augmented reality (the latter differs in that it displays a layer of computer-generated graphics over a view of the real world) will become an $80 billion industry by 2025.

The release of the Oculus Rift headset, owned by Facebook, last March and HTC’s Vive in April, allowed Rosedale’s team to give up on building their own headset and focus on developing the best possible version of High Fidelity. Finally, the right hardware is slowly catching on: According to SuperData Research, in New York City, Oculus will sell 355,088 Rifts this year, and HTC will sell 420,108 Vives.

The Rift was created by prodigy programmer Palmer Luckey, who’d been working on it since 2009, the year Rosedale left Second Life [See “Oculus Rift Takes Virtual Reality Mainstream,” IEEE Spectrum, December 2013]. A few days before the Rift’s release, I visited Luckey at Oculus’s office on the Facebook campus, in Menlo Park, Calif., to hear his thoughts on social VR. There, the 23-year-old showed me a two-player VR demo called Toybox.

For now, Oculus is focused on hardware and drivers to support gaming and other forms of entertainment. But there’s a reason why Facebook acquired the company in 2014 for $2 billion. Mark Zuckerberg saw Oculus as the means through which his company could create, as he put it, “a new communication platform” based in virtual reality. With Facebook’s support, Luckey thinks VR can become at least as popular as any social platform today. “I’m trying to convince people that virtual reality is going to be the thing that, in the long run, hundreds of millions and billions of people use,” he says.

At Facebook’s headquarters, Luckey and I wore large black Rift headsets tightly strapped to our faces. In each hand, we held a small Touch motion controller that was hooked up to the Rift. In contrast to the humanlike avatars of High Fidelity, each of us was represented by a featureless, blue, disembodied head and hands.

As I turned my head, I looked around a cavernous playroom. Assorted toys—blocks, robots, tanks, Ping-Pong paddles, and balls—were spread out on a table before us. As I clutched the controllers and moved my hands, two motion sensors across the room traced the controllers’ path through the air, which let the system transpose my movements in the virtual world. The click of a button allowed me to select an object, which I could then manipulate with a twist of the hand.

“Grab those lighters over there,” he told me, as he dropped dozens of cherry-red bottle rockets on the large gray table. I reached for two silver Zippos, and flicked open the caps to strike a flame. “Okay,” Luckey went on, holding up an M-80 firecracker and tossing it into a pile, “when I tell you, start lighting.”

As Luckey flung the last M-80 into the pile, he began his countdown. “Three, two, one,” he said, “Okay, light it up!” I brought my flame to the fuses, and when everything exploded I flinched. Despite my better judgment, my brain was duped. I didn’t feel like I was there. I was there.

Forty miles away, in San Francisco, Rosedale is betting his software is best positioned to make social VR a reality. And he’s persuaded others of this too. In 2015, High Fidelity secured an $11 million investment from Microsoft cofounder Paul Allen. In December, Rosedale said the company has raised an additional $22 million from investors, and a total of $37 million to date.

Jaron Lanier, the pioneer who popularized the term virtual reality, has tried High Fidelity and hopes it can deliver the kind of shared experiences he’d always imagined. “I like the idea of networked virtual worlds a lot,” he says. “I’d love to see one take off in a huge way.”

Back in the High Fidelity office, which has the kind of geeky funhouse vibe indigenous to a San Francisco startup, a team of 25 engineers, including several recruits from Linden Lab, have been working like elves to bring Rosedale’s dream to life. Scruffy coders rattle at their computers as multicolored balloons from a birthday party nudge the ceiling. A telepresence robot—a wheelie machine with an iPad affixed on top—rolls around so that the chief technology officer in Seattle can tune in.

Since 2013, when the company was founded, this team has focused largely on creating the software that provides the technology’s essential structure: its distributed client-server system. This means that people can readily deploy it on their own computer servers, without having to rely on a hosting service.

Such a client-server system allows people to essentially put up anything they want—just as if they were running a Web server. High Fidelity allows people to transform their own computers into servers that can support up to 20 people at a time. To run the program, a computer needs access to a broadband connection with a speed of 10 megabits per second or higher. That’s well within reach of many people in the United States, where broadband networks currently provide an average connection speed of 15.2 Mb/s (some other countries, such as Norway and South Korea, do considerably better). Over time, Rosedale thinks Internet service providers everywhere will readily offer enough bandwidth to support a massive virtual world because customer demand will be so great.

All of the content exchanged on these servers can be user generated, built on JavaScript. Users can design basic objects through graphical interfaces, or create more advanced objects and environments by writing code. Programmers wanting to tinker with the underlying High Fidelity client/server platform can do so: The platform is open source, written in the popular programming language C++, and licensed under the Apache 2.0 open source software license.

For ease of use, the High Fidelity platform supports two popular file formats called FBX and OBJ, which let people open 3D-vector graphics that were originally created in different formats. The platform will also be compatible with emerging standards like the GL Transmission Format (also known as “the JPEG of 3D”) so people can quickly load 3D scenes and move models around in their virtual worlds.

Alternatively, instead of wading into 3D coding, nonprogrammers can simply download materials from a marketplace. People can even earn virtual cash by selling their own digital objects through the marketplace (High Fidelity will not take a cut of these sales). This leaves High Fidelity, the company, to make money by providing services like domain name translation, user identification, and possibly advertising to all these personal worlds.

Back in-world, Rosedale shows me how someone can drag and drop objects from the marketplace into the world. In his virtual backyard, he calls up another menu which connects us to, a library of free 3D models. All I have to do is click on, say, a cactus, a conga, or a steam locomotive and drop it into my world, and it’s ready for use. For now, the marketplace is focused on digital objects—such as furniture and toys—but it could include textures and color palettes that people can use to build their own scenery at some point.

A key challenge here is keeping the appearance and behavior of objects consistent and synchronized among all the users visiting a given world. If a high school art teacher holds a distance learning class inside High Fidelity, students need to be able to smoothly hand each other their virtual sculptures.

To facilitate this, Rosedale’s team wrote their own “synchronization engine” software. It relies on an “entity server”—a database that maintains, tracks, and manages all virtual objects, or entities. Once a person has connected, the entity server performs lookups to find out what a client is able to see based on the person’s position and viewing direction. Then it sends information back to the client about which entities are in view and describes any changes as they occur. For example, if someone fires a gun, Rosedale says, the client will send a message to the entity server to request that it create a “bullet” entity at the client’s location and move it along a specific path through the air.

This is something Rosedale’s crew first had to tackle in Second Life but now aims to get working with a latency of under 100 milliseconds, so that the action feels smooth and cohesive. Rosedale admits High Fidelity can’t guarantee such low latency across all the worlds added to its online universe, but he expects that the builders of the most popular virtual locations will use high-performance servers and network connections to minimize delays. But, as in the early days of the Internet, the results may not always be perfect. “If one server is slower, information from that space might show up more slowly,” Rosedale says.

In addition to the delivery issues, High Fidelity’s software must also cope with a slew of demands inherent in any virtual reality within its distributed architecture. One of the biggest challenges, Rosedale says, is creating three-dimensional audio so that noises and voices in-world sound as though they are actually coming from the places where objects and people appear. To create an illusion of space—such as the sound of a ball dropping far across a room—High Fidelity employs its own patented process.

It starts with a database of thousands of standard WAV files, including, for example, the sounds of dropped balls of various sizes and shapes. An audio-mixing server measures the distance and angle of each sound relative to a person’s position inside the virtual world, and then mixes it, adjusting the frequency and delay for the desired effect before sending it out to the client.

By opting to release the platform under an open source license, Rosedale let developers outside the company help engineer solutions that work for everyone. And he’s not worried about competitors. “If we’re successful and I’m right,” he says, “it doesn’t matter that it’s open source because everybody’s going to want to deploy a standard system because they want to interconnect to each other.”

After using my garden gnome to play a xylophone in Rosedale’s virtual backyard, I wander toward a side wall, where a whiteboard hangs on the bricks. Rosedale shows me how I can pick up one of the virtual markers and draw a smiley face. “This is what I like best,” he tells me, through my headset, “just seeing people drawing on here.”

This seems like a far cry from the guy who spent the previous decade ruling the minions of Second Life. But while Rosedale is engineering a democratic social future of VR, he’s no less bombastic. By his calculations, if he can get his software on each of the billion Internet-connected machines deployed in the last four years, the virtual world—in sheer landmass—would exceed that of our actual world. “You’d have a space the size of Earth,” he says, eyes widening, “so talk about high fidelity. I mean, we’re going to go beyond planet Earth!”

But even though High Fidelity software is free, people will still need to buy their own headsets to join Rosedale’s dream world and find enough free time to dillydally around inside. A $600 Rift, for example, also requires a high-end PC with a central processing unit equivalent to an Intel Core i5-4590 processor and a powerful graphics card. Such a PC costs $800 or more.

If we join him, though, Rosedale’s convinced that we’ll never look at our old planet the same way again. He thinks we’ll spend most of our time immersed in VR, and drop back into reality only when we want to do things we simply can’t do in-world. “When you think about the Earth in 20 years,” he says, “you’re going to think of it as a museum.”

And, he goes on, the days of me coming out here to do this interview are numbered. “You won’t have to drive here. That’s nuts,” he says. He motions to the screen—“You’ll just be able to sit with me in there.”

About the Author

David Kushner, a Spectrum contributing editor, first experienced virtual reality in 1991, shooting arrows at boxy figures in the early virtual-reality game Dactyl Nightmare. When Kushner tries a VR demo, he isn’t too picky about the graphics, as long as it draws him in: “You just want to put on the glasses and be somewhere else.”