Dream Jobs 2011: Meet Weta Digital’s Master of Face Animation

This profile is part of IEEE Spectrum’s Special Report on Dream Jobs 2011.

Mark Sagar is taking a break from making faces for major motion pictures. It’s a sunny Tuesday in early summer, and he’s whipping up a couple of cappuccinos in a lounge at Weta Digital, the hot-as-the-sun special-effects house behind such blockbusters as Avatar, The Lord of the Rings trilogy, and the 2005 remake of King Kong. Based in a funky suburb of Wellington, New Zealand, the studio’s decor is all frowsy chic: This lounge we’re in houses a life-size stuffed gorilla, a pool table, and several racks of movie costumes. There’s a gourmet kitchen whose counters are heaped with delectables. The walls are covered with movie posters and brass dinosaurs (not life-size).

During my visit, Weta’s wizards are conjuring the dazzle for the highly anticipated DreamWorks/Steven Spielberg movie Tintin: The Secret of the Unicorn. But as a harried Weta publicist informs me weeks after I conclude my interviews, the effects house wants nothing—nothing—said about Tintin. Or else I’ll never eat lunch in that town again.

For now, though, breakfast is served. Sagar sprinkles cinnamon and marvels at the existence of a movieland and supercomputing powerhouse on this island, where all the other noteworthy industries involve sheep or fast sailboats. “What’s the chance of a high-tech film company existing on an island in the South Pacific?” Sagar wonders.

Left out of this small talk, somehow, is the fact that just moments ago he learned that he and three colleagues won an Academy Award. More on that later.

Sagar is as comfortable with art and aesthetics as he is with code and computers. The art/tech tug-of-war began in early childhood, in Kenya, where Sagar’s father was a systems analyst and programmer for the East African Railways. At home, his father would build radios and disassemble televisions, firing the youngster’s imagination with the wonders of modern electronics. His mother, a painter, took young Sagar to game parks, where she would point out the animals and sketch them for him. “From the beginning, my influences were half technical and scientific and half artistic,” he says.

He got a B.S. degree in physics and math in 1988 at the University of Auckland, in New Zealand, where the family had moved when Sagar was 4. He thought about grad studies in physics, but decided instead to “travel around the world painting and being a beach bum. I remember asking my mother for a quickie course in painting before I left.”

It came in handy. Over the next four years, he supported himself with odd jobs, bartending, and painting portraits of tourists in such places as China, Britain, and Nepal. “It was my first inkling that I could do something artistic for a living.”

Returning to New Zealand in 1992, he enrolled again at Auckland, in a mechanical engineering graduate program. For his master’s, he built a 3-D computer model of the human eye, for a system being developed to train doctors on surgical robots. That led to a Ph.D. project in which he wrote the software to let people build biologically accurate computer models of complex human anatomy.

Then came a postdoc at MIT where, in 1996, some Hollywood types came calling, looking for smart techies to work on the technology to make virtual (computer-synthesized) actors. By this time Sagar had: 1) gotten married, to a woman named Justine he’d met during grad school; and 2) put together a computer model of the human face. He and a colleague, Paul Charette, who had built a computer vision and tracking system, combined their work into something that was basically a forerunner of a modern motion-capture system: Charette’s optical arrangement tracked dots on a person’s face, and Sagar’s software connected the dots into a computer-generated face that could be manipulated by the computer.

“We ended up making what was at that time the most realistic computer face that had been done,” Sagar recalls. It was based on Justine’s face, and what this little animation did was ask, “Am I real or am I digital?”

It wasn’t much by today’s standards, but it was impressive enough then to be their ticket to Hollywood. So fast-forward a few years: It’s 1998, and Sagar is living in Santa Monica, wearing cool sunglasses and attending meetings that sometimes include the scurrying minions of such Hollywood overlords as Steven Spielberg. He and Charette have helped launch a special-effects company called Pacific Title/Mirage. They’re working on a Jim Carrey remake of The Incredible Mr. Limpet, for Warner Brothers. Their job is to create the title character, which has the body of a fish and Jim Carrey’s face.

Limpet never got made, a victim of the usual problems that doom big-budget movie projects: funding, contracts, and the actors’ competing schedules. Sagar and Charette pursued other opportunities for their face software. In 2000, as the dot-com rapture was climaxing, they started a software firm whose chief aim was to develop a program that made a realistic little digital head pop up that used text-to-speech software to read your e-mail to you. Just how daft did the dot-com mass delusion become? “The company had 50 employees, and at one time it was valued at around half a billion dollars, without a product on the market,” Sagar says. Needless to say, the company did not survive the subsequent bust.

Fortunately, Sagar had other irons in the fire. A year earlier, he and Charette had used their face technology to do an animated short, The Jester, that was a hit at the 1999 Siggraph conference. There, Sagar met Paul Debevec, then at Stanford University and well-known for his ground-breaking work on computer-generated lighting. “I said, how about we combine your lighting with our faces?” Sagar recalls. It turned out to be a splendid idea.

Pay attention now, because this is how Sagar won his Academy Award. He’d been trying to render realistic skin for years, so he was able to quickly introduce Debevec to all the difficulties in generating skin that looks convincing. This introduction helped Debevec develop a system to illuminate an actor’s face from dozens of angles, capture the image data, and use that data to synthesize how the face would reflect light under any lighting conditions.

The system, known as Light Stage, allows movie-effects specialists to take the image of an actor’s face and put it into any scene with vivid realism, regardless of the direction and color of the scene’s ambient lighting. Those scenes in Spider-Man 2 where Dr. Octopus and Peter Parker battle in the air and on the sides of trains? That’s Light Stage’s motion-picture debut. The system went on to major roles in Superman Returns, The Curious Case of Benjamin Button, and Avatar. For creating Light Stage, Sagar and Debevec, along with Tim Hawkins and John Monos, were honored last year with a Scientific and Technical Academy Award. “It was definitely the glitziest event I have ever been to,” Sagar says.

By then, Sagar had moved on to Weta and back to New Zealand. Weta dominates the art of motion capture, and Sagar has specialized in— no surprise—applying it to faces. He designed and set up the facial motion-capture system used for King Kong and later Avatar, vastly increasing the subtlety of expression that could be captured from the actors and transferred to the synthesized creatures.

He can’t remember all the actors he’s met, but the short list includes Tobey Maguire, Andy Serkis (whose facial expressions became those of Kong in King Kong), Naomi Watts, Alfred Molina, Kathleen Turner, Sigourney Weaver, and Maggie Gyllenhaal. His work with many of them involved applying the motion-capture dots to their faces. Some of the actors like it, others are baffled by it. “Andy Serkis loves it,” Sagar reports. “He’s really gung ho about it.”

Some of his brushes with celebrity can best be described as surreal. One morning in 2004, he was getting a cup of coffee in a studio in Culver City, Calif. “There was a huge African elephant next to the coffee stand,” he says. “It started crapping….I turned around, and there is Nicole Kidman in this utterly beautiful dress.” He smiles. “From elephant crap to high glamour. It wasn’t even nine o’clock yet.”

Lately he has been working on a computer-based system that generates face images based on the underlying structure of the face: the skull, jaw, cartilage, muscles, tendons, and so on. His office is littered with anatomical journals and textbooks: Gray’s Anatomy, Facial Expression, Surgical Anatomy of the Face. There’s a human skull on his desk. The walls are papered with anatomical drawings of the musculature of the face. The idea is that once that structure is part of the computer model, a smile, say, will cause the entire face to do what it would in real life: dimples might appear, the upper cheeks bulge, the eyes crinkle.

It’s part of a trend in movie effects toward higher and higher levels of simulation and less “hand animation,” in which individual frames are created from scratch or heavily tweaked. “From a long-term perspective, it’ll definitely happen,” Sagar says of simulated faces. “Cloth, water, and particle effects are all now simulated,” he notes. “Years ago, they were all hand animated.”

The work fascinates him because it pushes the limits of human knowledge. “No one’s 100 percent sure which muscles are activated during speech,” he says. “The muscles and fibers around the mouth have extremely complex structure and cross over each other, and are linked to each other and to many other structures in the mouth, such as ligaments,” he explains. “We’re building this model and trying to work [the anatomy] out at the same time.”

“That’s the crazy thing,” he adds. “Visual effects are getting to the point where we’re doing research that’s comparable to actual medical research.”

And there are less highfalutin reasons why he loves what he does. “The great thing about faces is that when you have a [software] bug, you can get a crazy-looking face out of it. With Jim Carrey, one bug caused his eyelashes to come off, and they were floating around like seagulls, independent of his eyelids. It was quite funny, actually.”

Sagar sees his work at Weta as a natural culmination of his varied interests and experiences—his youthful vagabonding, the postdoc at MIT, and the crazy years in L.A.

“Life’s a little like you’re a pinball in a pinball machine,” he says. “You’ve got to enjoy the game.”

An abridged version of this article originally appeared in print as “Putting a Face on Fantasy.”