The Making of The Beatles: Rock Band

The stage was dark. The curtain, drawn. And the crowd, ready to rock. A hundred lucky fans perched on the edge of their seats waiting for the invitation-only show to begin. Suddenly, the drapes parted as the joyful opening riff of ”I Want to Hold Your Hand” filled the room. The spotlight shone on a drum kit bearing the iconic black-and-white logo of the Beatles.

But the performers twisting and shouting on stage were not four young men from Liverpool. Nor were they members of any of the many Beatles tribute bands. The group performing at the Los Angeles Convention Center this past June consisted of six scruffy young geeks. A long-haired coder belted out the verse. A tattooed woman and a cheery guy added the harmonies. A stocky Asian-American played what appeared to be a tinier version of Paul McCartney’s familiar Hofner bass. Another guy held a likeness of George Harrison’s Gretsch guitar. And a really enthusiastic player smacked the drums.

All were employees of Harmonix Music Systems, a video-game company in Cambridge, Mass., and they weren’t actually playing the song; they were demonstrating the most hotly anticipated new game of the year, The Beatles: Rock Band, which hits stores worldwide this month.

The title, to be available for the Microsoft Xbox 360, Sony PlayStation 3, and Nintendo Wii, allows gamers to perform along with the Fab Four by singing and playing instruments that work as video-game controllers. The goal is to stay in sync with the music. As a song plays, color-coded dots, representing the musical notes, cascade down the TV screen. The guitarist and bassist must press the corresponding colored buttons on their instruments, and the drummer has to hit the right drum pads. Vocalists must sing on pitch as the lyrics scroll across the top of the screen.

The game, created by Harmonix and published by MTV Games, features 45 career-spanning songs and pixelated Beatles characters performing at locations like the Cavern Club in Liverpool and Shea Stadium in New York City. Paul McCartney and Ringo Starr, as well as Yoko Ono and Olivia Harrison, the widows of John Lennon and George Harrison, helped Harmonix pick the songs and tweak the graphics.

But the game also stands out for all the music technology behind it. To create the separate instrument and vocal tracks the game needs, Harmonix engineers had to use special audio filters to extract every note from the master tapes. They also developed a pitch-evaluating system that can monitor three players singing harmonies—sun, sun, sun, here it comes—a key part of the Beatles’ music.

Last year, word of the game’s impending release generated Beatlemania-like buzz. This is the Beatles’ first step into the digital entertainment domain. Apple Corps (not to be confused with the Apple of Macintosh and iPod fame), which manages the rights to the band’s original recordings, has long been reluctant about selling songs through online stores. Now the video-game deal might convince Apple Corps that it needs to explore other distribution channels to keep the band relevant.

The new Beatles game is the third and most ambitious title in the Rock Band franchise, one of the most successful video games ever made. Since debuting in 2007, Rock Band and its sequel, Rock Band 2, have earned more than US $1 billion in North America alone. The series is an outgrowth of Guitar Hero, the pioneering music game, which has grossed over $2 billion. The genre has become a phenomenon of its own, garnering fans of diverse ages and backgrounds. There are Rock Band parties and tournaments, and gamers flock to online forums to share tips and discuss the latest peripherals. It’s karaoke for Generation Net.

”These games are all about tapping into the community,” says Stephen Prentice, an Egham, England–based analyst for technology research firm Gartner. ”They’ve made games social.”

And they’ve made real-life stars out of the inventors of Rock Band and Guitar Hero, Harmonix founders Alex Rigopulos and Eran Egozy. The duo launched the company in 1995 when they were graduate students at MIT. In 2006, after the meteoric success of Harmonix’s music games, MTV purchased it for $175 million. Now the company employs 300 people, a mix of musicians and engineers.

”We’re trying to let people feel the awesome power of performing on stage,” says Egozy, who leads the company’s engineering staff.

Rigopulos, who as the CEO spends most of his time on the business side but still helps design the games, adds that Harmonix has been trying to ”raise the bar with every game that we release in terms of accessibility and emotional impact.” The challenge, he says, has been solving various technical problems along the way. And there have been plenty.

Rigopulos and Egozy first started experimenting with music and technology as members of the computer-music group at MIT’s Media Lab in the early 1990s. Rigopulos, who played in a Balinese gamelan orchestra, and Egozy, a clarinetist, studied hyperinstruments—interactive devices used to explore new forms of musical expression—under the tutelage of MIT scientist and composer Tod Machover. One project outfitted a Levi’s jacket with synthesizer keys and speakers. Another transformed drums into game-style controllers that could play an assortment of audio files in real time.

”It was really the work in that group that helped open our minds to the possibility of fundamentally reinventing people’s notion of what music was,” says Rigopulos, whose 1994 master’s thesis, ”Growing Music From Seeds: Parametric Generation and Control of Seed-Based Music for Interactive Composition and Performance,” explored how gestures might be used to trigger musical events.

Egozy had similar pursuits. In his 1995 master’s dissertation, ”Deriving Musical Control Features From a Real-Time Timbre Analysis of the Clarinet,” he looked at new methods of digital audio analysis, concluding with a prescient remark that such tools would ”shape the way we think about and experience the interplay of computer and musician.”

When they formed Harmonix, their goal was to develop games that could make people feel like rock stars—in their living rooms. The company’s first major project was Frequency. The game, released in 2001, introduced a feature that would become a key part of future Harmonix creations: colored ”gems” that represented musical notes. The gems rushed across the screen as the song played, and gamers had to hit the corresponding controller buttons. If you missed a gem, its sound would be muted from the song—and you’d lose points.

In 2003, the company released Karaoke Revolution. The idea was to create a game that would judge the pitch accuracy of a player’s singing. Previous karaoke systems had crude scoring functions, and the MIT grads wanted a technically rigorous solution.

”I went back to my work in academe to see how it could work in games—that was pretty cool,” says Egozy, who pored over his own work and also a Ph.D. thesis on pitch tracking by an MIT colleague.

His solution was based on a signal-processing algorithm that analyzed the player’s voice in real time. The algorithm used a digital filter to discover the frequency of the fundamental note and remove the harmonic overtones. It then compared the filtered signal to a musical instrument digital interface (MIDI) note from the original song. The better the match, the higher the player’s score.

In 2005, Rigopulos and Egozy were approached by a company called RedOctane, which had created a small guitar-shaped controller and wanted to build a game around it. The Harmonix founders decided to give it a go. The game was called Guitar Hero. The combination of an engaging user interface, innovative controller, and awesome music—including tracks from Jimi Hendrix, the Ramones, and Deep Purple—was a huge success. The game rocked.

A year later, just before Harmonix was acquired by MTV, RedOctane was purchased by Activision, which kept the Guitar Hero franchise. Now on its own, Harmonix decided to design new instrument controllers and to develop its most ambitious title yet: Rock Band.

”It was important that this thing was a real rock simulation, that it gave you a sense you were a rock star interacting with all your bandmates,” says Egozy.

Harmonix spent more than a year perfecting the instruments. The drums posed the biggest challenge. Each of the four drum pads contains a piezoelectric sensor that converts the mechanical vibrations into an electrical signal. The problem was that the four drums sat on the same supporting base, and when the player hit one pad, vibrations propagated and fired the other sensors. Isolating each drum physically would have solved the problem, but it would have made the device unwieldy and costlier to manufacture.

”It might look good on paper, but you have to build it,” says Daniel Sussman, Harmonix’s director of hardware development, who took long trips to China, where the instruments are produced, to iron out the wrinkles in various designs.

Another challenge was generating the gems—the graphic representations of the musical notes—for each song. The set list included tracks of varied styles, with songs from famed groups like Radiohead and Nirvana, but also from indie artists and even bands formed by Harmonix employees. The company tried to automate the process as much as possible, but to ensure greater accuracy, its sound engineers still arranged the gems by hand.

”Fights sometimes break out among our engineers about where to put the gems,” Rigopulos jokes.

Rock Band came out in 2007. Again, the result was a tremendous success, with rave reviews from gamers and critics. But even more significant, it opened the door to a deal with the greatest rock band ever.

The idea of a Beatles-based game first came up in 2007 during a chance conversation between MTV executives and George Harrison’s son, Dhani Harrison, who mentioned he was a big fan of Rock Band. The executives put him in touch with Harmonix, and a discussion with Apple Corps ensued.

But before things went far, Giles Martin, son of Beatles producer George Martin and the current engineering chief for the Beatles’ catalog, noticed a major hurdle standing in the way. Rock Band requires that each instrument—drums, bass, guitar, and vocals—be laid down onto a separate track. This works fine for a contemporary song recorded on a 48-track system. But roughly the first third of the Beatles’ catalog had been engineered using much simpler recording equipment.

Click to enlarge

Decoding the Beatles:

The Beatles recorded many songs using two- and four-track tape, with multiple instruments often combined on the same track. Harmonix worked with Giles Martin and sound engineers at Abbey Road Studios to isolate instruments onto separate tracks, which could then be converted into game data. Here's the process for "Taxman," from the 1966 album Revolver.

Though the Beatles were innovators in the use of two or four tracks, they often put all the instruments on a single track and then used the remaining tracks for ancillary sounds and effects. Apple Corps wanted the game to cover the span of the band’s career, so Harmonix would have to find a way to extract the different instruments from those early songs.

”We were thinking, oh crap, this is a daunting challenge,” recalls Rigopulos, ”but it’s a problem worth solving.”

Working with Martin, Harmonix first tried to use conventional audio engineering tools to filter specific sounds from the original master tapes, recorded at Abbey Road Studios in the 1960s. But the method required too many steps, and the results often weren’t accurate enough.

”You couldn’t just set up a single filter to remove the tambourine,” Egozy says.

A more hands-on process was necessary. Egozy and his team turned to audio forensic software normally used by law enforcement and in restoration projects. These tools, equipped with advanced digital-signal-processing capabilities, include more controls than those offered by most audio-filtering packages. Myriad audio parameters can be tweaked in order to zero in on specific sound elements.

Egozy says that although sounds with distinct frequency ranges were easily separated using equalizers, the work became much more difficult as the frequencies were intermingled.

”The forensic tools really helped in cases where we had several more full-frequency instruments on the same track,” he says. ”For example, guitar on the same track as drums, or vocals on the same track as guitar, or all three on a single track.”

Fine-tuning the filters was a painstaking process. Harmonix and Giles Martin spent several months going through each song, separating instruments and vocals onto different tracks and then identifying every bit of sound, note by note.

Still, the work was far from over. A big element of the Beatles’ musical success was the elaborate vocals—the harmonies that Lennon, McCartney, and Harrison performed in many songs. There couldn’t be a Beatles game without harmonies.

The problem now was that previous Rock Band games had featured only one microphone and one singer. Harmonix would have to redesign the user interface to fit three vocal lines and also reengineer the way the game evaluates pitch.

Egozy and his team designed a new system, which they called dynamic part assignment, that allows three different microphones to be used and three different vocal lines to be sung simultaneously. As the song unfolds, the vocals are represented on the screen as dashes next to the lyrics—for instance, the middle line of melody in ”Day Tripper” appears between the upper and lower harmonic notes.

At every time step in a particular song, the software analyzes each singer’s pitch and makes the best educated guess as to what vocal part a player is attempting so it can score each player’s singing. If one player is singing McCartney’s C and another is singing Lennon’s G, for example, the pitch is assessed and assigned automatically to the two players. The software can keep track of the player’s pitches even if they switch lines, as Lennon and McCartney sometimes did.

”Players can sing any part without having to decide who is singing what ahead of time,” Egozy says. ”The matching algorithm makes the whole puzzle work.”

Singing the harmonies requires some practice, and so Harmonix made them optional. If singers want to attempt the harmony lines, they get extra points, but if they all sing only the main line, they’re not penalized. And with three instruments and three microphones, a living-room Fab Four can actually be a Fab Six.

Click to enlarge

Engineering Music Games: Harmonix is responsible for the most advanced and commercially successful music video games ever made. Here are the main innovations in each title the company has released.

One day during the game’s development, Harmonix met even greater perfectionists than those on its own staff: the Beatles themselves. McCartney wanted to know why there was a whammy bar—the lever used to vary the string tension—on his Hofner bass (answer: so it could double as a guitar in the game). Starr wanted the Beatles drop-T logo on the bass drum. But Yoko Ono provided perhaps the most moving request: to give the digital Beatles more soul.

Harmonix took a long look at the video-game versions of the band members and realized that Ono was right. Despite all the computer graphics work, there was something dull about the characters.

The problem started with the eyes. In Harmonix’s earlier games, the band members would just stare off blankly into space, sometimes appearing aloof on stage. So this time the coders created a special graphics tool—they aptly called it eye tech—that keeps track of each avatar’s gaze. It allows animators to fix a character’s sight on one of dozens of points, like the back of the venue or the neck of the guitar or even another Beatle.

And then there were the grins—or the lack thereof.

”These guys were smiling all the time,” says creative director Josh Randall. ”We had to put more joy into the game.”

The coders then created another animation tool to allow artists to adjust the Beatles’ grins at different points in a song’s performance. The small tweaks made a joyful difference.

The final result is a fantastic journey through Beatles history. The game features famous locations where the band played and also imaginary psychedelic landscapes—Harmonix calls them ”dreamscapes”—like a mountaintop with a giant sun as a backdrop for ”Here Comes the Sun.” The game will also include unreleased studio chatter between the quartet and previously unseen archival photos.

All of which should make The Beatles: Rock Band another big hit for Harmonix. Critics note, however, that the company’s next task may be the most difficult—staying on top of its game.

”The biggest challenge [for Harmonix] is not letting this become a fad,” says Edward Woo, an analyst specializing in digital media at Los Angeles–based Wedbush Morgan Securities. ”What they have to do is come up with something innovative that gets people to keep buying the games.”

Indeed, if you ask the fans, there’s no shortage of ideas. Many are already clamoring for a follow-up game that simulates another greatly admired band, such as Led Zeppelin.

”That’s the first band everyone asks us about now that we’ve done the Beatles,” Rigopulos admits, but he declined to give more details.

In the meantime, Harmonix is counting on its exclusive deal with the biggest band in music history in order to maintain its edge. In addition to the initial 45 songs, buyers of the game will soon be able to pay to download additional playable tracks, including the album Abbey Road in its entirety.

For the Beatles, the embracing of the video-game generation is also a bright move. It’s an opportunity to present the band to a younger crowd that could be the next Fab Four fans. It’s a crowd that understands the language of cascading multicolored gems and digital avatars.

As McCartney quipped at the Los Angeles event, ”Who’d ever thought we’d end up as androids?”