Zenph Recreates Sound of Great Musicians

Software to Make Old, Scratchy Recordings Play Like New

Loading the podcast player...

Susan Hassler: This is Susan Hassler, Editor in Chief of IEEE Spectrum magazine. John Q. Walker II leads a high-tech start-up in Durham, North Carolina, that's doing some remarkable things with sound—specifically with the sound of great musicians, which has not always been recorded accurately. David Schneider investigates what Walker is doing—and what fun he's having in the process.

David Schneider: John Q. Walker's latest start-up—Zenph Sound Innovations—began with a single instrument: the piano.

[Scratchy piano music begins]

David Schneider: Walker and his team wrote software to take a piano recording—even an old, scratchy one—and determine not only which keys were struck but exactly how they were played, including all the subtle nuances that distinguish great pianists. The result was simply amazing.

[Improved version of same piano piece]

John Q. Walker: We saw it as a very hard signal-processing problem: Can we get backwards from recordings to the notes? We all know how to go forwards, but can we go backwards? And if we can crack that problem, the whole industry changes.

David Schneider: Walker's group indeed cracked that problem. And with their software, they were able to re-create great piano performances of the past. "Reperformance" is the word Walker uses. It may seem like remastering on steroids—but in fact it's a lot more. Think about turning a mono recording into stereo, for example, changing the acoustics of the room and positioning the microphones differently from where they were placed during the original recording session—even letting the listener experience what the pianist heard sitting on the bench. The commercial value was obvious, which is why Walker's company was able to strike a deal with Sony Music.

[Music: Glenn Gould playing Bach's Goldberg Variations]

John Q. Walker: This was the first album we recorded with Sony, and it's the Canadian pianist Glenn Gould. This was the first recording he made. He was 22 years old. He had had a very successful recital in New York, and Columbia Records said, "Let's sign him up." And normally when you sign up a young pianist, the first recording is, maybe, some Chopin or Beethoven sonatas or something. He picked a piece called the Bach Goldberg Variations, which had been rarely recorded on piano at all. It was thought you couldn't play it on one manual. And he said, "No, I want to do this piece" and made a phenomenal recording in 1955 that has been one of the best continuous piano recordings in classical music history.

[Music ends]

David Schneider: To demonstrate how well his system could capture even Gould's rapid-fire playing, Walker had me listen to the original recording while his robotic piano played along.

[Piano music combining original Gould recording of Goldberg Variations with Zenph reperformance]

David Schneider: Had the re-created notes been even a fraction of a second off in timing, you would hear the dissonance. There's none. Walker's piano re-created Gould perfectly.

[Piano music ends]

David Schneider: Next, Walker showed me another impressive feat, a piano reperformance based on a video recording of The Ed Sullivan Show from 1969.

Ed Sullivan: From Ferriday, Louisiana, Jerry Lee Lewis!

[Applause, music, and Jerry Lee Lewis's singing begins]

David Schneider: This really does take you back. But, as Walker explains, the trouble is, you can't hear the piano.

John Q. Walker: There's a bass player and a drummer and whatever, but there's no mic for the piano. Listen, you can see him playing, but you cannot hear any piano. Look—nothing.

David Schneider: But the video of Jerry Lee Lewis's hand movements provided Walker's team with enough information to reconstruct what he was playing. Listen to what happens when Walker's robotic piano—making up for the missing microphone—joins in.

[Re-created piano part added to Jerry Lee Lewis's singing]

David Schneider: Having conquered the piano, Walker and his team are turning their attention to other instruments, starting with the string bass. But instead of building a robotic bass to match their robotic pianos, they're using a computer and a high-tech speaker (which looks like something out of a sci-fi movie) to synthesize the sounds of the bass.

David Schneider: It's about 4-1/2 feet high, and it has the three-dimensional characteristics of a string bass. So in a room, in three dimensions, your ear goes, "Oh, it's a string bass because it's bouncing off the walls in the same way." This one is entirely a virtual instrument as opposed to robotic ones.

David Schneider: So let's see, piano, bass—do I detect a shift from classical to jazz?

John Q. Walker: Our first attempt at working with the bass was with a recording from the Oscar Peterson Trio. So it's Oscar Peterson at piano, Ray Brown on bass, and Ed Thigpen on drums.

[Music of the Oscar Peterson Trio begins]

David Schneider: This is a live recording of the Oscar Peterson Trio from 1960 at a bar in Chicago called the London House. Walker's software took that recording and determined exactly how the bass player—Ray Brown—was plucking his strings. Walker was able to re-create the bass part using that superhigh-tech speaker of his.

[Bass notes]

David Schneider: Listening to the bass alone wouldn't be that interesting, so Walker had one of his robotic pianos play along. Robots aren't yet able to improvise jazz on their own, of course. So this one re-created something Walker's colleague Eric Hirsh, a jazz pianist, played earlier.

[Piano joins in]

John Q. Walker: So we didn't take Oscar's original piano solo; we said, "Oh, look, now anyone can play along with the bassist." Hirsh was not here, nor was Ray Brown [laughs], all driven by data from a Macintosh that's sitting here clicking away.

[Music fades]

David Schneider: By this point it became very clear to me that Walker's group has command of technology that could really shake up the music industry in the same way that computer graphics has given the film industry the tools to create some amazing effects. But isn't the music industry as we know it on its way out?

John Q. Walker: We have heard on a regular basis the past few years that the big labels have been relatively flat or slightly on the down curve. So, I think all the growth is actually out in virtual worlds. You know, how do you think music will be played in Second Life and places like that? But the 3-D experience of, you know, hearing a concert and walking around the room and doing all that in a virtual world—I think we're building groundwork for that.

[Piano music builds]

David Schneider: And what better job for someone with a great love for both music and software?