Sunlight filters through the stained-glass windows as Francisco Ortigosa wanders around the Torre Girona chapel, his eyes taking in all the details. And what details they are: there are the thick beige stone walls, the Romanesque arches, the ornately carved wooden doors, and the sleek black cabinets housing the massively parallel supercomputer.
Yep, this is no ordinary chapel. Situated on the campus of the Technical University of Catalonia, in Barcelona, Spain, this chapel has been converted into the world’s most beautiful server room. It houses MareNostrum, the third most powerful supercomputer in Europe. The place is still inspiring, but these days visitors like Ortigosa come here for enlightenment not on spiritual matters but rather on the leading edge of high-performance computing.
Ortigosa is the director of geophysics at Repsol YPF, the Spanish oil giant. He heads an ambitious—and potentially stunningly lucrative—geophysical supercomputing initiative dubbed the Kaleidoscope Project. The goal of the project is to develop an entirely new class of seismic-imaging codes—the computer algorithms that transform raw seismic data into useful, detail-rich images of the Earth, kilometers below its surface. Ortigosa hopes those images will reveal oil and gas reservoirs that current codes can’t uncover.
Kaleidoscope will more fully unleash the power of supercomputers like MareNostrum, which was built by IBM and has 2560 computing nodes and a peak performance of 94.21 trillion floating-point operations per second (teraflops). Today’s most advanced seismic codes create color-coded three-dimensional maps of the subsurface realm by solving a mathematical construct known as the one-way wave equation, which describes seismic waves traveling in just one direction. But Kaleidoscope codes will solve the two-way wave equation, greatly improving the level of detail by taking into account waves propagating in multiple directions.
Repsol, based in Madrid, plans to use the new technology to locate hydrocarbons buried kilometers below the seafloor in the Gulf of Mexico—and below more than 2500 meters of ocean. That’s what oil companies call ultradeep water, and it’s the new frontier in petroleum exploration. Codes based on the one-way wave equation can’t accurately image the thick bodies of salt that typically trap hydrocarbons so far down. Repsol’s geophysicists are confident that two-way wave equation codes will overcome this limitation, allowing them to search for oil under 10 kilometers of sediment and hard rock where the salt bodies and underlying oil hide.
“Seismic imaging today uses lots of approximations,” Ortigosa says. “Our codes will create a closer representation of the actual physics of the Earth. We’re not taking shortcuts.”
To carry out its plan, Repsol recruited two partners: 3DGeo, a seismic software firm headquartered in Santa Clara, Calif., and the Barcelona Supercomputing Center (BSC), which operates MareNostrum. The American geophysicists are developing the codes to solve the two-way wave equation, and the Spanish computer scientists are figuring out how to run the codes efficiently on supercomputers—MareNostrum in the immediate future and later on a BSC system based on the Cell processor, the powerful number-crunching chip developed jointly by IBM, Sony, and Toshiba.
Geophysicists have been chasing the holy grail of the two-way wave equation for years. The problem is that to even think about solving it required more than 10 times the computing power and around 100 times the data storage capacity than was typical of available supercomputers. But more recently, rising processing power and oil prices have conspired to at last put the solution within reach.
And it’s happening not a moment too soon. In the Gulf of Mexico, for example, most of the hydrocarbon reserves in the relatively shallow shelf waters have been drained. The easy oil is gone. But in deep waters there’s plenty left: at least 56 billion barrels of oil equivalent—a measure that includes oil and natural gas—which at US $90 a barrel would fetch about $5 trillion and meet the entire U.S. demand for oil and gas for five years. The catch is that finding oil at such depths is extremely challenging and hugely expensive.
Oil exploration is a hit-or-miss business. Just drilling one well in the deep waters of the Gulf of Mexico to find out if it contains oil can cost $100 million. So oil companies do all they can to avoid hitting dry wells. That’s where seismic imaging comes in. Better images mean less risk. So Repsol is not alone in its quest to solve the two-way wave equation.
“Every major oil company and seismic contractor is going after this,” says William W. Symes, a computational seismology specialist at Rice University, in Houston, who is not involved with the Kaleidoscope Project. Access to MareNostrum may give the American-Spanish team “a bit of a leg up,” he says, adding, “The main thing they’ve got is some very smart people with a great deal of theoretical background—and they are crackerjack programmers.”
One of those hotshot coders is Dimitri Bevc (pronounced “BAYoats”), who is president and a cofounder of 3DGeo. From the picture windows of his fourth-floor office in Santa Clara, he can see the Diablo Range, and it’s a source of inspiration for Bevc, an experienced mountain climber.
But today he’s pondering deeper things. He taps his keyboard and opens two large images on the screen. Each shows a cross section of a cube of earth below the seafloor, 14 km on its sides, that contains a mushroom-shaped salt body. To the untrained eye, the two grayscale images are very similar. But there’s a key difference.
“Look at these vertical lines,” Bevc says, pointing to the stem of the salt mushroom, where its edges merge with the surrounding sediments. In one image, created using the one-way wave equation, the stem is blurry; in the other image, based on the two-way wave equation, it’s sharp. “This has huge implications in the drilling planning,” he says. “Here you can’t see very clearly where the target is. Here you can.”
Bevc explains that oil is less dense than the sediments, so it tends to flow up through the Earth’s layers. But it can’t flow through impermeable salt bodies. As a result, oil accumulates in pockets resting against the salt structures. When you drill, you want to reach the top of the reservoir so that the oil flows up into your pipe. And when planning where to make a $100 million hole, you don’t want a blurry image.
To appreciate how 3DGeo solves the two-way wave equation, it helps to understand how seismic imaging works. It begins with a marine seismic survey. A specially built ship cruises over an area of interest and fires an air gun that sends a powerful sound wave into the ocean. This wave propagates through the water and down through subseafloor layers of sandstone, shale, salt, and other materials, producing echoes that return to the surface. The ship tows a dozen or so cables, each up to 10 km long, carrying thousands of hydrophones that measure the minute pressure waves of the echoes. In a typical survey, the ship covers 3000 square kilometers, about three times the area of Hong Kong, and fires the air gun tens of thousands of times. Hard-disk drives on the ship record many terabytes of echo data.
Then comes the real challenge: transforming that data into images of the Earth’s interior. Today’s most advanced seismic-imaging codes rely on an ingenious technique devised by Stanford University geophysicist Jon Claerbout in the 1970s. Basically, Claerbout’s method takes the recorded echoes, runs them through the wave equation as a mathematical extrapolation tool, and tells you the depths at which the echoes originated. With enough echoes, you can get a detailed image of the subsurface realm.
The wave equation consists of a single expression—a second-order linear partial differential equation—that describes the propagation of a wave as a function of space and time. It is commonly used not only in geophysics but also in acoustics, fluid dynamics, and electromagnetism. It can describe the behavior of a vibrating string, sound in air, waves in water, and light waves. In geophysics, the equation gives you the pressure produced by a sound wave at a specific point and time.
To solve the wave equation in three dimensions for a large volume, you need a very powerful computer. You start by creating a large 3-D grid of numbers that represent the surveyed volume of ocean and subseafloor earth. Each point in the grid stores the pressure of one or more sound waves present at that spot. Seismic-imaging codes use the wave equation to extrapolate, or “push,” the echoes from the top of the grid, where they were recorded, to intermediary positions, where they originated. To keep things simple, this extrapolation assumes that the echoes traveled in only one direction: from the intermediary positions within the Earth straight to the surface—hence the name one-way wave equation.
The method worked beautifully for years in such areas as shelf waters, but geophysicists recently discovered that it can’t accurately image sites with more complex geological structures, such as salt bodies buried deep below the seafloor. The reason is that the one-way wave equation doesn’t account for the specific echoes ricocheting in multiple directions around those structures.
Now solutions for the two-way wave equation, which emerged in the 1980s, promise to overcome those limitations. The two-way wave equation method is different from its one-way counterpart because it accounts for cases in which a wave bounces a few times under, say, a salt dome before emerging as an echo. The two-way wave equation can retrace that propagation and thus image the area under the salt body.
The idea is to get rid of the extrapolations and instead use the complete wave equation to simulate the actual path of the echoes through the subsurface sediments. But how do you retrace those paths when all you have is information about the echo as it entered the hydrophones on the ship? Such a simulation would require going backward in time! The good news is, you can—in a computer, at least.
Here’s how the Kaleidoscope team solves the two-way wave equation. The first step consists of getting a kind of rough model of the subsurface layers; this model is obtained from some initial preprocessing of the echo data that reveals where the waves travel faster, where they are refracted, and so on. To get a good image, you need a good initial model, so Repsol geophysicists spend weeks and even months crafting it.
Next, the 3DGeo codes use that initial model—a 3-D grid of numbers, just as in the one-way method—to propagate the echoes, each step of the wave front calculated using the wave equation running backward in time. It may sound esoteric, but all this means is that time values plugged into the equation have a minus sign. (The method is also known as reverse time migration.)
The two-way wave equation codes also need to simulate the propagation of the air gun wave through the grid. That’s because you generate your image by comparing this grid of air gun data with the grid of echo data; wherever the two waves intersect, an echo originated at that point. These intersections reveal the contours and interfaces of the surveyed volume.
The Kaleidoscope codes created by 3DGeo consist of several components, written in C and Fortran, that basically solve the wave equation for each point in a spherical wave propagating within the 3-D grid. Computing each point’s next step in the simulation requires about 100 floating-point calculations. For a large seismic survey consisting of 10 000 subsurface cubes, each a 3-D grid with billions of points, and requiring tens of thousands of time steps, your simulation quickly shoots up close to 10 quintillion (1019) floating-point calculations. If you tried to run it on your desktop PC, it would go on for a century before you got an image like those Bevc was looking at.
Meanwhile, at the Barcelona Supercomputing Center, other Kaleidoscope researchers are using their expertise in fluid dynamics and computational mechanics to fine-tune the 3DGeo codes to run on MareNostrum. The machine, which comes in at No. 13 in the Top500 ranking of the world’s fastest computers, has 5120 dual-core PowerPC processors, 20 TB of central memory, and 400 TB of disk storage. Built in 2005 by the Spanish government, MareNostrum resides inside a glass box at the center of Torre Girona’s nave. (Latin for “our sea,” Mare Nostrum was the ancient Romans’ name for the Mediterranean.)
Other big oil companies and seismic-imaging firms probably have computers as powerful as MareNostrum—or even more powerful. They guard that kind of information as carefully as the National Security Agency would. “But those systems are busy with exploration projects, with not much time for R&D,” says Michael P. Perrone, a supercomputing expert at IBM, which collaborates with the Kaleidoscope efforts. “MareNostrum lets the Kaleidoscope partners test their big algorithms.”
What makes seismic imaging particularly challenging for supercomputers is the amount of data involved. The data for one subsurface cube 10 km on a side can reach several gigabytes, and a typical survey consists of thousands of such cubes. “We’re working with terabytes of data, and this means that in the supercomputer we must manage the input and output of data very carefully,” says José María Cela, a computer engineering professor at the Technical University of Catalonia and a BSC researcher.
To overcome this problem, the Kaleidoscope researchers adopted a divide-and-conquer approach. They divided the cubes into smaller chunks, each going to one of MareNostrum’s computing nodes. In one test, 3DGeo divided a 10-km cube into 512 chunks. MareNostrum took about a minute to process all of them. If the supercomputer were to process the cube as a whole using one node, it would require almost 6 hours.
To speed up the codes even more, the BSC experts came up with several other strategies. They improved the codes by manually verifying the source code for tasks that could run in parallel. They minimized the exchange of data between different tasks and hand-optimized all the calculation routines. Cela says that these changes have improved the processing speed of the original Kaleidoscope code by a factor of five and at the same time reduced memory usage by a factor of two.
But MareNostrum is just a test bed for the Kaleidoscope algorithms. The goal is to develop codes for the next generation of supercomputers. Oil prospectors replace their computers as fast as you replace your PC, and maybe even more frequently—about every two years. “It’s really a race,” says Ortigosa, Kaleidoscope’s project leader. “Before you finish coding your algorithm there’s already a new hardware, and you have to start coding again.”
Kaleidoscope’s goal is to develop the algorithm with tomorrow's hardware—the Cell processor—in mind. But programming the Cell is an entirely new world for most coders. The processor’s architecture—one main general-purpose PowerPC core and eight number-crunching units—is so extraordinary that it requires programmers to rethink their strategies. That’s why Repsol partnered with BSC, which has lots of experience with the Cell.
In one initiative, the Spanish researchers are developing a programming environment dubbed SuperScalar, which hides the parallelization task from programmers. It allows them to develop highly parallelized code without worrying about the data flows among processors. This past November, BSC and IBM formalized a partnership to develop a new supercomputer based on the Cell. Francesc Subirada, associate director of BSC, says that nobody knows at the moment what this computer will look like. “But we do have a name for it,” he says. “We call it MareIncognito.”
The Kaleidoscope Project had its largest software run late last year. From 3DGeo’s office in California, Bevc and his team loaded their wave equation codes into MareNostrum, more than 9000 km away, and turned them loose on some echo data. Then they waited.
Twenty days later, the supercomputer completed the task. It was a simulated seismic survey. Instead of using a real ship to gather real data, 3DGeo re-created that process in a computer. A virtual ship fired air gun shots, and virtual hydrophones recorded the echoes. In contrast with the conditions of a real survey, however, 3DGeo knew the exact geology of the subseabed volume, a model provided by Repsol. The idea was to apply the wave equation codes to the simulated echoes and then compare the resulting image with the known geology to see how well the codes performed.
The area surveyed was huge: 38 km by 30 km by 15 km, representing a geological setting much like the Gulf of Mexico, with complex salt bodies. The simulation generated 32 TB of data—one of the largest synthetic data sets in the industry, according to 3DGeo. “I remember folks at BSC said, ‘You cannot produce that much data,’ and we said, ‘Yes we can,’ ” Bevc says. 3DGeo considered bringing a copy to its own servers, but that much data would take two suitcases full of magnetic tapes.
Next 3DGeo used the data to test its codes. It ran both one-way and two-way wave equation codes. “We know exactly what the answer should be, so we can see if our code is right,” Bevc says. The result? “It’s pretty much dead on,” he says. “We’re able to image things [using the two-way wave equation] that we weren’t seeing before, steep salt flanks and such.”
Also last year, the Kaleidoscope Project began its first production run. It involved real seismic data for a 500-km2 area in the deep waters of the Gulf of Mexico. Repsol transported 15 TB stored in hard drives to Barcelona and loaded it into MareNostrum. How long did it take to image the area? Repsol won’t say.
The company is a bit cagey about the details because it doesn’t want to tip its hand to its competitors. Ortigosa says they’re still analyzing the results and that sometime this year, based on those images and other inputs, the company will decide whether to drill or not. “This is real, not synthetic data, so this time we don’t know the answer,” he says. “But I’m confident we’ll get it right.”
About the Author
Geophysicist Francisco Ortigosa was photographed by Francisco Guerrero for “Solving the Oil Equation”, one of our winners. Shooting the MareNostrum supercomputer in Barcelona, Guerrero says, was ”like discovering a hidden world. At the head of the path sat the old chapel structure, and within its ancient walls and stained-glass windows rests this fantastic piece of 21st-century technology. Imagine your desktop computer housed inside an 18th-century antique box.”
Two-Way Wave Equation Seismic Imaging
Goal: To develop advanced seismic-imaging codes based on the two-way wave equation and designed to fully exploit the power of supercomputers.
Why It's a Winner: The codes will generate images of oil and gas reservoirs in the deep waters of the Gulf of Mexico with more detail than current techniques.
Players: Repsol YPF, 3DGeo, and Barcelona Supercomputing Center
Where: Houston; Santa Clara, Calif.; and Barcelona, Spain
Staff: 28 geophysicists, mathematicians, and computer engineers
Budget: 8 million (about US $11.7 million)