13 April 2005--"Houston, we've had a problem."
Thirty-five years ago today, these words marked the start of a crisis that nearly killed three astronauts in outer space. In the four days that followed, the world was transfixed as the crew of Apollo 13--Jim Lovell, Fred Haise, and Jack Swigert--fought cold, fatigue, and uncertainty to bring their crippled spacecraft home.
But the crew had an angel on their shoulders--in fact thousands of them--in the form of the flight controllers of NASA's mission control and supporting engineers scattered across the United States.
To the outsider, it looked like a stream of engineering miracles was being pulled out of some magician's hat as mission control identified, diagnosed, and worked around life-threatening problem after life-threatening problem on the long road back to Earth.
From the navigation of a badly damaged spacecraft to impending carbon dioxide poisoning, NASA's ground team worked around the clock to give the Apollo 13 astronauts a fighting chance. But what was going on behind the doors of the Manned Spacecraft Center in Houston--now Lyndon B. Johnson Space Center--wasn't a trick, or even a case of engineers on an incredible lucky streak. It was the manifestation of years of training, teamwork, discipline, and foresight that to this day serves as a perfect example of how to do high-risk endeavors right.
Many people are familiar with Apollo 13, thanks to the 1995 Ron Howard movie of the same name. But as Howard himself was quick to point out when the movie was released, it is a dramatization, not a documentary, and many of the elements that mark the difference between Hollywood and real life are omitted or altered. For this 35th anniversary of Apollo 13, IEEE Spectrum spoke to some of the key figures in mission control to get the real story of how they saved the day.
First, A Little Refresher on moon-shot hardware: a powerful, 85-meter tall, three-stage Saturn V booster launched each mission from Cape Canaveral in Florida [see photo, To The Moon]. Atop the Saturn V rode the Apollo stack, which was composed of two spacecraft: a three-person mother ship to go to the moon and back, called the command and service module, or CSM; and a two-person lander, called the lunar module, or LM, to travel between the CSM and the surface of the moon.
The two spacecraft each, in turn, had two parts. The CSM divided into a cylindrical service module (SM) and a conical command module (CM). The service module housed the main engine and supplied all the oxygen, electricity, and water the crew needed for the long voyage--it took about six days for a round trip between the Earth and the moon. The crew lived in the cramped command module, which housed the flight computer and navigation equipment. The command module was the only part of the Apollo stack that was designed to come back safely to Earth. It would plummet through the atmosphere, the blunt end of its cone designed to withstand the immense heat generated by the descent, and then deploy parachutes and splash down in the ocean.
The lunar module consisted of an ascent stage and a descent stage. The descent stage had a powerful engine used to land the lunar module on the moon. After the lunar expedition was complete, it served as a launch pad for the ascent stage, which housed the astronauts, to blast off and rendezvous with the command and service module in lunar orbit.
For most of the way to the moon, the command and service module and the lunar module--dubbed the Odyssey and Aquarius, respectively, on the Apollo 13 mission--were docked nose to nose. But the astronauts generally remained in the command module, because the lunar module was turned off to preserve power.
Most of that power came from a cluster of three fuel cells in the service module. The fuel cells were fed hydrogen and oxygen from two pairs of cryogenic tanks, combining them to produce electricity and water [see diagram, Module Map].
There were some batteries on board the command module, but these were intended for only a few hours use during re-entry, after the service module was jettisoned close to Earth.
It was one of the cryogenic tanks that would reveal itself as the Odyssey's Achilles' heel. On 13 April 1970, around 9 p.m. Houston time, almost 56 hours into Apollo 13's flight, mission control [see photo, Calm Before The Storm] asked the crew to turn on fans in all the cryogenic tanks to stir the contents in order to get accurate quantity readings. Due to a series of pre-launch mishaps, turning on the fan sparked a short circuit between exposed wires within oxygen tank two [see sidebar, "The Devil's in the Details"].
The Odyssey was dying, but no one knew it yet.
Even The Crew were unaware of the gravity of the situation. In the Ron Howard movie, the oxygen tank two explosion is accompanied by a whole series of bangs and creaks while the astronauts are tossed around like ping-pong balls. But in real life, "there was a dull but definite bang--not much of a vibration though...just a noise," said Apollo's 13's commander, Lovell, afterward. Then the Odyssey's caution and warning lights lit up like a Christmas tree.
On the ground, mission control was initially unperturbed. During the cryogenic tank stir, the flight controller in charge of the fuel cells and the tanks, Sy Liebergot, had his attention focused on oxygen tank one. Liebergot was an EECOM, a job title that dated back to the Mercury program days of the early 1960s. It originally meant the person was responsible for all Electrical, Environmental, and COMunications systems onboard the CSM. The communications responsibilities had recently been split out of the EECOM's job, but the name remained.
In an unfortunate coincidence, oxygen tank two's quantity sensor had failed earlier, but the two tanks were interconnected, so Liebergot was watching the quantity that tank one reported to get an idea what was in tank two.
As he sat in mission control at his console, with its mosaic of push buttons and black-and-white computer displays, Liebergot wasn't alone in tending to the Odyssey's electronic and life support systems. He was in voice contact with three other controllers in a staff support room across the hall. Each flight controller in mission control was connected via so-called voice loops--pre-established audio-conferencing channels--to a number of supporting specialists in back rooms who watched over one subsystem or another and who sat at similar consoles to those in mission control.
Liebergot's wingmen that day were Dick Brown, a power-systems specialist, and George Bliss and Larry Sheaks, both life support specialists. As the pressure rapidly rose in oxygen tank two and then abruptly fell within seconds, their eyes were fixed on the other cryogenic tank readouts, and they all missed the signs that tank two had just exploded.
Suddenly The Radio Link from the crew crackled to life. "Okay Houston, we've had a problem here," reported command module pilot Swigert as he surveyed the Odyssey's instruments. "Houston, we've had a problem," repeated Lovell a few seconds later, adding that the voltage of one of the two main power-distribution circuits, or buses, that powered the spacecraft's systems, was too low. But a few seconds later the voltage righted itself, so the crew began chasing down what seemed to be the big problems: the jolt of the explosion had caused their computer to reset and had knocked a number of valves closed in the attitude-control system that kept the Odyssey pointed in the right direction.
In mission control though, things weren't adding up. The spacecraft's high-gain directional antenna had stopped transmitting, and the Odyssey had automatically fallen back to its low-gain omnidirectional antennas. Liebergot and his team were seeing a lot of screwy data, dozens of measurements out of whack. Fuel cells one and three had lost pressure, and were no longer supplying current, leaving only fuel cell two to pick up the load; oxygen tank two's pressure was reading zero; the pressure in oxygen tank one was rapidly failing; and Odyssey had completely lost one of its electrical distribution buses along with all the equipment powered by it. The crew connected one of their re-entry batteries to the remaining bus in a bid to keep the command module's systems up and running.
Liebergot's training kicked in. Simulation after simulation had taught controllers not to make rash decisions based on a few seconds of oddball data--measurements were made by imperfect sensors and had to pass through a lot of space, with a lot of opportunities to get mangled, before they turned up on a controller's screen. "Engineers that work in this business are well schooled to think first in terms of instrumentation," explains Arnold Aldrich, chief of the command and service module systems branch during Apollo 13. He was in mission control at the time of the explosion and recalls that "it wasn't immediately clear how one particular thing could have caused so many things to start looking peculiar."
So when Gene Kranz, the flight director in charge of the mission (referred to as "Flight" on the voice loops), pointedly asked Liebergot what was happening on board the Odyssey, the EECOM responded, "We may have had an instrumentation problem, Flight."
Thirty-five years later, Liebergot still ruefully remembers his initial assessment. "It was the understatement of the manned space program. I never did live that down," he chuckles.
To Kranz, the answer sounded reasonable, as he'd already had some electrical problems with the Odyssey on his shift, including one involving the high-gain antenna. "I thought we had another electrical glitch and we were going to solve the problem rapidly and get back on track. That phase lasted for 3 to 5 minutes," says Kranz. Then "we realized we'd got some problem here we didn't fully understand, and we ought to proceed pretty damn carefully."
Kranz's word was law. "The flight director probably has the simplest mission job description in all America," Kranz told Spectrum. "It's only one sentence long: 'The flight director may take any action necessary for crew safety and mission success.' " The only way for NASA to overrule a flight director during a mission was to fire him on the spot.
The rule vesting ultimate authority in the flight director during a mission was on the books thanks to Chris Kraft, who founded mission control as NASA's first flight director and who was deputy director of the Manned Spacecraft Center during Apollo 13. He had written the rule following an incident during the Mercury program when Kraft, as flight director, had been second-guessed by management. This time, as the crisis unfolded, no one had any doubts as to who was in charge. While other flight directors would take shifts during Apollo 13, as the lead flight director Kranz would bear most of the responsibility for getting the crew home.
Mission control and the astronauts tried various fuel cell and power bus configurations to restore the Odyssey to health, but anyone's remaining hope that the problem was something that could be shrugged off was dashed when Lovell radioed down: "It looks to me, looking out of the hatch, that we are venting something out into space." It was liquid oxygen spilling out from the wounded service module.
The problems were piling up at Liebergot's door. Although his voice is impressively calm throughout the recordings of the voice loops from mission control, Liebergot admits that he was almost overwhelmed when he realized "it was not an instrumentation problem but some kind of a monster systems failure that I couldn't sort out...It was probably the most stressful time in my life. There was a point where panic almost overcame me."
Liebergot gives credit to the endless emergency simulation training for getting him through the moment--as well as to the big handles that flanked each mission control console, intended to make servicing easier and jokingly dubbed "security handles" by the controllers. "I shoved the panic down and grabbed the security handles with both hands and hung on. I decided to settle down and work the problem with my backroom guys. Not to say that the thought of getting up and going home didn't pass my mind," he remembers.
The Emergency Simulations had also taught controllers "to be very careful how you made decisions, because if you jumped to the end, the sims taught you how devastating that could be. You could do wrong things and not be able to undo them," explains Kraft.
As controllers scrambled to track down the source of the venting, flight director Kranz echoed this thinking to all his controllers. "Okay, let's everybody keep cool...Let's solve the problem, but let's not make it any worse by guessing," he broadcast over the voice loops, practically spitting the word "guessing," and he reminded them that, just in case, they had an undamaged lunar module attached to the Odyssey that could be used to sustain the crew.
For now, Liebergot and his back room concentrated on ways to ease the ailing command module's power problem until they figured out what was wrong, and the crew started powering down nonessential equipment to reduce the load temporarily. The goal was to stabilize the situation pending a solution that would get the Odyssey back on track.
But Liebergot, who was starting to realize the full depth of the problem, unhappily told Kranz, "Flight, I got a feeling we've lost two fuel cells. I hate to put it that way, but I don't know why we've lost them."
Liebergot began to suspect that the venting Lovell had reported was coming from the cryogenic oxygen system, an idea bolstered when Bliss, one of Liebergot's backroom life support specialists, asked Liebergot worriedly, "are you going to isolate that surge tank?" The surge tank was the small reserve tank of oxygen that the crew would breath during re-entry, but the massive leak in the service module's cryogenic system meant that the remaining fuel cell was starting to draw on the surge tank's small supply of oxygen to keep power flowing.
Drawing On The Command module's limited reserves, such as its battery power or oxygen, was usually a reasonable thing to do in sticky situations--assuming the problem was relatively short-lived and the reserves could be replenished from the service module later. But Liebergot was now worried that the service module was running out of power and oxygen, permanently. Once he confirmed that the surge tank was being tapped, he revised his priorities, from stabilizing the Odyssey to preserving the command module's re-entry reserves. This caught Kranz momentarily off guard.
"Let's isolate the surge tank in the CM," Liebergot told Kranz. "Why that? I don't understand that, Sy," Kranz replied, noting that isolating that tank was the very opposite of what was needed to do to keep the last fuel cell running.
In effect, Liebergot's request was a vote of no confidence in the service module, and if the service module couldn't be relied on, the mission was in deep trouble. "We want to save the surge tank which we will need for entry," Liebergot prompted. The implication immediately sank in. "Okay, I'm with you. I'm with you," said Kranz resignedly, and he ordered the crew to isolate the surge tank.
For a few minutes more, Liebergot and his backroom guys fought the good fight to keep the remaining fuel cell on line, but it was looking grim. Without the fuel cell, he was going to have to power down even more command module systems in order to keep the most essential system running: the guidance system. The guidance system mainly comprised the onboard computer and a gyroscope-based inertial measurement system that kept track of which way the spacecraft was pointing. Without it, the crew wouldn't be able to navigate in space. But turning off nearly everything else in the command module was going to make it a pretty inhospitable place for the astronauts.
"You'd better think about getting into the LM," Liebergot told Kranz. It was now about 45 minutes since the explosion, and Liebergot's backroom team estimated that at the oxygen supply's current rate of decay, they would lose the last fuel cell in less than 2 hours. "That's the end right there," said Liebergot.
Kranz called Bob Heselmeyer on his loop. Heselmeyer sat two consoles over from Liebergot, and his job title was TELMU, which stood for Telemetery, Environmental, eLectrical, and extravehicular Mobility Unit. What that mouthful boils down to is that the TELMU was the equivalent of the EECOM for the lunar module, with the added responsibility of monitoring the astronaut's spacesuits. Like Liebergot, Heselmeyer had a posse of backroom guys--Bob Legler, Bill Reeves, Fred Frere and Hershel Perkins--and Kranz was about to hand them all a job. "I want you to get some guys figuring out minimum power in the LM to sustain life," Kranz ordered Heselmeyer.
It doesn't sound like a tall order--the lunar module had big, charged, batteries and full oxygen tanks all designed to last the duration of Apollo 13's lunar excursion, some 33 hours on the surface--so it should have been a simple matter of hopping into the Aquarius, flipping a few switches to turn on the power and getting the life-support system running, right?
Unfortunately, spaceships don't work like that. They have complicated, interdependent, systems that have to be turned on in just the right sequence as dictated by lengthy checklists. Miss a step and you can do irreparable damage.
What Follows Is A Little Known story, even to many involved in the Apollo 13 mission. While they have been complimented on rapidly getting the lunar module into lifeboat mode, stretching its resources to keep the crew alive for the journey back to Earth, few realize the lunar module controllers first had to overcome a basic problem: how to get the lunar module to turn on at all. Over the last 35 years, the incredible efforts of the lunar module flight controllers have been somewhat overlooked, ironically because the Aquarius performed so well. It did everything asked of it, whether designed to or not. So the attention has focused on the titanic struggle over the crippled Odyssey. But without the lunar module controllers' dedication, foresight, and years of work, Lovell, Haise, and Swigert wouldn't have had a chance.
A fundamental problem stood in the way of getting the lunar module on line. Call it the step-zero problem. They couldn't even turn on the first piece of equipment in the lifeboat checklist because of the way the Aquarius had been designed to handle the coast between the Earth and the Moon.
Remember that for most of this coast, the lunar module and the command and service module were docked, connected by a narrow transfer tunnel, with almost everything on the lunar module turned off to save power. A number of critical systems in the lunar module were protected from freezing by thermostatically controlled heaters. During the coast, these heaters were powered via two umbilicals from the command module, which in turn got its power from the service module.
Within the Odyssey, the umbilicals were connected to a power distribution switch that shifted the lunar module between drawing power from the Odyssey and drawing power from its own batteries, the bulk of which were located in the descent stage. Here was the hitch. The distribution switch itself needed electricity to operate, which the Odyssey could no longer supply.
With the last fuel cell running out of oxygen, the astronauts needed another way to get the lunar modules batteries on line, fast.
The lunar module controllers were already on the case when Kranz's order came through. Back in the staff support room, the lunar module consoles were right beside the EECOM's support controllers' consoles, separated by a paper strip chart that recorded the activity of the lunar module heaters. From the start of the crisis, they had front-row seats as Brown, Bliss, and Sheaks tried to save the command and service module with Liebergot. It hadn't been long before Brown turned to the lunar module controllers and said, "I'll bet anything that oxygen tank blew up," remembers lunar module controller Legler. "Bill Reeves and I put a lot of stock in what Dick Brown said, and if that was true, the CSM was going to be out of power before long and we were going to have to use the LM as a lifeboat."
Looking at their strip chart, Legler and Reeves could see the lunar module heater activity had flatlined--meaning the electrical bus in the Odyssey that was connected to the umbilicals was no longer supplying power to the Aquarius. "We had lost power to the switch that was used to transfer power from the LM descent batteries. So they would have been unable to turn on the LM," says Legler.
The large batteries in the descent stage were essential to powering up most of the lunar module's systems. They were connected to the lunar module's power distribution system via relays--relays that required power to operate, power that was no longer available from the command module. Fortunately, smaller batteries in the lunar module's ascent stage could be tapped independently of the switch in the Odyssey--but they could power only some systems for a limited amount of time. In order to get systems such as life support and the computer running, the ascent batteries had to be connected to the power distribution system, energizing the relays and so allowing the descent batteries to be brought on line.
Nobody had ever planned for this situation. Legler and Reeves began working out a set of ad hoc procedures--step-by-step, switch-by-switch instructions for the astronauts--that would coax some power through the maze of circuits in the Aquarius from the ascent batteries to the relays. Working from wiring and equipment diagrams of the lunar module, it took them about 30 minutes from the time of Brown's warning about the state of the command module to finish the list of instructions. The final list involved about "10 to 15" switch throws and circuit breaker pulls for the crew, remembers Legler. Once the relays had electricity, the crew could switch over from the Odyssey's now-dead umbilicals and start powering up the lunar module's life support systems in lifeboat mode, an even more complicated process.
Fortunately, somebody had already been working on that problem for months.
A Year Earlier, in the run-up to the Apollo 10 mission, the flight controllers and astronauts had been thrown a curveball during a simulation. "The simulation guys failed those fuel cells at almost the same spot," as when Apollo 13's oxygen tank exploded in real life, remembers James ("Jim") Hannigan, the lunar module branch chief, "It was uncanny."
Legler had been present for the Apollo 10 simulation when the lunar module was suddenly in demand as a lifeboat. While some lifeboat procedures had already been worked out for earlier missions, none addressed having to use the lunar module as a lifeboat with a damaged command module attached. Although Legler called in reinforcements from among the other lunar module flight controllers, they were unable to get the spacecraft powered up in time, and the Apollo 10 simulation had finished with a dead crew.
"Many people had discussed the use of the LM as lifeboat, but we found out in this sim," that exactly how to do it couldn't be worked out in real time, Legler says. At the time, the simulation was rejected as unrealistic, and it was soon forgotten by most. NASA "didn't consider that an authentic failure case," because it involved the simultaneous failure of so many systems, explains Hannigan.
But the simulation nagged at the lunar module controllers. They had been caught unprepared and a crew had died, albeit only virtually. "You lose a crew, even in a simulation, and it's doom," says Hannigan. He tasked his deputy, Donald Puddy, to form a team to come up with a set of lifeboat procedures that would work, even with a crippled command module in the mix.
"Bob Legler was one of the key guys," on that team, recalls Hannigan. As part of his work, Legler "figured out how to reverse the power flow, so it could go from the LM back to CSM," through the umbilicals, says Hannigan. "That had never been done. Nothing had been designed to do that." Reversing the power flow was a trick that would ultimately be critical to the final stages of Apollo 13's return to Earth.
For the next few months after the Apollo 10 simulation, even as Apollo 11 made the first lunar landing and Apollo 12 returned to the moon, Puddy's team worked on the procedures, looking at many different failure scenarios and coming up with solutions. Although the results hadn't yet been formally certified and incorporated into NASA's official procedures, the lunar module controllers quickly pulled them off the shelf after the Apollo 13 explosion. The crew had a copy of the official emergency lunar module activation checklist on board, but the controllers needed to cut the 30-minute procedure to the bare minimum.
The lunar module team's head start stood them in good stead. Although Liebergot and his team had initially estimated 2 hours of life left in the last fuel cell when Kranz had asked Heselmeyer and his team to start working up how to get life support running in the lunar module, the situation was rapidly worsening. By the time the crew actually got into the Aquarius and started turning it on, the backroom controllers estimated there were just 15 minutes of life left in the last fuel cell onboard the Odyssey.
This article is presented in three parts. For part two click here.