The Rescue of the Chandra X-Ray Observatory

24 August 2011—When the science and engineering team that operates the Chandra X-Ray Observatory celebrated the space telescope’s 12th year in orbit on 23 July, it was with a sense of real accomplishment—and also a collective sigh of relief. Just a few weeks earlier, the NASA satellite suffered the kind of mysterious failure that had the team worried that its long-lived instrument was about to sign off. It also had the astrophysics community concerned that its chief tool for taking X-ray images of the universe might be lost.

The nearly 4.8-metric-ton telescope is capable of imaging the hottest objects in the universe. Notably, it has provided astonishing insights into the anatomy and physiology of black holes and their crucial role in galaxy formation and destruction.

"It lets us see the freshly made elements in stars that have just exploded," says Jonathan McDowell, an astrophysicist with the Chandra X-Ray Center , in Cambridge, Mass. "We can look at supernova remnants and see, oh, here’s a lump of iron the mass of the sun that’s just been created." (Project scientist Martin Weisskopf acknowledged the telescope’s accomplishments with a "top 10" list of Chandra’s scientific achievements.) Though more specialized X-ray telescopes have been fielded over the years, Chandra remains the most sensitive.

The first sign of trouble came on the afternoon of 6 July, when Chandra failed to respond to a routine call. Running through a checklist for such events, the team reduced the data rate used to download telemetry data from Chandra. "Once telemetry was flowing, we found the critical onboard components had switched to their backup units," explains Michael Juda, Chandra’s mission operations manager. (As with most big satellites, Chandra carries a nearly complete set of redundant components, including gyroscopes, heaters, and the power electronics for its solar array.) The switch meant that Chandra was now in "safe mode," a kind of holding pattern it enters when it detects a possible failure. In safe mode, the satellite shuts down its scientific instruments, points its solar panels toward the sun so that it continues to collect power, transfers control from its main onboard computer to a backup unit called the control processing electronics, or CPE, and awaits further instructions. It was the first time the satellite had done so in 11 years.

"We had three safe modes during our first seven months in orbit [in 1999 and 2000] due to operational errors—we were still learning to drive the satellite. But this was the first time the safe-mode information indicated there was a problem onboard the satellite itself," explains Harvey Tananbaum, who with astrophysicist Riccardo Giacconi proposed the telescope back in 1976. Tananbaum now serves as director of the Chandra X-Ray Center, based at the Smithsonian Astrophysical Observatory, which operates the telescope for NASA.

The next two and a half days were "nerve-wracking," Tananbaum says. Under normal conditions, the operations center communicates with Chandra once every 8 hours or so via the Deep Space Network, the NASA system of large ground antennas that allows communication with space probes throughout the solar system. Realizing they’d need to stay in closer contact with the satellite, the operations center staff asked other users of the network to turn over some bandwidth to Chandra. "Other projects were really good about giving up time," Tananbaum says. That generosity allowed his team to remain in near continuous contact with Chandra over the next several days.

Working through the night, staff members discovered that the safe mode had been triggered at 8:28 eastern daylight time on 6 July, when Chandra’s computer detected an unusually large change in its angular momentum. Tiny changes are okay: Based on readings from the gyroscopes and the aspect camera (a visible-light star tracker used to help point the satellite toward its scientific targets), Chandra’s flywheels routinely make small corrections to maintain its alignment, which can be perturbed by things like the solar wind, Earth’s magnetic field, and the difference in the pull of gravity across the satellite’s body. But what the computer showed was a momentum change more than 25 times as great as the telemetry indicated by the gyroscopes and the flywheels, also known as reaction wheels.

Over the next two days, Tananbaum says, there was "a lot of analysis going on, a lot of scratching of heads." There didn’t seem to be anything wrong with Chandra’s hardware, so the working hypothesis was that the computer’s data had become corrupted.

Illustration: Harvard-Smithsonian Center for Astrophysics/NASA

Click on image to enlarge.

Even as staff members attempted to reconstruct the failure, they had a second potential problem on their hands. Chandra orbits Earth in a highly elliptical path that at its farthest point stretches a third of the way to the moon. On the evening of 8 July, Chandra was due to reach its perigee, the low point in its orbit, when it is most susceptible to torques caused by Earth’s gravitational pull. The reaction wheels normally counteract the torques by slowing down or speeding up. However, the maximum rotation rate of the wheels is limited, so the thrusters can also be fired in brief pulses to help out, too. But the CPE, which now had control of the thrusters, was programmed to fire them in longer pulses than Chandra typically used. Although the thrusters were designed to handle the longer pulses, they had a tendency to heat up, so the team worried that the lifetime of the thrusters might be shortened. In the worst case, there was the question of whether the thrusters might overheat and explode, says Tananbaum.

To help solve these problems, the Chandra team turned to an attitude control expert from Northrop Grumman/TRW, the contractor that had built the telescope, and to a retired propulsion engineer who had helped design the spacecraft’s thrusters. They calculated that the thrusters would not, in fact, explode. More important, they also found a way to deal with the satellite’s excess momentum. Without having to fire the thrusters, they stopped the spacecraft’s slow spin and placed it in a yaw orientation, allowing Chandra to complete its perigee by midmorning of 9 July without incident.

At about the same time, the operations center team announced that it had pieced together why Chandra had entered safe mode. It turned out to be the result of, well, multitasking.

Here’s what happened: Due to the gravitational pull of the sun and the moon, the telescope’s orbit shifts over time, on a roughly 15-year cycle, so that rather than coming within about 10 000 kilometers of Earth’s surface at its perigee, Chandra now dips down to 4500 km. So, over the years, the gravitational forces on the telescope have been getting a tiny bit stronger with each orbit.

Something else happens during perigee: The extremely high level of radiation from Earth prevents the telescope from capturing scientific imagery as it loops around the planet. Rather than wasting that time, the operations center had programmed Chandra to conduct some routine maneuvers, like moving the gratings that intercept the X-ray photons reflected off the telescope’s mirrors. But performing these maneuvers can give the spacecraft a jolt, so the angular momentum monitor is temporarily switched off.

On 6 July, Chandra had just completed its perigee maneuvers as it routinely did, then activated its aspect camera. But because of the large gravitational torques on the satellite, the camera took much longer than usual to lock into position—6 minutes instead of just 10 to 20 seconds, Juda explains.

Three minutes after the aspect camera was activated, the momentum monitor was also turned back on. To determine the spacecraft’s momentum, the onboard computer would ordinarily take the initial reading from the momentum monitor, wait a few seconds, and then ask the monitor for a second reading; it would then calculate the spacecraft’s change in momentum by comparing the two values. But the aspect camera was still busy, so the computer waited until it was done. So instead of comparing the current momentum reading with one gathered a few seconds earlier, the computer compared it with the reading from 3 minutes earlier. By then the spacecraft’s momentum had changed quite a bit, but the software didn’t take into account the actual amount of time that had elapsed. The result of its calculation was therefore much too high, and the computer sent Chandra into safe mode.

The good news, says Juda, is that "the spacecraft and software functioned as designed. The ’failure’ was in our understanding of the impact of scheduling a maneuver and grating motion so close to perigee." The gravitational torques on Chandra will continue to increase for the next several years, so the operations team has revised its procedures to prohibit such maneuvers during that portion of the orbit. Chandra resumed normal operations on the afternoon of 11 July.

This episode is also instructive as "yet another example of the way in which modern space technology is both so dependent on software and so vulnerable to software failures," says McDowell. "Modern spacecraft are basically very fancy computers with a bit of hardware attached, which will occasionally send you into conniptions."

On the other hand, McDowell adds, nothing about this latest episode suggests Chandra is in decline, and in fact it could well function for years to come. "Nothing is broken, the solar arrays are still providing enough power, the thrusters started with 60 pounds of propellant, and we have 50 pounds left," notes Tananbaum. "We hope it will go to the 20-year mark and even further. Barring some gremlin or a horrible mistake, it’s all very possible."

And if you’re an astrophysicist interested in X-ray views of the universe, or you’re just fond of black holes and quasars, that’s a good thing, because in April, NASA and the European Space Agency pulled back from a collaborative effort to build a successor to Chandra, the proposed International X-ray Observatory.