Automated to Death

The passengers and crew of Malaysia Airlines Flight 124 were just settling into their five-hour flight from Perth to Kuala Lumpur that late on the afternoon of 1 August 2005. Approximately 18 minutes into the flight, as the Boeing 777-200 series aircraft was climbing through 36 000 feet altitude on autopilot, the aircraft—suddenly and without warning—pitched to 18 degrees, nose up, and started to climb rapidly. As the plane passed 39 000 feet, the stall and overspeed warning indicators came on simultaneously—something that’s supposed to be impossible, and a situation the crew is not trained to handle.

At 41 000 feet, the command pilot disconnected the autopilot and lowered the airplane’s nose. The auto throttle then commanded an increase in thrust, and the craft plunged 4000 feet. The pilot countered by manually moving the throttles back to the idle position. The nose pitched up again, and the aircraft climbed 2000 feet before the pilot regained control.

The flight crew notified air-traffic control that they could not maintain altitude and requested to return to Perth. The crew and the 177 shaken but uninjured passengers safely returned to the ground.

The Australian Transport Safety Bureau investigation discovered that the air data inertial reference unit (ADIRU)—which provides air data and inertial reference data to several systems on the Boeing 777, including the primary flight control and autopilot flight director systems—had two faulty accelerometers. One had gone bad in 2001. The other failed as Flight 124 passed 36 571 feet.

The fault-tolerant ADIRU was designed to operate with a failed accelerometer (it has six). The redundant design of the ADIRU also meant that it wasn’t mandatory to replace the unit when an accelerometer failed.

However, when the second accelerometer failed, a latent software anomaly allowed inputs from the first faulty accelerometer to be used, resulting in the erroneous feed of acceleration information into the flight control systems. The anomaly, which lay hidden for a decade, wasn’t found in testing because the ADIRU’s designers had never considered that such an event might occur.

The Flight 124 crew had fallen prey to what psychologist Lisanne Bainbridge in the early 1980s identified as the ironies and paradoxes of automation. The irony, she said, is that the more advanced the automated system, the more crucial the contribution of the human operator becomes to the successful operation of the system. Bainbridge also discusses the paradoxes of automation, the main one being that the more reliable the automation, the less the human operator may be able to contribute to that success. Consequently, operators are increasingly left out of the loop, at least until something unexpected happens. Then the operators need to get involved quickly and flawlessly, says Raja Parasuraman, professor of psychology at George Mason University in Fairfax, Va., who has been studying the issue of increasingly reliable automation and how that affects human performance, and therefore overall system performance.

”There will always be a set of circumstances that was not expected, that the automation either was not designed to handle or other things that just cannot be predicted,” explains Parasuraman. So as system reliability approaches—but doesn’t quite reach—100 percent, ”the more difficult it is to detect the error and recover from it,” he says.

And when the human operator can’t detect the system’s error, the consequences can be tragic.

Photo: Win McNamee/Getty Images

In June of this year, a Washington Metropolitan Area Transit Authority (Metro) red line subway train operated by Jeanice McMillan rear-ended a stationary subway train outside Fort Totten station in northeast Washington, killing McMillan and eight others and injuring 80. The cause is still under investigation by the U.S. National Transportation Safety Board (NTSB), but it appears that a safety-signal system design anomaly was at fault, in which a spurious signal generated by a track circuit module transmitter mimicked a valid signal and bypassed the rails via an unintended signal path. The spurious signal was sensed by the module receiver, which resulted in the train not being detected when it stopped in the track circuit where the accident occurred. So the safety system thought the track was clear when it was not. When she saw the other train in her path, a surprised McMillan hit the emergency brake in an attempt to slow her train, which may have been traveling nearly 95 kilometers per hour (59 miles per hour), but it was too late.

To put this accident in perspective, however, it was only the second fatal crash involving Washington, D.C.’s Metro in its 33 years of operation. In 2008, customers took 215 million trips on the system. Not counting train-vehicle accidents, a total of 27 people were killed and 324 people were injured in train accidents in the United States in 2008. This compares with statistics from 1910, when W.L. Park, general superintendent of the Union Pacific Railroad, asserted that ”one human being is killed every hour, and one injured every 10 minutes.”

Not only has automation improved train safety, but travel by plane, ship, and automobile is safer too. According to Boeing, in 2000 the world’s commercial jet airlines carried approximately 1.09 billion people on 18 million flights and suffered only 20 fatal accidents. The NTSB estimates that traffic deaths in the United States may drop by 30 percent after electronic stability control becomes mandatory in 2012 for automobiles.

Charles Perrow, professor emeritus of sociology at Yale University and author of the landmark book Normal Accidents: Living With High-Risk Technologies (Princeton University Press, 1999), contends that ”productivity, successful launches, successful targeting, and so on, increase sharply with automation,” with the result being that ”system failures become more rare.”

One can see this in aviation. As automation has increased aircraft safety, the rarity of crashes has made it harder to find common causes for them, the NTSB says.

However, the underlying reason for this rarity, namely the ubiquity of increasingly reliable automation, is also becoming a concern for system designers and safety regulators alike, especially as systems become ever more complex. While designers are trying to automate as much as they can, complex interactions of hardware systems and their software end up causing surprising emergencies that the designers never considered—as on Flight 124—and which humans are often ill-equipped to deal with.

”The really hard things to automate or synthesize, we leave to the operator to do,” says Ericka Rovira, an assistant professor of engineering psychology at the U.S. Military Academy at West Point. That means people have to be alert and ready to act at the most crucial moments, even though the monotony of monitoring supposedly reliable systems can leave them figuratively or physically asleep at the wheel.

Photo: Steven Senne/AP Photo

That was the case in June 1995, when the 568-foot-long cruise ship Royal Majesty ran aground onto the sandy Rose and Crown Shoal about 10 miles east of Nantucket Island, off the coast of Massachusetts. Fifty-two minutes after leaving St. George’s, Bermuda, on its way to Boston, the Royal Majesty’s GPS antenna cable became detached from the GPS antenna. This placed the GPS in dead-reckoning mode, which does not take into consideration wind or sea changes. The degraded GPS continued to feed the ship’s autopilot. No one noticed the change in GPS status, even though the GPS position was supposed to be checked hourly against the Loran-C radio navigation system, which is accurate by roughly one-half to 1 nautical mile at sea, and a quarter-mile as a ship approaches shore. The Royal Majesty proceeded on autopilot for the next 34 hours until it hit the Rose and Crown Shoal.

Why hadn’t the watch officers noticed something was wrong? One major reason, the NTSB said, was that the ship’s watch officers had become overreliant on the automated features of the integrated bridge system.

Watch officers, who in less-automated times actively monitored the current environment and used this information to control the ship, are now relegated to passively monitoring the status and performance of the ship’s automated systems, the NTSB said. The previous flawless performance of the equipment also likely encouraged this overreliance. Checking the accuracy of the GPS system and autopilot perhaps seemed like a waste of time to the watch officers, like checking a watch against the Coordinated Universal Time clock every hour.

In many ways, operators are being asked to be omniscient systems administrators who are able to jump into the middle of a situation that a complex automated system can’t or wasn’t designed to handle, quickly diagnose the problem, and then find a satisfactory and safe solution. And if they don’t, the operators, not the system’s designers, often get the blame.

Adding another system to help detect the error and recover from it isn’t a straightforward solution either. In Flight 124, the fault-tolerant, redundant system design helped to mask the problem. In fact, such redundancy often merely serves to act as yet another layer that abstracts the human operator from the system’s operational control.

”In other words, the initial control loop is done by one system, and then you have a computer that is backing up that system, and another is backing up that one,” according to Parasuraman. ”Finally, you have to display some information to the operator, but the operator is now so far from the system and the complexity is so great that their developing a [mental] model of how to deal with something going wrong becomes very, very difficult.”

Economics also figures into the equation. The ADIRU in Flight 124’s Boeing 777 was designed to be fault-tolerant and redundant not only to increase safety but also to reduce operating costs by deferring maintenance.

”The assumption is that automation is not only going to make [what you are doing] safer but that it will make it more efficient,” says Martyn Thomas, a Fellow of the UK’s Royal Academy of Engineering. ”This creates a rather nasty feedback loop, which means that when adverse events become relatively rare, it is taken as an opportunity to deskill the people you’re employing or to reduce their number in order to reduce a cost.”

This erosion of skills in pilots was a major concern raised in the last decade as glass cockpits in aircraft became common [see IEEE Spectrum ’s article The Glass Cockpit, September 1995].

Peter Ladkin, a professor of computer networks and distributed systems at Bielefeld University, in Germany, is heavily involved in aircraft accident investigations and is a pilot himself. ”Line pilots are trained and told—and their procedures also say—to use the automation all the time. Many older pilots are really worried that when they get into difficulties, they aren’t going to [know how to] get out of them,” he says.

Photo: United Photos/Reuters

The crash in February of Turkish Airlines Flight 1951 just short of Amsterdam’s Schiphol International Airport, which killed 9 people and injured 86 others, raised this concern anew. As the aircraft passed through 1950 feet, the left radio altimeter failed and indicated an altitude of –8 feet, which it passed on to the autopilot, which in turn reduced engine power because it assumed the aircraft was in the final stages of approach. The pilots did not initially react to the warnings that something was wrong until it was too late to recover the aircraft.

”When we start removing active learning for the operator, the operator begins to overtrust the automation,” Rovira says. ”They’re not going back and gathering those data pieces that they need” to make an effective decision.

Another issue associated with overtrusting the automation is that it can encourage ”practical drift,” a term coined by Scott Snook in his book Friendly Fire: The Accidental Shootdown of U.S. Black Hawks over Northern Iraq (Princeton University Press, 2002). The phrase refers to the slow, steady uncoupling of practice from written procedures.

Illustration: Boswell/MCT/Landov

We see how that happened in the Royal Majesty incident, where the watch officers failed to follow established procedures. This was also the case in the October incident involving Northwest Airlines Flight 188 on its way to Minneapolis-St. Paul International Airport, which overshot the airport by 150 miles. The pilots claimed they were working on their laptops and lost track of the time and their location. The aircraft was on autopilot, which in normal circumstances leaves the pilots with little left to do other than monitor the controls.

 Again, when you are only a system’s monitor, especially for an automated system that rarely, if ever, fails, it is hard not to get fatigued or bored and start taking shortcuts.

The situation isn’t hopeless, however. For some time now, researchers have been working to address the ironies and paradoxes of automation. One new approach has been to address the issues from the human point of view instead of the point of view of the system.

”We draw a system’s boundary in the wrong place,” Thomas states. ”There is an assumption that the system boundary that the engineer should be interested in [sits] at the boundary of the sensors and actuators of the box that is being designed by the engineers. The humans who are interrelating with these systems are outside it. Whether they are operators, pilots, controllers, or clinicians, they are not part of the system.

”That is just wrong,” Thomas adds. ”The system’s designer, engineer, and overall architect all need to accept responsibility for the ways those people are going to act.”

Victor Riley, associate technical fellow in Boeing Flight Deck, Crew Operations, argues that there needs to be a two-way dialogue between the operator and the automated system.

”The operator-to-the-system part of the dialogue is more important than the system-to-the-operator part,” Riley says. ”People see what they expect to see, and what they expect to see is based on what they thought they told the system to do.”

Studies by Parasuraman, Rovira, and others have found that operators of highly reliable automated systems will often perform worse than if they were operating a lower-reliability system, which seems paradoxical.

Parasuraman explains that ”if you deliberately engineer anomalies into the automation, people rely less on it and will perform a little bit better in monitoring the system. For example, if the system is 90 percent reliable, operators will be better at picking up the 10 percent of the errors than if the system is 99 percent reliable.”

Rovira also says that operators need to be able to see how well the automation is working in a given context.

”The goal for us as designers is to provide an interface that allows a drill-down if the operator needs to query the system, in the event they have a different perspective of the decision than the automation has given them,” Rovira says. ”Or if not a drill-down, there should be some visibility or transparency right up front about what the underlying constraints or variables are that make this decision not totally reliable.”

Maybe one way to remind ourselves of the potential effects of the ironies and paradoxes of automation is to simply pull the plug.

”If we don’t want people to depend on automated systems, we need to turn them off sometimes,” Thomas observes. ”People, after all, are the backup systems, and they aren’t being exercised.”