It was a slow week in the land of IT-related “ooftas.” We start off with one that could have been much worse if the timing had been different.
Southwest Airlines Computer Failure Cancels 57 Flights, Grounds Another 250
Last Friday night at about 8 p.m. PDT, Southwest Airlines, the largest U.S. domestic carrier, with 3400 flights daily, experienced a system-wide computer failure that grounded the airline’s entire fleet not already in the air, the AP reported. Full service was restored by 11 p.m. PDT, but not before the airline had to cancel 43 flights (originally reported as 50) and delay about 250 others mostly west of the Mississippi River. Flights in Minneapolis, Chicago, Phoenix, Denver and San Diego were said to be affected. The airline also had to cancel another 17 flights early Saturday morning because crews and equipment were in the wrong place.
The computer failure, a Southwest spokesperson told the AP, “impaired the airline's ability to do such things as conduct check-ins, print boarding passes and monitor the weight of each aircraft.” Planes on the taxiways were recalled to the terminals although planes in flight were unaffected. The airline was able to get its back-up system operational, although the system's performance was said to be “sluggish.” Southwest got its primary system back up and operating normally by early Saturday, but the airline indicated it still wasn’t exactly sure of the source of the problem.
Southwest said it “sincerely apologized” for the “airport technology issue.” Luckily, the failure hit late on a Friday night, instead of first thing Monday morning. Southwest also seemed to do a good job in being able to get its customers who were scheduled to be aboard the canceled flights rebooked: there was little grousing in the press from disgruntled Southwest passengers.
As you may remember, American Airlines had to cancel 970 flights and delay another 1068 when a computer problem hit mid-morning on a Tuesday and lasted for only 90 minutes more than Southwest’s snafu. However, American's problem—whose cause has, to my knowledge, never been disclosed—affected its back-up system as well.
American Airlines, along with United, had another computer-related system problem last Wednesday, but it was localized to Philadelphia International Airport. The AP reported that an airport spokesperson said there was a “connectivity issue” that “started early Wednesday morning in a computer system at a ticketing counter. It caused problems for several hours before the issue was resolved by 9 a.m.” CBS News, on the other hand, reported that United Airlines said the problem, which took out their computer systems and phones, was a power outage that began at about 4:45 a.m.
Regardless of the true cause, the problem caused United to cancel six flights and delay several others. American delayed flights, but reportedly did not cancel any.
No other airline at the airport reported having any problems.
Utah and Colorado Motor Vehicle Systems Have Problems
Last week saw the motor vehicle systems in both Utah and Colorado suffer outages. Last Tuesday, the Salt Lake Tribune reported that a “database error” crashed the state’s drivers license records system. The Tribune stated that the incident was the result of “a code-reading error involving a daily transfer of motor vehicle records from the state Tax Commission to a DPS [Department of Public Safety] database.”
The Utah DPS was able to get its back-up system working, which allowed the state’s law enforcement officers limited access to records, but all processing of license applications and renewals came to a halt. The system was fixed by late Tuesday night, and normal operations resumed by Wednesday morning.
On Saturday, Colorado's lone state motor vehicle office open on weekends was “unable to provide new vehicle registrations, renewals or title work” because of a crash of the Colorado State Motor Vehicle system, the Colorado Springs Gazette reported. The office, located in Colorado Springs, was still “able to address drivers licenses, recording and elections,” the paper said. A news release at the state’s website seemingly blamed the problem on a network issue. A network problem affected motor vehicle offices across the state last month as well. The state said it appreciated everyone’s patience and apologized for the inconvenience.
1900 New York City High School Graduates Receive Post-it Note Diplomas
Some 57 000 New York City high school seniors graduated last week. For an unlucky 1900 or so, instead of receiving high school diplomas as they accepted congratulations from their principals, they each received a Post-it Note with their “name scrawled in Sharpie,” the New York Times reported. The reason for the less than thrilling recognition of achievement was because of an error at McGraw-Hill, the company that was in charge of scanning in the results of state’s Regents exams.
There is a bit of a backstory here that needs explanation. As the Times story described, for this school year, New York City school officials placed into effect a new Regents exam grading approach. In the past, officials felt teachers were grading their own students’ exams too easily, so they decided that the completed exams should be scanned and sent randomly to other teachers in the city to grade.
However, the company responsible for the exam scanning, McGraw-Hill Education in Connecticut, had what it called “intermittent slowdowns” which delayed the return of the exams to teachers be graded. The Times said the scanning system “broke down.”
New York City was forced to hire extra teachers over this past weekend to try to complete all the exam grading before school ends on Wednesday of this week. The New York Daily News says that McGraw-Hill will have to pick up the $42 an hour tab the extra teachers are going to be paid.
New York City Comptroller John Liu, who is running for mayor, wants McGraw-Hill to pay back the US $3 million the city spent on the scanning contract with the company. Liu is said by the Times to be considering an audit of the contract. In a related story, the state of Indiana announced on Friday that it plans to sue McGraw-Hill for its problems with administering standardized tests in that state. The announcement came on the same day company President Ellen Haley was apologizing to the state legislature for them.
An apology to New York City’s graduating seniors didn’t seem to be forthcoming from the city’s Education Department. A spokesperson there tried to minimize the issue by telling the Times that the department had expected some “bumps” and that “the problem affected fewer than 3 percent of the roughly 57,000 seniors”, and anyway, “each year there was a relatively small number of students who received their scores, and their diplomas, after graduation ceremonies.”
In other words, those graduating seniors who didn’t get their diplomas on time should quit griping and just be happy admiring the framed graduation Post-it Notes adorning their living room walls.
Also of interest…
“Technical Flaw” Exposed 6 million Facebook Users’ Phone Numbers and Email Addresses
Industrial and Commercial Bank of China System Upgrade Goes Awry
“Obama/Black” Color Listing a Computer Error Urban Outfitters Say
Photo: Karen Bleier/AFP/Getty Images
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.