Last week was a relatively quiet week for IT uffdas, with the possible exception of subscribers to AT&T’s U-verse bundled service. A “small proportion” of them saw their service go away for three days.
AT&T's U-verse Multi-Day Server Complex Outage
Last Monday night, 21 January, subscribers to U-Verse, the bundled digital TV, high-speed Internet and voice service throughout the Southeastern United States, discovered that none of the services worked. They were unable to, among other things, make emergency calls. AT&T subscribers without the U-Verse bundle had service as usual, however. At first, the outage seemed to be of the annoying but short-lived variety, as subscribers told the Miami Herald that they lost their service at 9:30 P.M. local time and had it back at around 8:00 Tuesday morning.
AT&T encouraged the thought that the outage was a minor glitch, telling the Atlanta Journal-Constitution that it believed that the outage affected only some 6000 subscribers out of its 7.4 million customers spread across 22 states who have the service. AT&T explained that it was a server-related problem, and apologized “for any inconvenience to our customers.”
However, by later Tuesday afternoon, it became apparent that the outage was affecting roughly 75 000 (and likely more) subscribers to the U-verse bundling package. Reports started to come in revealing that the outage hit 14 states and spread as far west as California. AT&T still tried to put a positive spin on the outage, saying that the “issue currently affects less than 1 percent of our U-verse subscribers.” It added that it was working hard to fix the problem at its “server-complex”, but offered little further insight as to when the outage would be fixed or exactly what caused it.
By Wednesday, frustration, especially on the part of small businesses that depended on U-verse, was ratcheted up several notches as it became clear that many of them wouldn't have their service restored until Thursday. That was the case despite AT&T's assurances on Wednesday morning that, “U-verse service has been restored for the vast majority of our customers affected by the outage. We expect any remaining customer issues will be resolved this morning.”
The New York Times reported that AT&T finally gave a reason for the problem: a faulty software upgrade.
AT&T announced on Thursday afternoon that, “U-verse service has been restored for all customers affected by the outage. The software problem causing the issues was resolved by AT&T engineers early this morning. We are not pleased it took so long to fix the issue. AT&T will provide a credit to customers who were affected.”
The outage was a major embarrassment for AT&T, which had just boasted about the reliability of U-verse’s fiber optic and fiber-to-the-premises (FTTP) network in early January. An AT&T spokesperson bragged that U-verse didn’t suffer outages like those of cable and satellite systems.
Just to make AT&T’s week complete, equipment failure in Cleveland knocked out 911 emergency service as well as some landlines and 2G wireless service across northern Ohio, for about four hours on Friday; and a North Carolina hospital blamed a problem with AT&T Thursday for knocking its electronic health record system offline for seven hours.
New York Cabbies Lose Money Because of Wireless Connectivity Problem
The New York Daily News reported on Wednesday that a glitch in thousands of wireless communications devices installed in New York City taxis to allow riders to pay by credit or debit card. The malfunctioning systems, provided by Creative Mobile Technologies and connected to Sprint’s network, affected at least 2400 taxis’ card readers as well as their electronic fare meters, televisions, and navigation systems.
Taxi cab drivers were understandably angry about the glitch. Many pay US $120 or more per day to rent their cabs, and the malfunctioning meters rendered them unable to make any money (even with cash-paying customers) for nearly two days because of the glitch. Prospective taxi customers were none too happy either.
There was no word on whether Creative Mobile Technologies would be offering any compensation for the outage.
BATS Trading Error Dialed Back
About three weeks ago, BATS Global Markets, the third-largest U.S. stock exchange, announced that it had discovered during internal system audits two situations where “its computers allowed trades that violated [U.S.] rules intended to ensure all investors get the best prices for equities.” BATS stated that some 436,528 trades involving $420,000 were affected over four years.
But after having studied the suspected incorrect trades further, BATS presented dramatically revised numbers on Friday. According to the Wall Street Journal, BATS reported that there were only 12 000 bad trades involving $17 000.
When the errors were first announced, BATS CEO Joe Ratterman blamed it all on the complexity of the trading environment, which he in turn blamed on government regulators. I guess the complexity of market regulation wasn’t so bad after all.
Where's My W-2? National Grid Workers Still Unhappy With Payroll System
Last month, I wrote about Massachusetts Attorney General Martha Coakley warning National Grid—transmitter and distributor of electricity and natural gas to customers in New York, New Hampshire, Massachusetts and Rhode Island—to begin paying its employees correctly (including all of the overtime hours they worked in the aftermath of Hurricane Sandy) or else face a fine. One of the reasons for the lack of pay was that the National Grid went live with a new accounting system just before Sandy hit. The changes needed in order to account for National Grid workers performing overtime in amounts and in locations outside of the ordinary created payroll havoc.
Coakley did impose a US $270 000 fine against National Grid earlier this month for its failure to pay its employees in a timely manner. Now it may be the Federal government’s turn to fine the company. According to the Boston Herald, National Grid has indicated to its employees that it might not be able to distribute the W-2 annual wage and tax statements needed to file taxes by 31 January as required, again because of problems with its accounting system. Failure to do so can mean a fine of up to $50 per W-2 statement from the U.S. Internal Revenue Service unless the IRS grants a waiver.
The Herald story says National Grid is insisting publicly that it will meet the deadline, but according to the employee union, the company is telling workers something different. We’ll see what happens come this Thursday, by which time the W-2 statements need to be mailed out.
Photo: Eric Gay/AP Photo
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.