Last week was an unusually busy week across the global landscape of IT-related snafus, snarls and peculiar system interruptions. For instance, last Wednesday, quick-drying cement from a nearby construction site accidentally flowed into the London Underground’s Victoria line signal control room, significantly disrupting Central London Tube service for the day. Then, on Thursday, a human error during system maintenance caused a power outage that took out the automated train signaling system for three of New York City’s Metro-North lines, stranding thousands of the city’s train commuters for a good part of the evening. In light of these two events, I decided to start this week’s IT Hiccups with a software-cum-human error that occurred late last year, but only lately has been explained.
In mid-December, the Amsterdam Heraldreported that Amsterdam’s tax office was trying to figure out how €188 million was mistakenly paid out in annual government rent subsidies to some 10 000 people instead of the expected €2 million or so. In some cases, people received as much as €34 000 in housing subsidies.
What made the error more disconcerting was that no one in Amsterdam’s tax office seemed to have noticed. The Amsterdam Herald quoted City alderman Pieter Hilhorst as saying, “How can it be that no alarms went off? ... It seems we’re able to pay out €188 million without realizing it.”
The investigation into the error ordered by Hilhorst recently disclosed that the software used by the Amsterdam government “calculates payments in cents rather than euros” and no one in the finance office seemed to have noticed the slight discrepancy. A story at Dutch News states that “all but €2.4m of the €188m in wrongful payments” has been recovered (while half of the remaining amount probably will never be paid back). Furthermore, says the Dutch News story, the city spent some €300 000 trying to understand and fix the situation. Other news reports state that the Amsterdam city council is putting more controls over its finance office to keep such an error from happening again.
It could have been worse: Amsterdam could have been launching a $125 million Mars Climate Orbiter and lost it because of a failure to convert from English to metric units.
China Suffers Web Outage It Blames on Hackers; Others Say it Was Self-inflicted
Last Tuesday, the New York Times and others reported that up to two-thirds of Internet traffic in China—text, audio, and video sent by hundreds of millions of people—was disrupted by what the Chinese government said was the work of hackers associated with the Falun Gong movement. The Times stated that, “The China Internet Network Information Center wrote on its official Weibo account that the outage was caused by a glitch in the Domain Name System servers that convert alphabetical website addresses into the numerical addresses of computers on the Internet. Instead of matching the names of popular Chinese sites with their proper addresses, the DNS servers instead redirected users to an address associated with the homepage of United States-based Dynamic Internet Technology.”
DIT, the Times states, “is best known for a software tool called Freegate that helps Internet users in China circumvent the government’s pervasive system of online censorship and filters.”
DIT denied any involvement in the outage, and said that it was more likely caused by a “misconfiguration” in China’s own Great Firewall Internet censorship program. DIT's contention was supported by Greatfire.org, which collects information pertaining to Internet censorship in China.
As of now, China is still claiming to be a hacking victim, although the government apparently is softening its accusations by saying it isn’t sure who is responsible.
Gmail and other Services Experience Outage
On Friday, Gmail and many other Google online applications including Calendar, Talk, Drive, Docs, Sites, Groups, Voice and Google+ Hangouts went down at about 1410 EST and didn’t return until 1520 EST, Computerworld reported. Google, says Computerworld, stated that for about 25 minutes, “most” users of its online services were unable to access them, thereby potentially affecting hundreds of millions of users around the world.
Google apologized for the outage, saying that “an internal system that generates configurations—essentially, information that tells other systems how to behave—encountered a software bug and generated an incorrect configuration. The incorrect configuration was sent to live services over the next 15 minutes, caused users’ requests for their data to be ignored, and those services, in turn, generated errors.”
On the same day, a different and pretty bizarre Google-related hiccup caused David S. Peck of Fresno, California, to receive thousands of no-subject, blank e-mails. According to this story at Time, “users who searched [in Google search on Friday for] ‘Gmail’ were led to a results page with a link that said ‘Email.’ Clicking that link created a new email with Peck’s address—email@example.com—already filled in.”
Tech Crunch, which first reported the story, has some interesting screenshots and other background information on the weird error.
Google, which fixed the problem by late Friday night, has apologized to Peck “for any inconvenience caused.”
Amsterdam Pays Out 100 Times More in Rent Subsidies Than Planned
Amsterdam Investigates Error that Causes €188 Million Benefits Overpayment
Software and Staff Blamed for Amsterdam Benefit Error
Amsterdam Error Caused by Software and Poor Staff Oversight
Two-Thirds of China's Internet Disrupted
China Blames Hackers for Internet Problems
China Internet Outage May be From Censorship Changes
Unclear What Caused China Internet Outage
Gmail, Other Google Services Experience Outage
Gmail and Other Services Go Down for Over an Hour
Google Apologizes and Explains Reasons for the Outage
Bizarre Google Bug Sends Unwanted E-mails to Hotmail Account
Google Glitch Sends Thousands of E-mails to One Man’s Hotmail Account
Of Other Interest …
Commuter Chaos as Quick-Drying Cement Fills Victoria Station Control Room
Human Error Blamed for Metro-North Train Delays
Auto Credit Cars Inadvertently Disabled by Software Problems
Telecommunications Services of Trinidad and Tobago Paychecks Delayed by Technical Error
Arkansas State Workers Receive Paychecks Early Due to Software Error
Russia’s Avia Center Refunds 800 Customers for Plane Ticket Glitch
Lloyds Banking Group Technical Issues Affect ATMs and Debit Cards Across UK
Apple Will Fix iOS 7 Random Reboot Issue
UK Screwfix.com Screw-up Gives Scrooge-Approved Bargains
Millions in Switzerland Charged Twice for Debit/Credit Card Purchases
Tech Problem Duplicates Visa Debit Payments at Bank of Ireland
Maine’s Unemployment System Payment Glitch Fixed
Software Issue Shuts Down Melbourne's Docklands Observation Wheel
Taiwan Demonstrators Protest Persistent Problems with eTag System
Illustration: Bjarn Kindler/Getty Images
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.