How a Misplaced Decimal Point Led to €188 Million in Unintended Gov't Largesse

Great Firewall of China causes national Web outage, while Gmail and other Google services go down

4 min read

How a Misplaced Decimal Point Led to €188 Million in Unintended Gov't Largesse
Bjarn Kindler/Getty Images

IT Hiccups of the Week

Last week was an unusually busy week across the global landscape of IT-related snafus, snarls and peculiar system interruptions. For instance, last Wednesday, quick-drying cement from a nearby construction site accidentally flowed into the London Underground’s Victoria line signal control room, significantly disrupting Central London Tube service for the day. Then, on Thursday, a human error during system maintenance caused a power outage that took out the automated train signaling system for three of New York City’s Metro-North lines, stranding thousands of the city’s train commuters for a good part of the evening. In light of these two events, I decided to start this week’s IT Hiccups with a software-cum-human error that occurred late last year, but only lately has been explained.

In mid-December, the Amsterdam Heraldreported that Amsterdam’s tax office was trying to figure out how €188 million was mistakenly paid out in annual government rent subsidies to some 10 000 people instead of the expected €2 million or so. In some cases, people received as much as €34 000 in housing subsidies.

What made the error more disconcerting was that no one in Amsterdam’s tax office seemed to have noticed. The Amsterdam Herald quoted City alderman Pieter Hilhorst as saying, “How can it be that no alarms went off? ... It seems we’re able to pay out €188 million without realizing it.”

The investigation into the error ordered by Hilhorst recently disclosed that the software used by the Amsterdam government “calculates payments in cents rather than euros” and no one in the finance office seemed to have noticed the slight discrepancy. A story at Dutch News states that “all but €2.4m of the €188m in wrongful payments” has been recovered (while half of the remaining amount probably will never be paid back). Furthermore, says the Dutch News story, the city spent some €300 000 trying to understand and fix the situation. Other news reports state that the Amsterdam city council is putting more controls over its finance office to keep such an error from happening again.

It could have been worse: Amsterdam could have been launching a $125 million Mars Climate Orbiter and lost it because of a failure to convert from English to metric units.

China Suffers Web Outage It Blames on Hackers; Others Say it Was Self-inflicted

Last Tuesday, the New York Times and others reported that up to two-thirds of Internet traffic in China—text, audio, and video sent by hundreds of millions of people—was disrupted by what the Chinese government said was the work of hackers associated with the Falun Gong movement. The Times stated that, “The China Internet Network Information Center wrote on its official Weibo account that the outage was caused by a glitch in the Domain Name System servers that convert alphabetical website addresses into the numerical addresses of computers on the Internet. Instead of matching the names of popular Chinese sites with their proper addresses, the DNS servers instead redirected users to an address associated with the homepage of United States-based Dynamic Internet Technology.”

DIT, the Times states, “is best known for a software tool called Freegate that helps Internet users in China circumvent the government’s pervasive system of online censorship and filters.”

DIT denied any involvement in the outage, and said that it was more likely caused by a “misconfiguration” in China’s own Great Firewall Internet censorship program. DIT's contention was supported by Greatfire.org, which collects information pertaining to Internet censorship in China.

As of now, China is still claiming to be a hacking victim, although the government apparently is softening its accusations by saying it isn’t sure who is responsible.  

Gmail and other Services Experience Outage

On Friday, Gmail and many other Google online applications including Calendar, Talk, Drive, Docs, Sites, Groups, Voice and Google+ Hangouts went down at about 1410 EST and didn’t return until 1520 EST, Computerworld reported. Google, says Computerworld, stated that for about 25 minutes, “most” users of its online services were unable to access them, thereby potentially affecting hundreds of millions of users around the world.

 Google apologized for the outage, saying that “an internal system that generates configurations—essentially, information that tells other systems how to behave—encountered a software bug and generated an incorrect configuration. The incorrect configuration was sent to live services over the next 15 minutes, caused users’ requests for their data to be ignored, and those services, in turn, generated errors.”

On the same day, a different and pretty bizarre Google-related hiccup caused David S. Peck of Fresno, California, to receive thousands of no-subject, blank e-mails. According to this story at Time, “users who searched [in Google search on Friday for] ‘Gmail’ were led to a results page with a link that said ‘Email.’ Clicking that link created a new email with Peck’s address—dsp559@hotmail.com—already filled in.”

Tech Crunch, which first reported the story, has some interesting screenshots and other background information on the weird error.

Google, which fixed the problem by late Friday night, has apologized to Peck “for any inconvenience caused.”

Amsterdam Pays Out 100 Times More in Rent Subsidies Than Planned

Amsterdam Investigates Error that Causes €188 Million Benefits Overpayment

Software and Staff Blamed for Amsterdam Benefit Error

Amsterdam Error Caused by Software and Poor Staff Oversight

Two-Thirds of China's Internet Disrupted

China Blames Hackers for Internet Problems

China Internet Outage May be From Censorship Changes

Unclear What Caused China Internet Outage

Gmail, Other Google Services Experience Outage

Gmail and Other Services Go Down for Over an Hour

Google Apologizes and Explains Reasons for the Outage

Bizarre Google Bug Sends Unwanted E-mails to Hotmail Account

Google Glitch Sends Thousands of E-mails to One Man’s Hotmail Account

Of Other Interest …

Commuter Chaos as Quick-Drying Cement Fills Victoria Station Control Room

Human Error Blamed for Metro-North Train Delays

Auto Credit Cars Inadvertently Disabled by Software Problems

Telecommunications Services of Trinidad and Tobago Paychecks Delayed by Technical Error

Arkansas State Workers Receive Paychecks Early Due to Software Error

Russia’s Avia Center Refunds 800 Customers for Plane Ticket Glitch

Lloyds Banking Group Technical Issues Affect ATMs and Debit Cards Across UK

Apple Will Fix iOS 7 Random Reboot Issue

UK Screwfix.com Screw-up Gives Scrooge-Approved Bargains

Millions in Switzerland Charged Twice for Debit/Credit Card Purchases

Tech Problem Duplicates Visa Debit Payments at Bank of Ireland

Maine’s Unemployment System Payment Glitch Fixed

Software Issue Shuts Down Melbourne's Docklands Observation Wheel

Taiwan Demonstrators Protest Persistent Problems with eTag System

Illustration: Bjarn Kindler/Getty Images

The Conversation (0)