After looking back at the project failures chronicled in the Risk Factor for our recent set of interactive features “Lessons From a Decade of IT Failures,” I became intrigued by the formulaic apologies that organizations put out in their press releases when something untoward happens involving their IT systems. For instance, below are three typical IT mea culpas:
“We would like to apologize for the inconvenience this has caused.”
“We regret any inconvenience this may have caused patients.”
“We apologize for the inconvenience to those affected.”
The first apology came about as a result of the Nationwide, the UK’s largest building society, charging 704,426 customers’ debit card twice for the same transaction, which it blamed on a batch file that was doubly processed. The second apology was in response to the crash of Northern California’s Sutter’s Health $1 billion electronic health record system crashing for more than a day across seven of its major medical facilities because of a faulty system upgrade. The last apology was prompted by the shambolic mess that marked the rollout of California’s EDD unemployment system that affected at least 185,000 claimants, many of them for weeks.
Apparently, regardless of the impact or duration of an IT failure, it is viewed by the offending organization as being merely an inconvenience to those experiencing them. Ontario’s Social Services Minister Helena Jaczek went so far as to liken the months of havoc resulting from the botched rollout of a new provincial CN$240 million welfare and disability system as reminding her “of when I have my BlackBerry telling me that there are issues and I need an update. . . it is often very inconvenient.”
Hmms, not receiving a desperately needed disability or subsistence check equates to a Blackberry software update. Who knew?
Most apologetic public statements by corporate or government officials at least attempt to provide the pretense that executive management feels bad for the consequences of their organization’s IT oofta. However, there have been other times where perhaps not apologizing would have been a better strategy than the apology given. Below are a few of my favorite “best of the worst apologies” from the Risk Factor blog files which clearly indicate that the organization would have been a lot happier if it didn’t have to deal with those pesky customers or taxpayers.
We start off with two of the largest organizations in Ireland that worked overtime to antagonize their respective customers in the wake of billing system errors. The first is Irish Rail, which discovered that a software upgrade to its vending machines caused tickets to be issued to some 9,000 passengers without actually deducting the fare amounts from their debit cards for several days. On the Friday, a week after the discrepancy was discovered, Irish Rail decided to notify the affected customers by way of a press release on its website, which mentioned that it would be deducting the fare amounts due it beginning on the following Monday.
Irish Rail’s press release also stated, “We apologies [sic] for any inconvenience this fault causes customers,” which for many could include incurring hefty penalty charges for unauthorized overdrafts on their debit cards. When asked why it couldn’t wait a week so its customers could ensure that their accounts had the funds to cover the charges, Irish Rail responded it had every right to collect the money immediately and was going to do so. Unsurprisingly, the affected Irish Rail customers didn’t think much of the company’s apology.
Another “show me the money” demand disguised as an apology came from Eircom, the largest telecom provider in the Republic of Ireland. It, like Irish Rail, had a billing “system error” that did not directly debit some 30,000 customer bank accounts correctly, even though its customers’ bills indicated otherwise. Eircom deemed the incident “regrettable” and further stated that “it’s embarrassing and we're very sorry that it's happened.” However, Eircom was neither too embarrassed nor sorry enough to insist that although it planned to reimburse customers the failed direct debit fee charge of €18.45, customers would still have to pay all monies owed the telecom in their next billing cycle. Ireland’s telecom regulator was as unhappy with Eircom’s payment demand as its customers were, even more so because the utility also failed to inform it of the billing error.
“Teething issues” also featured prominently in several apologies for IT fouls ups. Take, for instance, EnergyAustralia, which claimed that on-going “teething problems” with its newly introduced accounting system were why 145,000 of its customers had not been billed for their electricity or gas usage on time, including 21,000 who had never received a bill from the utility. In the apology the company issued, it tried to downplay the extent of the foul-up by saying, “We are sorry to the small number of customers who haven't had the best experience and we're working round the clock to improve our service.” However, for some 12,500 EnergyAustralia customers, the clock spun around for more a year before their billing issues were finally corrected.
Automotive manufacturer McLaren also apologized that its MP4-12C $229,000 supercar was suffering from “teething problems.” Ron Dennis, the executive chairman of McLaren Automotive and McLaren Group, sent out a letter to customers that stated in part, “As you will have already heard from my staff, we are experiencing some early software bugs resulting in unnecessarily sensitive warning lights, battery drainage in certain conditions and IRIS [infotainment] performance issues. My team and the McLaren retailers are working with the pace and intensity that the McLaren brand demands to fully resolve these bugs rapidly and effectively to ensure that any inconvenience to you is kept to a minimum.” Dennis, however, tried to make up for the inconvenience by promising customers that he was going to give them “a pre-release copy of the new McLaren: The Wins coffee-table book." I wonder how many software bugs it would take to get a personally signed copy of the book.
Additionally, there were a couple of organizations that had to make so many apologies that customers just stopped listening to them, deciding to head for the exits instead. Take, for example, UK’s RBS Group which had a major system meltdown in the summer of 2012 caused by a routine software-update gone bad that kept millions of its bank customers from accessing their accounts for days, and some even for months. At the time, then Chairman Stephen Hester apologized, saying, “Our customers rely on us day in and day out to get things right. On this occasion we have let them down… Once again I am very sorry for the inconvenience.”
Various RBS Group spokespersons had to apologize several more times that summer as they promised everything would soon be made right, which quickly turned out not to be true. At the time, RBS promised to invest hundreds of millions of pounds into upgrading its IT systems to keep major disruptions from happening again.
However, RBS has suffered significant glitches since, including in December 2013 on Cyber Monday and once more in June of this year. Although after each incident RBS management stated that it was “sorry for the inconvenienced caused” and that the incident was “unacceptable,” tens of thousands of its discomforted customers have decided to do their banking elsewhere.
While RBS may have seen droves of customers desert it over its IT failures, it is nothing compared to Australia’s telecom company Vodafone which has seen millions of its customers leave because of the company’s persistent IT ooftas. The root of the problem can be traced to 2009, when Vodafone merged its network with rival Hutchinson’s “3” network. Not surprisingly, the merger’s objective of creating a high-quality, unified network across Australia wasn’t as easy or seamless as envisioned. Customer complaints about poor Vodafone service grew throughout 2010, but really came to a head when a network software upgrade in late 2010 didn’t work as expected. Instead of speeding up network traffic, the upgrade slowed it down. That problem took weeks to fix, angering legions of Vodafone customers.
Then a different, concurrent software issue caused additional problems across the network. Vodafone, which by now was being referred to in the press as “Vodafail,” had to apologize to its angry customers multiple times that the company was “truly sorry” for the continued “dropped calls, delayed SMS and voicemails, slow data speeds, inconsistent coverage, and long waits when you called us.” For more than 2 million Vodafone fed-up customers who left the company between 2010 and 2013, the company didn’t improve fast enough. Finally, after spending AU$3 billion to upgrade its networks and customer support, Vodafone Australia announced earlier this year that it had started adding customers again.
There was also an interesting apologetic non-apology that happened in my home state of Virginia. In the summer of 2010, a server problem at the Virginia Information Technologies Agency (VITA) knocked out the IT systems used by 27 of Virginia’s 89 state agencies for several days, and a number of agencies were affected for over a week. At the time, the state’s IT infrastructure was in the midst of a $2.3 billion upgrade which was a constant source of contention between Virginia and its contractor Northrup Grumman.
When the server problem was finally fixed, Northrop Grumman vice president Linda Mills put out the expected pabulum and said the company “deeply regrets the disruption and inconvenience this has caused state agencies and Virginia citizens.” However, Grumman’s “regrets” were immediately undercut by a company spokesperson who, when asked by a Richmond Times-Dispatch newspaper reporter whether Mills’ statement was an apology, declined to comment. Whatever little goodwill that was left in state government for Northrop Grumman quickly vanished.
In May of 2011, after an investigation into the outage, NG agreed to pay a $4.7 million fine for the outage, which is an apology of the best kind, in my opinion.
Our final apology was given through firmly clenched teeth. For years, Her Majesty's Revenue and Customs (HMRC) in the UK worked to upgrade (pdf) its troubled computer systems. HMRC promised that when complete, the new PAYE (pay as you earn) system would significantly reduce both over- and underpayments of taxes. When it was fully introduced in 2010, the HMRC announced that some 4.3 million UK taxpayers were going to receive letters stating that they had paid on average £400 too much in taxes between 2008 and April of 2010. Additionally, another 1.5 million would be receiving letters just before Christmas that they had paid on average £1,428 too little over the same period, and HMRC wanted its money now. Furthermore, the HMRC indicated that another 6 million taxpayers prior to 2008 were likely owned money for taxes paid previous to 2008, and another 1.7 million possibly owed more taxes. This group would be receiving letters soon, too.
The underlying reason for the millions of over-and under-payments was that taxpayers were being placed in an incorrect tax bracket for years because of errors in the new HMRC PAYE computer system database.
Needless to say, the UK public was not a happy bunch at the news. Fueling their unhappiness was the attitude of HMRC Permanent Secretary Dave Harnett, who stated in a radio interview there was no need for him or his department to apologize for the errors or demand for quick payment of owed taxes, because at least to him, the PAYE system was working as designed: “I'm not sure I see a need to apologise... We didn’t get it wrong.”
Politicians of both parties were appalled by that statement, calling Harnett out of touch and arrogant, especially in light of all the reported PAYE system foul-ups. Harnett was forced by senior Conservative party leaders to retract his statement the following day, saying that he was “deeply sorry that people are facing an unexpected bill.”
A few days later, however, HMRC CEO Dame Lesley Strathie made it very clear Harnett’s apology was really a non-apology when she insisted that HMRC staff made “no mistakes,” and any and all errors were due to taxpayer mistakes. Dame Strathie also said the critiques of HMRC's performance was unfair since it couldn't pick and choose which customers to serve—it had to deal with everyone—whether her government department liked it or not. That bit of insight didn’t go over well, either.
HMRC’s PAYE system continues to cause grief to UK taxpayers, and the HMRC is taking yet another crack at updating its computer systems. Unfortunately, UK taxpayers don’t have a choice of which tax department to use, like customers of RBS or Vodafone do with banks or telecom companies.
If you have some “worst IT foul-up apologies of all time” stories to add, please let me know.
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.