Aside from the billion or so 17-year brood cicadas which all seem to be singing directly outside my office window, it was very quiet last week in regard to IT-related snafus, problems and outages. We start off this week with the Royal Bank of Scotland, which can always be counted on to liven up a slow week.
This Time, RBS Mobile Banking App Fails
Two million customers of the UK government nationalized Royal Bank of Scotland and its subsidiaries, NatWest and the Bank of Ulster, were frustrated yet again by an IT problem. According to the BBC, customers were unable to log into their accounts through their mobile phone app for about two and half hours last Friday just before the long bank holiday started. RBS customers reported that the problems began around 7:15 a.m. London time. When they tried to access their bank accounts through the app, they received various error messages telling them that the app couldn't find an Internet connection even though other apps were working fine. The problem was fully cleared up by 1:00 p.m., the BBC stated.
This was the second IT failure for RBS in two months, and follows the disastrous bank IT system meltdown of last year which the banking group is still trying hard to put behind it. Just two weeks ago, RBS announced that it was going to spend an extra £450 million to fix its problem-plagued IT systems. The Financial Times quoted bank chairman Sir Phillip Hampton as saying that, “As the IT incidents over the past year have shown, building and maintaining a top-class infrastructure is fundamental. Our customers deserve banking services that work 100 per cent of the time.”
Friday's failure was embarrassing not only because of Sir Phillip’s comments, but because about a week ago RBS announced that it was eliminating another 1400 jobs in its retail banking sector as part of its move to encourage its 17 million customers to move to online and mobile banking. The latest gaffe may instead encourage RBS customers to decide to move to a rival's online and mobile banking app.
RBS offered up an apology to its customers, who probably aren't listening anymore.
Everyone Into the South Lake Tahoe Jury Pool
A few weeks ago, my Risk Factor colleague Will Jones wrote about how a software system upgrade/update error effectively excused 23 000 residents of Polk County, Florida from jury duty, which has led defense lawyers to claim that unrepresentative juries were being selected.
Well, not having a representative jury pool wasn’t a problem last week in South Lake Tahoe, California, which is east of Sacramento. According to the Tahoe Daily Tribune, a “computer glitch” caused an extra 300 people above the normal 100 typically called to show up for jury duty for a murder trial at the El Dorado Superior Court.
The Tribune reported that, “The [jury duty] system, which is designed to shut down and reboot nightly, failed to restart,” but no one at the Superior Court noticed until all the potential jurors started to showed up. Apparently, the first 100 potential jurors waiting in line were selected to be part of the jury pool. There's a lesson about being the early bird somewhere in there.
In a related case of erroneously being called to court, the BBC reported that 6 000 Stoke-on-Trent residents received letters instructing them to appear in court for failing to fully pay their council tax. While the residents were indeed behind in their payments, they were “not yet facing legal action,” the BBC said.
Council officials blamed the erroneous letters on a “technical error,” but apparently aren't falling over themselves to apologize for the mistake. According to the BBC, council officials said the error actually had a “beneficial impact” since some of the residents with unpaid bills hurried to pay them or made payment arrangements after receiving the letters, “which is good news for the council and the individual because they have avoided costs.”
Makes one wonder whether there will be another "technical error" the next time council taxes come due.
Multiple Failures Defeated Spokane County’s 911 System
Last November 26, the Spokane County, Washington, 911 emergency call center experienced an outage that lasted some 38 minutes. The emergency center handles calls requesting police or fire help within Spokane County, including within the cities of Spokane and Spokane Valley, the Spokane Spokesman-Review stated. At the time, it was unclear what caused the outage, which required operators to physically relocate to the emergency center’s backup location, which is 14 kilometers away, to resume taking 911 calls. No one was seriously affected by the outage.
A Spokesman-Review recently ran a story on the outage that caught my eye. According to the paper, the system that handled 911 emergency calls shut down, but the equipment that “that was supposed to automatically switch the calls to a backup system didn’t work. A second link also failed to transfer calls to the backup system. Emergency communications workers then tried to activate another backup system, but that effort also failed.” Once that happened, the emergency operators drove to the backup center where 911 service was quickly restored.
I’ve heard of a couple of back-up systems failing, but three? What’s the probability of that happening?
Emergency center operators had to work out of the backup center for 58 hours before service was restored in the county’s main emergency center and they could return.
Montreal’s Métro System Shuts Down for the Seventh Time Since Last July
And finally, riders of Montreal’s Métro System were frustrated again last Tuesday as the entire Métro system shut down during afternoon rush hour for 40 minutes, once more apparently from a computer problem, the Montreal Gazette reported. So far, this latest problem marks the fourth time that the Métro has suffered a system-wide outage this year, and brings the total to seven since last July. A software patch planned for installation last month in the main computer system to fix a known problem was supposed to keep such outages from happening, but it apparently hasn’t worked.
Also of interest…
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.