It’s been another relatively quiet week on the IT glitch front. We start off, again, with news of errors involving a stock exchange—this time ones that have avoided detection for four years before being discovered.
BATS' Undiscovered Errors
According to Bloomberg News, BATS Global Markets, the third-largest U. S. stock exchange, announced on the 9th of January that it had discovered during internal system audits two situations where “its computers allowed trades that violated [U.S.] rules intended to ensure all investors get the best prices for equities.” In one case, a programming error in BATS’ Matching Engine that matches “orders for two Bats equity exchanges and an options venue allowed some trades to occur at prices inferior to the best available bid or offer.” In the second case, another programming error allowed short sales that violated trading rules. You can read the details in the BATS press release.
A story at the Financial Times said that the errors first occurred in October 2008 and weren’t discovered until about a week ago. Some 436,528 trades were affected, and “customers lost more than $420,000 from inferior prices they received,” the FT reported. BATS CEO Joe Ratterman defended how BATS tests its systems, telling the FT that the affected trades were “anomalies,” and implying that, given the complexity of the exchange operating environment, no one should be surprised by these issues. At least he didn’t call his programmers “knuckleheads.”
Ratterman did try to deflect responsibility, placing the blame on government regulators for creating the complexity. “The regulatory environment is getting layered with additional guidelines and requirements and what you end up with is a fair amount of rules . . . as you layer on over time, it gets more complex,” Ratterman explained. Ratterman conveniently neglected to mention that if there weren’t so many technical issues cropping up (as well as unbridled growth in complex and risky financial engineering instruments few understand), maybe there wouldn’t be so many regulations being piled on the exchanges.
While in artful dodger mode, Ratterman added that customers hadn't complained about being disadvantaged, as if that is an excuse. That said, at least Ratterman told the FT that BATS wants to compensate customers affected by the programming errors.
BATS is understandably defensive about this latest issue—even though it was small from a financial standpoint—given the major hit the exchange took to its reputation last March when it botched its own IPO. That screw-up helped Standards & Poor's decide to assign BATS a BB- corporate credit rating late last year; the S&P in its announcement said it believed “that BATS is highly vulnerable to operational risk.”
Forewarned is forearmed.
RIM Takes a Hit
Speaking of reputation hits, the last thing that RIM wanted last week was being associated with yet another service outage, even if it was relatively minor. According to a Wall Street Journal report, an apparent Vodafone U.K. router error caused e-mail and instant-messaging services throughout Europe, the Middle East and Africa to be affected for about five hours on Friday. Vodafone U.K. BlackBerry customers were the worst hit. Voice and text messages were not affected, however. Vodafone said in an emailed statement that, “We apologize to customers for any inconvenience caused.”
RIM is hoping that its new BlackBerry 10, scheduled for launch by the end of the month, will turn around its fortunes. With that in mind, it certainly doesn’t need any visits from ghosts of problems past.
NAV Canada Regrets the Inconvenience
Finally, last Thursday evening, flights into and out of Toronto’s Pearson International Airport were delayed and some were canceled when a flight planning system computer belonging to NAV Canada, the country's civil air navigation services provider, crashed.
Everything was back to normal by 0300 local time Friday morning, a Canadian Press story reported. NAV Canada said that it “regretted the inconvenience,” and although it wasn’t sure what caused the problem, it was “looking into the matter to make sure it never happens again.”
That's most reassuring, given that they don't know what caused the crash yet.
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.