We were blessed with another impressive week of IT-related burps, belches and eructs. This time, stock market officials are reaching for the antacid.
Nasdaq Suffers Three-Hour Trading “Glitch”
Well, opinions vary about whether it was or wasn’t a big deal. Last Thursday, the Nasdaq suffered what the AP called a “mysterious trading glitch” that suspended trading on the exchange from 12:15 p.m. to 3:25 p.m. EDT. After trading resumed, the market closed up 39 points.
The trading suspension was the longest in Nasdaq history and was a major embarrassment for the exchange, which is still trying to recover from its Facebook IPO screw-up. The exchange blamed the problem on a “connectivity issue” involving its Securities Information Processor, which Reuters describes as being “the system that receives all traffic on quotes and orders for stocks on the exchange.” When the SIP doesn’t work, stock quotations cannot be disseminated.
Nasdaq Chief Executive Robert Greifeld has refused to discuss the cause of the problem in public. Greifeld did darkly hint, however, that the problems were someone else’s fault. The Guardian quoted him as saying, “I think where we have to get better is what I call defensive driving. Defensive driving means what do you do when another part of the ecosystem, another player, has some bad event that triggers something in your system?”
He then went on to say, “We spend a lot of time and effort where other things happen outside our control and how we respond to it.”
Greifeld’s statement immediately triggered further speculation that the other “player” was rival exchange NYSE Arca, which was known to have had connectivity issues with Nasdaq. Greifield refused, however, to elaborate further on his statements.
Today’s Wall Street Journal published a lengthy story that has shed more light on what happened—although why it happened is still being investigated. According to the WSJ, between 10:53 a.m. and 10:55 a.m. EDT Thursday, “NYSE Arca officials tried and failed to establish a connection with Nasdaq about 30 times, according to people familiar with the events of that day. Nasdaq, for its part, was having its own problems regarding its connectivity to Arca, the people said.”
The WSJ goes on to say that: “What remained unclear Sunday was how that connectivity problem—which Nasdaq officials privately have called ‘unprecedented’—could have had so catastrophic an effect on Nasdaq's systems that the exchange decided to stop trading in all Nasdaq-listed shares, causing ripples of shutdowns across the market and spreading confusion.”
The Journal said NYSE Arca went to a backup system, and after several failed attempts, finally re-established connection with Nasdaq at 11:17 a.m. However, once the two exchanges were reconnected, “Nasdaq's computers began to suffer from a capacity overload created by the multiple efforts to connect the two exchanges.”
As a result, other markets also started to report problems in receiving and sending quotes from Nasdaq. Officials at Nasdaq decided they had better pull the plug in order to figure out how to get back to a normal operating state, which they did at 12:15 p.m.
Many traders viewed the episode as a non-event, while other interested observers, like U.S. Securities and Exchange Commission Chairman Mary Jo White, were more concerned. Given the complexity of the systems involved, no one should be surprised to see more hiccups in the future. In Charles Perrow’s terminology, they are now just “normal accidents.”
The folks at Goldman Sachs and Everbright were probably very happy about the distraction created by Nasdaq’s difficulties. Last Tuesday, Goldman “accidentally sent thousands of orders for options contracts to exchanges operated by NYSE Euronext, Nasdaq OMX and the CBOE, after a systems upgrade that went awry. The faulty orders roiled options markets in the opening 17 minutes of the day’s trading and sparked reviews of the transactions,” the Financial Timesreported.
Bloomberg News reported that Goldman had placed four senior technology specialists on administrative leave because of the programming error, but Goldman declined to discuss why. Probably a smarter move than when Knight Capital Group CEO Thomas Joyce blamed “knuckleheads” in IT when a similar problem a year ago this month resulted in loss of US $440 million in about 45 minutes. Goldman's losses were expected to be less than US $100 million.
Knight Capital was sold last December to Getco Holdings Co.
Everbright Securities, the state-controlled Chinese brokerage, is also likely happy at the timing of Nasdaq’s and Goldman’s problems. On 16 August, a trading error traced to the brokerage significantly disrupted the Shanghai market. The error, which cost the brokerage US $31.7 million and the brokerage’s president his job, was blamed originally on a “fat finger” trade. However, a “computer system malfunction” was the real cause, the Financial Timesreported. Needless to say, the China Securities Regulatory Commission is investigating Everbright and says “severe punishments” might be in order.
Finally, a real fat finger incident hit the Tel Aviv Stock Exchange (TASE) yesterday. The Jerusalem Post reported that, “a trader from a TASE member intending to carry out a major transaction for a different company's stock accidentally typed in Israel Corporation, the third-largest company traded on the exchange. The disparity in prices cause the company's stock value did a nose-dive, from an opening value of NIS 1690 down to NIS 2.10,” or a 99.9 percent loss. The trader quickly realized his typo, and requested the transaction be canceled. However, by then, the error had already triggered a halt in the exchange’s trading.
Helsinki’s Automated Metro Trains Rough First Half Day
Last week, Helsinki's Metro tried out its three driverless Seimens-built trains for the first time. However, in a bit of irony, after a few hours, problems developed with the ventilation system in the trains' drivers cabins, and the trains had to be taken out of service. Drivers were aboard the automated trains for safety reasons. The Metro didn’t indicate whether the trains would have been pulled out of service (or the problem even detected) if they had been running in full automatic mode without drivers.
Indian Overseas Bank Back to Normal
The Hindu Timesreported on Friday that the Indian Overseas Bank announced that the problem with the bank’s central server had been finally fixed. For three days, hundreds of thousands of bank customers were unable to deposit checks or use the bank’s ATM network. A story at Business Standard said that the problem was related to annual system maintenance of the core banking system, which instead ended up creating what the bank said was a “complex technological malfunction.”
Of Other Interest….
Chrysler Delaying Launch of 2014 Cherokee Jeep Due to Transmission Software Issues
Network Issues Stop Marines from Using Unclassified Network
Tesco Pricing Glitch Lowers Price of Ice Cream by 88 Percent
Xerox Releases Scanner “Error” Software Patch
Photo: Seth Wenig/AP Photo
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.
