A Bloomberg News story yesterday shed a bit more insight into what caused the uncontrolled electronic trading by the market making brokerage Knight Capital a few weeks ago. It seems a dormant legacy program was somehow "inadvertently reactivated", and then interfered with (or took over?) the firm's trading on 1 August, when a new software trading program Knight had installed began operation. "Once triggered on Aug. 1, the dormant system started multiplying stock trades by one thousand,” Bloomberg was told by two unnamed sources who were briefed on the matter.
Hmm, dead software becomes reanimated, takes over a computer system, and then runs amok. I think I've seen that movie somewhere.
Also, according to the sources, “Knight’s staff looked through eight sets of software before determining what happened.” Almost sounds like there was a graveyard full of dead software ready to be reanimated.
Unfortunately, the article doesn't say anything more about how the dormant software awakened and interposed itself when it came to executing trades that were supposed to be initiated by the new software Knight had installed. It also doesn’t say why Knight would keep “eight sets of software” apparently resident in its execution environment. We’ll probably have to wait until the SEC finishes its investigation to find out what actually happened as well as, presumably, some juicy details about Knight's software development and system testing practices.
Nevertheless, the so-called “Knight-mare glitch” (among others) has spurred regulators in Asia and Australia to “clamp down” on high frequency trading firms, the Financial Times reported this week. The regulators are “unveiling sweeping proposals that would require traders to have controls on their systems and test them annually to prevent market disruption,” the paper said. Regulators want “pre-trade” risk controls in place to keep “aberrant” trading from happening, as well as trading “kill switches” when the risk controls fail.
In related news, the FT also reported that the recent glitch at the Tokyo Stock Exchange was traced to a bad “router in its Tdex+ derivatives trading system.” For reasons not yet explained, a backup router failed to kick in. This is the second time this year that a TSE backup system did not kick in when it was needed.
A router problem also caused problems for several hours yesterday morning at California’s Department of Motor Vehicles (DMV). According to the Associated Press, a problem with a California state router caused the DMV’s computers to become disconnected from the state’s network from about 0800 to noon local time, no doubt exasperating and angering many customers. According to television station CBS Sacramento, the California DMV has apologized to its customers via Twitter 26 times alone since February for technical difficulties. The station has been investigating without much luck DMV outages that have apparently been taking the system down with some regularity since 2007.
By coincidence, several other California state agencies had computer networking problems yesterday as well, which the state blamed on an unexplained “circuit reconfiguration” issue. This unrelated issue, the AP reports, was also corrected by mid-day yesterday.
Finally, Manganese Bronze, the company that makes London’s familiar black taxi, announced this week that “it is delaying the release of its unaudited half-year results for the six months ended 30 June 2012 ... due to the need to restate prior years’ financial results because of accounting errors that have come to light.” The errors could probably be labeled computer-related errors, though, rather than accounting ones.
According to a company statement, in August 2010, a new integrated IT accounting system, which was installed to help manage the company’s “complex global supply chain,” missed some key transactions during the cut-over: “Due to a combination of system and procedural errors, a number of transactions relating to 2010 and 2011 and some residual balances from the previous system were not properly processed through the new IT system. This problem led to the over-statement of stock and under-statement of liabilities in the financial statements of previous years.” As a result, the company understated by £3.9 million its historical losses.
Manganese Bronze has been under heavy competitive pressure, or in its own words, “Trading in the first seven months of the year has been difficult and remains challenging with the Group continuing to trade at a loss.” Although it expects the situation to eventually improve, the current strong competition from Mercedes-Benz (which now provides nearly a third of the London taxi fleet) and some expected new competition from Nissan’s new low emission taxis may not bode well for the company’s future financial health. The FT reported that Manganese Bronze stock fell 34 percent on the news of the results restatement and now only has a market value of some £5 million.
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.
