Risk Factor

Market maker Knight Capital Group announced this week that it is taking steps to improve its operational risk management after an electronic trading glitch in August cost it $440 million in about 45 minutes. It initially blamed “old dormant software.” The losses forced the firm sell more than 70 percent of its preferred stock for $400 million in order to stay in business.

The Wall Street Journal reported that Knight’s CEO Thomas Joyce will be appointing a chief risk officer to oversee both market credit and operational risk issues. In addition, Joyce said that the firm hired IBM in late August to investigate its software development practices and processes and report its findings to the board later this autumn.

The WSJ article was more cloudy than clear when it came to the cause of the glitch. Joyce was reported as saying that the trading glitch was caused by an “undetected bug” in old software that allowed other software to generate trading orders unrestricted by volume caps.  The bug was apparently triggered when Knight’s new electronic trading software was installed improperly.

To say the least, this explanation is a bit bizarre. A latent bug in what once was operational software was triggered by a poor installation of new operational software? IBM will earn its money if its report explains how that happened.

Joyce, saying he was “deeply embarrassed” by the episode, then blamed it all on the company’s IT group. According to the Journal, Joyce said, “People do stupid things… A small team of people made a grievous mistake.” Joyce later went on to call them “knuckleheads.”

Maybe before top management blames internal IT "knuckleheads,' they should be looking inward. I wonder how much pressure Joyce—who has been an “unapologetic advocate” of automated trading—and his management team placed on the company's IT group to hurry up and get that new trading software installed. I also wonder how open Joyce would have been to his IT group telling him that they needed more testing time. I suspect not much.

The SEC has tried to keep rogue algorithms from creating havoc by mandating “circuit breakers” to halt sudden and inexplicable spikes in a stock’s price, but as this case showed, that isn't sufficient. In an attempt to forestall more regulatory oversight, the stock exchanges and the market markers are now looking at the possibility of installing “speed bumps” that “would stop all orders from one market maker at an exchange if the situation called for it, and also possibly across multiple exchanges,” a Money News article yesterday reported. In addition, they are discussing the creation of a “kill switch” to “shut down order flow” when a trading algorithm begins to go rogue.

Joyce is not enamored with market kill switches, however, since they might shut down all of a firm’s trading when only one trading area is being affected, he says. I find this an interesting position to take as there was no effective kill switch in the company’s own software to turn off the errant trading that almost sunk his firm. If there had been a market kill switch, he might have still been "deeply embarrassed" but the firm also may have lost a lot less money.

Joyce went on to say that he expects the debacle to eventually make the company smarter and stronger. Given that Knight Capital’s stock closed yesterday at $2.62, the lowest it has been since the bailout (and down from $10.33 before the glitch), investors seem to think it won't be soon.

General Motors confirmed over the weekend that it would be temporarily halting the U. S. sales of eight of its 2013 model year cars including the Buick Verano sedan, Cadillac XTS and ATS sports sedans, Chevrolet Cruze, Equinox, and GMC Terrain crossovers, and the Volt hybrid because it discovered a software issue in its popular OnStar in-vehicle security, communications, and diagnostics system. According to Reuters, when an OnStar equipped vehicle is involved in a collision where the airbags do not deploy, the OnStar system fails to send out the requisite alert to an OnStar operator triggering a call to the vehicle to check whether the driver and passengers are okay or need assistance. OnStar does work as designed when a collision involving the deployment of airbags occurs, Reuters says.

Approximately 60 000 vehicles, most of which are already in dealer lots, are affected. Each car will require about 30 minutes of the dealer’s time to get the software fixed; GM expects the issue to be fully taken care of in about a week.  Because problems with automotive electronics are becoming the biggest complaint of new car buyers, it isn’t surprising that GM would halt the sales of its new model cars for a week or so. Five years ago, I doubt GM would have taken such an action.

GM also announced last week that it “is hiring software developers, project managers, database experts, business analysts and other information technology professionals to staff the first of several new IT Innovation Centers in the United States. The centers are intended to drive breakthrough ideas into GM vehicles and business processes globally.” The hiring, which may reach 10 000 according to a story in ComputerWorld, is part of a GM plan to “rebalance [its] employment model over the next three years so that the majority of [its] IT work is done by GM employees focused on extending new capabilities that further enable [the] business,” the company said.

In other words, GM, which currently outsources 90 percent of its IT work, is starting the process to bring that work back inside its corporate structure. The automaker began outsourcing its IT operations to EDS (and others) beginning in the late 1980s. GM originally bought EDS for US $2.55 billion in 1984; HP then acquired it from GM in 2008 for $13.8 billion.  Early this year, HP wrote off $8 billion against its EDS acquisition, acknowledging that it was pretty much the failure it was predicted to end up being.

The first GM innovation center will be located in Austin, Texas, where the company is looking to immediately hire 500 IT folks. Why Austin? GM says that is where the skills it requires reside. It is reported that GM is talking with several other cities about opening innovation centers, but won't reveal which cities these are.

—On 4 September, the New York Times reported that members of a hacking group known as AntiSec posted information online that it says is evidence that the government regularly uses cellular handsets as tracking devices. The file AntiSec uploaded contains a million unique device identifiers (UDIDs) for Apple iPhone, iPad, and iPod Touch devices, plus phone numbers and other personal data on the owners of these devices. AntiSec, a joining of forces between the hacker collectives known as Anonymous and LulzSec, says it obtained the information—which it claims is but a small sample of the 12 million UDIDs it has in its possession—by hacking into the computer of an FBI agent who is a member of the bureau’s Cyber Action Team.

The FBI quickly responded with a statement saying that “At this time there is no evidence indicating that an FBI laptop was compromised or that the FBI either sought or obtained this data”—which is an extremely artful way of avoiding making an admission or denial. Apple immediately chimed in with a denial that it has been helping the government spy on its customers. Security experts said that the released information wouldn’t necessarily put the Apple customers at risk. But according to the Times article, a security researcher from New Zealand showed last year that the 40-character UDIDs, in combination with other data, could be used to discover the device owners’ user names, e-mail, addresses, and Facebook profiles, and to track their locations. Still, despite the FBI’s and Apple’s protestations that they haven’t the foggiest notion of how this could have happened, the data has been shown to be legitimate. 

—Feeling confident that your laptop secured by a fingerprint reader will discourage thieves from taking your machine—or at least keep them from gaining access to sensitive data? Ars Technica reported on 4 September that Elcomsoft, a Russian developer of password-cracking software, has pinpointed a weakness in fingerprint reading software used by Dell, Sony, IBM/Lenovo, and 13 other computer makers. It turns out that the software in question, UPEK Protector Suite, makes computers less secure than if they didn’t require a finger swipe. When the software is activated, it automatically writes Windows account passwords to a registry and encrypts them with a relatively weak key. According to an advisory issued by Elcomsoft, a hacker with physical access to a laptop running the UPEK software could acquire passwords to all user accounts on a machine in a matter of minutes. By contrast, a machine not running the fingerprint-reading software leaves hackers with access only to one-way password hashes; if they’re based on a strong password, brute force cracking could take years.

—Ars Technica reports that the U.S. Department of Homeland Security has issued a notice to power utilities, railroad companies, and other large industrial firms there, warning them of a feature/flaw in a widely used line of mission-critical network routers. The GarrettCom routers, hardened against dust and extremes in temperature and moisture, each contain a so-called “factory account” with a default password; anyone able to figure out the password would have greatly enhanced access and control privileges. DHS is concerned that even a user authenticated as “guest”—in a worst-case scenario, a terrorist or a disgruntled former employee—could have the power to sabotage a power plant or a rail system. The vulnerability was discovered by Justin W. Clarke, a self-schooled expert in industrial control system security. Clarke, who told Ars Technica that he bought one of the routers on eBay for US $12 and noticed the undocumented account during his analysis of the way the router works, found the same type of account with a default password in switches made by GarrettCom rival RuggedCom. A cursory search turned up nine such devices connected to the Internet with U.S.-based IP addresses.

The U.S. Federal Aviation Administration’s En Route Automation Modernization (ERAM) program, which started ten years ago at an estimated cost of $2.1 billion, was scheduled to go into full operations last year. However, the modernization effort ran into major software and system difficulties (pdf) as outlined by the Department of Transportation Inspector General Calvin Scovel in late 2010. To address the myriad of problems, the FAA decided by early 2011 to add another US $300 million and three years to the modernization effort. This amount, and schedule, change was at the low end of the range the IG said the effort might take; Scovel estimated it might need three additional years and another $200 million more than what the FAA predicted.  

With this difference in mind, I was struck by something in a recent Bloomberg News article on the status of the Federal Aviation Administration’s (FAA) En Route Automation Modernization (ERAM) program that was published just before the holiday weekend and thereby didn’t garner much notice. ERAM is the replacement for the 45-year-old En Route Host computer and backup system (pdf) used at 20 FAA Air Route Traffic Control Centers across the U.S. and is, the FAA states with an odd mix of literary flourish and anacronyms, “the heart of the Next Generation Air Transportation System (NextGen) and the pulse of the National Airspace System (NAS).”

In the article, Michael Huerta, acting chief of the FAA, says that ERAM is now on schedule and budget and that he feels “very good about where we are.”  Additionally, Jim Ullmann, regional vice president for the National Air Traffic Controllers Association (NATCA) union, which has been critical of ERAM in the past, now supports Huerta’s position.

That’s great news, right?

Yet, there was also this statement in the article: “Even after Huerta’s assessment, Calvin Scovel ... is sticking with his view expressed last October that there may be more cost overruns and delays, his spokesman, David Wonnenberg, said in an interview.”

Huh?

Last October, testifying before the House Subcommittee on Aviation, Scovel outlined then-existing problems with ERAM  (pdf) that, if the FAA's and NATCA’s current assessments are to be believed, have been generally addressed. So why is Scovel still holding to his previous pessimistic opinion?

One clue might be found in an April 2012 IG report on the risks to NextGen (pdf) that notes that Scovel has been performing a new audit of ERAM, to be published in the next few months. Perhaps Scovel's next depiction of the ERAM program won't be as pretty as the one the FAA and NATCA have been painting.

Checking the U. S. government IT dashboard, the Transportation Department's CIO shows ERAM has made significant progress since the beginning of the year, but, as of August 31st, he still rates ERAM as a medium risk for the fifth month in a row in both cost and schedule (it was high risk at the end of 2011). That's surprisingly, given the FAA chief’s comments, which would lead you expect at least a better rating of moderately low risk.

Scouting around the web for more information, I came across this blog by an anonymous veteran En Route Center controller by the name of George who, at least as of June, was claiming that from his personal experience ERAM was still buggy and barely ready for prime time.  

The blogger also made an interesting argument that “both the FAA and NATCA have too much political capital at stake with ERAM to be impartial with respect to the project.” He argues that in the past, NATCA could be critical of ERAM because the FAA didn’t desire controller input, but now that it is a partner with the FAA, NATCA has to be shall we say, more circumspect with its criticism. The Bloomberg article, similarly notes that with the settlement of a labor dispute three years ago, "[c]ontrollers and the FAA now have an 'unheard of' level of collaboration."

So, we apparently have a Goldilocks situation where the FAA and NATCA are all smiles about ERAM’s current state, the DOT IG is still frowning, and the DOT CIO and the anonymous NATCA controller are somewhere in the pursed-lips middle.

Maybe someone in the Risk Factor readership can shed some insight as to which bear is right?

Oscar Wilde once defined a cynic as someone who knows the price of everything but the value of nothing. Ulster Bank chief executive Jim Brown has put a cynical price on the inconvenience his 600 000 customers suffered from the bank's IT system outage that denied them access to their accounts for at least six weeks.

After inexplicably delaying for over a month, Ulster Bank finally announced its long promised and “simple” customer compensation plan (pdf) last Friday. Wilde would have surely questioned the bank's earnestness.

First, the plan states that personal customers “who visited and transacted at a branch during the period of the incident (19 June–18 July 2012) more frequently than in the equivalent period before the incident (19 May – 18 June 2012)” would receive a whopping €25 for their effort. Those who didn’t visit their branch more often during the time period (or spent their time calling the help lines instead) get nothing; the same is apparently true for those folks who aren’t customers of Ulster Bank but got caught up in the IT meltdown because their funds had to be processed by the bank’s systems.

The mid-July cutoff is as cynical as the compensation amount. It was only in late August that Ulster Bank management declared the bank’s operations were finally back to “business as usual,” and even then many customers were still having problems. The compensation period only covers the time when the IT system wasn’t working—the “inconvenience” of the aftermath doesn’t figure into the calculation. I suspect customers visited Ulster Bank branches a lot more after the 18th of July to try to straighten out their accounts than during the period when it was well known the bank’s IT systems were still on the fritz.

The bank went on to say, as before, that “all fees, charges and debit interest charged in error and correcting any credit interest owed as a result of this incident” will be refunded. However, it admitted it may take another six weeks for the refunds to finally show up on customer accounts.

In addition, personal and small and medium enterprise (SME) customers who incurred out of pocket expenses will get an inconvenience fee amounting up to 20 percent of their proven expenses up to €600 (that is, for a maximum payment of €120). Personal and SME customers will also see a minor 3-month interest rate boost to their accounts, and a likewise short waiver of some account fees.

Interestingly, the bank has been quiet on what it is doing to compensate large enterprise customers.

For customers worried about their credit being affected by the prolonged outage and not being able to pay their bills on time, the bank will help them get a credit report. However, you have to give your details to a bank staff member who will then send it to the Irish Credit Bureau. Customers should not expect to be compensated for their time spent doing this, or for the time needed straightening out their credit if it indeed got hosed.

And as a final "have a nice day," the bank closed the letter with a reminder that the additional payment of 20 percent on top of out of pocket expenses and the automatic one-off payment of €25 “are treated by the Revenue Commissioners as capital receipts and may, depending on your personal circumstances, be taxable.”

As I said, reaction hasn’t been exactly one of dancing in the streets.  Customers and politicians alike called the plan “underwhelming,” “too little too late,” “miserly,”  “an insult”—well, you get the picture.  CEO Brown, however, seems genuinely puzzled by the outcry. He defended the plan, telling the Independent, “I'm happy with it. I think it goes far enough.” He also indicated that the bank’s compensation expenses may reach €100 million, up from the original estimate of €35 million currently set aside. From his perspective, the bank is no doubt bending over backwards to put a fair value on its customers' troubles.

The total cost of the outage may go a little bit higher now that the consulting company PWC has been selected by Ulster Bank’s parent company RBS Group to investigate the cause of the outage and to ensure that all RBS Group banks (RBS and NatWest are the others) have contingency plans in place to deal with potential outages in the future.

Monday's Financial Times reported that on 22 August the U.K. government’s Financial Services Authority sent a letter to the chairman and boards of the nine largest banks and building societies asking them to “explicitly detail efforts” to avoid problems like that which hit the RBS Group, as well as to provide “the names of senior managers who could be held personally responsible if information technology systems go awry.”

There may well be more than a couple of U.K. bank CIO positions opening up soon. Interested? Make sure the job includes superb executive liability insurance.

Adding pressure on the banks, the FT noted, is the Parliament’s Treasury Select Committee, which will also be looking into the issue and wants to be convinced “that such a failure cannot happen again.”

I bet the banks are thinking that's a lot of government oversight for only 25 euros worth of customer inconvenience.

For the past several years I have covered the on-going payroll system problems at Australia’s Queensland Health Service which saw an estimated A$6.19 million (fixed price) IT project morph into one that will likely end up costing over A$530 million to get right. The debacle, which helped spur the defeat of Queensland’s long-ruling Labor government earlier this year, also unsurprisingly prompted an audit of Queensland’s governmental IT projects by the incoming LNP-led government.

This week, LNP’s Information Technology Minister, Ros Bates, announced that the audit has so far uncovered 997 governmental IT applications that should have been replaced long ago but haven't been.  In many cases, replacement plans for these obsolete systems don’t even exist.

In a story in the Brisbane Times, Bates compared the state of Queensland’s government IT to a “1972 Ford Falcon” clunker, in terms of the need for daily repair. However, many of its obsolete legacy systems are mission critical, and could “melt down” at any time with disastrous effects. The audit report estimates that getting  Queensland's IT back up to acceptable standards will cost at least A$3.6 billion and as much as A$6 billion. The challenge is going to be how to reach an agreeable level of acceptable IT risk since the government, already facing a nearly A$100 billion deficit, “cannot afford” the sums identified by the audit.

Bates says that as part of its efforts to solve the IT mess the government will “be going out to the market and looking for solutions from the market.” Well, she may want to first read Auditor General for Scotland Caroline Gardner’s audit (pdf), published last week. It found “significant weaknesses” in three major IT government projects that involved major IT systems upgrades (and market solutions) totaling some £130 million; each had business cases the audit termed, with typical UK understatement, of “variable quality.”

As reported by UK Press Association, the three projects reviewed were “the Crown Office and Procurator Fiscal Service (COPFS), Registers of Scotland (the body responsible for compiling and maintaining registers relating to property and other legal documents), and Disclosure Scotland, which provides information about the criminal record of those applying for certain jobs.”

According to the audit and the UKPA story, a £10 million project to improve COPFS case management was originally slated to start in August 2009 and to be completed by June 2011. However, warnings arose soon after the business case was approved that the project was much more complicated than estimated and therefore would cost significantly more than budgeted. So after spending £2.3 million but before anything of substance was created, the project was terminated, and all the costs written off.

The audit also reported that in 2004 the Registers of Scotland signed a ten year, £66 million agreement with telecom company BT to maintain and update its IT systems. However, for six years there was no formal peer review of the project's overall status. Now, with the contract's cost reaching £112 million (including having to write off two projects at a cost of £6.7 million) due in part to incorrect strategic assumptions in the original business case and subsequent poor contract management and oversight, the government has belatedly decided to terminate the contract 20 months early. How much the government will pay BT for early contract termination is still under negotiation.

In Disclosure Scotland’s case, its IT project was supposed to cost £31 million, running from June 2009 to May 2011. However, when it went live in February 2011, the “system experienced significant problems and did not perform as required,” meaning the old system had to be kept in operation. One reason for the difficulties was traced back to the business case which did not make it clear who in government was in charge of the program; the IT supplier viewed its client as being the government of Scotland, because that's who signed the contract, not Disclosure Scotland’s management team. This in turn helped create “different views of risks and [the] priorities” for managing them, which eventually led to system development issues. The project has cost £19 million so far, as payments to the supplier have been held up until the system is put right, which is planned to be by the end of this year. The legacy system it is meant to replace is expected to be finally shut down next year.

A story in the BBC summarized the findings of the Scottish IT audit in this way, saying that it found “ ‘weaknesses in financial control and progress reporting,’ claimed that ‘risk management was inadequate’ and the findings of independent reviews were ‘not always acted on.’ ”

All of which should serve as a warning to the government of Queensland as it tries to sort out its own IT messes.

Evasion techniques that let cybercrooks bypass companies' antivirus software, firewalls, intrusion prevention systems, and other first lines of defense are becoming more prevalent, says a CSO.com article citing a study released on 29 August. The study, from FireEye, a security vendor focused on advanced persistent threats, says that in the first half of 2012, the incidence of advanced malware successfully evading signature-based detection such as blacklisting technology and AV software was 225-percent as great as for the previous six-month period. One such evasion tactic, says FireEye, is the use of "throwaway" domains in spearphishing e-mails, in order to keep technologies that rely on domain reputation analysis from sniffing out the sender’s intentions. According to CSO.com, the number of domains used fewer than 10 times rose 45 percent from the second half of 2011. "These numbers make clear that cybercriminals are changing their malware more quickly, and reproducing malware and morphing it in an automated fashion," the report said.

On 29 August, Computerworld reported that hackers have added two new zero-day exploits that take advantage of Java vulnerabilities to Blackhole, a veritable Swiss Army knife of exploits. Blackhole is designed to try each of its malware tools until it finds one that will work against a particular computer. The head of research at security firm Websense said that by that morning, his team had found more than 100 unique domains serving the Java exploit. He predicted that numerous sites would be successfully attacked using these new exploits over the next few days. According to Computerworld, Michael Coates, director of security assurance at Mozilla, maker of the Firefox Web browser, is urging Firefox users to disable the browser's Java plug-in because Oracle has not issued fixes. Others, including the United States Computer Emergency Readiness Team, seconded Mozilla’s suggestion or recommended uninstalling Java entirely.

What is the global economic impact of cybercrime? A NetworkWorld article reports that U.S. government officials including the president have parroted reports pegging the figure at US $1 trillion. But in a recent ProPublica report, several security experts and analysts call that trillion-dollar-cybercrime estimate grossly inflated. They remind the public that any estimate from a security vendor should be taken with a grain of salt. It is understood that IT security firms stand to gain from an atmosphere where the perceived security risks and costs are greater. "I don't beat them up for it," Jason Healey, director of the Atlantic Council’s Cyber Statecraft Initiative, told NetworkWorld. "Experts have long had trouble agreeing on estimates that are within even two orders of magnitude of each other," says Healey. Why? For one, industry reports are not peer reviewed the way articles in academic and professional journals are. And even the most even-handed reports suffer from the fact that, as the authors of a cybercrime assessment done at the behest of the UK Ministry of Defense noted in the paper, "There are over 100 different sources of data on cybercrime, yet the available statistics are still insufficient and fragmented; they suffer from under- and over-reporting.”

SecurityWeek reported this week that Toyota has filed a lawsuit against an ex-contractor for sabotaging the company’s supplier network and downloading confidential information. According to Toyota’s court filing, Ibrahimshah Shahulhameed, who was working for the automaker at a facility in Kentucky, gained access to the company’s online supplier network after he was fired on 23 August; he spent that night downloading trade secrets and other proprietary information then sabotaging the network. Toyota’s computer security officials told SecurityWeek that they weren’t immediately sure how much damage Shahulhameed had done; his sabotage efforts were seemingly meant to cover his tracks. For a moment, Toyota officials were afraid that Shahulhameed, who SecurityWeek says is in the United States on an H-1B visa, would escape punishment for his actions. After he was released on $2500 bond, he told corporate investigators that he was planning on returning to his native India. But he subsequently agreed not to travel during the court proceedings.

Yesterday afternoon at about 14:20 CDT, United Continental Holdings suffered a “network outage” lasting about two hours that affected its reservation and online systems. At least 200 flights around the world were affected, United acknowledged. At one point, United requested that the U.S. Federal Aviation Administration place ground-stops “to prevent flights from taking off to some of its hub airports, including San Francisco, Newark and Houston,” the Chicago Sun-Times reported. United Continental Holdings is the parent of United Airlines and, since a May 2010 merger, Continental Airlines.

Per usual, there were long check-in lines as boarding passes had to be written out by hand and jammed reservation telephone lines as frustrated and often angry passengers tried to change their flights. United allowed, with some restrictions, those traveling yesterday to make changes without incurring a penalty once the system came back up.  

United has been struggling to regain customer trust since it botched the introduction of its “new” integrated reservation system in early March. This outage won’t help. Many news articles today speak of how poorly United communicated with its passengers as being a special sore point.

Last year, United suffered a similar “network connectivity issue,” the cause of which it never explained, at least as far as I can find out. According to the Sun-Times, “United said its IT department is reviewing the cause of the outage, and said it could not comment on whether the integration of the [reservation] systems caused the problem.” Don’t hold your breath waiting for additional details.

When last year’s outage hit, before the reservation system merger, only United Airline’s passengers were affected. As I noted then, “once the two airlines computer systems are fully merged, any similar glitch will likely have increased flight and passenger consequences.” You would think that the “world’s largest airline” would have better network back-up capability, especially given the experience of last year.

In a bit of irony, the Wall Street Journal ran a story yesterday on how airlines are pushing hard to implement the completely “self-service airport” where passengers won’t see an airline representative until they are greeted by the flight attendant on their plane. The story talked about airlines like Alaskan and American, who are introducing self-tagging of baggage as well as JetBlue, which became the first U.S. airline to introduce self-boarding gates at MaCarran Airport at Las Vegas.

Air transport communications and information technology providers like SITA claim the move to self-service technology is, “more about throughput with the resources you have than getting rid of humans,” while the airlines say it is to allow them to give more attention to passengers with questions. Yeah, right. Just like what happened at banks when they introduced ATMs and online banking technologies.

Of course, I do suppose there are opportunities for employment for at least some number of airline workers being displaced by the self-service technology who could be retrained to help fix it when it inevitably breaks down.

Still, it will be interesting to see what happens when a self-service airline has a “network outage.” Have you been in a large supermarket when the checkout scanners stop working?

Update: 30 August 2012

Yesterday afternoon, United announced in an email to major news organizations that, "A piece of communication equipment in one of our data centers failed and disabled communications with our airports and website. We have fully redundant systems and we are working with the manufacturers to determine why the backup equipment did not work as it was supposed to."

The failure of backup IT systems to take over for a primary system that isn't working correctly has been a recurring theme this year in the Risk Factor, especially in the IT-related failures plaguing the world's stock exchanges.

What I found interesting about the United disclosure of the cause of its outage is how uncharacteristic it is. In the past, United has been very reticent to talk about any of its computer problems. One reason for the disclosure seems to be that United wanted to make it absolutely clear that the problem wasn't with its Shares reservation system and that its troubles were the result of something outside of its control. If it had been a reservation system issue, both passengers and investors would without a doubt have severely punished the airline's reputation, each in their own way.

In the end, 580 flights were delayed and 9 were canceled because of the two hour outage.

The recent rash of “technical problems” hitting the world’s stock exchanges continues unabated. When Indonesia’s Stock Exchange (IDX) went to open yesterday morning, only 84 of the 114 listed security companies were able to connect to the exchange. As a result, the exchange decided to cancel its usual 15-minute pre-trading session and delay its opening for 30 minutes until 10:00 local time to diagnose and repair the problem which the Jakarta Globe reported was centered at the IDX’s main remote trading system.

The severity of the problem, however, led IDX to move trading to its revamped Disaster Recovery Center (DRC) which was built partly in response to previous reputation-tarnishing outages, but apparently that move did not totally solve the problem.  The exchange did open at 10:00 as planned, but by 10:15, the connection problem reoccurred, and the exchange was shut down. Trading later resumed at 13:00, but the exchange then unexpectedly was shut down again 30 minutes early at 15:30. The premature closure was later blamed on the aftermath of the earlier technical problems as traders weren’t getting timely stock price information.  

There were some doubts last night whether IDX would open as normal today, but it seems to be up and running without incident.

IDX trading volume ended yesterday down two-thirds of what would be a normal day. Trading volume was especially low given that the IDX had been closed for a portion of last week on account of the Eid al-Fitr Muslim holiday. While IDX apologized for the outage, it has not explained the exact cause of the “technical problems” other than to deny speculation that the problems were a result of it being hacked.  Not unexpectedly, many traders were unhappy with the exchange's problems, but so were many investors; some are reportedly contemplating a class action lawsuit over the loss of trading time.

In other exchange-related news, last Friday Japanese regulators sanctioned officials of the Japanese Stock Exchange for an outage that occurred earlier this month. A defective router in its Tdex+ derivatives trading system was the cause, but the real anger was that for a second time this year, the backup systems didn’t kick in as expected (or promised).  According to Bloomberg News, regulators said that the TSE did not check its systems thoroughly enough after the February outage. TSE’s CIO promised that this time it “will figure out a solution for the system trouble” and quickly.

 Outside consultants are going to be brought in to examine the system, the Wall Street Journal reported. The consultants will “inspect all of [the TSE’s servers and network devices as well as its emergency backup systems.”

The regulators indicated that they will be keeping a close eye on TSE’s efforts, especially since it and the Osaka Securities Exchange are planning to merge in January 2013. TSE management said it will be again docking senior executive pay some 30 percent for one to two months in light of the latest incident.

Then last Wednesday, the Financial Times reported that NASDAQ and other exchanges were forced to cancel trades in Peet’s Coffee and Tea “after erroneous orders triggered a steep rise in its share price in a matter of seconds.” The share prices for Peet’s Coffee and Tea jumped 5 percent on “unusually high volume” within two minutes of the opening bell.

The FT further reported that, “Citing guidelines, NASDAQ said it could not name the firm or firms from where the trading error might have occurred,” so the culprit and the exact cause of the error will go unrevealed, at least for the moment.

Also on Wednesday, the Dow Jones Newswire reported that NASDAQ is preparing a report for the U.S. Security and Exchange Commission (SEC) because of a “software problem that may have caused violations of short-selling rules.” The Dow Jones story states that the software, which ensures the SEC short-selling rules are being followed, was inadvertently deactivated as other changes were being made to NASDAQ's systems. The situation apparently lasted a week before anyone noticed. The error is now causing financial firms and traders to review their trades during the deactivation period for violations of the SEC rules.

In addition, the Dow Jones story reported that the NASDAQ "market-data feed called ITCH (pdf) experienced service interruptions Tuesday and Wednesday, the exchange said in notes to customers. Minutes after Wednesday's issue, a backup version of the feed kicked in. Nasdaq on Wednesday also experienced an issue with technology meant to streamline automated trading.” Again, details on the causes of the problems are lacking.

All this is just more fodder for discussion at the SEC roundtable next month which it is holding “to discuss ways to promote stability in markets that rely on highly automated systems.”

-A Computerworld article reports that on 21 August, Internet security firm Prolexic revealed that it has found vulnerabilities in the tools hackers use to launch distributed denial of service (DDoS) attacks. In a written online statement, the company, which specializes in providing protection against DDoS attacks targeting corporate networks, said that flaws in the command and control component of the Dirt Jumper DDoS toolkit that has been associated with recent DDoS attacks make it possible for "counter-attackers to obtain access to the Command and Control database backend, and potentially server-side files.” That level of access, says Prolexic, would allow a network operator to halt an attack in real time.

-Thinking of downloading an “unofficial” app for your cellphone or tablet? News from Security vendor Arxan, which makes tools for protecting apps from tampering, may give you pause. The company has released the details of a study reporting that more than 90 percent of the apps being sold at Apple’s App Store (and all of the top 100 apps originally found in Google’s Android app marketplace, Google Play) have been hacked by cybercriminals. The hacked versions, which are subsequently posted in various online outposts, contain modifications that, from the consumer’s standpoint, range from the seemingly benign (extrication of ads) to the nefarious (malware that could steal data or turn a device into a zombie used to attack other machines). An Arxan spokesman says that for an experienced hacker, reverse engineering an app is trivial. “Android Java apps can be easily and trivially decompiled back to source code. Native Android and iOS apps are relatively easy to reverse-engineer as well,” the Arxan study says.

-Saudi Aramco, Saudi Arabia’s national oil and gas company, reported on 15 August that some of its systems had been hacked. Saudi Aramco insisted that the attack had not affected any core business systems, nor its petroleum production operations. On 17 August, a team of hackers calling itself the Arab Youth Group claimed responsibility for the online attack. Security experts note that the corroborating details the group provided to prove that it was behind the disturbance suggest a link between the Aramco attack and a new bit of destructive malware called Shamoon that is being used to target energy companies. Though Aramco has admitted that the network disruption was caused by a computer virus, it has not revealed the extent of the damage or whether its computers have been disabled. Shamoon reportedly covers its tracks by overwriting files and a PC’s master boot record, making it impossible to boot up the machine. According to a Computerworld article, the Arab Youth Group said the attack was its way of lashing out against the Saudi government’s support of Israel and the United States.

-According to the British Retail Consortium, cybercrimes cost U.K. retailers £205.4 million over the past year. Though that sounds like a pittance compared with the overall revenues that businesses rake in, the survey reports that retailers lost 0.75 percent of the value of online sales to theft or fraud, more than twice the loss rate they suffer in their brick-and-mortar operations. Because e-commerce sales increased by 15 percent in 2011 and now account for 10 percent of total retail spending in the U.K., the BRC concludes that that e-crime is the “biggest emerging threat” to retailers.