Risk Factor iconRisk Factor

Boeing Crash: Speculation Continues Unabated

Boeing.gif

The cause of last week's crash at London Heathrow's airport of a British Airways Boeing 777 is still unclear. Crash investigators promise a preliminary report within a month.

Speculation about the cause currently run from a problem with the airplane's electrics, avionics system and/or engine control automation (reported in the Sunday Times and yesterday's London Guardian) to something wrong with either the aircraft's fuel system or the fuel itself that led to fuel starvation (Sunday Express). Just about every British paper has a theory, it seems.

What is known that about 2 miles from the airport and 600 feet up, the "the autothrottle demanded more thrust. It was a normal procedure, a small adjustment intended to keep the plane at the correct speed and height. Nothing happened. The computer system again ordered more thrust. Again, no response." The pilots apparently then tried to increase the throttle manually, and again, no response. Skilled airmanship brought the 777 into what one could called a semi-controlled crash, which fortunately, didn't result in any loss of life.

The plane's wreckage is being moved to British Airway's Hatton Cross engineering facility about 500 meters from the crash site for further investigation. If a rare software anomaly is found to be the problem - as it was in the Malaysian 777-200 incident of 2005 (see the Australian Transport Safety Bureau incident report, and a brief description of it in today's Sunday Times) - then expect there to be some additional fall out towards the Boeing 787 development.

UPDATE: Peter Ladkin point out that a preliminary crash report is required within 30 days (I wrote promised, which implies something else). As Peter noted, the UK is an International Civil Aviation Organization (ICAO) signatory, and ICAO signatories are required to produce accident reports according to a general standard format; they are also required to issue a preliminary report within 30 days of the accident.

UPDATE 1: Today's London Times is claiming that, "British Airways technical staff believe that the Boeing aircraftâ''s computerised control system caused both engines to fail during its final descent towards Heathrow on Thursday." We shall see.

Boeing B787 network certification requirement

Greetings, folks. I am Peter Ladkin and hope to be contributing on safety matters, especially in transportation.

Bob wrote recently about the FAA's new certification requirement on the Boeing B787 "Dreamliner" networks. I checked it out.

The FAA makes regulatory requirements (which are administrative law) by publishing a Notice of Proposed Rulemaking (NPR) in the Federal Register (FR) , collecting comments, and implementing the rule in the light of comments. The NPR was published in FR 72(71) on April 13, 2007, eight months ago. The FAA received comments from Airbus and from the Air Line Pilots Association, and issued the rule, unchanged, with answers to the comments, in FR 73(1) on January 2, 2008, whence the brouhaha in Wired.

So far, this all looks routine. Let's look at what the rule does.

There are three "domains" for networks in the B787: the Aircraft Control Domain (ACD), the Airline Information Domain (AID) and the Passenger Information and Entertainment Domain (PIES). The ACD is the safety-critical bit. The PIES is the passenger network. The rule says "the design shall prevent all inadvertent or malicious changes to, and all adverse impacts upon, all systems, networks, hardware, software, and data in the Aircraft Control Domain and in the Airline Information Domain from all points within the Passenger Information and Entertainment Domain." It is harder to get any more stringent than that.

Why are the FAA doing this now? Because they have perceived a gap in existing regulation which needs to be filled. And it needs to come now because Boeing are certifying the aircraft now. Airbus wanted more generally applicable conditions along with guidance on how to comply. The FAA replied that they are working on that, but the B787 needs it right now.

A colleague suggested the least expensive way of fulfilling this criterion might be to separate the domains physically. Well, I am not sure that can be done, since some of the AID as well as PIED are wireless. In some current fleets, for example, sensor data and other data in the aircraft control networks is siphoned off to go to, amongst other things, the Quick Access Recorder (QAR), which records data on the flight for airline flight quality control and maintenance. At least one major airline downloads the QAR data at the end of each flight directly through the local cell phone network at the destination. So one already has potential interconnections between public networks and aircraft control networks in which all the bad stuff must be controlled (and is, by obvious means).

Why aren't the FAA requiring similar for ACD/AID interaction? They are; they say this is covered by existing regulation as well as other special conditions (which I haven't yet seen).

So this looks all routine admin stuff. I don't see anything below the surface. Except, of course, for the monster question of how one does assure absolute security of the sort that looks to be required. I don't know who can answer that question, and I doubt if Boeing's answer will enter the public domain.

Bank Software Problem Shreds Customers' Credit History

Shred.gif The Orlando Sentinel reports that a computer problem at Cincinnati-based Fifth Third Bank related to the recent acquisition of the former R-G Crown Bank of Casselberry, Florida "spilled false information into 'several thousand' customer accounts, in some cases generating credit-history errors and incorrect credit scores."

The problem started in December when Fifth Third converted files of R-G Crown Bank customers to its own system. In at least one case, a customer found that he had an account showing three loans that were not his, with one showing a history of 19 late payments, all of which trashed this person's credit history. Others found that they were denied credit because of the false information put into their bank records.

Fifth Third Bank seems to have been very slow in notifying customers negatively affected by the problem, as well as shown a pretty cavalier attitude towards the whole episode. It won't discuss details of the problem, citing the old canard of "customer privacy."

Fifth Third claims that it has notified the credit reporting bureaus, and that everyone's credit rating is as good as before, but I seriously doubt this. Once poor credit history information gets out there, even if false and later "corrected," it is extremely difficult to put that genie back into the bottle, especially in this time of tightening credit.

Another Big Data Loss

I thought we'd be able to ring the bell, but only the records of 650,000 J. C. Penney (and up to 100 other retailers') customer records were lost when a computer tape went missing. In a Chicago Tribune story, GE Money which handles the the credit card operations for Penney's and the others said btoh customer credit card and social security records were on the missing tape.

GE Money says that it will be paying for 12 months of credit-monitoring services for those on the missing tape.

The tape went missing last October, so I guess this loss wouldn't have counted towards the first million record data loss in the US of 2008 anyway.

UK Chinooks: $150 million for Hope over Experience Software?

Chinook.gif While Boeing may be having troubles with the Dreamliner, according to a story in the UK Computing it is to receive a £90m contract to rectify software and avionics problems for eight brand-new Chinook helicopters that have been sitting in hangers at RAF Odiham for the past nine years.

This has been one strange defense program from the beginning, which goes back some 13 years. Below are excerpts from the 2005 UK Select Committee on Public Accounts report that gives some background to the story:

"In July 1995, the Department [UK Ministry of Defence] decided to upgrade eight of the 14 Chinook Mk2 helicopters it was procuring as part of its requirement for a Medium Support Helicopter. The upgrade to an enhanced Mk3 standard would include improvements in range, night vision, and navigation capabilities. The project was scheduled to cost more than £250 million and the forecast in­service date was November 1998. A subsequent change to the requirement led to an avionics upgrade programme being put to contract in 1997, which entailed a hybrid solution, incorporating elements of the existing analogue cockpit and new digital systems and displays. The need to test the airworthiness of the aircraft together with some programme slippage led to the setting of a new In-Service Date of January 2002. When the aircraft were accepted from the contractor in December 2001, the Department found that it was unable to demonstrate that the flight instruments met United Kingdom Defence Standards, as this requirement had not been specified in the contract. Consequently, the aircraft could not be used other than for limited flight trials."

"The Department said that there were three main reasons why the helicopters remained grounded and were unfit for their operational task. First, without access to the source software codes held by the United States, the safety parameters of the aircraft could not be tested in its current configuration. One of the main contractors has now indicated that it would allow access to some software data. The process of analysis is, however, time consuming and expensive and there is no guarantee of success because the legacy software is not amenable to the techniques required to confirm the robustness of the software design. Secondly, the specialist role envisaged for the aircraft had changed since they were acquired. Finally, the aircraft needed to be fitted with Health and Usage Monitoring Systems, a range of systems that seek to monitor the progressive wear of engines, and better Defensive Aids Suites."

"Despite the fact that all the aircraft accepted from the contractor met, and in some cases exceeded, the contract, the Department accepted that the taxpayer had not been well served by the procurement of the Chinook Mk3."

According to Computing, the Chinooks should be ready by 2009, 11 years late. I guess that is a bit longer than the Australian Super Seasprite avionics upgrade program. At least the Chinook program hasn't been a continuous cock up like the Seasprite, though.

Boeing Delays 787 Dreamliner Again

Boeing.gif

Just a month after delaying the first-flight of the 787 Dreamliner and promising that things were on track, Boeing once again delayed first flight by at least three months. First customer delivery subsequently slipped from the end of this year into early next.

Boeing admitted that it had underestimated the amount of time needed to complete the work done by suppliers of key 787 components. During the analyst conference call yesterday, Boeing management was asked, "Supply chain aside, you have a slip here in first flight of three months, and how much of that and the delay in power on is related to problems in getting systems to play to one another? We still hear rumors of problems with the flight control computer, the common core, etc." Boeing management gave a long-winded, roundabout answer to the question without really answering it, so one suspects that there may be more computer issues than it is letting on to or the FAA is asking about.

One analyst says that "Boeing's credibility is shot." Some customers like Qantas, Nippon Airways and Japan Airlines are thinking of pressing for compensation as well. Boeing's credibility may not be shot, but it is pretty thin.

If Boeing has to slip again, things could start to get very, very interesting.

Microsoft Wants to Patent Spying on You

Spymaster.gif

In today's London Times, there is a story about Microsoft developing "Big Brother-style software capable of remotely monitoring a workerâ''s productivity, physical wellbeing and competence."

The story goes on, "The Times has seen a patent application filed by the company for a computer system that links workers to their computers via wireless sensors that measure their metabolism. The system would allow managers to monitor employeesâ'' performance by measuring their heart rate, body temperature, movement, facial expression and blood pressure."

"Microsoft submitted a patent application in the US for a 'unique monitoring system' that could link workers to their computers. Wireless sensors could read â''heart rate, galvanic skin response, EMG, brain signals, respiration rate, body temperature, movement facial movements, facial expressions and blood pressure', the application states."

"The system could also 'automatically detect frustration or stress in the user' and 'offer and provide assistance accordingly'."

This last part made me laugh pretty hard. Let's see, my Windows XP machine had to reboot twice yesterday because of Windows errors, and my frustration level got fairly high. What could it do to "provide assistance accordingly?" The best Microsoft could do was to offer to send an error message to its gnomes. A better solution would have been to offer to buy me a new ultra-thin Mac laptop.

If Microsoft's "monitoring software" works as well as Windows, Vista, or whatever else is in their future labs, no one has anything to worry about, except maybe the continuous mis-reading your state of well-being.

And I do wonder what will be so unique about such a system when NASA has been monitoring its astronauts since the early 1960s. I would like to see the patent section on prior art and why their software is so "unique."

Now Where Did I Leave That Sponge?

bunny.gif

A short time ago, the Chicago Tribune ran a very interesting story on the use of bar codes as well as Radio Frequency (RF) detection as a means to keep track of surgical sponges during operations. Sponges are left in about 1,500 people a year during their operations in the US. In a 2003 study published by the New England Journal of Medicine, leaving sponges and other surgical instruments in patients happens most often during emergency surgery or because of some unexpected change in the surgical procedure.

One system by SurgiCount uses a bar-coding approach. "Essentially, the system works much like a grocery store check-out counter â'' every laparotomy and gauze sponge is pre-labeled with an individual and unique bar code and a scanning SurgiCounter is used to read the labels.

"When using the system, staff concurrently scan sponges during their manual counts or can scan the items before or after the manual count. The SurgiCounters can be held by the circulator, or can be placed on a holster on an IV pole in a hands-free mode. By scanning in the unique labels, the system builds a database of items used in that particular procedure. At the end of the procedure when the circulator is counting out the sponges, the circulator will again swipe the sponge under the SurgiCounter, this time in order to 'count' the sponge out of the database. Because each sponge has a unique bar code, the system automatically alerts the staff in case they have accidentally tried to count the same sponge twice. This assists the staff in validating that they have an accurate count in case the there was a manual counting error."

Another approach is that developed by Medline called RF-Detect. Here, "a sterile radio frequency chip, (the size of a grain of rice) is embedded in the surgical disposables. With the RF Detect system, a Blair-Port wand is waved over the patient accurately alerting the user when an RF-tagged surgical disposable remains in the patient before surgical closing procedures."

Of course, it isn't only surgical sponges, towels or surgical gauze that is left in patients. A 2003 story involving the New England Journal of Medicine study that I mentioned above in the New York Times describes a patient who had a six inch metal clamp left in him. This was left as a result of a previous operation to remove a previously lost metal clamp. Talk about being unlucky.

In a BBC story from 2005, a man left the hospital with a two inch scalpel blade in him after heart by-pass surgery, which wasn't found until months later by an x-ray for a different condition.

Hospitals are working harder to ensure that these types of medical "never events" don't occur, not only because of the patient's health and the cost of lawsuits, but both Medicare and insurance companies like Aetna and Wellpoint are now refusing to pay for fixing these problems.

Legacy Computer Models & Decision Making

Wood-Model.gif My post from yesterday about the false warning to prepare to evacuate based on an outdated computer model/data that a dam was in danger of bursting made me wonder about the flip side: how many outdated computer models are being used to make decisions that are too optimistic?

Recently, there was a story in the Washington Post on an Environmental Protection Agency (EPA) review that "found that a computer model of the Chesapeake, used by the EPA's Chesapeake Bay Program to gauge improvements in the estuary's health, tended to inflate the impact of some cleanup measures."

"Tom Simpson, a University of Maryland professor who led the review requested by the bay program, said there was no evidence that the EPA had been purposefully deceitful."

The Post story goes on:

"Simpson and other researchers were asked by the bay program to review some of the calculations plugged into its computer model. These equations described the impact of certain save-the-bay tactics: plant X amount of cover crops to hold fertilizer on farm fields, thus achieving a decline of Y in fertilizer-polluted runoff."

"But Simpson said his review found that many of the equations were based on small-scale experiments that might not predict what would happen on a large farm. Others were based on the educated guesses of experts."

"Fifteen assumptions were found to be accurate, and three were found to underestimate the benefit to the bay, according to the bay program"

This story reinforces the notions that essentially, all models are wrong, but some are useful, as George Box said (there is a nice paper here by John Sterman, Director, System Dynamics Group at MIT on the usefulness of models that is worth reading). However, I think we may need to update Box's saying to something along the lines of:

All models are wrong but old models that aren't reviewed are more wrong; make decision based on them at your own risk.

Run! The Dam is Breaking! Oh, Never mind

The National Weather Service issued a flash flood warning last Tuesday morning stating that failure of the Norway Dam on the Tippecanoe River north of Monticello, Indiana "is becoming more likely."

But it really wasn't.

According to the Louisville Courier-Journal, "Michael Lewis, the warning coordination meteorologist with the National Weather Service's Northern Indiana bureau, said the erroneous warning was based on bad information that may have been entered into the agency's computer system up to two decades ago."

Two hours after it issued the alarm, the Weather Service rescinded it.

Lewis went on to say, "The problem is being fixed and the office's entire warning system will be reviewed."

Advertisement

Risk Factor

IEEE Spectrum's risk analysis blog, featuring daily news, updates and analysis on computing and IT projects, software and systems failures, successes and innovations, security threats, and more.

Contributor
Willie D. Jones
 
Load More