Risk Factor iconRisk Factor

Anatomy of a Software Project Fiasco: LAUSD Payroll System


A very sad, but very unsurprising, story appeared in yesterday's Los Angeles Times about the LA Unified School District (LAUSD) payroll system fiasco that I have been blogging about for the past several months. The story shows how the system was in trouble from the start, tells of the warnings of trouble along the way, and how the warnings were ignored anyway until it was too late.

"In the weeks leading up to the launch of a new payroll system, Los Angeles Unified School District officials had plenty of warning that the $95-million technology project would have serious problems."

"Critical hardware had failed numerous times. Flawed data collected over decades proved difficult to clean up and input into the new system. Payroll clerks complained that training had fallen far short -- more than 60 schools didn't have a single staff member who'd received any training."

"Still, consultants hired to implement the system urged the district to proceed as scheduled in early January 2007. Three days before the system was to begin, they urged the district in a report to 'Go! Proceed . . . and go-live on January 1!' "

The story goes on to say that LAUSD officials admitted that they were too inexperienced - I say incompetent - to understand the issues, risks, and problems popping up all around them.

"Supt. David L. Brewer oversaw the district's clumsy recovery effort. It has taken a year to stabilize the system. The ordeal has weakened the superintendent, opening him to criticism that he has been ineffective."

" 'We were not ready to go live with this system, but we didn't have the internal expertise to know that,' Brewer said in a recent interview."

The LAUSD chief operating officer responsible for pushing through the payroll system also admitted that "he knows little about computer systems."

Helping matter along, the original payroll data was not cleansed before it was put into the new system, so the new system used inaccurate information. "The dirty data had produced bogus employee rosters. How, clerks wondered, do you cut a paycheck for a person the computer system says does not exist?"

The contractor responsible for the payroll, however, maintains that "throughout the project we not only did our part, but we in fact went above and beyond." I am sure, therefore, that they are going to use the project as a proud customer reference site in their future proposals.

All one can say is, sad, sad, sad, and another software project to add to the case study pile of what not to do - not that anyone will read it.

London 777 Crash - Likely Caused by Ice in Fuel System?


In today's Wall Street Journal, there is a story about American and United Airlines taking precautions against the possibility ice accumulation in the fuel systems of Boeing 777s, which seems to be the leading theory for the crash of the British Airways 777 on 17 January. According to the paper, "The moves come amid growing indications that a buildup of ice crystals or slush simultaneously restricted fuel flow and reduced the thrust of both engines of the Boeing 777."

The paper goes on, "U.S. and British investigators are focusing on whether ice crystals may have clogged the plane's dual oil-cooler systems, according to people familiar with the details. The radiator-like devices use fuel flow from each of the wing tanks to cool engine oil, and fuel then flows from there to the nearby engine during flight. .... [it is] believed to be the first time ice contamination in fuel brought down a large, state-of-the-art jetliner with no apparent mechanical or computer malfunctions."

So much for it being software.

Crackberry, err, Blackberry Users Get Their Juice Back


For the second time in 10 months, Research in Motion's (RIM) Blackberry network suffered another outage affecting some 8 million of its customers. This one lasted only about 3 hours, while the one last April lasted more than 12 hours.

While many Blackberry users were forgiving of the first outage (a typical comment of the time was ""These things happen. Life goes on."), this time, Blackberry owners seem more irritated ("I don't know what happened, I don't care what happened. They need to save their excuses for someone who cares").

Part of the reason for the angst, other than the Blackberry addiction theory, is that RIM promised after the April fiasco - which was caused by poor software testing - that something like this wouldn't happen again claiming that it would put in back-up systems. So user expectations were naturally raised.

RIM also said that they would be more open about any problems in the future. Some of you may remember that RIM was very closed-mouth about the April outage, providing the press with a constant stream of "no comments," not only about its cause but during the event itself. Surprisingly, there is nothing about this latest one on their website this time either. Not a smart move, guys.

RIM now has two strikes against it, and if a third outage hits soon, expect the company to take a big share hit as it will get tagged with a reputation for unreliability. Not a good thing for a device that health, safety and security professionals and government officials depend on, nor with alternatives available to tempt Blackberry users.

I recently traveled around the world without either a cell phone or a laptop. It was very refreshing, with the only downside being a boatload of emails when I got back that I had to wade through. Still, it was a good trade.

Silver-lining to Boeing 787 Schedule Slip


" 'We have taken advantage of the delays to make sure our system level maturity is coming along at a rate that will avoid problems as we enter flight test.' " So said Boeing's Scott Carson, chief executive officer of Boeing Commercial Airplanes last Wednesday in regard to the year-long slip in the Boeing 787 Dreamliner's flight testing. As I noted a few months back, the software folks were praying for some other part of the aircraft's development to be blamed for the slip, and not them. They got their wish - and they better make the most of it.

Carson says that the 787 is now on track - although we've heard that a couple of times before. There still has been no public disclosure of the penalties Boeing will have to pay airlines such as All Nippon Airways for the late delivery of the 787, but I doubt it will be minimal.

On a side note, I had a chance to see Boeing's major competitor the Airbus 380 arriving in Singapore last week. Others may disagree, but it looked to me like a cargo plane on steroids. It definitely looks better in pictures than up close.

Unfortunately - or maybe fortunately - I was outbound to Germany and did not get to see how baggage, customs and immigration handled the plane's passenger volume. The newly opened and spectacular terminal at Changi Airport (which I'll blog about soon) has been designed to handle Singapore Airline's future fleet of A380's (Singapore Airlines currently has two A380s in service, a further 17 on firm order and options on six more), but even though the new terminal is spacious, I don't think I would want to be around when a couple of A380s off-load at the same time.

It is not known how much Airbus had to pay in penalty costs to Singapore Airline for delivering the A380 two years late (operations started last October), but given that Singapore Air had very publicly committed its future business around it, the amount had to be pretty high.

Computer Gives Away Money, Sparks Brawl

In-the-money.gif This is a story from last Thanksgiving time that slipped under the radar. Apparently, a computer glitch at a Kmart store in Wauwatosa, Wisconsin helped spark a brawl. It seems that the store was running a promotion in which it would give away $10 to anyone applying for its credit card. However, a computer error allowed everyone's application to be approved, thereby immediately giving $850 to $4,000 in instant credit to anyone who applied, regardless of credit history.

Naturally, word spread pretty quickly (claims of "free money at Kmart" was how it was phrased) and soon the store was overrun with people trying to sign up for credit cards. Things got ugly as credit card applications ran out, and a dozen police were need to restore order. A couple of folks were arrested as well.

Some innovative folks went to another Kmart when the credit card applications ran out, and started to sell them in the parking lot for $20 apiece. Now that's good old American enterprise at work.

KMart said that it was a localized, not national, problem - which is interesting in itself. Local stores can grant their own credit approvals?

Is $40,000 Too Much for A Good Night's Sleep?


The London Guardian published an article a few weeks back about the Starry Night Bed from Leggett & Platt, a computer-controlled bed that claims to "monitor your sleeping patterns, regulate your temperature and even intervene to stop you snoring."

The story goes on: "The Starry Night Bed contains vibration sensors that can monitor the sleeper's breathing. If the breathing stops, a built-in computer can call 999. If it detects snoring, the head of the bed automatically elevates by seven degrees to unblock the sleeper's airways. If the snoring continues, it goes up by a further seven degrees. When the snoring stops, the bed flattens out again."

"A thin layer of liquid sealed into the mattress allows the bed to provide heating or cooling anywhere between 20C and 47C (68 to 117 degrees Fahrenheit). Built-in ambient lighting switches on if the user goes wandering in the middle of the night. The bed's headboard contains a media centre with 1.5 terabytes of memory, enough to store 400,000 songs or 2,000 hours of video. There are also surround-sound speakers and an LCD-based projector in the headboard that casts a 10ft screen on the wall, projecting films, books or the internet."

You can also connect the bed to something called Life|ware, which "connects the diagnostic and entertainment features of Starry Night with other electronic components in the bedroom and throughout the house," according to the website.

And yes, there is an iPod docking station included, but no, there is no espresso maker.

Who Invented What?


There is a new book out by Seth Shulman titled, "The Telephone Gambit" which argues that Alexander Graham Bell didn't invent the telephone after all, and that he stole the critical technology from Elisha Gray. He appears to make a pretty compelling case. There are extensive reviews of the book on line, such as here and here.

The thrust of Shulman's thesis lays in his discovery that Bell was trying out all sorts of approaches that weren't panning out, as noted in Bell's notebooks of early 1876. Entries in the notebook about his different attempts continue until 24 February 1876, and then don't resume until 7 March 1876 when Bell tries a novel approach. A day and a half later, "Mr. Watson, come here. I need you," is supposedly transmitted.

The gap is important, Shulman argues, because Bell had just returned from a trip from Washington, DC, where it seems that Bell got improper access to Gray's patent application. Shulman also contends that Bell's subsequent reticent behavior indicates that he struggled to come to terms with his theft of Gray's invention.

The book reminds one a bit of the recent flap over the MS-DOS and how much credit for it is owed to Gary Kildall and CP/M. I wonder if there will be books about it a hundred and thirty years from now, or will anyone then even remember MS-DOS, Windows or even Microsoft.

Worrying ID Theft

A few weeks ago, the Naval Surface Warfare Center Dahlgren Division began contacting as many as 10,000 current and former federal employees who worked at the Naval Bases in Dahlgren, Va., Silver Spring, Md., and Panama City, Fla., on or before July 7, 1994, to warn them of potential identity theft, NSWC urged them to contact their creditor bureaus in the wake of a reported attempt to illegally obtain a credit card using an employeeâ''s personal information.

NSWC was notified on Jan. 8 that four individuals had been arrested in Bensalem Township, Pa., on Jan. 5, 2008, for attempted identity fraud. The police informed a Dahlgren employee that someone had stolen his identity and was about to use his credit card to buy a big-screen TV at Sears.

The four had in their possession two pages of a hard copy report dated July 7, 1994, containing personally identifiable information (PII) â'' names, social security numbers and dates of birth â'' of nearly 100 individuals with the last name beginning with â''B.â''

So far, there is still no information as to how the individual(s) came to be in possession of employee personal information. The episode is currently under Secret Service investigation.

This theft shows how even older data records can pose a major threat. What is concerning is whether this was a one-off, aberration, or whether id thieves are now targeting archival information. It also points out that thieves might be very willing to hold on to stolen data - especially if it is a "high profile theft" - for a very long time, since its value doesn't decrease much.

Electronic medical records: Does Clinton Overstate Potential Savings?

From Spectrum senior associate editor Steven Cherry at Tech Talk:

Would electronic medical records save the United States $77 billion?

Hillary Clinton, Senator from New York and one of the leading candidates for the 2008 presidency, said so on Thursday night.

You can hear it for yourself. Itâ''s about five and half minutes into this YouTube video.

If you don't want to listen, hereâ''s the key soundbite:

According to the Rand Corporation, hardly a bastion of liberal thinking, they have said that we would save $77 billion dollars a year. That money could be put into prevention. It could be put into chronic care management. It can be put into making sure that our health care system has enough access so that if you are in a rural community somewhere in California or somewhere in Tennessee or somewhere in Georgia, youâ''ll have access to health care. If youâ''re in an inner city area, and you see your hospital, like the Drew Medical Center, closed on you, then youâ''re going to have a place once again where you can get health care in the immediate area.

Clinton wants to pay for universal health care, in part, with these savings. And sheâ''s been talking about it for a while. She mentioned the $77 billion figure in a key policy speech on the eve of the New Hampshire primary that reinvigorated her campaign. Itâ''s obviously an important matter, yet Iâ''m not sure the press has taken even the quickest look at the RAND study on which so much of Clintonâ''s health plan depends.

If they had, theyâ''d notice that the study was released back in 2005. Not only that, but it was two years in the making, according to a RAND press release issued at the time. So the savings might be greater, adjusted for inflation, and might be greater, or a lot less, depending on how outdated the data is.

The study actually claims $81 billion in annual savings, according to a press release issued at the time. For some reason Clinton isnâ''t counting $4 billion that â''would be saved each year because of improved safety, primarily by reducing prescription errors as computerized systems warn doctors and pharmacists of potential mistakes.â''

Leaving aside those safety savings, what would the $77 billions in savings result from? Richard Hillestad, a senior management scientist at RAND who led the study,

estimates that if 90 percent of doctors and hospitals successfully adopt health information technology and use it effectively, resulting efficiencies would save $77 billion annually. The biggest savings would come through shorter hospital stays prompted by better-coordinated care; less nursing time spent on administrative tasks; better use of medications in hospitals; and better utilization of drugs, labs and radiology services in outpatient settings.

In other words, itâ''s not as if we save $77 billion from eliminating the manual operations of paper records, and then can plunge the savings into improved care. The savings come from the very improvements in heathcare that electronic health records make possible. So the question arises: Is Clinton double-counting the benefits of electronic health care records, once in the saving of the money, and then again in the spending of it? Look at one specific instance the RAND release gives:

For example, health information technology could make a major contribution to improving care for patients with chronic conditions such as diabetes, who account for 75 percent of the nation's medical care costs, according to researchers.

That sounds an awful lot like the â''chronic care managementâ'' that Clinton cited as something that â''that money could be put into.â''

To be sure, the estimated cost of Clintonâ''s proposed changes to health care run about $110, half of which she says can come from ending the Bush tax cuts, which are set to expire soon. â''The other $55 billion,â'' she explained, â''would come from the modernization and the efficiencies,â'' of which, presumably, electronic health care is only one. It is, though, the only one she discussed at length in the debate.

Thereâ''s also, then, a question of timing. The RAND study says

â''It's going to take 10 to 15 years to achieve wide adoption of electronic medical information, even if all the ongoing efforts are successful,â'' Hillestad said.

Does Clinton plan to wait for the savings to materialize before reforming health care? Surely not.

Health care in general is a serious issue, as is the specific one of electronic health records. It was given quite a bit of debate time this week. Unfortunately, Wolf Blitzer, who moderated the CNN-sponsored debate, arrived unprepared to challenge Clintonâ''s airy claims about it, despite their having been made more than three weeks earlier in an important speech.

For those who want more substance than air, Spectrum has plenty to offer. Way back in 2002, we published "Welcome To The (Almost) Digital Hospital,"

More recently, contributing editor Robert N. Charette looked specifically at the promises and problems of electronic medical records in â''Dying for Data.â''

In fact, Bobâ''s been a little obsessed by the topic. Last summer he blogged about it three times in one month, here, here, and here. The last two are about the critical issue of privacy.

And earlier this month, Bob wrote about a fascinating 3-D visualization tool for electronic health records being developed at IBM, â''Visualizing Electronic Health Records With â''Google-Earth for the Bodyâ''.â''

Weâ''ll continue to follow this and other tech-related issues as the presidential campaign continues. Some of the claims made by the candidates involve some pretty interesting and complicated technologies. For this one, though, all you had to do was read a two-year-old press release.

UK ID Cards: Delayed But Not Dead

News reports are surfacing that the plan to implement UK ID cards by 2010 has been slipped to 2012, i.e., after the next election, according to a leaked government strategy paper. The plan, however, is still that by 2015, access to certain government services will require use of the ID Card, like here with REAL ID, thereby "encouraging" everyone to get one.

All the bad publicity from the data scandals has probably helped influence the new strategy.

However, the government, while not exactly denying the delay, said that it was fully committed to introducing the cards, regardless. Unless, of course, the current government loses the next election.


Risk Factor

IEEE Spectrum's risk analysis blog, featuring daily news, updates and analysis on computing and IT projects, software and systems failures, successes and innovations, security threats, and more.

Willie D. Jones
Load More