Risk Factor iconRisk Factor

The VA August EHR Meltdown: The Reasons Why

Last week, ComputerWorld published a lengthy story about the disruption of the US Department of Veterans Affairs' VistA electronic health record (EHR) system in Northern California last August. According to the story, the outage was caused by "a simple change management procedure that wasn't properly followed."

It turns out that one group of maintainers asked another to make a change to a network port configuration without having the proper authorization to do so, which the second team did. In other words, the system was done in by poor configuration management.

For reasons left better explained by the ComputerWorld article, the VistA back-up systems that were supposed to kick in, didn't.

The outage caused the VistA system to be down for a good part of a day, which caused healthcare workers to revert to paper and pencil. Patient safety was increasingly put at risk, because the VA health system is almost completely electronic. In the VA's words, the outage was "the most significant technological threat to patient safety (the) VA has ever had.â'' It has taken months to put all the paper-based information created that day back into electronic format.

The VA experience provided a glimpse of what may happen if a major outage and back-up systems fail once EHR systems are fully up and running. System designers of EHR systems need to think a bit harder about what happens when the "unthinkable" does indeed happen.

Unintended Consequences: Human-Medical Equipment Computer Interfaces

Spectrum's Senior Associate Editor Samuel Moore sent me a note on an interesting news release titled, "Design of Patient Tracking Tools May Have Unintended Consequences" about a study by researchers at the University of Buffalo regarding the replacement of dry-erase patient status boards by electronic patient tracking systems. The researchers studied how new electronic patient-status boards were functioning in the emergency departments of two busy, university-affiliated hospitals.

What the researchers found was while there were surface similarities between the manual and electronic systems, there were subtle differences in the design of the latter that affected how health-care providers communicated and tracked patient care, sometimes not for the better. As one of the researchers noted,

"The manual whiteboard allows flexibility in tracking patients. For example, maybe the first time the provider sees a patient, she initials the name on the whiteboard, then the next time she circles the initials, then when the patient is discharged, she might put an 'x' in the circle, signals that are a means of communicating with her colleagues in the ER."

"With a computerized system, providers have to find an available computer terminal and log-in. The providers can't just walk up to the whiteboard and make a notation."

Whiteboards also provided immediate visual clues that the electronic tracking system did not, like how busy the emergency room was and how critical resources were allocated.

The researchers note that future electronic patient tracking systems need to investigate workflow and communication issues more carefully, and hope their study will encourage designers to better meet user needs.

Wisconsin Prison Software System Misses Fourth Deadline

The first phase of a new $25 million computer system project to keep track of Wisconsin's 23,000 prisoners will miss its December 2007 deadline - the fourth such schedule slip since the project started in 2003. The project is now at least 18 months late in its first phase: it has three more stages to go. It was originally scheduled for completion in May of 2009, but it is more like sometime in 2011 before it will be finished, assuming the other three stages don't have problems.

The project is fixed price, so the state Department of Corrections claims it hasn't overspent their contracting budget. However, the Department of Corrections did admit it didn't know how much the total project will really cost, since it didn't include the cost of state workers in the project's budget.

If the project slips a fifth time, it may be time for the IT Mercy Rule.

Hope They Match Your Name to Your DNA

In the wake of the great UK ID scandal comes another bit of slightly jarring news from the UK. It turns out that discrepancies, albeit small in number, have been discovered in the UK National Criminal Intelligence DNA Database. As reported by the London Telegraph, the errors include "incorrect spellings, dates, police crime codes and duplications that have left many records compromised."

With 30,000 or so DNA profiles being added in each month, errors are to be expected. The worry is that people (which in the future may include visitors to the UK) may be falsely arrested based on faulty information in the database. Again, while the statistical risk to any individual is very small, given the lack of trust in the current UK government because of its cavalier attitude towards protecting personal data and its reticence to talk about security problems, the perceived public risk looms large.

A Few More Shoes Hit the Floor in the UK

A few more shoes hit the floor in the UK id scandal.

According to the London Telegraph, the cost to secure those missing CDs containing the personal details of 25 million UK citizens was a whopping £5,000. HM Revenue and Custom senior officials didn't want to spend that amount of money to filter out sensitive personal data because to do so would "overburden the business by asking them to run additional data scans/filters that may incur a cost to the department". The current estimated cost of mitigating the risk of losing the data may reach £200 million, even if no fraud is committed. Nove cost/benefit ration, don't you think.

The Chancellor of the Exchequer Alistair Darling claimed that senior HMRC managers were not informed for three weeks that the 2 CDs went missing. Yet, in fact, HMRC was told within 6 days of the CDs being sent that they were missing by their intended recipient, the National Audit Office (NAO). The children's chant of liar, liar, pants on fire seems most apropos here.

The BBC is now reporting that instead of just four CDs, there now appear to be six HMRC CDs containing UK citizen private information that are missing. No one should be surprised that this number steadily increases over the coming week.

In the same BBC report, there is now a growing row between the UK government and the banks over who will pay for any fraud that might be committed. The UK government says that the banks are responsible in making their customers whole, and the banks naturally are saying, wait a minute, the government should be the ones paying since it caused the mess.

Anyone want to bet that the government will win in shifting its moral if not legal financial obligation to the banks, and the banks in turn will soon jack up their fees as an excuse to pay for "future fraud payouts," as well as play hardball with any customer who claims id theft?

The Sounds of Shoes Dropping Everywhere

In regard to the massive loss of personal data by the UK government earlier this week, it has emerged that senior UK government officials had been repeatedly warned that sensitive data was at risk of being compromised months ago because of slack security procedures. However, even after being told this, officials insisted that the data protection approached being used were "fit for purpose" - i.e., acceptable. Shoe Number 1.

An almost exact replica of this problem happened in 2005 involving HM Revenue and Customs and UBS customers. At the time, HMRC said, "This is a one off incident in a single office which receives thousands of pieces of post per week. We are urgently reviewing our procedures to make sure this does not happen again." Yeah, right. Shoe Number 2.

Seems that senior officials at HM Revenue and Customs knowingly refused taking even minimum security measures to protect the data being sent to the NAO because it was seen as being too expensive to do so. Shoe Number 3.

These senior officials - not the "junior official" whom the government blamed for the mess (who in fact looks like an administrative clerk) - apparently also authorized the method of data security (password protection, not data encryption) and the means of getting the information to the NAO (on CD sent by unregistered post). The junior official was merely following orders. Shoe Number 4.

It has now come out that HM Revenue and Customs has had over 1,211 - yes, 1,211 - data protection breaches in the past year, but as I mentioned earlier - this was apparently seen as being perfectly acceptable. HMRC has refused to talk about them. Shoe Number 5.

It was also disclosed that there are at least two other CDs that are missing on top of the two that are currently missing. Again, HMRC refuses to comment. Shoe Number 6.

The Chancellor of the Exchequer Alistair Darling informed parliament that he delayed announcing the loss of the CDs for 10 days after being told about it on November 10 because banks wanted more time to prepare anti-fraud measures. The banks vehemently dispute this. Shoe Number 7.

The government has told people not to worry - if any fraud is committed because of the breach, they will be covered. Now, all you have to do is prove a causal link. Good luck and God speed on that one. Shoe Number 8.

The "junior official" involved has been at least suspended (and some say sacked), and is at a safe house under 24 hour protection, supposedly for the person's own safety. More likely the government doesn't want this person talking to the press. Future Shoes Number 9 to ???

Hmm, its looks like UK Government's closet is as full of shoes as Imelda Marcos' closet.

LAUSD Payroll Debacle Explained - and Still Not Over

David Brewer, superintendent of the L.A. Unified School District, gave an interview to the LA Times in which he gave his reasons as to why the LAUSD payroll system blew up:

"The failure was this: That first of all there was no contractor oversight. That there was no real person in charge of this thing, at least the person who was in charge of it was not technically smart enough to know how to work the system. There was no separate chief information/technology officer dedicated to this. That was the first thing. We were depending on people who frankly speaking did not know how to interpret the problems that the system had technically."

I wonder why the project risk assessment didn't catch those pretty glaring risks/problems - wait, maybe there was no risk assessment. Does anyone out there in cyberspace know if there was any risk assessment done for this project?

Also, Brewer added in the interview that the payroll system "cannot account for about 500 people inside of the system who do not work to a standard calendar, even though we were told that we could. And now my contractor oversight says if that doesn't happen, they can't get paid." Two weeks ago, Brewer claimed that the payroll system was essentially fixed - I guess it isn't, after all, is it?

UK Government Security Blunder Continued

Details are now emerging on the lost confidential details of 25 million UK citizens. It appears that HM Revenue and Customs had established a practice of sending unencrypted data to the National Audit Office since March of 2007 to support its independent checks on the child benefit data, and would have likely continued if the CDs containing the information hadn't been lost in the mail last month.

Of course, the UK government is blaming the whole sorry affair on a "junior person" for not following procedures, that it wasn't an indication of a systemic failure (even though the same governmental agency had very similar security violations earlier this year), that an urgent review was being conducted to make sure it wouldn't happen again, that no one should panic (but do keep an eye on your bank account), yadda, yadda, yadda.

Prime Minister Gordon Brown told Parliament that, "I profoundly regret and apologise for the inconvenience caused"; the Chancellor of the Exchequer Alistair Darling said the episode was "catastrophic", "unprecedented" and "unforgiveable"; while the chairman of HM Revenue and Customs Paul Gray resigned, saying it was "a substantial operational failure." I do love British understatement, don't you?

Just to increase the sense of peace of mind of UK citizens, Richard Jeavons, director of IT implementation at the Department of Health admitted, when asked this week by a Commons Home Affairs Committee member about the security of the NHS Care Records Service database, i.e., "How confident are you that there won't be problems over [NHS] data and privacy?" responded that "You cannot stop the wicked doing wicked things with information and patient data..."

As a footnote, the UK government denied requests just last week from the Commons Health Select Committee to make information about NHS data security breaches public, saying that the information would, "add no value to the public understanding." I bet it wouldn't.

UK Government Mislays Half the Country's Personal Details

Reuters is reporting that the UK government Chancellor of the Exchequer Alistair Darling informed parliament that "two discs containing information on 25 million Britons had disappeared after being sent through HMRC's courier, Dutch mail and parcel company TNT NV, and a police investigation was underway."

"The missing information contains details of all child benefit recipients: records for 25 million individuals and 7.25 million families," according to Darling. It was a "serious failure" he said - no kidding?

Hmm, let's see. The UK government desires every citizens' and travelers' DNA, every person's travel related details, has created a national registry of all children under 18, is developing a national ID card, etc., etc., and yet it can't guarantee basic protection to any of the information it collects.

Nice, very nice.

Why Government Needs Sarbanes-Oxley - and its Penalties

This past week a $31 million property tax refund scam conducted by members of the Washington DC Office of Tax and Revenue was revealed by the FBI. The scam has been running for at least the past seven years, and allegedly involved two tax office employees (so far) and their families. The perpetrators were so unconcerned about getting caught, they sent a phony $346,700 check to a fictitious company named "Bilkemor LLC."

The employees were able to get away with the scam because their activities weren't supervised, nor extensively if at all audited. A "breakdown of internal controls" were blamed by DC officials - something that Sarbanes-Oxley reviews of computer system controls would have made much more difficult. The District's CFO hasn't resigned, and has indicated that he sees no reason to do so. Basically, he stated that it wasn't his fault, that he has already fired the wayward employees' managers, and that it wasn't a big deal anyway, since it didn't materially affect the District's finances: "It is important to emphasize that this unfortunate incident does not compromise the financial stability and viability of the District."

Public corporations would love to operate under that definition of materiality. If the CFO or CEO were in the same position of utter and absolute ignorance of their company's finances, they would be fired, sued by shareholders, and face possible criminal charges. I guess shareholder money is more important to protect than that of taxpayers.

This week, the Security and Exchange Commission (SEC) also admitted once again that it still couldn't meet Sarbanes-Oxley requirements either - more than a bit ironic for the agency whose job is to administer it to public companies and punish those who transgress its requirements. No one at the SEC is losing their job because of material weaknesses found there either, it appears.


Risk Factor

IEEE Spectrum's risk analysis blog, featuring daily news, updates and analysis on computing and IT projects, software and systems failures, successes and innovations, security threats, and more.

Willie D. Jones
Load More