Risk Factor

The Office of the Information and Privacy Commissioners of Alberta and Calgary, Canada, released a report today on the TJX data breach. Not surprising, the report found that TJX skimped on privacy safeguards. It appears that initial access into TJX computer systems was via the wireless local area networks at two of its US stores.

The Wall Street Journal (subscription required) ran a story today on the implementation of a new Oracle payroll system for Arizona State (there is an earlier story here from California State University). The approach taken was a bit different, or in the words of Emeritus Professor Colin Tully, a planned failure.

On the advice of Arizona State's IT department, instead of a designing and planning for a project that would take an originally estimated 4 years and cost $70 million, the University would instead try to do it for $30 million ($15 million for development, $15 million for maintenance over 5 years) in just 18 months. The reasoning was that since there would be operational glitches whether the project took the 4 years or 18 months, so what the hey, let's go for short approach and fix everything in the wash.

To hit their target date, the University increased the number of programmers and consultants on the project, as well as opted for a very "vanilla" payroll system (i.e., absolutely minimal customization) and if required, Arizona State business processes would be changed to conform to the payroll system requirements rather than vice versa. For instance, instead of being paid on the 15th and 30th of the month, employees would now be paid every other Friday.

Well, the University got exactly what it hoped for - a buggy payroll system in need of fixing.

The minor change in pay dates meant that 26 paychecks would be issued per year instead of 24, which has created recurring problems. There is a more detailed of explanation of why this seemingly simple change has caused recurring issues here. At least 700 out of 15,000 Arizona State faculty and staff have experienced paycheck problems.

Of course, the people are really responsible for all the "cost savings" have been the 700 Arizona State workers, who, like the LA Unified School District employees, paid the price in terms of incorrect or no paychecks. This hidden but very real sacrificial cost was not factored into the equation, and employee morale is not in the best of shape, to say the least. Affected Arizona State employees got an apology from management and a $25 give card for their trouble.

However, the new payroll system has been declared by one and all (or at least the Board of Trustees, University Management, the IT Department, and Oracle) as a major IT success for saving so much money. It is also being touted as the "wave of the future."

I am a bit puzzled, however, if the 4-year, $70 million originally proposed system was also a vanilla payroll system. If it wasn't, then I am a bit skeptical that this project can be declared a success, since it's an apple and oranges comparison.

Furthermore, using a vanilla approach to installing payroll systems to keep costs and schedule down is not a new idea by any stretch. Software vendors have been arguing that organizations should conform to their software for at least twenty years.

In addition, the project didn't seem to have a very robust risk management effort in place, given some of the reported errors. I guess you don't need to have one if the problems are expected.

I also wonder why they didn't try to do the project in 16 months or 14 months instead of 18 months, given their cost/benefit logic. Seems to me that by targeting say 20% of the faculty and staff for payroll problems, ASU could have saved millions more.

The situation reminds me some of what King Pyrrhus supposedly said after defeating the Romans, "One more such victory and we will be undone."

Speaking of the IT mercy rule, for the US Department of Homeland Security (DHS) Secure Border Initiative's SBInet Project 28, they now appear to be 8 runs down and its the bottom of the fifth inning.

DHS Secretary Michael Chertoff told the House Committee on Homeland Security that payment to the Boeing Company, the prime contractor, was being suspended until it can prove that the "virtual fence" can be made to work. Seems that there is a "software glitch" and some integration issues that are causing problems.

Chertoff, however, is reportedly confident now that Boeing has "retooled their team on the ground and replaced some of the managers. ... They are now working through the problems of system integration as we speak. I think they put their A-team in place to do it."

Ahem, did this mean that Boeing has been using its B-Team?

Given the supposed importance of the SBInet project to Boeing, a somewhat surprising winner of the SBInet contract, you would have thought that Beoing's A-Team was on the project from the beginning.

But, in looking back to just a few weeks ago, the A-Team was supposedly on the project, and that the change in management was just routine ("The [Boeing] program management change was planned several months ago'). Maybe what Chertoff meant to say that Boeing now has the A-plus team on the project.

Yet... this project was said from its inception to be a technologically-low risk project according to both Boeing ("Boeing's solution concentrated on using proven, low risk, off-the-shelf technology..") and the DHS ("... we believe that the selection of the Boeing proposal validates the approach for acquiring a low-risk technological solution.")

While low-risk doesn't mean no-risk, Project 28 was declared low-risk because it is using common, off-the-shelf system components that has all been used before, albeit not altogether at one time.

Sort of like saying I am going to build a car using a Honda Accord engine, a Ford F-150 chassis, a GM Saturn Vue interior , etc., and then claiming the resulting effort is low-risk. Makes sense to me, although there is this small issue of integration, eh?

Curiously, the SBInet project is on OMB's current high risk list. This must be what OMB means when it says that just because a project is on its high risk list does not mean that it is in reality high risk. But Project 28 is now 20% over schedule, and if it were a defense program, it would be considered a failure in the making. So, is SBInet high risk - or is it low risk? Or do we just split the difference and call it medium risk?

What makes this all a bit more puzzling, of course, is that back in April, the mobile tower (i.e., the integrated, mobile sensor tower that houses cameras, radar, wireless data access points, communications and computer equipment, and a security system) element of Project 28 passed all of its tests with flying colors. Seemed to indicate a low-risk project, right?

But in June, DHS admitted to Congress, after oops forgetting to disclose it during testimony, that the Project 28 go live date had slipped from 13 June to 20 June. Ah, better make that September (maybe). So, what did the April test actually test, and what insight in the level of project risk did it produce?

As time goes on, I assume all this will become clear. And assuming that DHS is a learning organization, I can expect that DHS has at this very moment an on-going learning review of Project 28 to help DHS understand why a supposedly low-risk project has turned out not to be one, and how it can avoid this situation in the future. I am very anxious to read DHS's Project 28's lessons learned report, too, since maybe this high-risk/low-risk thing will become clear to me.

Boeing, an avowed learning organization, must be doing the same thing, especially now that at least two of its recent high-visibility, publicly declared low-risk, commercial off-the-shelf system programs (the other being Wedgetail) have proven anything but. Interestingly, Wedgetail also supposedly passed its early tests with flying colors. A risk pattern, perhaps?

Rule 4.10 (e) of Little baseball states that: "If one team has a lead of 10 runs or more after the game becomes a regulation game, the game is over."

Sen. Thomas Carper, D-Del. has suggested during last week's Senate Homeland Security and Governmental Affairs Subcommittee on Federal Financial Management, Government Information, Federal Services and International Security hearings on High Risk IT: Is Poor Management Leading to Billions in Waste?, that something akin to the mercy rule needs to be invoked on government IT projects, according to Government Executive magazine. Carper reportedly said that, "Some of these [IT] projects can be extremely difficult to manage, and mistakes may be made along the way. But there are times when maybe we should accept our losses and end a failing project before we waste even more hard-earned taxpayer dollars."

According to the article, Karen Evans, the Administrator of the Office of Electronic Government and Information Technology (IT) at the Office of Management and Budget (OMB), told the Senators that OMB list, tracking expensive IT projects that require special attention from top management due to their complexity or potential risk, went from 447 to 553 projects since February.

Evans, trying to positively spin the 19% uptick in this way: "Those figures can be misleading because not all projects on the High Risk List are technically 'at risk.'"

"A successfully performing project may still be classified as high-risk due to exceptionally high costs and or complexity. For example, all e-government initiatives have been determined to be 'high risk' and therefore are reported on agency quarterly reports."

Let me clue OMB in on something - if these high risk IT projects aren't "really" high-risk but OMB shows them as high-risk, OMB at the very least has a big risk measurement problem that needs to be immediately fixed.

Furthermore, given how OMB assesses high-risk IT projects, it is much more likely that OMB is significantly under-counting versus over-counting the actual number of high-risk government IT projects.

It has been absolutely clear for the past several years that OMB is clueless when it comes to IT project risk, and that Sen. Carper is on the right track: kill off under-performing IT projects sooner, not later. It is the only merciful thing to do both for the project participants and taxpayers.

The Washington Post has a deeply disturbing article on the six wayward nuclear cruise missiles of a few weeks ago. A cascading chain of not followed safety procedures led to the nuclear missiles to be loaded onto a B-52 bomber and flown unnoticed across the country in direct violation of 40 years of national policy.

As I mentioned in my previous post on the subject, it appears that risk management had become routine and therefore incredibly sloppy, even though weapons of mass destruction were involved. If some rightly worried military personnel had not leaked the episode to the Military Times, the whole thing may have never seen the light of day. The US Air Force even thought that the event was not going to cause much of public furor, hence "No press interest anticipated." I think the Air Force Public Affairs Office needs a bit of a reality check if they thought that loose nukes were a non-public interest item.

From a risk management standpoint, what irritates me most is that this was a classic "predictable surprise." The article describes how that the Air Force was warned in 1998 of "diminished attention for even 'the minimum standards' of nuclear weapons' maintenance, support and security;" the Air Force Inspector General found in 2003 found that half of the "nuclear surety" inspections conducted that year resulted in failing grades, the worst performance in probably 50 or more years; and; in 2006, the Air Force eliminated a separate nuclear-operations directorate known informally as the N Staff, which closely tracked the maintenance and security of nuclear weapons in the United States and other NATO countries.

The Air Force claims that the N Staff functions were still being done by other Air Force units, but I doubt that these other units viewed their newly acquired mission as a high priority, given the daily stress of dealing with the wars in Iraq and Afghanistan.

The Associated Press reported that Secretary of Defense Robert Gates has asked for an outside review of the incident by the Defense Science Board be conducted on top of the one being conducted internally by the Air Force. While the outside review is said not to be a reflection on whether the Air Force will conduct an honest review, it is hard to read it as other than a "trust, but verify" decision.

PS - Happy 60th Anniversary to the US Air Force.

The San Jose Mercury News (subscription may be required) reported that a US Department of Commerce agent used government computers to spy on the travel movements of his ex-girl friend at least 163 times between mid-2003 and mid-2004. He used the Treasury Enforcement Communications System (TECS) to perform his spying.

According to the US Internal Revenue Service website, "TECS is a computerized information system designed to identify individuals and businesses suspected of, or involved in violation of federal law. The TECS is also a communications system permitting message transmittal between Treasury law enforcement offices and other Federal, national, state, and local law enforcement agencies. The TECS provides access to the FBIâ''s National Crime Information Center (NCIC) and the National Law Enforcement Telecommunication Systems (NLETS) with the capability of communicating directly with state and local enforcement agencies. The NLETS provides direct access to state motor vehicle departments."

The agent faces up to five years in prison, a fine of $250,000 and a three-year term of supervised release. There is no word if his ex-girl friend is going to file stalking charges.

Just in case you missed it, by the end of May 2008 paper tickets will virtually be no more. According to the Associated Press, on June 1, the International Air Transport Association that handles ticketing for most major airlines will stop issuing paper tickets. Some small regional or foreign airlines will continue to issue paper tickets, but they will be a small minority of regional carriers.

I can hardly wait for the day when several major airline reservation and ticketing systems like what happened to All Nippon Airways in July have software problems simultaneously, which will no doubt happen on a day where there is bad weather everywhere.

In Allan Holmes's Tech Insider blog over at Government Executive magazine, he quotes part of the testimony of John Glaser, vice president and CIO for Partners Healthcare in Boston given at Senate Committee on Veterans' Affairs regarding how easy it is to share electronic health records (EHRs). When Glaser was asked what the private sector experience was with sharing EHRs at the scale of what the VA and Defense are trying to do, he said:

"A common EHR? That's interesting to me. That's a codeword for, 'You got to be kidding me.'"

This was undoubtedly a splash of cold water on those Senators who think creating inter-operable EHRs is just a matter of a few lines of software code.

"Unfathomable."

That's how Gov. M. Jodi Rell of Connecticut described the incident involving a computer backup tape that was stolen in June from a car in Ohio that contained bank account among other financial data for nearly all Connecticut state agencies as well as sensitive information on 1.3 million Ohio residents, according to an article in the New York Times.

The tape was in a car of an intern working for Accenture, which was hired by both Connecticut and Ohio to develop a computer systems integrating payroll, accounting, personnel and other fiscal functions. According to the Times report, "Rich Harris, a spokesman for Governor Rell, said yesterday that Accenture seemed to have used the program it created for Connecticut as a template for its project in Ohio, 'and itâ''s our understanding that this is how the data got mixed up' on the tape."

Whoops.

Also, Gov. Reid said,â''The depth and breadth of the bank account data breach is shocking. In essence, the stateâ''s banking information has been laid bare.â''

To make matters worse, in August, a laptop containing 106,000 Connecticut resident social security numbers was stolen from a Connecticut state employee while he was on vacation. Connecticut residents have to think they are snake-bit.

The back story of the whole sordid stolen tape mess can be found in a report by the Ohio Inspector General. It describes schedule and cost pressures on the $158 million computer development that encouraged a culture of shortcuts, poor risk management decisions, as well as cover-up of mistakes. Sounds like another blunder in the making to me.

I wonder if Accenture was giving Connecticut a reduced price on their computer system development given that it seemed to be reusing a lot of work it had been doing on the Ohio system.

As reported in ComputerWeekly, a UK National Health Service (NHS) primary care trust admitted that some 50 staff members viewed the the electronic records of a celebrity who had been admitted into its care. At least it wasn't like what happened to a baseball player in New York a while ago, who had over 150 hospital staff looking at his records.

It has been been argued by electronic health record advocates that medical records are more secure because you will be able to tell who had access to them, therefore this would provide a deterrent to snoops, but as the report above notes, this may be less effective than proclaimed.

On the same day as this story hit (an interesting coincidence), the non-profit group called the E-Health Vulnerability Reporting Program (EHVRP) released their 15-month study assessing the security risks associated with electronic health record (EHR) systems. Quoting from its executive summary:

â'¢ In all cases, evaluated EHR system vulnerabilities could be identified using standard tools and techniques. Subsets of these vulnerabilities were exploited to gain control of the application and access to data to demonstrate the potential consequences.

â'¢ EHR vendors are either not disclosing or inadequately disclosing system vulnerabilities to customers, preventing organizations from appropriately managing risk or implementing compensating controls.

â'¢ No industry organization could be identified that has established guidelines or practices to appropriately mitigate and manage risks associated with ehealth systems.

â'¢ No industry organization could be identified that has the responsibility, charter or mission to address security vulnerabilities in ehealth systems.

The bottom-line: there is a lot more work to do to ensure EHR security and hence privacy.