Last year, the Virginia Information Technologies Agency (VITA) experienced a server problem that caused havoc across some 26 Virginia state agencies for a week. The outage was initially caused by a hardware error that was then exacerbated by human error in trying to remedy it.
At the time, VITA said that it would be looking for compensation for the outage from Northrop Grumman, which has a $2.3 billion IT outsourcing contract with the state. Well, it has taken 8 months, but last week Virginia and Northrop Grumman finally negotiated a settlement.
"Northrop Grumman will provide $4.748 million in financial compensation and operational improvements to the Commonwealth for losses from the computer outage last August that impacted several state agencies, including the Department of Motor Vehicles. In addition, Northrop Grumman has agreed to implement a corrective action plan that addresses the findings and recommendations in the independent third-party audit report."
Furthermore, the press release says that:
"... the financial package being provided by Northrop Grumman includes monthly invoice credits over the next 24 months totaling $1.908 million. This represents the entire amount of direct costs incurred by the Commonwealth as a result of the disruption.
"Northrop Grumman also will expend $2.09 million making operational improvements to database backup, storage systems, system monitoring, database monitoring, network monitoring and capacity management."
"Northrop Grumman also will implement and pay for 'point in time' or 'snap/clone' technology to protect the data held by the state, and will address additional recommendations made in the Agilysys review of the outage. These remedial actions are valued at approximately $750,000."
"In addition, the company previously paid $250,000 for the cost of the independent third-party audit performed by Agilysys."
The Agilysys audit report summary is here (PDF).
Which brings me to this statement in the Governor's press release:
"Virginia Chief Information Officer Sam Nixon added, 'I believe that the risks and problems identified in the audit have been satisfactorily addressed. Moving forward, I am confident that the improvements will make our infrastructure more reliable and resilient.' "
I find the wording in the above - especially the phrase "moving forward" - interesting. You see, since the outage last September, there have been at least four more VITA-related computer problems, one in January, two in March and another just last month. The first three involved the Department of Motor Vehicles and caused outages lasting from twenty minutes to most of a day. The last incident, however, crippled Virginia's State Police email system and lasted for several days.
It will be interesting to see whether VITA will now seek new compensation from Northrop Grumman for these post-outage outages, or whether last week's compensation agreement covers these four as well. Alas, the phrase "moving forward" tends to make me (and probably most other Virginia taxpayers) think the latter will be the case, unfortunately.
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.