Last year, the Virginia Information Technologies Agency (VITA) experienced a server problem that caused havoc across some 26 Virginia state agencies for a week. The outage was initially caused by a hardware error that was then exacerbated by human error in trying to remedy it.

At the time, VITA said that it would be looking for compensation for the outage from Northrop Grumman, which has a $2.3 billion IT outsourcing contract with the state. Well, it has taken 8 months, but last week Virginia and Northrop Grumman finally negotiated a settlement.

According to Virginia Gov. Bob McDonnell'spress release:

"Northrop Grumman will provide $4.748 million in financial compensation and operational improvements to the Commonwealth for losses from the computer outage last August that impacted several state agencies, including the Department of Motor Vehicles.  In addition, Northrop Grumman has agreed to implement a corrective action plan that addresses the findings and recommendations in the independent third-party audit report."

Furthermore, the press release says that:

"...  the financial package being provided by Northrop Grumman includes monthly invoice credits over the next 24 months totaling $1.908 million. This represents the entire amount of direct costs incurred by the Commonwealth as a result of the disruption.

"Northrop Grumman also will expend $2.09 million making operational improvements to database backup, storage systems, system monitoring, database monitoring, network monitoring and capacity management." 

"Northrop Grumman also will implement and pay for 'point in time' or 'snap/clone' technology to protect the data held by the state, and will address additional recommendations made in the Agilysys review of the outage. These remedial actions are valued at approximately $750,000."

"In addition, the company previously paid $250,000 for the cost of the independent third-party audit performed by Agilysys."

The Agilysys audit report summary is here (PDF).

According to the Washington Post, Northrop Grumman was pleased with the deal

Which brings me to this statement in the Governor's press release:

"Virginia Chief Information Officer Sam Nixon added, 'I believe that the risks and problems identified in the audit have been satisfactorily addressed.  Moving forward, I am confident that the improvements will make our infrastructure more reliable and resilient.' "

I find the wording in the above - especially the phrase "moving forward" - interesting. You see, since the outage last September, there have been at least four more VITA-related computer problems, one in January, two in March and another just last month. The first three involved the Department of Motor Vehicles and caused outages lasting from twenty minutes to most of a day. The last incident, however, crippled Virginia's State Police email system and lasted for several days.

It will be interesting to see whether VITA will now seek new compensation from Northrop Grumman for these post-outage outages, or whether last week's compensation agreement covers these four as well. Alas, the phrase "moving forward" tends to make me (and probably most other Virginia taxpayers) think the latter will be the case, unfortunately.

The Conversation (0)

Why Functional Programming Should Be the Future of Software Development

It’s hard to learn, but your code will produce fewer nasty surprises

11 min read
A plate of spaghetti made from code
Shira Inbar

You’d expectthe longest and most costly phase in the lifecycle of a software product to be the initial development of the system, when all those great features are first imagined and then created. In fact, the hardest part comes later, during the maintenance phase. That’s when programmers pay the price for the shortcuts they took during development.

So why did they take shortcuts? Maybe they didn’t realize that they were cutting any corners. Only when their code was deployed and exercised by a lot of users did its hidden flaws come to light. And maybe the developers were rushed. Time-to-market pressures would almost guarantee that their software will contain more bugs than it would otherwise.

Keep Reading ↓Show less