The Virginia Information Technologies Agency (VITA) server problem that began a week ago Wednesday afternoon was finally "fixed" late this past Wednesday night, thus allowing all state agencies to operate normally, including Virginia's Department of Motor Vehicles (DMV) was hard hit by the outage.
To try to reduce the backlog of some 35,000 to 45, 0000 drivers trying to renew their licenses as well as get new ones, Governor Bob McDonnell ordered many DMV offices to stay open over the Labor Day holiday weekend. That has no doubt "improved" the working relationship of Northrop Grumman, the state's IT outsourcing contractor, with DMV employees. The relationship wasn't too great before this latest outage.
The DMV has also extended the validity of licenses and id cards by 20 days in wake of the outage. State and local law enforcement agencies have also been notified of the extension.
As I predicted, the mess has turned into a political football, with Gov. McDonnell trying to keep the fallout to a minimum. He has called for an independent review to be conducted together with the Joint Legislative Audit and Review Committee (JLARC) - which will be paid for by Northrop Grumman - to look into the incident. No timetable for the review has been announced.
[Update: The review is to be completed within three months.]
By including the JLARC in the review, the governor hopes to keep the state's legislature from conducting its own investigation. That hope may go by the wayside if a still unresolved problem isn't able to be fixed.
It turns out that the between 12,000 and 16,000 photographs for licenses and identity cards cannot be found, the Richmond Times-Dispatch says. If they cannot be recovered, and Virginians like myself have to be called back in to the DMV to have new id cards and licenses made, then the political football will have gotten greased.
[Update: At least 4,240 driver's license and ID card applicants are being sent letters by the DMV to come back and have their pictures taken again. These unfortunate folks had their photos taken on the day of the server meltdown.]
The Governor said in a press release yesterday that:
"I have spoken personally with Northrop Grumman CEO Wes Bush. I expressed to him that extended lapses in state computer services was an unacceptable hardship on our citizens and employees. I made clear that I expected the best around-the-clock recovery efforts possible in order to reclaim and restore all missing files and data. I appreciate the company's commitment to a full and comprehensive recovery from this system failure, and the agreement that Northrop Grumman would pay for the reasonable cost of the independent review that must take place immediately."
"This recent computer failure is unacceptable. I look forward to learning exactly how this occurred, how we can prevent such a disruption in the future, and how we can improve responsiveness and data reclamation if future interruptions occur. The report produced by our external vendor will be thorough and detailed. And, we will move forward quickly and resolutely once it has been received and reviewed by the executive and legislative branches."
Northrop Grumman is looking at a hefty bill for the outage. According to this story in the Richmond Times-Dispatch, a single 31-hour outage at one agency - the DMV - resulted in labor costs of $200,000 alone. This time there were 26 state agencies, several which were out for a week. Northrop will also be charged for the cost of those DMV folks that now have to work this holiday weekend.
Northrop Grumman, which has not looked too good throughout this affair, probably made things worse yesterday.
The Times-Dispatch reported that Linda A. Mills, a Northrop Grumman vice president, said in a written statement that:
"Northrop Grumman deeply regrets the disruption and inconvenience this has caused state agencies and Virginia citizens."
However, the Times-Dispatch also said that, "Company spokeswoman Christy Whitman declined to comment on whether the statement is an apology.
Huh?
That refusal is about as dumb as it gets from a PR perspective.
I expect some in Virginia state government to ask very soon and very loudly how deeply Northrop Grumman actually regrets the outage.
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.
