Last Friday, I blogged about a Virginia Information Technologies Agency (VITA) server problem that occurred two days before which was still affecting 24 Virginia state agencies. Some agencies were knocked off-line, including VITA itself. VITA stated that how badly a state agency was affected was dependent upon where its information was stored on the state's servers.
Virginia's Department of Motor Vehicles (DMV) was the most visibly affected, with no in-person license renewals possible at its 74 locations across the state.
I also mentioned in my blog post that I thought the incident would likely raise questions once more about whether Virginia was getting value from its contentious IT outsourcing contract with Northrop Grumman. Those questions have gotten significant louder since Friday, with Virginia's governor now asking for a probe into the (still on-going) incident.
One reason this incident has turned into a political hot potato is that on Friday afternoon, VITA basically promised that everything would be working normally by yesterday morning (Monday). An article in Saturday morning's Richmond Times-Dispatch reported that:
"Samuel A. Nixon Jr., head of the Virginia Information Technologies Agency, said VITA and contractor Northrop Grumman 'absolutely' expect affected agencies will be operating by 8 a.m. Monday."
The Times-Dispatch story also noted that 25 (not 24) agencies and the governor's office were hit by the outage. By Saturday morning, a dozen state agencies were still being affected, the Times-Dispatch said.
However, by Monday morning, three state agencies were still having major problems: the Department of Motor Vehicles, the Department of Taxation and the State Board of Elections. The Department of Taxation's website says that:
"We are currently experiencing technical problems that prevent taxpayers from being able to file tax returns and make payments. If you need to file your withholding return and/or make your withholding payment, please be assured that we are working quickly to resolve the problem. In addition, if your withholding filing is late because of this problem, you will not be assessed penalty and interest."
With the state hurting for revenue, not being able to collect taxes in a timely fashion doesn't go over well in the governor's office.
Other agencies also are still experiencing "minor issues," VITA acknowledges.
While few Virginia residents are likely to get too worked up over the Department of Taxation having problems, the DMV is another matter. VITA said Monday morning that the DMV would be working by Tuesday, but by Monday evening had to backtrack. The current VITA timeline for the problems to be resolved is tomorrow morning (Wednesday) - maybe.
And if things are fixed tomorrow as promised, the DMV says to expect very long lines.
Late yesterday, VITA finally gave out a bit more detailed information about what happened:
"On Wednesday, August 25, at approximately 3 p.m., the Commonwealth of Virginia experienced an information technology (IT) infrastructure outage that affected 27 of the Commonwealth’s 89 agencies and caused 13 percent of the Commonwealth’s file servers to fail. The failure was in the equipment used for data storage, commonly known as a storage area network (SAN). Specifically, the SAN that failed was an EMC DMX-3."
"According to the manufacturer of the storage system, the events that led to the outage appear to be unprecedented. The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999 percent and no similar failure has occurred in more than one billion hours of run time. A root cause analysis of the failure is currently being conducted."
VITA indicated late Friday that Northrop Grumman would be paying a financial penalty of "at least $100,000" for the outage. I think the financial figure associated with the term "at least" has gotten much bigger since then.
Yesterday afternoon, the Virginia's Governor Bob McDonnell stated that he now wants an "independent third party" investigation of what happened and why. The Times-Dispatch in this story quotes the governor as saying,
"I am not pleased that our employees and citizens have experienced this disruption in service."
I expect others in the state legislature who haven't been happy with the VITA contract to Northrop Grumman will be calling for their own investigation and or hearings soon as well.
How many state residents have been affected is not known, and hopefully the governor's investigation can come up with a good estimate. A Washington Post story today says that, "About 5,000 Virginians who cannot renew their licenses or state-issued identification cards online have had them expire since the meltdown ..."
Of course, that number doesn't include the thousands of others who want to get their licenses for the first time but can't.
In addition, "The Virginia Association of Chiefs of Police asked local and state law-enforcement agencies Monday to stop issuing tickets for those who have been unable to renew their licenses" between August 25 and August 31.
However, the Post noted, if you were received a ticket for an expiring license before the stop ticketing request came into force, then you will have to "go through the court system" to get the ticket resolved.
And also sit in long DMV lines to boot.
That should be punishment enough for anyone.
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.