After a couple of quiet weeks, IT related snafus, snarls and ooftas reappeared with a vengeance last week. We start off with several U.S. state governments’ IT systems that have had better weeks.
Oregon, New Mexico, Kansas, North Carolina, New Jersey and Iowa Experience IT Problems
Last Monday evening at 7 p.m. Pacific time, government contractor Hitachi initiated a planned hardware upgrade to add storage capacity to Oregon’s State Data Center (SDC) computer systems located in Salem, Ore. The work, according to Matt Shelby, spokesman for the Department of Administrative Services, “was not supposed to cause any disruptions,” a story at the Oregonian reported. However, Shelby told the paper, “During the course of that work, a catastrophic failure and major connectivity issues arose.”
As a result, 90 state agencies were unable to connect with the data center and thus to each other. The outage knocked out a range of services including Oregon’s Department of Transportation TripCheck road cameras, government e-mail and websites, and the processing of some 70 000 state unemployment checks. The hardware problem was fixed Tuesday morning, and state services were restored by midday Tuesday.
Next, New Mexico continues to have trouble with its new unemployment computer system as well, an AP story from last week reports. An audit (pdf) of the $48 million system—which was originally supposed to cost half that—was released last week by the state's Legislative Finance Committee. The report stated that the new system, which went into operation in January, “is complex and can be difficult for users to navigate.” According to the AP story, businesses and individuals who've used the system say that the phrase “can be difficult” should be changed to “is exceedingly difficult”—so much so that a New Mexico state senator said that the state should consider going “straight back to paper because the system doesn’t work.”
The head of the state agency responsible for the system, Workforce Solutions Secretary Celina Bussey, told the AP that she considers the unemployment system implementation a success, but concedes that maybe the system interface could be more user-friendly. That may be a bit of an understatement: for example, it previously took unemployed workers 15-minutes to make a claim; now, with the new system, it takes up to an hour, the audit reported. Unemployed workers calling the state's help lines with their applications face long telephone wait times as well, the audit states.
In addition, the audit reports that the unemployment system suffers from “data conversion defects, limited application testing and the lack of a contingency and disaster recovery plan,” the latter of which it says creates “conditions of risk.” In other words, don't be surprised if the new unemployment system suddenly keels over.
Moving on to Wichita, Kansas, where television station KWCH reported last week that the unemployment system in Kansas has been having problems of its own. According to the story, the Kansas Department of Labor “was upgrading its servers when it discovered a software problem” that caused a delay in the issuing of an unknown number of unemployment checks for over a week. On top of not being able to tell how many Kansans were affected by the software problem, the Department of Labor didn't offer an explanation on why it kept the information about the problem quiet until the television station made inquiries about it.
North Carolina got a double dose of government IT system problems last week. First, the state’s new family assistance system NC FAST (North Carolina Families Accessing Services through Technology) reportedly experienced difficulties, according to a story Fox News Channel WGHP in Highpoint, North Carolina. The story reported that, “several Department of Social Services [offices] across the state [were] reporting glitches” that were keeping families from receiving food stamps. The state said it was trying to determine what was causing the problems and fix them.
North Carolina also continued to have problems with its new, expensive and controversial computerized Medicaid billing system NCTracks, which went live on 1 July. Some businesses have been claiming the new system has been a nightmare to deal with. I'll be writing more about the issues with NCTracks later this week.
Then on Friday, the New Jersey Motor Vehicle Commission’s computer system was offline for the entire day. According to a story at the New Jersey Journal, a “fire alarm was activated at one of the state’s data centers, causing the state’s website, including the MVC division, to shut down automatically” at about 2 a.m. Friday morning. While data center technicians were able to get most other state agencies back online by mid-morning Friday, they weren't successful in getting the MVC and “one section of the Department of Labor and Workforce Development” up and running, an article at the Record reported. MVC offices stayed open an extra hour on Saturday to compensate for Friday's outage.
Finally, last Friday night, a “faulty piece of equipment on the Iowa Communications Network, the state-owned fiber-optic system,” caused emergency 911 calls made from cell phones across Iowa to be “routed to out-of-town” call centers, a story at the Des Moines Register reported. An AP story stated that “a vendor tried to install new software to fix the problem, but that made it worse.” In addition, the “backup system also failed to activate,” the AP stated. The routing problem was finally fixed late Saturday morning.
PayPal: What's $92 Quadrillion Between Friends?
Back in 2007, Joe Martins closed a bank account at Wachovia Bank and got a letter inquiring about when he was going to pay off his US $211,010,028,257,303.00 outstanding balance. Well, if that happened to Chris Reynolds, it would not be a problem.
According to a story last week at the Philadelphia Inquirer, thanks to the generosity of PayPal, Reynolds was worth $92,233,720,368,547,800.00. Heck, even after paying off Martins’ $211 trillion debt, Reynolds would still have a nice $92 quadrillion and change left over to maybe buy himself a nice country or three.
Alas, it was all a mistake. PayPal told Reynolds that his real account balance was a measly $0.00, but it would “donate an unspecified amount of money to a cause of Reynolds' choice,” as a way to make up for the error, CNN reported.
Texas Lubbock Power & Light Apologizes for Billing Error
While Reynolds and the folks at PayPal were having a good laugh at the quadrillion dollar accounting mistake, customers of Lubbock Power & Light were less amused at the billing error they were told about last week.
According to the Lubbock Avalanche-Journal, Lubbock Power & Light officials held a news conference last Friday where they confirmed that there was a software “billing glitch” with its new customer billing system. The error meant some 44 000 customers were undercharged for their June electric bills, and the difference was applied to their July bills. The simultaneous addition of a 9.7 percent rate hike that went into effect on 1 June, as well as an increase in LP&P’s standard service charge, caused the July bills to, in the words of the utility's spokespersons, “appear disproportionately high.”
LP&P apologized for the “unfortunate and regrettable” billing error and its “communication errors” in not informing customers about the problem before the higher bills started to arrive in customer mailboxes.
Also of Interest…
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.