Okay, how many of you would willingly sign a multi-billion dollar IT outsourcing contract that didn't include a contract deliverable requiring that redundancy be provided for your organization's critical infrastructure networks?

None of you?

Well, we the taxpayers of Virginia need you to go work for our Virginia Information Technologies Agency (VITA).

Apparently, when VITA negotiated its 10-year, $2.3 billion outsourcing contract with Northrop Grumman to modernize Virginia's 85 state government agencies' IT systems and networks, it forgot to require network that backup capability be provided in case of network failure, the Richmond Times-Disptach reported over the weekend.

The new Virginia state CIO, George F. Coulter, was quoted as saying,

"The first thing I noticed was that the network that Northrop Grumman rolled out didn't have redundancy, backup. .. The contract does not call for redundancy in carriers . . . in the network. .. Why that wasn't put into the network, I don't know. This is a service we have to have."

During the first six months of 2009, Virginia's Department of Transportation (VDOT) experienced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One outage, the Times-Dispatch said, took 360 hours to correct. The state's Department of Motor Vehicles (DMV) has experienced over the course of 5 weeks this autumn some 12 outages that put individual DMV offices out of business for a total of more than 100 hours the paper says.

Before the outsourcing contract, VDOT had network redundancy.

So much for the benefits of IT modernization.

The state is now planning an emergency meeting in early December to try to figure out how to negotiate this capability into its outsourcing contract with Northrop Grumman. CIO Coutler says he doesn't having pricing on it yet, nor a timeline on how it would take to roll-out out.

Somehow, I don't think Northrop Grumman is going to provide it cheaply or quickly.

I also wonder how much the lack of providing network redundancy made up some of the original cost savings the IT outsourcing contract supposedly would generate.

You can read more about this troubled contract here and here.

Kind of makes Texas's outsourcing problems look manageable in comparison.

Various news reports like this one in the Wall Street Journal said that eBay suffered a "search glitch" over the weekend. Apparently, a problem Saturday morning meant that users couldn't search out products on US and on some international web sites. The problem was fixed by Sunday morning.

The WSJ quoted an eBay spokesperson as saying that the problem "resulted from a surge in live listings as sellers ramp up for the holiday season."

eBay said that it has more than 200 million listings, some 33% greater than last year at this time.

The company also said it plans to compensate sellers for the problem.

A computer problem with the FAA automated flight planning system in Atlanta caused major delays at Hartsfield-Jackson Atlanta International Airport, the world's busiest, this morning as well as other airports up and down the East Coast of the US. The computer problem started at about 0500 EST and ended at about 1000 EST. Delays are expected to continue through the rest of today, which were already climbing because of bad weather in the US.

A similar problem happened a year ago August. At that time, the FAA promised that it was working to make sure it never would happen again, just as they promised in 2007.

This article from July in eWeek says that the FAA was transitioning then to a new flight planning system that was supposed to end these types of outages and implied that the transition was almost complete. It is unclear, however, whether this latest problem was in this new system or not.

The BBC yesterday reported that T-Mobile has informed the UK Information Commissioner that company employees had illegally sold millions of records relating to thousands of British T-Mobile phone owners to data brokers for "substantial sums." The brokers in turn sold the information which contained contract details to other phone companies, who then cold-called T-Mobile customers as their contracts were expiring.

According to this story in the London Telegraph, T-Mobile said it had alerted the Information Commissioner's Office (ICO) about the illegal sales and had been asked to keep quiet about the incident to help avoid prejudicing an investigation. T-Mobile said it was surprised when the incident was disclosed yesterday by the BBC.

Fines in Britain for breaching of the Data Protection Act amount to only £5,000.

T-Mobile said the actions by its employees was "deeply regrettable" and that "T-Mobile takes the protection of customer information seriously."

A press release by the ICO on the incident can be found here.

In what the company hoped would be seen as good news, this week T-Mobile resumed the sale of its Sidekick smartphone. Sales of the Sidekick had been halted last month because of a massive loss of customer data due to a server problem.

This latest incident isn't likely to increase consumer confidence in T-Mobile, which took a hit over October's contretemps.

The new CIO of the US Department of Homeland Security, Richard Spires, told NextGov in an interview last week that he is personally reviewing 79 large DHS IT programs. According to the current US Government IT dashboard, CIO Spires has initially rated 11 of the 79 as being "red" or have significant concerns, 26 as being yellow, meaning they need attention, and 42 as being "green" or normal.

CIO Spires was quoted as saying, "We'll assess the program, and if it's fatally flawed, we'll stop the program. We're not going to continue to spend money for no benefit. [But] we haven't seen that yet." 

So far, CIO Spires has reviewed some dozen programs, and implies that the risks on those programs need to be better managed. He is also looking for systemic risks in managing or initiating DHS IT programs. He said that, "As we see issues and risks, I'm pushing that we address risks. We're doing that, and we're documenting those, and helping these programs improve on a tactical basis. But I also want to use that as a mechanism to discern where we have systemic weaknesses in managing programs."

Previously, CIO Spires was the CIO of the US Internal Review Service from 2007 through 2008.

Maybe this is the beginning of a trend. As we noted here, the CIO of the US Veterans Affairs Roger Baker earlier this year undertook a review of 45 IT projects there that were in trouble, then canceled 15 of them, and placed the rest on notice that they faced cancellation too if they didn't meet their commitments.

My oh my, holding government IT projects accountable: what a concept!

Okay, we have a little mystery.

Australian newspapers and other media reported late Monday and today that Qantas suffered a three hour check-in system meltdown at about 1700 AEDT on the 15th of November. A computer problem in the airline's Amadeus check-in system reportedly affected Qantas flights world-wide, forcing the airline to use manual check-in procedures for domestic and international passengers. Some flights were delayed as a result.

Okay, it happens.

However, in a couple of the stories, like this one in ZDnet Australia, "Along with Qantas, other global carriers which used Amadeus were also affected. Qantas was working with Amadeus to prevent a similar event occurring again, according to the spokesperson."

This article at new.com.au said that 485 airlines across the world including major airlines such as British Airways, Air France, South African Airways, Thai Airways, Lufthansa and United Airlines were all affected in the global check-in chaos.

However, I have not seen any other news articles on this supposed world-wide passenger check-in chaotic event, which I would expect if 485 airlines across the planet were affected. It would have made for a nice story if true.

So, was this just a local Qantas problem, or something more widespread that mysteriously didn't make the news?

Anyone know?

A year ago October, Texas Gov. Rick Perry suspended the transfer of state records to an IBM data management program saying that serious glitches in Texas's privatized computer system had put state agencies "in danger."

At the time, IT problems had plagued some 20 plus Texas government agencies since IBM won the state's $863 million, seven-year IT outsourcing contract in late 2006. IBM was given 30 days by the Texas Department of Information Resources to come up with a plan to correct the deficiencies, which the state eventually accepted.

However, a highly critical report released appropriately on Friday the 13th by the Texas Department of Information Resources, states that:

"... the Data Center Consolidation Project has experienced serious issues and challenges, and that these issues and challenges have proven very difficult to remediate, [and] that the contract does not align the original business intent of the project with the organizational, financial and operational realities of the State’s complex operating environment. This misalignment establishes dysfunctional relationship dynamics that are enshrined in the very processes prescribed in the contract for governance, oversight and service management, effectively driving the delays and performance issues experienced to date and at the same time assuring that tactical remediation initiatives will fail."

As a result, the morale on the project is low (turnover is running at 30 percent, which is very high given the current US IT job market), there is a large and growing backlog of unresolved contract disputes and unfulfilled requests for services, and there is a growing risk of major disruption of state government business because server back-up issues - the same ones that helped cause cure notice to be sent to IBM last year - have not been mitigated. Other critical problems also exist.

The report also in essence states that both Texas and IBM are at fault in the on-going fiasco and was caused by the same disease: abject naivety and excessive optimism about what could be accomplished and how to go about it.  Yet another case of a state's risk mismanagement of a large IT outsourcing contract.

The report states that the contract is "unsustainable" in its current form, but is salvageable, mostly because the outsourcing idea is a good one. 

The outsourcing idea may be a good one for Texas (and I think many people may debate that), but I believe from what I read in the report that claiming the contract is salvageable is more than a bit optimistic, and that Indiana's remedy to its recent outsourcing problem also involving IBM is probably a more realistic solution. Contract termination fees, however, may preclude this option from being considered. Anyone know?

That said, the report goes on to recommend that the contract be re-examined and fixed by February 2010 (at no extra cost to Texas taxpayers, of course) if things are not to become even worse.

A related article in Dallas Morning Star says that so far, the state has saved only $9.7 million through the contact over its first 23 months; the contract is intended to save the state $178 million by 2014.

In addition, the paper notes that in September, Texas Secretary of State Hope Andrade pulled "the state elections system out of the contract, weeks after a 13-day hardware outage denied the public access to records in her business-filings office." The Secretary of State's office felt that Texas's ability to conduct fair and credible elections was at risk if it did not.

The Star quotes Tony Befi, IBM's senior state executive for Texas, as saying that IBM "remains committed" to making the outsourcing contract successful.

Good luck.

Also, IBM announced last week "the opening of the sixth in a network of analytics solution centers - this one dedicated to helping federal agencies and other public sector organizations extract actionable insights from their data."

IBM said that, "The center's staff will work with federal agencies and other clients to apply breakthrough streaming technologies, mathematical algorithms, and modeling. Using these tools, IBM will help clients optimize individual business decisions, processes and even entire business models, as well as manage risk and fraud and, ultimately, improve the delivery of public services."

Sounds like to me that maybe they need to apply the staff's knowledge to figuring out how to make their federal as well as state IT outsourcing contracts work first.

The Japan Times is reporting that Toyota Motor Corporation will be offering to fix the gas pedals of the 3.8 million cars involved in its floor mat recall. The recall is expected to be voluntary rather than mandated by the U.S. National Highway Traffic Safety Administration (NHTSA).

According to the Times, "Toyota opted to offer to fix the gas pedals rather than issue a recall because it says the pedal's shape is not faulty."

The cost to Toyota is expected to be in the tens of billions of yen.

While helpful, I do not think this action will necessarily dampen the suspicion that something other than floor mats or the gas pedal are the root cause of the spate of sudden unintended acceleration accidents involving Toyota vehicles.

Toyota's view on the subject can be found here.

Image of commemorative Sentinel project coin from Mountain Skies catalog

The FBI Sentinel System, the replacement for the infamous Virtual Case File (VCF) fiasco that Spectrum's senior editor Harry Goldstein wrote about in exquisite detail in his report, "Who Killed the Virtual Case File," has slipped its schedule another three months but has managed to stay within its $456 million budget, according to an audit of the Justice Department's Inspector General (IG). The partially redacted IG report can be found here.

Sentinel, according to the FBI, "... strengthen[s] the FBI’s capabilities by replacing its primarily paper-based reporting system with an electronic system designed for information sharing."

The IG audit report states that the project is currently in Phase 2 of its 4 phases, with an expected completion schedule of September 2010; the original schedule was December 2009. The project slipped three more months since the last audit in December 2008. However, the overall cost has not changed, nor the final expected functionality.

That said, the functionality expected out of Phase 2 was less than expected. As stated in the audit report, "The FBI and Lockheed Martin encountered significant challenges deploying new electronic versions of forms used by FBI agents during investigations that functioned as intended and met user requirements."

In addition, users found Sentinel to be slow, which the audit report attributed to the FBI's aging internal network infrastructure, which is receiving a $39 million upgrade that is expected to be completed by the end of the year.

The IG also noted that there were still some risks that had to be dealt with, especially given Sentinel's "aggressive schedule, scope, and importance of Sentinel’s implementation."

The audit report stated that, "... due to the aggressive schedule, scope, and importance of Sentinel’s implementation, the project requires a highly skilled and integrated project management staff. We have concerns with the staffing of the project because of a recent increase in turnover among project staff members, vacancies within the Sentinel PMO, and because the Sentinel PMO Staffing Plan does not reflect the current staffing levels or skills needed for the project."

As Harry Goldstein's article noted - and the IG does again - turnover helped kill VCF and is something that needs to be addressed as a priority. The IG says that, "The Sentinel PMO [Project Management Office] lost staff in key positions, including Deputy Program Manager, Contracting Officer’s Technical Representative, Program Support Unit Chief, and Quality Manager. While the Program Support Unit Chief and the Contracting Officer’s Technical Representative positions were filled by existing Sentinel PMO staff, the replacement staff’s previous positions were left vacant."

Finally, and partly as a result of these risks, the IG says that "the FBI has limited funding for Phase 3 until Phase 2 is completed in order to ensure program continuity and retention of contractor personnel."

Read into that as you will - I see a not too subtle threat to Lockheed Martin, the contractor responsible for implementing Sentinel, not to lower the internal corporate priority of this project.

A year ago May, I wrote about British Gas suing Accenture for £182 million ($365 million) over an IT project called Project Jupiter which British Gas claimed reduced its customer-billing process to "shambles."

There are two nice stories from 2008 on the dispute here and here in the London Times.

At the time, an Accenture spokesman was quoted by the Times as saying that Centrica [the parent company of British Gas] "conducted extensive testing" on the system before it was handed over. 

The spokesperson added: "We are confident, based on the facts of the situation, that this claim is baseless and without merit. Accenture will vigorously defend the High Court proceedings."

Well, a UK High Court agreed last week that maybe the British Gas claims weren't so baseless after all.

According to a story in this week's London Times,

"On Friday [November 5] the High Court issued a judgment on the preliminary issues raised by Accenture, relating to the wording of a contract between the two companies and a warranty claim. The judgment from Mr Justice Field at the High Court in Winchester was in favour of British Gas on all six points" of alleged contract breaches by Accenture.

The court didn't say that the claims made by British Gas were correct, but that the company had a right to test them in court.

The Times reported that Accenture disagreed with the ruling and planned to appeal. The paper quotes an Accenture spokesperson saying, "There has been no final determination of the case or any of the detailed underlying facts of this case. We remain confident that Centrica’s claim is baseless and that Accenture will prevail when the factual issues are examined in detail at trial."

Hmm, I noticed that the Accenture spokesperson didn't say the British Gas claims were "without merit" this time around.

I wrote in 2008 that I thought the affair would be settled out of court before a trial occurs - I think that even more so now.