Last week saw an overflowing cornucopia of IT problems, challenges and failures being reported. From these rich pickings, we decided to focus this week’s edition of IT Hiccups first on a multi-day computer problem affecting the US Department of State’ passport and visa operations, followed by a quick rundown of the numerous US and UK government IT project failures that were also disclosed last week.
According to the Associated Press, beginning on Saturday, 21 July, the U.S. Department of State has being experiencing unspecified computer problems including “significant performance issues, including outages” with its Consular Consolidated Database [pdf], which has interfered with the “processing of passports, visas, and reports of Americans born abroad.” A story at ComputerWorld indicates that the problems began after maintenance was performed on the database. State Department spokeswoman Marie Harf told the AP that the computer problem effects were being felt across the globe.
The AP story says that a huge passport and visa application backlog is already forming, with one unidentified country already reporting that the backlog of applications had reached 50,000 as of Wednesday. The growing backlog has also “hampered efforts to get the system fully back on line,” Haff told AP.
The rapidly expanding backlog is easy to understand, as the Oracle-based database, which was completed in 2010, “is the backbone of all consular applications and services and supports domestic and overseas passport and visa activities,” according to a State Department document [pdf]. In 2013, for example, the database was used in the issuing of some 13 million passports and 9 million visitor visas.
Department spokeswoman Harf was quoted by the AP as saying, “We apologize to applicants and recognize this may cause hardship to applicants waiting on visas and passports. We are working to correct the issue as quickly as possible.” However, she did not give any indications when the problems would be fixed or the backlog would be erased. Stories of families stuck overseas and not able to return to the US are rapidly growing.
Earlier this summer, the UK saw a similar passport backlog develop over the mismanagement of the closures of passport offices at British Embassies during the past year. The backlog, which blossomed into a political embarrassment to Prime Minister Cameron’s Government, is still not fully under control. It remains to be seen whether the U.S. passport and visa problems will do the same for the Obama Administration—if it lasts for a couple of weeks, it very well could.
More likely to cause embarrassment to the Obama and the Cameron administrations are the numerous government IT failures reported last week. For example, the AP reported that the U.S. Army had to withdraw its controversial Distributed Common Ground System (DCGS-A) from an important testing exercise later this year because of “software glitches.” DCGS-A, the Army website says, “is the Army’s primary system for posting of data, processing of information, and disseminating Intelligence, Surveillance and Reconnaissance information about the threat, weather, and terrain to all components and echelons.”
The nearly $5 billion spent on DGCS-A so far has not impressed many of its Army operational users in Afghanistan, who have complained that the system is complex to use and unreliable, among other things. They also point out there is a less costly and more effective system available called Palantir, but the Army leadership is not interested in using it after spending so much money and effort on DCGS-A.
The AP also reported last week that a six year, $288 million U.S. Social Security Administration Disability Case Processing System (DCPS) project had virtually collapsed, and that the SSA was trying to figure out how to salvage it. DCPS, which was supposed to replace 54 legacy computer systems, was intended to allow SSA workers across the country “to process claims and track them as benefits are awarded or denied and claims are appealed,” the AP said.
The AP story says that the SSA may have tried to keep quiet a June report [pdf] by McKinsey and Co. into the program’s problems so as to not embarrass Acting Social Security Commissioner Carolyn Colvin who President Obama recently nominated to head the SSA. The McKinsey report indicates that one reason for the mess is that no one could be found to be in charge of the project. The report also states that “for past 5 years, Release 1.0 [has been] consistently projected to be 24-32 months away.” Colvin was deputy commissioner for 3½ years before becoming acting commissioner in February 2013, the AP says, so the DCPS debacle is squarely on her watch.
Then there was a story in the Fiscal Times concerning a Department of Homeland Security (DHS) Inspector General report [pdf] indicating that the Electronic Immigration System (ELIS), which was intended to “provide a more efficient and higher quality adjudication [immigration] process,” was doing the opposite. The IG wrote that, “instead of improved efficiency, time studies conducted by service centers show that adjudicating on paper is at least two times faster than adjudicating in ELIS.”
Why, you may ask? The IG states that, “Immigration services officers take longer to adjudicate in ELIS in part because of the estimated 100 to 150 clicks required to move among sublevels and open documents to complete the process. Staff also reported that ELIS does not provide system features such as tabs and highlighting, and that the search function is restricted and does not produce usable results.”
Hey, what did those immigration service officers expect for the $1.7 billion spent so far on ELIS, something that actually worked? DHS is now supposed to deploy an upgraded version of ELIS later this year, the IG says, but he is also warning that major improvements in efficiency should not be expected.
As I mentioned, reports of project failure were the story of the week in the UK as well. Computing published an article concerning the UK National Audit Office’s report into the 10-year and counting Aspire outsourcing contract for the on-going modernization and operation of some 650 HM Revenue & Customs tax systems. While the NAO has said that the work performed by the consortium led by Capgemini has resulted in a “high level of satisfactory implementations,” the cost to do so has been a staggering amount.
HMRC let the Aspire contract in 2004, after ending a ten-year outsourcing contract with EDS (now HP) when the relationship soured. HMRC said at the time that the ten-year cost of the Aspire contract would be between £3.6bn and £4.9bn; however, the NAO says the cost has topped £7.9 billion through the end of March this year, and may reach £10.4 billion by June 2017 when the contract, which was extended in 2007, expires. Public Accounts Committee (PAC) chair Margaret Hodge MP says the cost overrun is an example of HMRC’s management of the Aspire contract being “unacceptably poor.”
On top of being unhappy about the doubling in contract costs, and the high level of profits the suppliers made on it, the NAO also warned HMRC that it needs to get serious about a replacement contract when the Aspire contract ends. Hodge says that while HMRC has started planning Aspire’s replacement, “its new project is still half-baked, with no business case and no idea of the skills or resources needed to make it work.”
Apparently the NAO found another half-baked UK government IT project as well. According to the London Telegraph, the NAO published a report [pdf] describing how the UK Home Office has managed to waste nearly £347 million since 2010 on its “flag ship IT programme” called the Immigration Case Work system, which is intended to deal “with immigration and asylum applications.” The NAO says that the Home Office has now abandoned the effort, thereby, “forcing staff to revert to using an old system that regularly freezes.”
In addition, the NAO says that the Home Office is planning to spend at least another £209 million by 2017 on what it hopes to be a working immigration case work system. Until that new system comes on line, however, the Home Office will need to spend an undetermined amount of money trying to keep the increasingly less reliable legacy immigration system from completely falling over dead. The legacy system support contract ends in 2016, the NAO states, so that Home Office doesn’t have a lot of wiggle room to get its new replacement immigration system operational.
Finally, the London Telegraphreported that the UK National Health Service may have reached a deal to pay Fujitsu £700 million as compensation for the NHS unilaterally changing the terms of its National Program for IT (NPfIT) electronics health record contract with the Japanese company. The changes sought by the NHS led Fujitsu to walk off the program (as did Accenture) in 2008. The NPfIT project, a brain child of then Prime Minister Blair in 2002, was cancelled in 2011 after burning through some £7.5 billion so far.
In Other News…
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.