2018’s IT Failures Already Have a Familiar Look

The more things change, the more things seem to stay the same, at least for international travelers arriving in the United States over the New Year’s holiday period. For a second year in succession, the U.S. Customs and Border Protection (CBP) computer systems experienced an outage that left thousands of passengers across the United States waiting in long lines to clear customs. This time, the outage was only for about two hours, while last year’s lasted four hours and affected more than 13,000 passengers on 109 flights, according to a Department of Homeland Security Inspector General report released last November that investigated the disruption. The DHS IG report indicated that the 2017 New Year’s problem was caused by an inadequately tested software change related to CBP’s long-running IT modernization effort.

No official cause or total number of passengers or flights affected has been given for the latest CPB computer hiccup. However, another IT modernization-related issue is a likely culprit given that a September 2017 Homeland Security IG report assessing the state of the Customs Department’s IT systems and infrastructure indicated that the main CPB computer system used to screen international passengers has seen its performance “greatly diminished over the past year as a result of ongoing efforts to modernize (its) underlying system architecture.” Before this latest outage, there were three other service disruptions in 2017, according to the IG report.

The few hours of distress suffered by international travelers, however, is minisucle in comparison to that of the tens of thousands of Canadian federal government workers who are now facing a third year of payroll system torment. In what is quickly moving into contention as one of the worst government-managed IT implementations ever, over half the 290,000 plus civil servants paid through the IBM-developed Phoenix pay system have been underpaid, overpaid, or not paid at all since its rollout began in February of 2016. Government records show that, as of November 2017, there were some 589,000 payroll-related transactions still awaiting processing, meaning many government employees are contending with several pay issues. For instance, in Canada’s Department of National Defense, 63 percent of its workers  had outstanding pay issues as of 1 November 2017, with 15 percent having three or more outstanding problems to contend with. According to Canada’s Auditor General, nearly 50,000 government workers have had to wait over a year to get their pay straightened out.

A major objective of the Phoenix system—which traces its history back to 2009—was to save the government C$70 million per year through reductions in payroll processing overhead and staffing costs.  However, things have not turned out as planned. While the original cost of the project was pegged at C$309.5 million, Public Services and Procurement Minister Carla Qualtrough, who is now in charge of the project, admitted in November that it might cost as much as C$1 billion and three or more additional years to completely fix the system. The added costs include hiring hundreds of new payroll staff—including some of the 2,700 laid off when Phoenix was introduced—to try to sort out the mess.

The pain has been acute for the thousands of public workers who have received less than their correct salary, but those thousands who have been overpaid have not escaped misery, either.  For this latter group, the government delivered a New Year’s surprise: notice that they will have until 31 January 2018 to pay back any overpayments they received. If they don’t, they have been told they will have to pay back not the net salary overpayment after taxes were taken out, but the gross salary overpayment the workers erroneously received.  They then will have to wait to claim the difference back on their personal income tax filings in May, with refunds coming who knows when. To say the least, demanding  money that they did not receive and further complicating their tax returns has not made the affected government employees very happy, given that most have already spent months trying to get their pay straightened out to no avail.

UHIP Ain’t Hip

The Canadian Phoenix payroll system debacle is not the world’s only long-running government IT fiasco. The state of Rhode Island has been having its own troubles with the $364 million Unified Health Infrastructure Project (UHIP) public assistance program that it rolled out in September 2016 to great fanfare. Like Phoenix, UHIP was intended to save the state millions of dollars per year by again reducing processing and staffing costs. However, because of myriad operational problems, the cost of UHIP is now pegged at $492 million, not counting the $85.6 million credited back to the state by prime contractor Deloitte. As a point of reference, UHIP was originally slated to cost between $110 million and $135 million and be ready to go live in April 2015.

Since its debut, the barrage of significant errors in UHIP has meant thousands of Rhode Island’s neediest families have not received the public assistance payments they were eligible for. Many benefit-eligible families have waited three months or longer (and some over a year) before finally getting public assistance. Flaws in the system still keep showing up. Last October, for example, it was discovered that thousands of applications for benefits were never processed. The Rhode Island American Civil Liberties Union, which has sued the state multiple times over problems with UHIP, claimed this month that one in three eligible families are still not getting benefits in a timely manner, nor are eligible recipients being informed why they were losing their Medicaid coverage as required by federal law. Due to the one of lawsuits, Rhode Island finally had to accept, last autumn, a Federal District Court appointed special master to oversee the system and try to make it work properly. The hope is that later this year, UHIP will finally begin to operate reliably, although no one in government is willing to stake their reputation on that happening.

Phoenix and UHIP have a few more things in common, beyond being costly, long-lived operational IT disasters. Both went live against expert advice according to investigative reports into both Phoenix and UHIP. The reports indicate that senior government officials were not only naïve about the inherent complexity of their systems’ development and overly optimistic about their respective systems’ readiness to go live, but their decision making was largely driven by a desire to generate those beneficial cost savings as soon as possible. Now, predictably, those cost savings will never, ever be realized.  

In addition, the poor operational performances of both systems have caused material public harm which, in turn, has created political firestorms, forcing senior politicians in both governments (here and here) to publicly apologize for the fiascos. It has also forced resignations of senior government officials (here and here) who were originally in charge. The surviving politicians in both governments have also naturally tried to shift the blame for the problems—in Canada’s case, to the previous government, and in Rhode Island’s, to the contractor. In truth, both operational failures were the end product of a dedicated team effort. And finally, both governments’ politicians have solemnly vowed (here and here) that real lessons have been learned and these errors won’t be repeated, which means, of course, they haven’t and will be, respectively.

There is little doubt that new problems with Phoenix and UHIP will be rich Risk Factor fodder for the coming year, if not years. They will also likely be joined in the near future by other government IT fiascos that are now in their planning stages. The reason I am so confident is that the U.S. federal government, for instance, is about to embark on a plethora of IT system modernization efforts, now that the bi-partisan Modernizing Government Technology Act has been signed by President Trump. The MGT Act creates a centralized pool of $500 million over the next two years managed by the Government Services Administration. Agencies can draw from the pool when they apply for help with modernizing their IT systems. The fund allows each agency a lot more flexibility in paying for modernization efforts.

There is no question that U.S. government IT modernization is desperately required. However, given the past dubious track record of federal IT modernization efforts—and the CPB modernization effort is yet another on-going example—I think I’ll end as I began: plus ça change, plus c'est la même chose.