With no compelling IT system snafus, snags, or snarls last week to report on, we thought we’d return to an oldie but goodie project failure of the first order: the disastrous U.S. Air Force Expeditionary Combat Support System (ECSS) program.
The reason for our revisit is the public release a short time ago of the U.S. Senate staff report [pdf] into the fiasco. Last December, Senators Carl Levin and John McCain, respectively the chairman and ranking member of the Senate Armed Services Committee, requested the report. The request was made in the wake of the Air Force’s publication of the executive summary [pdf] of its own investigative report which apparently the Senators were not altogether happy with. You may recall that Levin and McCain christened the billion-dollar program failure—which the Air Force admitted failed to produce any significant military capability after almost eight years in development—as being “one of the most egregious examples of mismanagement in recent memory.” Given the number of massive DoD IT failures to choose from, that is saying something.
Not surprisingly, the Senate staff report identified basically the same contributing factors for the debacle as the internal Air Force report, albeit with different emphasis. Whereas the Air Force report listed four contributing factors for the ECSS program’s demise (poor program governance; inappropriate program management tactics, techniques, and procedures; difficulties in creating organizational change; and excessive personnel and organizational churn), the Senate staff report condensed them into three contributing factors:
- Cultural resistance to change within the Air Force
- Lack of leadership to implement needed changes; and
- Inadequate mitigation of identified risks at the outset of the procurement.
The Senate report focused much of its attention on the last bullet concerning ECSS program risk mismanagement. In large part, the report blamed the calamity on the Air Force’s failure to adhere to business process reengineering guidelines “mandated by several legislative and internal DOD directives and [that] are designed to ensure a successful and seamless transition from old methods to new, more efficient ways of doing business.” From reading the report, one gets the image of an exasperated parent scolding a recalcitrant child: Congress seemed as miffed at the Air Force for ignoring its many IT-related best practices directives as for the failure itself.
Clearly adding to the sense of frustration is that the Air Force “identified cultural resistance to change and lack of leadership as potential [ECSS] problems in 2004” when the service carried out a mandated risk assessment as the program was being initially planned. Nevertheless, the risk mitigation approaches the service ended up developing were “woefully inadequate.” In fact, the report said that the Air Force identified cultural resistance as an ongoing risk issue throughout the program. However, the lack of action to address it permitted the “potential problem” to become an acute problem.
To its credit, the ECSS program did try to set out an approach in 2006 to try to contain the technical risks involved in developing an integrated logistics system to replace hundreds of legacy systems then in use across the Air Force. Two key risk reduction aspects of the plan were to “forego any modifications” to the Oracle software selected for ECSS and to “conduct significant testing and evaluation” of the system. However, by the time the ECSS project was canceled in 2012, the report notes, Oracle’s software was not only being heavily customized, but it also wasn’t being properly tested.
Several things contributed to this 180 degree turn in project risk reduction, according to the report. One was partially a problem of the Air Force conducting what can only be called bait-and-switch procurement. As the report states:
"In its March 2005 solicitation, the Air Force requested an “integrated product solution.” The Air Force solicitation stated that it wanted to obtain “COTS [commercial off-the-shelf] software [that is] truly ‘off-the-shelf’: unmodified and available to anyone.” Oracle was awarded the software contract in October 2005, and provided the Air Force with three stand-alone integratable COTS software components that were “truly off the shelf.” Oracle also provided the Air Force with tools to put the three components together into a single software “suite,” which would “[require] a Systems Integrator (SI) to integrate the functions of the three [components].” Essentially, this meant the various new software pieces did not initially work together as a finished product and required additional integration to work as intended.
"In December 2005, the Air Force issued its solicitation for a systems integrator (SI) … portrayed the three separate Oracle COTS software components, as a single, already-integrated COTS product which was to be provided to the winning bidder as government funded equipment (GFE). Confusion about the software suite plagued ECSS, contributing significantly to program delays. Not only was time and effort dedicated to integrating the three separate software components into a single integrated solution, but there were disagreements about who was responsible for that integration. While CSC [the system integrator] claimed in its bid to have expertise with Oracle products, the company has said that it assumed, that the products it would receive from the Air Force would already be integrated. Among the root causes of the integration-related delay was the Air Force’s failure to clearly understand and communicate program requirements.
Adding to the general confusion was the small issue of exactly how many legacy systems were going to be replaced. The report states:
"When the Air Force began planning for ECSS, it did not even know how many legacy systems the new system would replace. The Air Force has, on different occasions, used wildly different estimates on the number of existing legacy programs, ranging from “175 legacy systems” to “hundreds of legacy systems” to “over 900 legacy systems.”
Curiously, the Senate report doesn’t note that even if the Air Force was trying to get rid of “only” 175 legacy systems, that was still some 20 times more than the Air Force’s last failed ERP attempt a few years earlier. The staff report seems to assume that such a business process engineering undertaking was still feasible from the start (and during a period of conflict as well), which is a highly dubious assumption to be making.
Probably the most damning sentence in the whole report is the following:
"To date, the Air Force still cannot provide the exact number of legacy systems ECSS would have replaced."
Two years after ECSS was terminated, after two major investigations into why ECSS failed, and while the Air Force is actively engaged in planning for another try, this fact is still rather amazing.
I’ll let you read the report to dig through the other gory details involving the risk-related issues involving cultural resistance and lack of leadership, but suffice to say you have to wonder where top Air Force and Department of Defense leadership was during the eight years this project blunder unfolded. As I have noted elsewhere, the DoD CIO at the time claimed to be “closely” monitoring the program, and up to the day ECSS was terminated, the CIO viewed it as being only a moderately risky program.
There was the same lack of curiosity on the part of Congress as well, however. DoD ERP system developments have been well-documented by the US Government Accountability Office [pdf] for over two decades as being prone to self-immolation. But Congress has kept the money flowing to them anyway without bothering to perform much in the way of oversight. Predictably, the Senate report avoids looking into Congress's own role in permitting the ECSS failure to occur.
The Senate report goes on to list several other DoD ERP programs that are trying their best to imitate ECSS. In this time of tight government budgets, that list might actually move Congress to quit acting as a disinterested party to their future outcomes. In fact, Federal Computer Week ran an article last week that indicated the Senate Appropriations Defense Subcommittee was slicing $500 million dollars off of DoD’s IT budget, which is clearly a warning shot across DoD’s bow.
Another warning shot of note is that both Senators Levin and McCain have noted that: “No one within the Air Force and the Department of Defense has been held accountable for ECSS’s appalling mismanagement. No one has been fired. And, not a single government employee has been held responsible for wasting over $1 billion dollars in taxpayer funds.” The Senators have stated they plan to introduce legislation to hold program managers more accountable in the future.
I suspect—and dearly hope—that if another ECSS happens in defense (or in other governmental agencies or departments, for that matter), more than a few civil and military careers will be, like ECSS, terminated.
In Other News …
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.