The U.S. Air Force Explains its $1 Billion ECSS Bonfire

Master Sgt. Jack Braden/U.S. Air Force

“We learn from failure, not from success!”

Well, if we apply Dracula author Bram Stoker's maxim to the U.S. Air Force, it could make the case that it has learned the most of all the U.S. military services.

A few weeks ago, the Air Force finally released the executive summary [pdf] of its investigation into its Expeditionary Combat Support System (ECSS). The system was a development blunder that the service mercifully terminated last year after spending US $1.03 billion over seven years and producing a system—if you can even call it that—without “any significant military capability.”   The  ECSS project  began in 2004 as an ambitious and risky effort to replace some 240 outdated Air Force computer systems with a single integrated enterprise resource planning  (ERP) system aimed at modernizing the service's global supply chain. It was also meant to help provide the core financial information required to meet a Congressional mandate that demanded an auditable set of books by 2017.

The investigation was demanded last December by Senators Carl Levin and John McCain, respectively the chairman and ranking member of the Senate Armed Services Committee, who wanted to know the root causes of what was “one of the most egregious examples of mismanagement in recent memory.” The Air Force only released the executive summary,  stamping the full document written by its Acquisition Incident Review (AIR) Team “For Official Use Only.” After reading through the summary, you can understand why: too many people would be at risk from having a heart attack either from anger, laughing themselves silly, or both simultaneously at the host of blunders highlighted.

The AIR Team’s executive summary begins by providing a short history of the program, including the several unsuccessful attempts at trying to resuscitate it. It goes on to identify four contributing causes along with six root causes as to why the ECSS project was cancelled. The four contributing causes listed are:

  1. governance
  2. tactics, techniques, and procedures
  3. difficulty of change
  4. personnel and organizational churn

In terms of governance, the AIR Team says that there existed “a confusing and, at times, ineffectual governance structure” which was “evident throughout the life of the program.” Different and competing acquisition guidance methods were used throughout the program’s development, sometimes concurrently, leading to “needless” delay, frustration, and decision uncertainty at the program executive level.

As far as tactics, techniques, and procedures, the AIR Team found that the Air Force’s ECSS management team was in over their heads. The ECSS team underestimated “the sophistication needed in tackling the enormous and complex effort." This led it to inappropriate use of, for example, a firm fixed price contract for a “significant” development effort that “lacked defined requirements.” Firm fixed price contracts are most appropriate when the acquisition and technical risks are low, not sky high, as in the ECSS program. According to the AIR Team, the ECSS acquisition was at least 28 times larger [pdf] than any similar ERP system development ever attempted by the Defense Department. (And it was probably 100 times larger than the Air Force has ever successfully delivered.)

In addition, the objectives and enormity of the ECSS program meant that virtually all U.S. Air Force logistics organizations (and their contractors) would need to change fundamentally how they did business, which few were prepared for or anxious to do. This led to the AIR Team’s third contributing factor, the difficulty of (implementing organizational) change. Not only did ECSS represent major, disruptive change, but the lack of early program success only served to amplify the doubts the affected Air Force logistics organizations had that moving to ECCC would still allow them to achieve their operational missions.

The final contributing factor was personnel and organizational churn, which actually resembled organizational chum. According to the AIR Team, the ECSS program: 

experienced six program manager changes in eight years; five Program Executive Officers in six years; ten different organizational constructs; and the Expeditionary Combat Support System Logistics Transformation Office was staffed with term positions, not permanent positions, leading to high turnover. [emphasis mine]

A Martian asking someone on the ECSS team, “take me to your leader,” would be left standing around for a good long time. This constant churn, the AIR Team says in a bit of understatement, made the ECSS program’s success "elusive.”

What the high turn-over rate actually looks like is that many Air Force officers and managers viewed the ECSS program as a ticket-punch station on their way to future promotion. Or, alternatively, they knew it was an inevitable disaster and couldn't wait to find a way to escape it.

While the contributing factors the AIR Team listed were the heat and flames generated from the Air Force billion dollar bonfire, the real fuel is found in the six root causes of failure the team exposed. The first root cause discussed and strongly emphasized in the summary was that the ECSS acquisition team did not thoroughly understand the data. The summary states:

All of the data must be understood, not just the data we thought we had, and not just the data we wanted to address, but all of it. This matter cannot be solved by doing Legacy Deconstruction at the same time as blueprinting, at the same time as building the new ‘To-Be’ solution.

In other words, it would have helped if someone actually understood the business processes and data across all those 240 or so systems that were to be replaced, before the hardware, software, and prime contractor were selected.

The second root cause was that the ECSS team did not define or comprehend in any meaningful way the current as well as future architecture. The AIR Team reported that, “the Air Force didn’t understand the ‘As-Is’ or the ‘To-Be’ architectures.”  The team found that even after they were through with their investigation, “the number of systems [the] Expeditionary Combat Support System was to replace is unknown"  [italics in the original]. Ponder that statement for a few moments.

A third root cause was not being able to define a transition plan from the “As-Is” to the “To-Be” system. The AIR Team points this is no great surprise, since without knowing what the data are, or the “As-Is” to the “To-Be” architectures, “arriving at a transition plan was impossible.”

The fourth root cause was a lack of the right execution plan. “Even had the Air Force had a transition plan from the ‘As-Is’ to the ‘To-Be' architecture," the AIR Team reports, the Air Force “lacked a way to properly execute it.”

The fifth root cause listed was an unrealistic development environment that didn’t “mirror the reality of the operational environment.” The importance of this particular root cause wasn’t, and still apparently isn’t, appreciated by the Air Force, the AIR Team states to its apparent bewilderment.

The final root cause—or puntilla—was not having the “right culture” of trust within the Air Force logistics organizations to accept ECSS and what it represented as the new way of doing business. In addition, ECSS program management failed to assure these logistic organizations that the program would try as hard as possible to minimize the impacts to operational mission effectiveness, which made the task of getting organizational buy-in to ECSS difficult if not impossible. In other words, mistrust was rampant, regardless of all the public happy talk about the program.

The AIR Team’s summary also contains a series of short-term, intermediate, and long-term recommendations on how to address each contributing factor and root cause. You can read them for yourself, but I will just point out that many of the recommendations addressing the root causes fall in the intermediate and long-term categories—in other words, they are never to be heard of again until the next bonfire celebrating the Air Force’s vanity.

Why a bonfire to its vanity, you ask? Well, if the Air Force had ever bothered to learn the lessons from the debacle that preceded ECSS—namely the massively over-budget, late, and incomplete Depot Maintenance Management System (DMMS) that started in the mid-1980s and eventually ran out of steam in the late 1990s, or its predecessor fiasco the Advanced Logistics System (ALS), which was also massively over-budget, late, and ineffective—the Air Force by now might have had at least a fighting chance. If one digs through the Government Accountability Office reports from the time of those troubled projects, you’ll find nearly identical contributing and root causes identified, as well as very similar recommendations. The AIR Team might have pointed this out as a root cause as well—the Air Force’s vain belief that it can successfully build an even larger, more complex integrated logistics system than the last one that failed miserably.

One final point: the summary has the mandatory upbeat and positive ending that all internal DoD reports critical of the Department are obliged to finish with. This one, which I wonder how many on the team would privately agree with, states:

The Acquisition Incident Review Team doesn’t want to leave the reader with a completely bleak outlook.  Much of the work that was done on the Expeditionary Combat Support System effort can be reused.  The progress made on legacy deconstruction and the spin ups to blueprinting can be the basis for the data work ahead. While reluctant to put a percentage on potential reuse, the Acquisition Incident Review Team suspects reusable data will be more than people think. Expeditionary Combat Support System wasn’t the failure people think it was; it was the first step to truly understanding the enormous task the Air Force has ahead of itself.

I guess by that measure, neither DMMS nor ALS were failures either, just useful, albeit expensive, initial learning experiences. I, for one, find cold comfort in the very real fact that it took over a billion dollars to get to the “first step” of “understanding” how difficult it is trying to build an ERP system 28 times larger than ever attempted before by an organization with a demonstrated track record of failure involving similar systems.

Photo: Master Sgt. Jack Braden/U.S. Air Force

Related Stories

Risk Factor

IEEE Spectrum's risk analysis blog, featuring daily news, updates and analysis on computing and IT projects, software and systems failures, successes and innovations, security threats, and more.

Contributors

 
Contributor
Willie D. Jones
 

Newsletter Sign Up

Sign up for the ComputerWise newsletter and get biweekly news and analysis on software, systems, and IT delivered directly to your inbox.

Advertisement
Advertisement