Lessons From a Decade of IT Failures

The takeaways from tracking the big IT debacles of the last 10 years

4 min read

Ten years ago, IEEE Spectrum published “Why Software Fails,” an article that examined the underlying causes of notable project failures. A couple of years later, we started the Risk Factor blog, with the goal of tracking technology failures both large and small.

To commemorate the last decade’s worth of failures, we organized and analyzed the data we’ve collected. We cannot claim—nor can anyone, really—to have a definitive, comprehensive database of debacles. Instead, from the incidents we have chronicled, we handpicked the most interesting and illustrative examples of big IT systems and projects gone awry and created the five interactives featured here.

Each reveals different emerging patterns and lessons. Dive in to see what we’ve found. One big takeaway: While it’s impossible to say whether IT failures are more frequent now than in the past, it does seem that the aggregate consequences are worse.

Lesson 1: The Staggering Impact of IT Systems Gone Wrong

The world has relied on large-scale IT systems for decades, but we still haven’t learned how to prevent and avoid major glitches and failures. Here at IEEE Spectrum, we’ve been writing about such failures for 10 years (first in the oft-cited article “Why Software Fails,” and later in the Risk Factor blog).

Now we’re taking a step back to look at the bigger picture. We’ve scoured our archives to create a rogues’ gallery of the most notable, interesting, and emblematic failures from the past decade. We’ve included a diverse assortment of failures, which means there’s no single metric for measuring their impact. Some, like failed IT system upgrades or modernization projects, have straightforward financial consequences. Others, like operational outages and disruptions, are better measured by the time wasted and the number of people affected.

Keep in mind that the failures below are just the tip of the iceberg. They’re just a tiny fraction of the hundreds of incidents we’ve covered in Risk Factor, and an even smaller fraction of the global total. A complete list would be several orders of magnitude larger.

LESSON 1 INTERACTIVE

Explore the many ways in which IT failures have squandered money, wasted time, and generally disrupted people’s lives

Lesson 2: Overcomplexifying, Underdelivering

As hard as it is to build IT systems in the first place, it’s arguably even more difficult to maintain them properly over time. In many government agencies, decades of neglect have resulted in a tangled mess of poorly understood and poorly implemented systems that limit operational effectiveness and efficiency. In the past decade, we’ve seen numerous attempts to combine the functionality of such legacy systems into a single modern replacement system.

That’s easier said than done. In nearly every case, such an effort turns out to be more difficult than originally thought. It’s not surprising, because each additional legacy system that needs replacing comes with its own unique challenges and hidden traps. We were curious to see if there’s a limit to the number of systems that can be practically combined.

Below, we’ve plotted the starting expectations and end results of several modernization efforts from the past few years. Nearly all exceeded their initial budget estimates, and many delivered a tiny fraction of their expected functionality (or failed to provide any functionality at all). Longer lines generally indicate greater relative failure.

LESSON 2 INTERACTIVE

See how complex projects trying to replace multiple systems with one can lead to none

Lesson 3: The Life Cycles of Failed Projects

IT projects rarely fail all at once. Instead, these failures tend to snowball, growing larger and more hopeless as time goes on. Along the way, the definition of success is prone to mutation, as deadlines get pushed back and budgets increase. That’s how a project can launch “ahead of schedule” even if it’s more than three years late.

To illustrate this evolution toward catastrophe, we’ve reconstructed the budget and deadline history of a handful of the most egregious IT project debacles of the last decade.

LESSON 3 INTERACTIVE

Explore cases when even more money and more time couldn't prevent project disasters

Lesson 4: The IT Failure Blame Game

When IT systems fail, there’s always a reason. If you dig deep enough, at the root of any problem are human decisions: sloppy code, insufficient testing, poorly understood dependencies, and incorrect assumptions. Yet when we read about (and report on) failures, the language we use tends to assign blame to inanimate technology that can’t defend itself or get fired.

From our archive of failure coverage we’ve extracted verbatim excerpts that tried to assign blame or identify the cause of a failure. You’re likely to notice some trends about blame and accountability.

LESSON 4 INTERACTIVE

Try to match failures and glitches with their reported causes

Lesson 5: Monuments to Failure

The final lesson to take away from our 10-year look back is how little has changed since IEEE Spectrum’s 2005 special issue on software failure. The project risk factors that can quickly lead to project death haven’t changed. This graveyard is a testament to unrealistic and unarticulated project goals, badly defined system requirements, unbounded project complexity, poorly designed human interfaces, sloppy development practices, poor project management, vicious stakeholder politics, and unbridled commercial pressures, to name but a few.

If you’re an IT executive, programmer, or project manager, you might find yourself whistling past this graveyard of some of the most expensive IT project failures of the last decade. There’s about $70 billion worth of project deaths commemorated here, just a small subset of all the dearly departed projects we have covered in the Risk Factor blog. We haven’t even tried to compute the costs involving those projects which may be, like Frankenstein’s monster, technically alive, but would be better off dead for the all the negative value they create.

Explore the graveyard by hovering over tombstones to bring up basic info about the failed system or project. You can take a deeper dive by clicking on the headline to follow the link to the original post.

LESSON 5 INTERACTIVE

The cadavers of dead IT projects are buried under mounds of cash

Read More

The Making of “Lessons From a Decade of IT Failures”

Why and how we’re looking back at a decade’s worth of IT debacles

We Need Better IT Project Failure Post-Mortems

It’s hard to find trustworthy data about IT debacles

The Conversation (0)

The Future of Deep Learning Is Photonic

Computing with light could slash the energy needs of neural networks

10 min read

This computer rendering depicts the pattern on a photonic chip that the author and his colleagues have devised for performing neural-network calculations using light.

Alexander Sludds
DarkBlue1

Think of the many tasks to which computers are being applied that in the not-so-distant past required human intuition. Computers routinely identify objects in images, transcribe speech, translate between languages, diagnose medical conditions, play complex games, and drive cars.

The technique that has empowered these stunning developments is called deep learning, a term that refers to mathematical models known as artificial neural networks. Deep learning is a subfield of machine learning, a branch of computer science based on fitting complex models to data.

Keep Reading ↓ Show less