I enjoyed Samuel Arbesman’s first book, The Half-Life of Facts, which was a discussion of the exponential pace of change, as exemplified by Moore’s Law, among other things. When I saw the title of his new book, Overcomplicated, I assumed that it would be a warning that we technologists had gone too far in creating complex systems. It would advocate moving to simpler systems, just as a doctor might advise an overweight person to go on a diet. I was prepared to argue against such a conclusion, but as I discovered upon reading the book, Arbesman does not say that complexity is necessarily bad or that we should seek simplicity. Instead, he maintains that systems are now unknowably complex, that they will become even more so, and we should…just get over it.
Much of the book is spent in discussing the reasons why complexity is inevitably increasing. Arbesman writes that “almost everything we do in the technological realm seems to lead us away from elegance and understandability, and toward impenetrable complexity and unexpectedness.” He cites three main factors driving this increase—“accretion,” “interconnection,” and “edge cases.” Accretion is the result of large systems being built on top of smaller and older systems, often via the incorporation of legacy code, producing what we call kludges. As these subsystems become interconnected, the resulting entanglement can change what was simply intricate to truly complex. Finally, complexity is exacerbated by the inevitable existence of edge cases—that myriad of individually negligible exceptions and rarities that yet constitute the long tail of cases that must all be accounted for in system design. The complexity resulting from these factors has passed a tipping point where no single person can fully understand a complete system.
At this point in reading the book I’m thinking that this is all familiar territory to us engineers. The question is: What do we do about it? Arbesman passes quickly by our usual strategies to manage complexity—the abstraction of subsystems to hide the complexity of lower levels, and “good hygiene” in coding. Good things to do, but they won’t solve the ultimate problem, argues Arbesman. So what else is there?
Arbesman suggests two approaches. The first is that we need to create more generalists—ideally people with “T-shaped” knowledge bases, those with a deep specialty combined with more superficial knowledge of a broad area. This has become particularly important now that most systems involve divergent specialized fields. However, Arbesman acknowledges that generalists are relatively useless unless accompanied by narrow specialists. Moreover, the market does not at present support generalists. It was easier to be a renaissance man in the Renaissance!
The other approach, which is dealt with at some length, is that we need to think more like biologists than physicists. A physicist would be inclined to attempt a mathematical analysis of the system, hoping for an elegant solution that explains and predicts behavior. On the other hand, a field biologist, accustomed to the overwhelming complexity wrought by evolution, would work from the bottom up, cataloging the appearance and behavior of the systems found, hoping perhaps to identify a new species or ecosystem.
I’ve been thinking about this biological approach, but I’m not sure how much insight it would provide. Some of our bugs are like distinct new species, like the “Goto” trap in programming, but the overwhelming majority might be one-offs, with little general relevance.
I agree, however, with Arbesman’s final conclusion that we should celebrate the functionality and sophistication of our creations. Our achievements will necessarily move beyond the understanding of a single human. After all, evolution has created us, creatures with astounding functionality and resilience alongside esoteric fragilities. Perhaps it will be the same in technology.
This article appears in the January 2017 print issue as “Cozying Up to Complexity.”