Disruption, Disrupted

A report in this week’s Nature says that the disruptive content of research papers and patents has been dropping for decades. That counterintuitive finding is not from a grumpy pundit fed up with CES press conferences that seem little different from last year. Instead, it comes from management researchers studying the process of innovation whoanalyzed several decades’ worth of papers and patents. And it raises questions about how the innovation process may be changing.

The Nature study divides innovation into two categories: disruptive concepts that make old technologies obsolete, and concepts that consolidate knowledge based on existing technologies. A disruptive concept like the invention of the transistor is needed to launch a new field, but consolidation is needed to put that new knowledge to practical purposes, such as using transistors to build a control system. “Any field needs a healthy balance of the two, but they have different roles in innovation,” says lead author Michael Park, of the University of Minnesota.

“Economists have been studying economic growth and seeing it [can] slow down and stagnate in certain countries,” says Russell Funk, of Minnesota’s management school, and senior author of the Nature paper. Earlier studies lacked a standard metric to study a broad range of fields, so a few years ago he developed the C-D (consolidating-disruptive) index. He demonstrated it by classifying the disruptive and consolidating elements of references in U.S. patents.

Many studies had credited disruptive ideas with causing the explosive growth of science and technology since the mid-20th century. However, Funk’s study found that the fraction of disruptive ideas in patents declined over time. That might explain observations that in recent years the pace of innovation has slowed, particularly in fields such as drug discovery, which has stalled in some areas.

To get a high-level view of disruptive innovation across a broader range, Funk, Park, and Erin Leahey of the University of Arizona launched a computer analysis of some 25 million research papers published from 1945 to 2010 and 3.9 million U.S. patents from between 1976 and 2010. They searched for disruption and consolidation present in references in the papers and patents at the time of publication and for the five years that followed. They covered four fields in research papers: life sciences and biomedicine, physical sciences, social sciences, and technology. And they covered five patent categories: chemical, computers and communications, drugs and medical, electrical and electronic, and mechanical categories.

Their conclusion: The fraction of disruptive references and other input dropped steadily over the the period studied for all groups studied.

“There seems to be something in common across these fields,” says Funk. Exactly what is happening remains unclear. He hopes his work will stimulate others to look more systematically at common factors that may be behind the decline.

One important insight is that because the number of research papers overall has skyrocketed since 1945 while the number of disruptive papers has stayed roughly constant, the fraction of all papers that are disruptive has declined. “We are not saying [the difference reflects] the quality of the work,” says Park. The difference is that the more recent papers describe consolidating rather than disruptive inventions.

Funk says the broad range of fields with disruption decreasing across the board makes it unlikely that a few fields have passed their heyday after harvesting all the “low hanging fruit” of novelty, leaving little more to discover. However, he does think the rapid growth of research publishing may be producing another factor that limits disruption—a volume of publications so huge that researchers cannot keep up with the literature in their own fields, leaving them to miss the chance to discover disrupting ideas.

When asked if starting his study at the start of the post-WWII science and technology boom—which produced discoveries from transistors to the double helix—might have produced the decline seen now, Funk says it’s possible. But he also points to a related factor: Today’s scientific establishments have grown gigantic and, as the title of a 2019 Nature paper says, “Large teams develop and small teams disrupt science and technology.”

Thinking how Charles Kao disrupted the communications world by introducing fiber optics and the giant Bell Labs had to catch up, I see his point.

Updated 6 January 2023