The Evolution of Innovation: An Insight from Nature's Playbook
Constraints to the novelty of innovation
I’ve been informally polling friends on the topic of innovation. Specifically, whether it is becoming more or less difficult to achieve. It’s a tricky question to answer because intuition can point you in conflicting directions, largely due to a number of countervailing forces acting on the ecosystem:
The on-the-shoulders-of-giants effect: As discoveries accumulate, they unlock a greater overall body of knowledge. This overall body of knowledge has many more permutations of combinations, allowing for new combinatorial discoveries with less difficulty.
The genealogical tree effect: Innovations naturally have a parent/child relationship, forming a family-tree-like structure over time, with more tiny branches than big trunks. This leads to a higher volume of innovations, but many more are incremental than radical.
The new tools effect: New technologies have brought about powerful tools that shorten the time and cost of new discoveries.
The low-hanging-fruit effect: Every new discovery pushes the technological frontier forward, and the frontier is shaped like a fruit tree. More discoveries mean climbing higher to achieve the next discovery, each presenting greater difficulty (perhaps exponentially so).
Opinions have been generally split equally between the more difficult and less difficult camp. And neither observable data nor expert opinion help to bring much clarity:
Patent data shows an ever-increasing volume being issued each year. However, this does not present much of a smoking gun, as it is hard to tease out the different types of innovations (radical, incremental, copycat, etc.).
Academic research suggests that the quality of innovation has been decreasing, despite a larger volume of published, peer-reviewed research today versus yesteryear.
But we are not entirely at a loss. We can better understand the evolving innovation ecosystem by comparing it to a more familiar realm: biological evolution. This analogy provides us with a richer history and deeper insights into similar dynamics and competing forces.
What can we learn from evolution?
Evolution “acts” via the mechanism of perpetuating advantageous mutations and results in speciation over long periods of time. However, we rarely stop to think about how evolution itself has changed.
Evolution is a dynamic system and a many-round (if not infinite) game. But each new round of the game is different from the last. This creates a form of path dependency, limiting options at each stage rather than allowing infinite open-endedness.
Why? As a species progresses through the game, those organisms do what biologists call ecological niche filling—dominating their micro-environment through specialization. This changes the game’s rules: what was once sufficient for survival becomes inadequate. Niche domination makes it difficult, if not impossible, for a novel organism to appear and compete in that niche (absent an external shock).
This makes intuitive sense: imagine two teams competing in an obstacle course with many challenges such as crossing a river, climbing a wall, etc. Team A gets the opportunity to practice the obstacle course multiple times and equip themselves with tools they can use to surmount obstacles. Team B competes without knowledge of the course. You’d expect Team A to win.
How evolutionary pressures affect innovation
The same competitive, multi-round-game dynamic exists in the realm of innovation. To start, nearly all of the resources engaged in the innovation ecosystem (except those earmarked for Basic Research) focus on solving observed problems. And once a problem is first solved, the incentive to solve that problem in a different way reduces by at least an order of magnitude.
Thus, path dependency arises and generally takes hold until an external shock acts as a nudge from the path. Think of the computer mouse: operating systems faced a problem with user interfaces—the need to select items on the screen. Once the mouse was invented and proved to be a generally satisfactory solution, all other solutions sat on the shelf for nearly half a century (including the touch screen). Existing solutions block creativity and paths to other innovations.
Another force against novelty arises: the concentration of resources. There is a strong incentive to pour resources into solving the first-instance problem when it prevents accessing a large market (e.g., unlocking the home computing market due to complexities of interacting with a command prompt interface versus a point-and-click interface). Once solved, there’s a huge shift in resource allocation overnight: toward marketing, distribution, and incremental innovation, and away from radical innovation. Existing solutions block investment into novel approaches to solving problems—sometimes even better ones.
Why it matters
The problem with ecological niche filling in innovation is not necessarily the cycle of explore and exploit (or, invent and commercialize). During periods where the equilibrium seems stable and well functioning, incrementally innovating to optimize is an efficient and utility-maximizing use of resources. However, in unstable and dysfunctional environments, the opposite approach is called for: a shifting of resources towards radical innovation.
Once again, we find parallels to evolution and natural systems. After each of the great extinction events (end-Permian, end-Triassic, etc.), when the environment was at its most chaotic, evolution embraced1 radical innovation. After millions of years of incrementalism, it explored novel solutions to survival that ultimately ensured life would continue.
Today, the realm of technological innovation is a mismatch of these scenarios: an overwhelming dedication of resources to optimization and incrementalism in an environment with an unstable and unattractive equilibrium—one where discovering radically new innovations to solve our greatest crises is paramount.
To shift this current, it’s crucial to avoid a faulty logic trap: highly-directed innovation, attractive on its facade for lower costs and immediate commercializability, cannot alone solve the major problems of this century. Evolution has no ability to decide between the pursuit of incrementalism versus novelty, but, thankfully, we do.
While radical innovation produces unpredictable results—either spectacular failures or groundbreaking discoveries—it has the potential to produce solutions that are 10x, if not 100x, better. Doing so often requires open-ended research without specific, short-term goals. Why? Because once you put the blinders of path dependency on, and double down with a concentration of resources, it is very difficult to veer off the path.
Further exploration
Wikipedia: Ecological niches
Wikipedia: Competitive exclusion principle
Research: Papers and patents are becoming less disruptive over time
Book: The Rise and Reign of the Mammals, The Rise and Fall of the Dinosaurs
Podcast: Expert: Kenneth Stanley - AI researcher on how innovation occurs
I’m going to get in trouble with some readers for using language that suggests evolution has agency. While improper, I hope the small transgression provides additional clarity.