The Lindy Effect: Why Some Data Engineering Technologies Stand the Test of Time
Ever wonder why, despite all the hype cycles, you still see “old” tech like SQL, Kafka, or S3 at the heart of modern data stacks? I used to chase every shiny new tool—until I learned about the Lindy Effect. This simple idea changed how I evaluate technology choices for my data projects.
What is the Lindy Effect?
The Lindy Effect, popularized by Nassim Nicholas Taleb, says:
The longer a non-perishable thing (like a technology or idea) has survived, the longer it’s likely to stick around.
So, if SQL has been around for 50 years, odds are it’ll be around for at least another 50. The same goes for other “boring” but essential tech.
Read more about the Lindy Effect here
Why Do Some Technologies Become “Lindy”?
Battle-tested: They’ve survived countless use cases, edge cases, and production disasters.
Ecosystem: They have huge communities, tons of documentation, and a wealth of best practices.
Interoperability: New tools are often built to work with them, not replace them.
Simplicity and Robustness: They do one thing well and don’t try to be everything to everyone.
Lindy Technologies in Data Engineering
Let’s look at some “Lindy” tech in the data world:
*The longer a technology has survived, the longer it’s likely to remain relevant.*
Why Betting on Lindy Tech Makes You Future-Proof
Lower risk of obsolescence: You won’t have to rewrite your stack every two years.
Easier hiring: More engineers know these tools.
Better support: More docs, more Stack Overflow answers, more battle-tested solutions.
Ecosystem stability: New tools are built to integrate, not disrupt.
Key Takeaway
The Lindy Effect explains why “boring” tech is often the best bet for your data stack. If it’s survived decades of change, it’s probably not going anywhere soon. So next time you’re tempted by the latest hot tool, ask yourself: is it Lindy?
What’s your favorite Lindy technology? Have you ever regretted betting on something “too new”? Share your stories in the comments!