203 lines
6.2 KiB
Markdown
203 lines
6.2 KiB
Markdown
# Why Enterprises Overfund Failure and Underfund Prevention
|
||
|
||
- **期号**: SRE Weekly Issue #510（2026-03-29）
|
||
- **作者**: Florian Hoeppner
|
||
- **链接**: https://techaccelerationandresilience.com/blog-posts/why-enterprises-overfund-failure-and-underfund-prevention
|
||
|
||
## 简介
|
||
|
||
> Enterprises rarely fail because they don’t care about reliability.They fail because:failure is loud,prevention is quiet,and budgeting systems are wired to respond to noise.
|
||
|
||
## 正文
|
||
|
||
# Why Enterprises Overfund Failure and Underfund Prevention
|
||
|
||
Most reliability debates start with technology and end with frustration.
|
||
|
||
——————————————-
|
||
|
||
Want to try it out? Take the 5-minute Reliability U-Curve Assessment → reliabilityeconomics.com/benchmark
|
||
|
||
———————————————
|
||
|
||
Do we need more redundancy?
|
||
|
||
More automation?
|
||
|
||
More “nines”?
|
||
|
||
But after working with large enterprises for years, I’ve come to a different conclusion:
|
||
|
||
*Most organizations don’t have a reliability problem. They have a failure-funding problem.*
|
||
|
||
|
||
### The hidden bias in how reliability gets funded
|
||
|
||
In theory, organizations want stability, resilience, and predictable delivery.
|
||
|
||
In practice, money flows very differently.
|
||
|
||
There are two fundamentally different cost buckets:
|
||
|
||
- **Failure cost (reactive):** incidents, war rooms, hotfixes, customer impact, escalation overhead
|
||
- **Prevention cost (proactive):** SLOs, automation, resilience patterns, testing, observability, compliance-by-design
|
||
|
||
Only one of these is *visible and urgent*.
|
||
|
||
Failure cost:
|
||
|
||
- shows up as outages,
|
||
- triggers executive attention,
|
||
- creates immediate pressure to “do something.”
|
||
|
||
Prevention cost:
|
||
|
||
- is mostly invisible,
|
||
- pays off over time,
|
||
- competes with feature delivery and short-term KPIs.
|
||
|
||
So organizations do what humans and systems always do under pressure:
|
||
|
||
they **optimize for what hurts now**, not for what compounds later.
|
||
|
||
### Why “we’ll fix it in incident response” feels rational (but isn’t)
|
||
|
||
From a budgeting perspective, failure remediation feels safe:
|
||
|
||
- Incidents are real.
|
||
- Customers are angry.
|
||
- Regulators are watching.
|
||
- Action is justified.
|
||
|
||
Prevention, on the other hand, requires belief:
|
||
|
||
- belief that future incidents will be avoided,
|
||
- belief that automation will pay off,
|
||
- belief that today’s effort reduces tomorrow’s cost.
|
||
|
||
That belief is hard to defend in quarterly planning cycles.
|
||
|
||
The result is a predictable pattern:
|
||
|
||
- incident response teams grow,
|
||
- processes accrete,
|
||
- coordination overhead increases,
|
||
- and yet reliability outcomes improve only marginally.
|
||
|
||
This is how organizations end up **spending more every year on failure without ever feeling “done.”**
|
||
|
||
### The reliability U-curve (in one sentence)
|
||
|
||
As reliability improves:
|
||
|
||
- **failure cost goes down** ,
|
||
- **prevention cost goes up** ,
|
||
- and **total cost forms a U-shape** .
|
||
|
||
The bottom of that curve is the point where **total spend is minimized**.
|
||
|
||
Most enterprises never intentionally look for that point.
|
||
|
||
They drift along the curve driven by incidents, not economics.
|
||
|
||
### Why this is not a failure-mode or probability model
|
||
|
||
A common (and valid) objection is:
|
||
|
||
*“Failure costs depend on likelihood, failure modes, SLAs, and contracts. You can’t aggregate this.”*
|
||
|
||
|
||
That’s true, at the *failure-mode* level.
|
||
|
||
But this is not a failure-mode model.
|
||
|
||
It’s a **portfolio-level diagnostic** designed to answer a simpler question:
|
||
|
||
Are we structurally overpaying for failure compared to prevention for this service or journey?
|
||
|
||
|
||
At that level:
|
||
|
||
- recurring operational failures already “price in” likelihood,
|
||
- black-swan events should be treated separately and selectively,
|
||
- and perfect modeling is often the enemy of usable decisions.
|
||
|
||
This is not about precision.
|
||
|
||
It’s about **direction**.
|
||
|
||
### What happens when you make both sides visible
|
||
|
||
When organizations put **failure cost and prevention cost side by side**, something interesting happens.
|
||
|
||
They realize that:
|
||
|
||
- incident labor and coordination time dominate downtime cost,
|
||
- release delays and context switching are real economic drag,
|
||
- compliance overhead is often paid manually instead of being automated,
|
||
- and prevention is often far cheaper than the failures it could eliminate.
|
||
|
||
In many large environments, it’s not unusual to see:
|
||
|
||
*monthly failure cost 5–10× higher than prevention spend*
|
||
|
||
for a single tier-1 service.
|
||
|
||
|
||
At that point, the conversation changes.
|
||
|
||
Not because of SRE ideology, but because of economics.
|
||
|
||
### The shift that actually matters
|
||
|
||
This isn’t about chasing “five nines.”
|
||
|
||
It’s about shifting from:
|
||
|
||
- **funding failure because it’s visible** ,
|
||
to
|
||
- **funding prevention because it’s cheaper** .
|
||
|
||
That shift only happens when:
|
||
|
||
- engineering brings data about failure drag,
|
||
- finance helps frame recurring cost,
|
||
- leadership sets explicit risk tolerance.
|
||
|
||
Reliability improves not because teams try harder, but because **the system starts rewarding the right investments**.
|
||
|
||
### A practical starting point
|
||
|
||
You don’t need a perfect model to begin.
|
||
|
||
Pick:
|
||
|
||
- one service or customer journey,
|
||
- estimate monthly failure cost,
|
||
- estimate monthly prevention cost,
|
||
- sanity-check outcomes via SLOs or error budgets.
|
||
|
||
The goal is not to be “right.”
|
||
|
||
The goal is to stop being **blind**.
|
||
|
||
### Final thought
|
||
|
||
Enterprises rarely fail because they don’t care about reliability.
|
||
|
||
They fail because:
|
||
|
||
- failure is loud,
|
||
- prevention is quiet,
|
||
- and budgeting systems are wired to respond to noise.
|
||
|
||
Until we change that, we’ll keep getting better at fixing incidents, and worse at preventing them.
|
||
|
||
→ If you want a practical starting point, message me **“INVERSION”** and I’ll share a lightweight diagnostic to estimate your current position on the U-curve and where the optimum likely sits.
|
||
|
||
→ We explore these ideas in much more depth in our book, *Mastering Site Reliability Engineering in Enterprise,* a complete guide to building resilient, chaos-tolerant systems, available on Amazon and Springer.
|
||
|
||
Book (Amazon): [Mastering Site Reliability Engineering in Enterprise now on](https://www.amazon.com/Mastering-Site-Reliability-Engineering-Enterprise/dp/B0DYHQXZBQ/) [amazon.com](http://amazon.com/)
|
||
|
||
Book (Springer): [Mastering Site Reliability Engineering in Enterprise on Springer](https://doi.org/10.1007/979-8-8688-1448-8)
|