I’m using this post to gather together some common threads I’ve noticed after reading write-ups of major cloud software incidents. By cloud software, I’m referring to software-as-a-service (do people even say that anymore? in the cloud. This doesn’t just apply to cloud providers, although it does apply to them as well.
+ + + +Here’s an outline of the topics in this post:
+ + + +-
+
- problem areas
+
-
+
- saturation
+
-
+
- example: databases +
+
+
+
+ - networking (traffic routing failure)
+
-
+
- example: DNS +
+
+
+
+ - security (deny valid access)
+
-
+
- example: SSL certificates +
+
+
+
+
+ - saturation
+
- essential non-standard changes
+
-
+
- mitigating an operational issue + + + +
- migration +
+
+
+
+ - essential increase in essential complexity
+
-
+
- reliability subsystem + + + +
- migration +
+
I think of all of these as omnipresent availability risks: I think these are fundamentally unavoidable, and will be contributing to software incidents until the end of time; or, at the very least, until the end of my own career in software.
+ + + +There are three general areas that most major incidents seem to fall into: saturation, networking, and security. So, let’s start with those.
+ + + +Saturation
+ + + +Saturation is probably the topic I talk about most frequently, both on this blog and elsewhere (e.g.,: the saturation post I wrote for the Resilience in Software Foundation and my saturation talk at the Software Should Work conference). The system becomes saturated when it reaches a limit. That’s a pretty generic description, but there are many limits!
+ + + +Databases
+ + + +Many major incidents involve some system component becoming saturated in one form or another. I personally worry about database saturation the most. That’s because it’s difficult to recover from an overloaded production database. In addition, because database systems are such complex beasts, it can be quite difficult to even determine what the specific performance issue actually is. This is why having in-house database operational expertise is critical.
+ + + +Saturation is an omnipresent risk because the finite nature of resources is a hard constraint in the world that we live in. Eventually, some resource in your system is going to run out.
+ + + +Example: GitHub Incident, Aug 26, 2026
+ + + +Networking
+ + + +While I prattle on endlessly about saturation, not every major incident involves saturation. You can encounter scenarios where all of your internal subsystems are reporting healthy, but from your customer’s point of view, your site is down: they can’t use it. One way this can happen is if your users can’t even reach your site, and that’s where the networking problem area comes in.
+ + + +A networking problem can lead to packets being misrouted. These requests might be black-holed (i.e., silently dropped), or they might be incorrectly routed to a service that doesn’t have the capacity to respond to all of these requests, in which case you’ve got both a network routing issue and a saturation issue.
+ + + +
DNS
+ + + +DNS issues are an example of this kind of network-related failure mode. There’s no way those packets are going to make it to their destination if the client can’t even determine which IP address to send them to. And when DNS breaks, that’s what happens.
+ + + +I bring DNS up because it’s bitten folks enough times that there’s a famous haiku:
+ + + +
More generally, networking is an omnipresent risk because cloud software is inherently distributed, so networking is always a critical service. Now, I don’t work in networking, but from the outside, networking just feels like a dangerous domain to do operational stuff in. The blast radius of a networking issue can be very large. And, because network behavior is inherently distributed, reasoning about the behavior of operational changes is just inherently difficult. Honestly, that’s probably why I don’t work in networking.
+ + + +And, so, I predict we’ll continue to see networking issues contribute to large-scale incidents.
+ + + +Example: Buildkite incident, Aug 25, 2026
+ + + +Security
+ + + +There’s a fundamental tradeoff between availability and security: availability is about ensuring that the good people can access the system. Security is about ensuring that the bad people cannot access the system. This means that there’s always a risk that a security system designed to prevent bad actors from accessing the system can lead to good actors also being blocked. Consider this scenario: there’s an internal security subsystem that goes unhealthy (possibly due to saturation). Is your policy to fail closed or fail open in the event that this subsystem is erroring? Answering that requires making an availability-security tradeoff.
+ + + +SSL certificate expiration
+ + + +Another example of this failure mode, which keeps biting our industry again and again, is SSL certificate expiration. Here you have the behavior of a security system that is preventing legitimate access because the cert wasn’t renewed.
+ + + +
And so, my claim is availability incidents that involve security subsystems will continue to be a thing forever.
+ + + +Example: Bazel incident, Sep 27, 2025
+ + + +Essential uncommon changes
+ + + +Your system is constantly undergoing change. Heck, if you stopped making changes, the system would eventually stop working properly. Now, there are some changes that your org does very frequently. Hopefully, you’re deploying often, flipping feature flags a lot, and so on. But there are other changes that your org has less experience with, because they happen much less often. That means that there hasn’t been as much investment in tooling to support these sorts of changes, and it means that the people making these changes don’t have the same level of expertise as they do with the more common changes. That makes these sorts of changes more dangerous: less mature tooling and less experienced humans.
+ + + +Mitigating an operational issue
+ + + +A few years ago, I wrote a post titled a conjecture on why reliable systems fail where I speculated on two common contributors to major incidents. One of those contributors was a manual intervention that was intended to mitigate a minor incident. Now, it may be that you frequently have to do manual interventions to mitigate system issues, in which case you’ll have a lot of experience with those sorts of interventions. But you’ll also be more motivated to put in the engineering effort to automate away those sorts of common issues.
+ + + +It’s exactly the uncommon issues that require a human operator to intervene to mitigate that are dangerous, because they are uncommon. But they’re essential: there’s a problem in the system, and you need to fix it! But because all practitioner actions are gambles, the manual mitigation carries risk that you could make the problem even worse. And, eventually, this will happen to you.
+ + + +Example: Azure Regional Outage, Jul 23, 2026
+ + + +Migration
+ + + +If you’re at a tech company, unless it’s a start-up, you’ll be dealing with migrations, as old tech gets replaced by newer tech that is better suited to the problems that your org is currently facing. While migrations as a general category are extremely common, each migration is itself a snowflake. This means that the specific details of the migration work is an uncommon change. The work of migration involves making a kind of change to your system that you haven’t made before.
+ + + +To make things worse, one of the dangers of migration is that, as you go along, you start to build confidence that your changes are safe, but there are actually hidden dangers lurking in the system for the next migration. The confidence in the safety of the work exceeds the actual safety. I mean, you made n-1 changes as part of the migration, and none of those changes had negative consequences. It’s natural to assume that the same outcome will occur with the nth change.
+ + + +Example: Rogers Network outage, Jul 8, 2022
+ + + +Essential increase in essential complexity
+ + + +The late American computer scientist Fred Brooks wrote a famous software engineering essay titled No Silver Bullet where he drew a distinction between accidental complexity and essential complexity. The general idea was that there was some amount of complexity in a software system that didn’t need to be there (accidental complexity) and some amount that was just inherent to the nature of the problem space and solution space and so could not be removed (essential complexity).
+ + + +Reliability subsystem
+ + + +We’ve developed multiple techniques to improve the reliability of software systems, including retries, concurrency limiting, autoscaling, automated failover, circuit breakers, health checks, canaries, outlier detection, the list goes on and on. There’s one thing that all of these techniques have in common: they increase the complexity of the overall system! And they do this because they have to increase complexity in order to do their job. This is a consequence of Ashby’s Law, which states that if you want to build a control system that handles more scenarios, you have to increase the complexity of the controller itself.
+ + + +This means that reliability subsystems result in a complexity trade-off. On the one hand, our system can now automatically recover from failure modes that previously required manual intervention. On the other hand, as we all know, increase in complexity is itself dangerous because it can introduce entirely new failure modes that weren’t there before.
+ + + +Going back to my conjecture blog post, the second contributor I posited was: unexpected behavior of a subsystem whose primary purpose was to improve reliability. And this is exactly why. Adding reliability subsystems improves the robustness of our system, but it adds essential complexity to our system, which can lead to novel incidents.
+ + + +Example: OpenAI incident, Dec 11, 2024
+ + + +Migration
+ + + +Like all engineers, I’m a big fan of giving the answer “it depends” if somebody asks me a question about whether they should do X or Y. However, if someone came up to me and said, “Lorin, I’m preparing to do a migration at my company, and I’m trying to decide whether to do a big-bang migration or an incremental one”, then I would almost certainly say, “For the love of God, please do an incremental migration!”. Sometimes big-bang migrations are unavoidable, but when given a choice, I’m going to go for the incremental migration as the safer option.
+ + + +However, when you do an incremental migration, it means that you need to simultaneously support the old system and the new system at the same time while you’re doing the migration. This means that even if the new system yields a net decrease in overall complexity over the old system, while the migration is happening, you’re going to see an increase in system complexity. And that means that you’ll see incidents arise as a byproduct of this increased complexity.
+ + + +Example: Cloudflare incident, Jul 14, 2025
+ + + +Incidents are inevitable, so you’d better be ready
+ + + +To reiterate, I think all of the risks mentioned here are omnipresent: they are fundamental to the nature of cloud software. I don’t think that any of these risks can be eliminated. That’s why I believe so strongly in the value of getting better at incident response. Because, if you prepare, you can get better at dealing with problems that arise as a result of these risks.
+
+
+
+
+
+
+
diff --git a/sreweekly/articles/536/03-master-background-jobs-with-smarter-queue-architecture-and-reliability.html b/sreweekly/articles/536/03-master-background-jobs-with-smarter-queue-architecture-and-reliability.html
new file mode 100644
index 00000000..022b6b33
--- /dev/null
+++ b/sreweekly/articles/536/03-master-background-jobs-with-smarter-queue-architecture-and-reliability.html
@@ -0,0 +1,498 @@
+ 
Step 2 — Let dependency detection build the protection scope
Step 3 — Continuous, isolated recovery points
Step 4 — Test on demand, then recover in one click
Digitata’s Result
+
+
