119 lines
10 KiB
HTML
119 lines
10 KiB
HTML
<!DOCTYPE html>
|
||
<html>
|
||
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||
<title>SRE Toolkit: Failure Domains — willett dot io</title>
|
||
|
||
<link rel="stylesheet" href="/assets/css/tailwind.css?v=1788356482">
|
||
<link rel="stylesheet" href="/assets/css/fonts.css?v=1788356482">
|
||
<link rel="stylesheet" href="/assets/css/markdown.css?v=1788356482">
|
||
<link rel="stylesheet" href="/assets/css/syntax.css?v=1788356482">
|
||
<link rel="alternate" href="/feed.xml" type="application/rss+xml">
|
||
<link rel="preload" href="/assets/fonts/wotfard-extralight-subset.woff2" as="font" type="font/woff2" crossorigin>
|
||
<link rel="preload" href="/assets/fonts/wotfard-regular-subset.woff2" as="font" type="font/woff2" crossorigin>
|
||
<link rel="preload" href="/assets/fonts/wotfard-semibold-subset.woff2" as="font" type="font/woff2" crossorigin>
|
||
<link rel="preload" href="/assets/fonts/roboto-mono-latin-400.woff2" as="font" type="font/woff2" crossorigin>
|
||
<link rel="preload" href="/assets/fonts/roboto-mono-latin-700.woff2" as="font" type="font/woff2" crossorigin>
|
||
|
||
<!-- Begin Jekyll SEO tag v2.8.0 -->
|
||
<title>SRE Toolkit: Failure Domains | Brandon Website</title>
|
||
<meta name="generator" content="Jekyll v4.3.3" />
|
||
<meta property="og:title" content="SRE Toolkit: Failure Domains" />
|
||
<meta property="og:locale" content="en_US" />
|
||
<meta name="description" content="This post is first in a short series I’m calling the “SRE Toolkit”, each entry being a friendly introduction to one concept I’ve consistently found useful in my quest to make software sturdier. Up first – how to be good at failing!" />
|
||
<meta property="og:description" content="This post is first in a short series I’m calling the “SRE Toolkit”, each entry being a friendly introduction to one concept I’ve consistently found useful in my quest to make software sturdier. Up first – how to be good at failing!" />
|
||
<link rel="canonical" href="https://www.willett.io/posts/domains/" />
|
||
<meta property="og:url" content="https://www.willett.io/posts/domains/" />
|
||
<meta property="og:site_name" content="Brandon Website" />
|
||
<meta property="og:type" content="article" />
|
||
<meta property="article:published_time" content="2021-10-14T00:00:00+00:00" />
|
||
<meta name="twitter:card" content="summary" />
|
||
<meta property="twitter:title" content="SRE Toolkit: Failure Domains" />
|
||
<script type="application/ld+json">
|
||
{"@context":"https://schema.org","@type":"BlogPosting","dateModified":"2021-10-14T00:00:00+00:00","datePublished":"2021-10-14T00:00:00+00:00","description":"This post is first in a short series I’m calling the “SRE Toolkit”, each entry being a friendly introduction to one concept I’ve consistently found useful in my quest to make software sturdier. Up first – how to be good at failing!","headline":"SRE Toolkit: Failure Domains","mainEntityOfPage":{"@type":"WebPage","@id":"https://www.willett.io/posts/domains/"},"url":"https://www.willett.io/posts/domains/"}</script>
|
||
<!-- End Jekyll SEO tag -->
|
||
|
||
</head>
|
||
|
||
<body class="text-stone-900 bg-[#fefcfa] dark:text-stone-100 dark:bg-[#181410] antialiased">
|
||
<div class="w-screen px-6 md:px-12 flex flex-col items-center">
|
||
<!-- NAVBAR -->
|
||
<div class="w-full h-16 flex justify-between items-center">
|
||
<div class="text-2xl text-stone-600 dark:text-stone-300">
|
||
<a href="/">brandon willett</a>
|
||
</div>
|
||
<div class="flex items-center">
|
||
<div class="hidden md:block">
|
||
<a class="px-2 hover:text-orange-300" href="/">home</a>
|
||
<a class="px-2 hover:text-orange-300" href="/posts/">blog</a>
|
||
<a class="px-2 hover:text-orange-300" href="/contact/">contact</a>
|
||
</div>
|
||
<a class="pl-4 hover:text-orange-300" href="https://github.com/pickledish">
|
||
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="currentColor">
|
||
<path d="M12 0c-6.626 0-12 5.373-12 12 0 5.302 3.438 9.8 8.207 11.387.599.111.793-.261.793-.577v-2.234c-3.338.726-4.033-1.416-4.033-1.416-.546-1.387-1.333-1.756-1.333-1.756-1.089-.745.083-.729.083-.729 1.205.084 1.839 1.237 1.839 1.237 1.07 1.834 2.807 1.304 3.492.997.107-.775.418-1.305.762-1.604-2.665-.305-5.467-1.334-5.467-5.931 0-1.311.469-2.381 1.236-3.221-.124-.303-.535-1.524.117-3.176 0 0 1.008-.322 3.301 1.23.957-.266 1.983-.399 3.003-.404 1.02.005 2.047.138 3.006.404 2.291-1.552 3.297-1.23 3.297-1.23.653 1.653.242 2.874.118 3.176.77.84 1.235 1.911 1.235 3.221 0 4.609-2.807 5.624-5.479 5.921.43.372.823 1.102.823 2.222v3.293c0 .319.192.694.801.576 4.765-1.589 8.199-6.086 8.199-11.386 0-6.627-5.373-12-12-12z"/>
|
||
</svg>
|
||
</a>
|
||
</div>
|
||
</div>
|
||
<!-- CONTENT -->
|
||
<div class="w-full max-w-[692px] min-w-0 my-12">
|
||
<div class="text-5xl font-light leading-tight text-center mb-4">SRE Toolkit: Failure Domains</div>
|
||
|
||
|
||
<div class="text-xs text-center mb-8">October 14, 2021 -- 4 minutes</div>
|
||
|
||
<div class="text-center font-bold mb-8">· · ·</div>
|
||
<div id="content"><p><em>This post is first in a short series I’m calling the “SRE Toolkit”, each entry being a friendly introduction to one concept I’ve consistently found useful in <a href="/about">my quest to make software sturdier</a>. Up first – how to be good at failing!</em></p>
|
||
|
||
<hr />
|
||
|
||
<p>So, you’re at the train station.</p>
|
||
|
||
<p><img src="/assets/domains/badstation.png" class="pxl" /></p>
|
||
|
||
<p>This is a big station, think <a href="https://en.wikipedia.org/wiki/Grand_Central_Terminal">Grand Central Terminal</a> in Manhattan (which services 67 tracks and hundreds of trains per hour during peak times), so we’ve got a lot of trains coming through. And these trains travel along several different routes; let’s say there are distinct 4 major lines (North, South, East, and West) that all terminate at our big station in the middle of town. Let’s also add that, like the real Grand Central, there is no consistency about which train lines arrive on which tracks. If your home is along the North line, be prepared to run to any individual track once you’ve left work, they’re all fair game.</p>
|
||
|
||
<p>Everything’s humming along well enough, until one unlucky Friday afternoon, a snowstorm hits / cow falls over / alien colony invades and takes out a big section of the West line.</p>
|
||
|
||
<p><img src="/assets/domains/cow.png" class="pxl" /></p>
|
||
|
||
<p>This… creates a bit of an issue at our station. Inbound West trains are arriving, but then the controller tells them they can’t leave, since the rails need to be kept clear for workers. So they start to build up, taking space in any available track at the station, until eventually, all the tracks are full of West trains, and <em>poof</em> – service for the North, East, and South lines is totally out, since there’s no place for them to go. Even though there was no disaster on those lines, 100% of folks are now stuck, even though 75% of them could be getting home if we had just kept a few tracks open.</p>
|
||
|
||
<p>What we’re experiencing is the problem with <strong>not isolating your failure domains</strong>.</p>
|
||
|
||
<hr />
|
||
|
||
<p>In the world of software development, a “failure domain” is an abstract concept which refers to a subset of your application that might fail.</p>
|
||
|
||
<p><img src="/assets/domains/horiz.png" class="pxl" /></p>
|
||
|
||
<p>This can certainly mean an entire subsystem of your service – all of your API servers or databases or the like. I like to call these <strong>horizontal</strong> failure domains. However, more interesting (in the context of this blog post, at least) is the <strong>vertical</strong> failure domain, which will span multiple subsystems, but only occupies a slice of each one. Think at a grocery store – the cash registers, little conveyor belts, and employees are each horizontal domains, but the checkout lanes (each consisting of one register, one belt, and one employee) make up a vertical domain.</p>
|
||
|
||
<p><img src="/assets/domains/vert.png" class="pxl" /></p>
|
||
|
||
<p>The most common examples in tech of vertical failure domains are at the cloud infrastructure level (for instance, <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html">Availability Zones</a> in AWS), but they’re usually just as applicable at the application level (for instance, different customers signed up for your SaaS).</p>
|
||
|
||
<hr />
|
||
|
||
<p>When we say it’s good to “isolate” the failure domains from each other, it just means that we should strive to make sure that one independent part blowing up doesn’t take down all the other parts with it. So to reference our previous examples, the grocery store is doing it well, because when one register fails or an employee doesn’t show up for work, all the other lanes are unaffected! But your SaaS might not be, since one customer suddenly sending a whole bunch of malformed requests might take down your platform for everyone else (unless you’ve done your architectural diligence).</p>
|
||
|
||
<p>Or to settle our imaginary train station’s problems, we’d just need to set up <strong>dedicated tracks for each line</strong> in our terminal. Then, a catastrophic issue on the West line couldn’t cause those trains to overwhelm the terminal, because the hindered trains would fill up about 1/4 of the tracks (the West allotment), and then no more, allowing the other lines to continue operating normally – and most people to make it home.</p>
|
||
|
||
<p><img src="/assets/domains/goodstation.png" class="pxl" /></p>
|
||
|
||
<p>Accomplishing this can be tricky, since it almost always involves taking nice, easy, shared infrastructure, and breaking it up into a bunch of different pieces to manage (I’ve heard them called “pools”, “shards”, “tracks”, or “cells”). But when every outage starts being a partial outage instead of a full-blown dumpster fire, you’ll be glad you did!</p>
|
||
</div>
|
||
</div>
|
||
<!-- FOOTER -->
|
||
<div class="w-full h-24 mt-12 flex items-center justify-end border-t border-stone-300">
|
||
<div>Made with <a href="https://jekyllrb.com">Jekyll</a> | Last updated 02 Sep 2026</div>
|
||
</div>
|
||
</div>
|
||
|
||
|
||
</body>
|
||
|
||
</html>
|