Files
nexus/sreweekly/articles/259/02-increment-reliability-everything-is-broken-and-it-s-okay.html
2026-09-12 17:23:01 +08:00

19 lines
30 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!doctype html><html><head><meta charset=utf-8><title>Everything Is Broken, and It’s Okay - Increment</title><meta name=description content='By accepting failure as part of the process, engineering teams can build more resilient systems and prevent technical incidents from becoming catastrophes.'><link rel=canonical href=http://localhost:3000/reliability/failure-is-okay/ ><link rel=apple-touch-icon-precomposed href=/img/icon-571805a1.png><meta property=og:title content='Everything is broken, and it’s okay – Increment: Reliability'><meta property=og:url content=http://localhost:3000/reliability/failure-is-okay/ ><meta property=og:description content='Accepting that imperfect things still work is fundamental to preventing failures from becoming catastrophes.'><meta property=og:image content='https://images.ctfassets.net/3njn2qm7rrbs/9E8PKfEUDqU1ur4DZlfgs/a1b9ad228574391bf174caae38da6f43/cover-2000-b105ba75.jpeg?w=1000'><meta name=twitter:card content=summary_large_image><meta name=twitter:image content='https://images.ctfassets.net/3njn2qm7rrbs/9E8PKfEUDqU1ur4DZlfgs/a1b9ad228574391bf174caae38da6f43/cover-2000-b105ba75.jpeg?w=1000'><meta name=twitter:site content=@IncrementMag><meta name=twitter:title content='Everything is broken, and it’s okay – Increment: Reliability'><meta name=twitter:description content='Accepting that imperfect things still work is fundamental to preventing failures from becoming catastrophes.'><link rel=alternate type=application/rss+xml title=Increment href=/feed.xml><meta name=viewport content='width=device-width,initial-scale=1'><link rel=preload href=/fonts/baton-turbo/400-30a55d66.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=/fonts/baton-turbo/500-1603c0e8.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=/fonts/tiempos-text/400-c4810745.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=/fonts/tiempos-head/700-383ede62.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=stylesheet type=text/css href=/css/bundle-289d885f.css><link rel=stylesheet type=text/css href=/css/issues/16-c2b7b183.css><script>// Don't fade art if it loads ~instantly
setTimeout(()=>{document.documentElement.classList.add('fadeArt')},250);</script><script defer src=/js/defer-737fbd90.js></script><script>const INCREMENT_META={issueNumber:16,issueSlug:'reliability',articleSlug:'failure-is-okay'};</script></head><body class='Issue_reliability Article_failure-is-okay'><nav class=PageNav><div class=u-Container><div class=column><h1 class=logo><a href=/ ><img src=/img/logo-ae2c55d5.svg alt=Increment></a></h1><a class=out-now href=https://store.increment.com/ style=color:#595959><div class='IssueTitle tiny'><div class='t-Caps meta tiny'><span>NEW</span></div><h3 class='t-IssueTitle title'>Buy the print edition</h3></div></a><ul class=nav><li><a href=/issues/ ><span>Issues</span></a></li><li><a href=/topics/ ><span>Topics</span></a></li><li><a href=https://store.increment.com/ ><span>Store</span></a></li><li><a href=/about/ ><span>About</span></a></li></ul></div></div></nav><div class=ArticlePage itemscope itemtype=http://schema.org/Article><script type=application/ld+json>{
"@context": "http://schema.org",
"@type": "Article",
"headline": "Everything is broken, and it’s okay",
"image": " https://images.ctfassets.net/3njn2qm7rrbs/9E8PKfEUDqU1ur4DZlfgs/a1b9ad228574391bf174caae38da6f43/cover-2000-b105ba75.jpeg?w&#x3D;1000",
"datePublished": "Thu, 25 Feb 2021 19:00:00 GMT",
"dateModified": "Thu, 25 Feb 2021 19:00:00 GMT",
"publisher": {
"@type": "Organization",
"name": "Increment",
"logo": {
"@type": "ImageObject",
"url": "https://increment.com/img/logo.png"
}
},
"description": "Accepting that imperfect things still work is fundamental to preventing failures from becoming catastrophes.",
"mainEntityOfPage": "http://localhost:3000/reliability/failure-is-okay/"
}</script><header class='u-Container ArticleHeader'><div class='u-Container art'><div class=u-Art><div class=placeholder style=background:#f2f2f2></div><picture><source srcset='https://images.ctfassets.net/3njn2qm7rrbs/9E8PKfEUDqU1ur4DZlfgs/a1b9ad228574391bf174caae38da6f43/cover-2000-b105ba75.jpeg?w=2000 2000w, https://images.ctfassets.net/3njn2qm7rrbs/9E8PKfEUDqU1ur4DZlfgs/a1b9ad228574391bf174caae38da6f43/cover-2000-b105ba75.jpeg?w=1000 1000w, https://images.ctfassets.net/3njn2qm7rrbs/9E8PKfEUDqU1ur4DZlfgs/a1b9ad228574391bf174caae38da6f43/cover-2000-b105ba75.jpeg?w=500 500w, https://images.ctfassets.net/3njn2qm7rrbs/9E8PKfEUDqU1ur4DZlfgs/a1b9ad228574391bf174caae38da6f43/cover-2000-b105ba75.jpeg?w=100 100w' sizes=100vw><img class=j-SmoothLoad src='https://images.ctfassets.net/3njn2qm7rrbs/9E8PKfEUDqU1ur4DZlfgs/a1b9ad228574391bf174caae38da6f43/cover-2000-b105ba75.jpeg?w=1000' sizes=100vw alt='Everything is broken, and it’s okay'></picture></div></div><div class=column><div class=u-Grid><div class=main><h4 class='t-Byline byline large'><a href=#authors class=j-SmoothScroll itemprop=author><span>Heidi Waterhouse</span></a></h4><h1 class='t-TitleSerif large title' itemprop=name>Everything is broken, and it’s okay</h1><div class='t-BodySans large intro' itemprop=description>Accepting that imperfect things still work is fundamental to preventing failures from becoming catastrophes.</div></div><a class=issue href=/reliability/ ><span class='t-Caps tiny part-of'>Part of</span><div class='IssueTitle small' style=color:#863051><div class='t-Caps meta'><span>Issue 16</span> <span>February 2021</span></div><h2 class='t-IssueTitle title'>Reliability</h2></div></a></div></div></header><div class='u-Container ArticleContent'><article class='ContentBody column' itemprop=articleBody><div class=ArticleLayout><p>Everything is a little bit broken. Nothing made by human hands or minds is perfect. Every car you’ve ever ridden in, every elevator you’ve ever taken, every safety-critical computer program you’ve ever trusted your life with was flawed in some way. It doesn’t matter how much money you spend—your system’s uptime will always be measured in 9s, a percentage of perfection.</p><p>But lots of imperfect things are still perfectly operational, including human bodies and many of our systems. This fact—that imperfect things still work—is integral to understanding how to prevent failures from escalating into catastrophes.</p><p>A failure is one part of a system breaking; a catastrophe is when many failures accumulate to a point beyond recovery.<sup>1</sup> When a catastrophe happens, it often seems like something very safe failed suddenly. But when we analyze the contributing causes, we find it wasn’t really sudden at all: The warning signs were present, the early failures, but we didn’t predict how they’d combine.</p></div><div class='ArticleLayout flipped'><h2>Control is an illusion</h2><p>The way we control computers’ inputs and outputs has evolved radically over the past two centuries. When inventor <a href=https://www.britannica.com/technology/Analytical-Engine target=_blank rel='noopener noreferrer'>Charles Babbage designed his Analytical Engine</a> in the mid-1800s and mathematician <a href=https://www.britannica.com/biography/Ada-Lovelace target=_blank rel='noopener noreferrer'>Ada Lovelace programmed it</a>, they were manipulating physical objects. Now, we work with digital objects like software and functions rather than punch cards and readers. This shift, from manually altering a machine’s structure to using layers of computer languages and abstraction, has changed our conception of how we work and what we imagine we can control.</p><p>None of the changes we’ve made through the decades have reduced the amount of work that happens in a system, though. From chip design to developer tools, we’re not saving any time or effort, just redistributing labor. Developers now routinely reuse and conceptually compress the work of others; we assume the work underlying our own—the electrical engineering, the manufacturing precision, the code we build on top of, the tools we use to make new tools—is a given and focus only on the tasks in our scope of influence.</p><p>If we want to build stable and reliable systems, we have to admit that there are parts we don’t understand. Otherwise, we’ll go on assuming the systems we’re using are fixed, immutable, or at least stable. We’ll assume past performance is an indicator of future performance. But that’s not necessarily true, nor within our control—we just act like it is so we can get our work done.<sup>2</sup></p><p>One of the clearest examples of our dependence on the unseen work of others is the <a href=https://meltdownattack.com/ target=_blank rel='noopener noreferrer'>Spectre vulnerability</a>, officially made public in January 2018, which affects microprocessors that perform branch prediction. Mitigating it required a change in how our CPUs function—not working ahead of actual commands—and caused a worldwide slowdown in execution speeds. Few developers consider the chips their software will run on. Why would they?</p><h2>Failure is inevitable</h2><p>We know complex systems can fail. But what makes failure itself complex is that we don’t always know when it will happen, why, or how it will affect other parts of the system.</p><p>The real question is whether that failure will become a catastrophe. As an example, every airplane you’ve ever flown on has had many tiny problems. Most of them are known and will be fixed at some point—things like a sticky luggage latch or a broken seat or a frayed seatbelt. None of these problems alone are cause to ground the plane. However, if enough small problems compound, the plane may no longer <a href=https://www.icao.int/safety/airnavigation/OPS/CabinSafety/Pages/ICAO-Requirements-related-to-Cabin-Safety.aspx target=_blank rel='noopener noreferrer'>meet the requirements for passenger airworthiness</a> and the airline will ground it. A plane with many malfunctioning call buttons may also be poorly maintained in other ways, like faulty checking for turbine blade microfractures or landing gear behavior.</p></div><div class=ArticleLayout><h2>Responding to fragility</h2><div class='Sidebar large'><blockquote><p>If everything is the most important thing, then nothing is.</p></blockquote></div><p>Once you accept failure is inevitable and commit to avoiding the concatenation of failures that causes catastrophes, you—and your organization—need to decide what risk looks like and what to prioritize. If everything is the most important thing, then nothing is.</p><p>We all want to believe our software or services do something vital for our users, but some industries bear more significant risks. Life-critical industries such as aviation, medical devices, and nuclear energy have their own standards, while leisure-based industries like gaming, gambling, or fashion have standards more to do with loss of market share than immediate, life-limiting effects. Assessing the error’s impact, such as the number of people affected or the monetary consequences of failure, can help you make reasoned choices about allocating your human and financial resources for mitigation.</p><h2>Designing against disasters</h2><p>When designing software, risk reduction and harm mitigation patterns can reduce the frequency and severity of negative events. The goal of risk reduction is to prevent negative events and make systems harder to break. Vaccines, antilock brakes, and railroad crossing arms are everyday examples of risk reduction. Harm mitigation strategies acknowledge that negative events will still happen and try to make their impact as mild as possible. Seatbelts, antiviral medications, and cut-proof gloves are examples of harm mitigation.</p><p>In software development, risk reduction patterns might include restricting access to a system, abstractions to manage changes, approval workflows, automated acceptance testing, and pair programming. However, any strategy that hardens a system to make it less risky might also make it less flexible. A perfectly optimized system is often fragile because it has no built-in expectation of failure or redundancy: If something goes wrong, there’s little or no room for recovery. For example, if a subway system has exactly the capacity it needs to accommodate peak commuter traffic, a single out-of-service train can cause delays and overcrowding throughout the system. This is where harm mitigation comes in.</p><p>Harm mitigation patterns in software development include circuit breakers to remove problem processes or inputs, the isolation of suspect parts of a system, having many control points or breakpoints (such as on microservices), and the ability to rapidly roll back recently implemented changes. While adding more points of control to a system could also introduce more potential points of failure, it’s often a worthwhile trade-off to be able to isolate and diagnose a subset of the system without taking it all down.</p><p>One of the ways we express harm mitigation is with error budgets, as described in <a href=https://sre.google/workbook/table-of-contents/ target=_blank rel='noopener noreferrer'>O’Reilly’s 2018 <i>SRE Workbook</i></a>. Systems reliability engineering accepts and anticipates that some parts of a system may fail and budgets for these failures. The teams running the system deliberately take parts offline to test responses, workarounds, and restoration.</p><p>This “deliberate disaster” practice was popularized by Netflix and its <a href=https://netflix.github.io/chaosmonkey/ target=_blank rel='noopener noreferrer'>Chaos Monkey tool</a>, released in 2011, but organizations have used disaster recovery practices for much longer. When the practice is a regular part of an organization’s resilience response, it’s sometimes called chaos engineering. Practicing systems failure in real time not only teaches teams how to respond to known known–style failures, where everyone is aware of what’s offline, but also unknown or unpredicted failures, because the team has learned to work together to solve a problem and has practice resolving rapidly moving situations.</p><p>Preparing for disasters with multiple layers of safety protection is sometimes known as the <a href=https://en.wikipedia.org/wiki/Swiss_cheese_model target=_blank rel='noopener noreferrer'>Swiss cheese model</a>. Each layer of preparation, prevention, and mitigation is insufficient by itself—but, when stacked together, they can provide good-enough protection from worst-case scenarios.</p></div><div class='ArticleLayout flipped'><h2>Accept imperfection, within limits</h2><p>The more complex the system, the more likely it is that some part of it is broken. As engineers, we’re at the apex of literally millions of hours of design and engineering time to help us, and our users, with everything from finding our phone to continuously integrating our code. We have to assume that some of that infrastructure, and some of our work, is broken.</p><p>If we accept these imperfections, we can work toward building resilient systems that can handle a little static and deal with flawed foundations without falling over. We don’t stop working toward perfection just because it’s impossible.</p></div><div class='ArticleLayout flipped'><p class=small><sup>1</sup> Dr. Richard Cook touches on these concepts in the first part of his famed 1998 treatise <a href=https://how.complexsystems.fail/ ><i>How Complex Systems Fail</i></a>, in which he observes medical errors and the systems that cause them. The theory applies to all sorts of complex systems, especially software.</p><p class=small><sup>2</sup> Ruby on Rails creator David Heinemeier Hansson expands on this point in “<a href=https://youtu.be/zKyv-IGvgGE>FIXME</a>,” his keynote talk at <a href=https://railsconf.com/ >RailsConf 2018</a>.</p></div></article></div><div class='u-Container ArticleFooter'><div class=column><div class='u-Grid ContentBody small content'><div class=authors id=authors><div><div class=photo><figure style='background-image:url(https://images.ctfassets.net/3njn2qm7rrbs/5u1AhCdA3q6WQ8O9jI38Xp/9c26e360cc84baa3118c0f22c0bf5f9d/Heidi_Waterhouse.jpeg?w=500)'></figure></div><div class=text><h4>About the author</h4><p><b>Heidi Waterhouse</b> is a principal transformation advocate for LaunchDarkly, where she works on explaining the intersections of feature control, reliability, and sociotechnical systems.</p><p><a href=https://twitter.com/wiredferret target=_blank>@wiredferret</a></p></div></div><div><div class=photo><figure style='background-image:url(https://images.ctfassets.net/3njn2qm7rrbs/9E8PKfEUDqU1ur4DZlfgs/a1b9ad228574391bf174caae38da6f43/cover-2000-b105ba75.jpeg?w=500)'></figure></div><div class=text><h4>Artwork by</h4><p><b>Janice Chang</b></p><p><a href=http://www.janicechangart.com/ target=_blank>janicechangart.com</a></p></div></div></div><div class=topics><div class=text><h4>Topics</h4><p><a href=/topics/opinion/ >Essays &amp; Opinion</a></p><p><a href=/topics/culture/ >Workplace &amp; Culture</a></p></div></div></div></div></div></div><div class=SubscribeBox><a id=newsletter class=anchor href=#newsletter></a><div class='ContentBody inverted store'><div class=u-Container><div class='u-Grid column'><div class=box style=background:#4c70b1><div class=text><h2>Buy the print edition</h2><p class=j-TextBalance>Visit the Increment Store to purchase print issues.</p><p><a class='t-Caps u-Arrow' href=https://store.increment.com/ >Store</a></p></div><a href=https://store.increment.com/ class=magazine><figure class=j-MaskedImage data-fill=/art/19/19-cutout-1000-7ffb5dba.png data-mask=/art/19/fill-cms-1000-83522f38.png></figure></a></div></div></div></div><div class='ContentBody email'><div class=u-Container><div class='u-Grid column'><div class='j-EmailForm box' style='box-shadow:0 -5px 0 #4c70b1'></div></div></div></div></div><div class=ContinueReading><div class=u-Container><div class=column><h2 class='t-Caps xlarge'>Continue Reading</h2><ul class='u-Grid articles'><li class='ArticleBlock footer' style=color:#863051><a class=j-Preload href=/reliability/technical-incident-command/ ><div class='IssueTitle tiny' style=color:#863051><div class='t-Caps meta'><span>16</span></div><h3 class='t-IssueTitle title'>Reliability</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Tanya Reilly</span></h4><h3 class='t-TitleSans title'>Embrace your inner incident commander</h3></div><div class='t-BodySerif small intro'>The way we fight fires affects how quickly we can resolve outages. Appointing an incident commander can help—and you (yes, you) can be&nbsp;one.</div></div></a></li><li class='ArticleBlock footer' style=color:#8e65bf><a class=j-Preload href=/teams/doing-a-little-bit-of-the-impossible/ ><div class='IssueTitle tiny' style=color:#8e65bf><div class='t-Caps meta'><span>11</span></div><h3 class='t-IssueTitle title'>Teams</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Anil Dash</span></h4><h3 class='t-TitleSans title'>Doing (a little bit of) the impossible</h3></div><div class='t-BodySerif small intro'>A commitment to trust, fairness, and inclusion is at the heart of a strong company—and healthy&nbsp;teams.</div></div></a></li><li class='ArticleBlock footer' style=color:#863051><a class=j-Preload href=/reliability/how-to-build-organizational-resilience/ ><div class='IssueTitle tiny' style=color:#863051><div class='t-Caps meta'><span>16</span></div><h3 class='t-IssueTitle title'>Reliability</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Ryn Daniels</span></h4><h3 class='t-TitleSans title'>How to build organizational resilience</h3></div><div class='t-BodySerif small intro'>By encoding resilience into an organization’s culture, engineering teams can be better equipped to tackle the unknown and unexpected.</div></div></a></li><li class='ArticleBlock footer' style=color:#863051><a class=j-Preload href=/reliability/brain-on-progress/ ><div class='IssueTitle tiny' style=color:#863051><div class='t-Caps meta'><span>16</span></div><h3 class='t-IssueTitle title'>Reliability</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Lara Hogan</span></h4><h3 class='t-TitleSans title'>Your brain on progress</h3></div><div class='t-BodySerif small intro'>Strategies for nurturing that feel-good sense of accomplishment when doing largely invisible&nbsp;work.</div></div></a></li><li class='ArticleBlock footer' style=color:#8e65bf><a class=j-Preload href=/teams/do-engineering-managers-need-to-be-technical/ ><div class='IssueTitle tiny' style=color:#8e65bf><div class='t-Caps meta'><span>11</span></div><h3 class='t-IssueTitle title'>Teams</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Will Larson</span></h4><h3 class='t-TitleSans title'>Do engineering managers need to be technical?</h3></div><div class='t-BodySerif small intro'>And what do we even mean when we say “technical”?</div></div></a></li><li class='ArticleBlock footer' style=color:#8e65bf><a class=j-Preload href=/teams/pay-fair/ ><div class='IssueTitle tiny' style=color:#8e65bf><div class='t-Caps meta'><span>11</span></div><h3 class='t-IssueTitle title'>Teams</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Lara Hogan</span></h4><h3 class='t-TitleSans title'>Pay fair</h3></div><div class='t-BodySerif small intro'>An introduction to correcting—and preventing—compensation&nbsp;inequity.</div></div></a></li><li class='ArticleBlock footer' style=color:#8e65bf><a class=j-Preload href=/teams/trust-the-process/ ><div class='IssueTitle tiny' style=color:#8e65bf><div class='t-Caps meta'><span>11</span></div><h3 class='t-IssueTitle title'>Teams</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Paul Ford</span></h4><h3 class='t-TitleSans title'>Trust the process</h3></div><div class='t-BodySerif small intro'>Though they run on a mixture of paper and lore, effective editorial organizations ship like clockwork. Can engineering teams learn from their enduring processes?</div></div></a></li><li class='ArticleBlock footer' style=color:#8e65bf><a class=j-Preload href=/teams/the-path-to-management/ ><div class='IssueTitle tiny' style=color:#8e65bf><div class='t-Caps meta'><span>11</span></div><h3 class='t-IssueTitle title'>Teams</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Kaya Thomas</span></h4><h3 class='t-TitleSans title'>The path to management</h3></div><div class='t-BodySerif small intro'>Insight for developers considering making a move from individual contributor to engineering&nbsp;manager.</div></div></a></li><li class='ArticleBlock footer' style=color:#8e65bf><a class=j-Preload href=/teams/the-epistemology-of-software-quality/ ><div class='IssueTitle tiny' style=color:#8e65bf><div class='t-Caps meta'><span>11</span></div><h3 class='t-IssueTitle title'>Teams</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Hillel Wayne</span></h4><h3 class='t-TitleSans title'>The epistemology of software quality</h3></div><div class='t-BodySerif small intro'>Studies show that human factors most influence the quality of our work. So why do we put so much stake in technical solutions?</div></div></a></li></ul><h3 class='t-Caps large'>Explore Topics</h3><ul class='t-BodySerif large topics'><li><a href=/topics/learn/ >Learn Something New</a></li><li><a href=/topics/scaling/ >Scaling &amp; Growth</a></li><li><a href=/topics/ask-an-expert/ >Ask an Expert</a></li><li><a href=/topics/interviews/ >Interviews &amp; Surveys</a></li><li><a href=/topics/guides/ >Guides &amp; Best Practices</a></li><li><a href=/topics/opinion/ >Essays &amp; Opinion</a></li><li><a href=/topics/culture/ >Workplace &amp; Culture</a></li></ul><h3 class='t-Caps large'>All Issues</h3><ul class=issues><li><a href=/planning/ ><div class='IssueTitle large' style=color:#4c70b1><div class='t-Caps meta'><span>Issue 19</span> <span>November 2021</span></div><h1 class='t-IssueTitle title'>Planning</h1></div></a></li><li><a href=/mobile/ ><div class='IssueTitle large' style=color:#439eab><div class='t-Caps meta'><span>Issue 18</span> <span>August 2021</span></div><h1 class='t-IssueTitle title'>Mobile</h1></div></a></li><li><a href=/containers/ ><div class='IssueTitle large' style=color:#443d79><div class='t-Caps meta'><span>Issue 17</span> <span>May 2021</span></div><h1 class='t-IssueTitle title'>Containers</h1></div></a></li><li><a href=/reliability/ ><div class='IssueTitle large' style=color:#863051><div class='t-Caps meta'><span>Issue 16</span> <span>February 2021</span></div><h1 class='t-IssueTitle title'>Reliability</h1></div></a></li><li><a href=/remote/ ><div class='IssueTitle large' style=color:#29386a><div class='t-Caps meta'><span>Issue 15</span> <span>November 2020</span></div><h1 class='t-IssueTitle title'>Remote</h1></div></a></li><li><a href=/apis/ ><div class='IssueTitle large' style=color:#00afbe><div class='t-Caps meta'><span>Issue 14</span> <span>August 2020</span></div><h1 class='t-IssueTitle title'>APIs</h1></div></a></li><li><a href=/frontend/ ><div class='IssueTitle large' style=color:#5ebe92><div class='t-Caps meta'><span>Issue 13</span> <span>May 2020</span></div><h1 class='t-IssueTitle title'>Frontend</h1></div></a></li><li><a href=/software-architecture/ ><div class='IssueTitle large' style=color:#40af9e><div class='t-Caps meta'><span>Issue 12</span> <span>February 2020</span></div><h1 class='t-IssueTitle title'>Software Architecture</h1></div></a></li><li><a href=/teams/ ><div class='IssueTitle large' style=color:#8e65bf><div class='t-Caps meta'><span>Issue 11</span> <span>November 2019</span></div><h1 class='t-IssueTitle title'>Teams</h1></div></a></li><li><a href=/testing/ ><div class='IssueTitle large' style=color:#e89e00><div class='t-Caps meta'><span>Issue 10</span> <span>August 2019</span></div><h1 class='t-IssueTitle title'>Testing</h1></div></a></li><li><a href=/open-source/ ><div class='IssueTitle large' style=color:#4a5ad3><div class='t-Caps meta'><span>Issue 9</span> <span>May 2019</span></div><h1 class='t-IssueTitle title'>Open Source</h1></div></a></li><li><a href=/internationalization/ ><div class='IssueTitle large' style=color:#c096ca><div class='t-Caps meta'><span>Issue 8</span> <span>February 2019</span></div><h1 class='t-IssueTitle title'>Internationalization</h1></div></a></li><li><a href=/security/ ><div class='IssueTitle large' style=color:#4dbac5><div class='t-Caps meta'><span>Issue 7</span> <span>October 2018</span></div><h1 class='t-IssueTitle title'>Security</h1></div></a></li><li><a href=/documentation/ ><div class='IssueTitle large' style=color:#f5684d><div class='t-Caps meta'><span>Issue 6</span> <span>August 2018</span></div><h1 class='t-IssueTitle title'>Documentation</h1></div></a></li><li><a href=/programming-languages/ ><div class='IssueTitle large' style=color:#d69336><div class='t-Caps meta'><span>Issue 5</span> <span>April 2018</span></div><h1 class='t-IssueTitle title'>Programming Languages</h1></div></a></li><li><a href=/energy-environment/ ><div class='IssueTitle large' style=color:#d6658e><div class='t-Caps meta'><span>Issue 4</span> <span>February 2018</span></div><h1 class='t-IssueTitle title'>Energy & Environment</h1></div></a></li><li><a href=/development/ ><div class='IssueTitle large' style=color:#53a88e><div class='t-Caps meta'><span>Issue 3</span> <span>October 2017</span></div><h1 class='t-IssueTitle title'>Development</h1></div></a></li><li><a href=/cloud/ ><div class='IssueTitle large' style=color:#707aed><div class='t-Caps meta'><span>Issue 2</span> <span>July 2017</span></div><h1 class='t-IssueTitle title'>Cloud</h1></div></a></li><li><a href=/on-call/ ><div class='IssueTitle large' style=color:#ef766e><div class='t-Caps meta'><span>Issue 1</span> <span>April 2017</span></div><h1 class='t-IssueTitle title'>On-Call</h1></div></a></li></ul></div></div></div><footer class=PageFooter><div class='u-Container ContentBody small'><svg style=display:none><symbol id=twitterIcon viewBox='0 0 32 32'><path d='M32.1 6c-1.2.5-2.5.9-3.8 1 1.4-.8 2.4-2.1 2.9-3.6-1.3.8-2.7 1.3-4.2 1.6a6.8 6.8 0 0 0-4.8-2c-3.6 0-6.6 3-6.6 6.6 0 .5.1 1 .2 1.5-5.5-.3-10.4-3-13.6-6.9-.6 1-.9 2.1-.9 3.3 0 2.3 1.2 4.3 2.9 5.5-1.1 0-2.1-.3-3-.8v.1c0 3.2 2.3 5.9 5.3 6.5-.6.2-1.1.2-1.7.2-.4 0-.8 0-1.2-.1.8 2.6 3.3 4.5 6.2 4.6-2.3 1.8-5.1 2.8-8.2 2.8-.5 0-1.1 0-1.6-.1C2.9 28 6.3 29 10 29c12.1 0 18.7-10 18.7-18.7v-.9c1.4-.9 2.5-2 3.4-3.4z' fill=currentColor /></symbol><symbol id=facebookIcon viewBox='0 0 32 32'><path d='M30.2 0H1.8C.8 0 0 .8 0 1.8v28.5c0 1 .8 1.8 1.8 1.8h15.3V19.6h-4.2v-4.8h4.2v-3.6c0-4.1 2.5-6.4 6.2-6.4 1.8 0 3.3.2 3.7.2v4.3h-2.6c-2 0-2.4 1-2.4 2.4v3.1h4.8l-.6 4.8H22V32h8.2c1 0 1.8-.8 1.8-1.8V1.8c0-1-.8-1.8-1.8-1.8z' fill=currentColor /></symbol><symbol id=rssIcon viewBox='0 0 32 32'><path d='M10.7 25.6c0 2.4-2 4.4-4.4 4.4S2 28 2 25.6s2-4.4 4.4-4.4 4.3 2 4.3 4.4zM6.1 2c-.6 0-1.3 0-2 .1-1.3.1-2.2 1.2-2.1 2.4.1 1.2 1.2 2.2 2.4 2.1.6 0 1.1-.1 1.6-.1 10.7 0 19.4 8.7 19.4 19.4 0 .5 0 1-.1 1.6-.1 1.2.8 2.3 2.1 2.4h.2c1.2 0 2.1-.9 2.2-2.1.1-.7.1-1.4.1-2C30 12.7 19.3 2 6.1 2zm-.7 9.6c-.4 0-.8 0-1.3.1-1.3.1-2.2 1.2-2.1 2.5.1 1.3 1.2 2.2 2.5 2.1h.9c5.7 0 10.4 4.7 10.4 10.4v.9c-.1 1.3.8 2.4 2.1 2.5h.2c1.2 0 2.2-.9 2.3-2.1 0-.4.1-.8.1-1.2-.1-8.4-6.9-15.2-15.1-15.2z' fill=currentColor /></symbol><symbol id=linkedInIcon viewBox='0 0 32 32'><path d='M29.6,0H2.4C1.1,0,0,1,0,2.3v27.4C0,31,1.1,32,2.4,32h27.3c1.3,0,2.4-1,2.4-2.3V2.3C32,1,30.9,0,29.6,0z M9.5,27.3H4.7V12 h4.8V27.3z M7.1,9.9c-1.5,0-2.8-1.2-2.8-2.8c0-1.5,1.2-2.8,2.8-2.8c1.5,0,2.8,1.2,2.8,2.8C9.9,8.7,8.6,9.9,7.1,9.9z M27.3,27.3 h-4.7v-7.4c0-1.8,0-4-2.5-4c-2.5,0-2.8,1.9-2.8,3.9v7.6h-4.7V12H17v2.1h0.1c0.6-1.2,2.2-2.5,4.5-2.5c4.8,0,5.7,3.2,5.7,7.3V27.3z' fill=currentColor /></symbol></svg><div class='column main'><section class=social><a href=https://twitter.com/incrementmag class=twitter><svg viewBox='0 0 32 32'><use xlink:href=#twitterIcon x=0 y=0></use></svg> <span>@incrementmag</span> </a><a href=https://facebook.com/incrementmag class=facebook><svg viewBox='0 0 32 32'><use xlink:href=#facebookIcon x=0 y=0></use></svg> <span>incrementmag</span> </a><a href=/feed.xml class=rss><svg viewBox='0 0 32 32'><use xlink:href=#rssIcon x=0 y=0></use></svg> <span>RSS Feed</span></a></section><section><h4>About</h4><p><em>Increment</em> is a print and digital magazine about how teams build and operate software systems at scale. <a href=/about/ >Learn more</a></p></section><section><h4>Work with us</h4><p>Interested in joining the team at Stripe? <a href=https://stripe.com/jobs>View job openings</a></p></section></div><p class='column copyright'><span>&copy; 2022 <em>Increment</em></span> <a href=https://stripe.com>Published by Stripe</a> <a href=https://stripe.com/privacy/media-policy>Privacy policy</a></p></div></footer></body></html>