Files
nexus/sreweekly/articles/134/07-post-mortems-to-the-rescue.html
2026-09-12 17:23:01 +08:00

19 lines
32 KiB
HTML
Raw Permalink Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!doctype html><html><head><meta charset=utf-8><title>Post-mortems to the rescue – Increment: Documentation</title><meta name=description content='Effective post-mortem documentation can improve the incident response process—and save on-call developers lost sleep.'><link rel=canonical href=http://localhost:3000/documentation/post-mortems-to-the-rescue/ ><link rel=apple-touch-icon-precomposed href=/img/icon-571805a1.png><meta property=og:title content='Post-mortems to the rescue – Increment: Documentation'><meta property=og:url content=http://localhost:3000/documentation/post-mortems-to-the-rescue/ ><meta property=og:description content='Effective post-mortem documentation can improve the incident response process—and save on-call developers lost sleep.'><meta property=og:image content='https://images.ctfassets.net/3njn2qm7rrbs/59VWoUz0IXdAuGFkvRU05m/cd7dbc2083124ad9939616f62e9f3bce/post-mortems-2000-796d6faf.jpeg?w=1000'><meta name=twitter:card content=summary_large_image><meta name=twitter:image content='https://images.ctfassets.net/3njn2qm7rrbs/59VWoUz0IXdAuGFkvRU05m/cd7dbc2083124ad9939616f62e9f3bce/post-mortems-2000-796d6faf.jpeg?w=1000'><meta name=twitter:site content=@IncrementMag><meta name=twitter:title content='Post-mortems to the rescue – Increment: Documentation'><meta name=twitter:description content='Effective post-mortem documentation can improve the incident response process—and save on-call developers lost sleep.'><link rel=alternate type=application/rss+xml title=Increment href=/feed.xml><meta name=viewport content='width=device-width,initial-scale=1'><link rel=preload href=/fonts/baton-turbo/400-30a55d66.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=/fonts/baton-turbo/500-1603c0e8.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=/fonts/tiempos-text/400-c4810745.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=/fonts/tiempos-head/700-383ede62.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=stylesheet type=text/css href=/css/bundle-289d885f.css><link rel=stylesheet type=text/css href=/css/issues/6-117b9270.css><script>// Don't fade art if it loads ~instantly
setTimeout(()=>{document.documentElement.classList.add('fadeArt')},250);</script><script defer src=/js/defer-737fbd90.js></script><script>const INCREMENT_META={issueNumber:6,issueSlug:'documentation',articleSlug:'post-mortems-to-the-rescue'};</script></head><body class='Issue_documentation Article_post-mortems-to-the-rescue'><nav class=PageNav><div class=u-Container><div class=column><h1 class=logo><a href=/ ><img src=/img/logo-ae2c55d5.svg alt=Increment></a></h1><a class=out-now href=https://store.increment.com/ style=color:#595959><div class='IssueTitle tiny'><div class='t-Caps meta tiny'><span>NEW</span></div><h3 class='t-IssueTitle title'>Buy the print edition</h3></div></a><ul class=nav><li><a href=/issues/ ><span>Issues</span></a></li><li><a href=/topics/ ><span>Topics</span></a></li><li><a href=https://store.increment.com/ ><span>Store</span></a></li><li><a href=/about/ ><span>About</span></a></li></ul></div></div></nav><div class=ArticlePage itemscope itemtype=http://schema.org/Article><script type=application/ld+json>{
"@context": "http://schema.org",
"@type": "Article",
"headline": "Post-mortems to the rescue",
"image": " https://images.ctfassets.net/3njn2qm7rrbs/59VWoUz0IXdAuGFkvRU05m/cd7dbc2083124ad9939616f62e9f3bce/post-mortems-2000-796d6faf.jpeg?w&#x3D;1000",
"datePublished": "Mon, 06 Aug 2018 19:00:00 GMT",
"dateModified": "Mon, 06 Aug 2018 19:00:00 GMT",
"publisher": {
"@type": "Organization",
"name": "Increment",
"logo": {
"@type": "ImageObject",
"url": "https://increment.com/img/logo.png"
}
},
"description": "Effective post-mortem documentation can improve the incident response process—and save on-call developers lost sleep.",
"mainEntityOfPage": "http://localhost:3000/documentation/post-mortems-to-the-rescue/"
}</script><header class='u-Container ArticleHeader'><div class='u-Container art'><div class=u-Art><div class=placeholder style=background:#f2f2f2></div><picture><source srcset='https://images.ctfassets.net/3njn2qm7rrbs/59VWoUz0IXdAuGFkvRU05m/cd7dbc2083124ad9939616f62e9f3bce/post-mortems-2000-796d6faf.jpeg?w=2000 2000w, https://images.ctfassets.net/3njn2qm7rrbs/59VWoUz0IXdAuGFkvRU05m/cd7dbc2083124ad9939616f62e9f3bce/post-mortems-2000-796d6faf.jpeg?w=1000 1000w, https://images.ctfassets.net/3njn2qm7rrbs/59VWoUz0IXdAuGFkvRU05m/cd7dbc2083124ad9939616f62e9f3bce/post-mortems-2000-796d6faf.jpeg?w=500 500w, https://images.ctfassets.net/3njn2qm7rrbs/59VWoUz0IXdAuGFkvRU05m/cd7dbc2083124ad9939616f62e9f3bce/post-mortems-2000-796d6faf.jpeg?w=100 100w' sizes=100vw><img class=j-SmoothLoad src='https://images.ctfassets.net/3njn2qm7rrbs/59VWoUz0IXdAuGFkvRU05m/cd7dbc2083124ad9939616f62e9f3bce/post-mortems-2000-796d6faf.jpeg?w=1000' sizes=100vw alt='Post-mortems to the rescue'></picture></div></div><div class=column><div class=u-Grid><div class=main><h4 class='t-Byline byline large'><a href=#authors class=j-SmoothScroll itemprop=author><span>Sweta Ackerman</span></a></h4><h1 class='t-TitleSerif large title' itemprop=name>Post-mortems to the rescue</h1><div class='t-BodySans large intro' itemprop=description>Effective post-mortem documentation can improve the incident response process—and save on-call developers lost&nbsp;sleep.</div></div><a class=issue href=/documentation/ ><span class='t-Caps tiny part-of'>Part of</span><div class='IssueTitle small' style=color:#f5684d><div class='t-Caps meta'><span>Issue 6</span> <span>August 2018</span></div><h2 class='t-IssueTitle title'>Documentation</h2></div></a></div></div></header><div class='u-Container ArticleContent'><article class='ContentBody column' itemprop=articleBody><div class=ArticleLayout><p>Once upon a time, I was on call every three weeks for a rotation during which I was reliably paged at 9 p.m. each Tuesday through Thursday night. New deployments were done at this time, and with the company still using some legacy tooling, something almost invariably went wrong. Often, the fix was simple: Restart the tool, rerun a script, and—voilà!—things would work.</p><p>But one fateful night, we ended up with about 100 people on the incident call before the fix was made, and everyone went to bed at 2 a.m. It was an engineer’s worst nightmare, a once-in-a-blue-moon fluke… Until the next night, when it happened again—and then again the night after that. The cycle continued for months, and we got so buried in dealing with the incidents themselves that we could barely pause to reflect on what we could have improved or done differently to prevent these issues.</p><div class='Small Sidebar CrossReference'><a href=/on-call/when-the-pager-goes-off><div class='u-Art art'><div class=placeholder style=background:#f2f2f2></div><picture><source srcset='https://images.ctfassets.net/3njn2qm7rrbs/3buorQfLVwQiD58sXNILJE/d59d2cbaf87ce0d40e499f26f586ad77/pager.png?w=2000 2000w, https://images.ctfassets.net/3njn2qm7rrbs/3buorQfLVwQiD58sXNILJE/d59d2cbaf87ce0d40e499f26f586ad77/pager.png?w=1000 1000w, https://images.ctfassets.net/3njn2qm7rrbs/3buorQfLVwQiD58sXNILJE/d59d2cbaf87ce0d40e499f26f586ad77/pager.png?w=500 500w, https://images.ctfassets.net/3njn2qm7rrbs/3buorQfLVwQiD58sXNILJE/d59d2cbaf87ce0d40e499f26f586ad77/pager.png?w=100 100w' sizes='(min-width: 670px) 300px, 100vw'><img class=j-SmoothLoad src='https://images.ctfassets.net/3njn2qm7rrbs/3buorQfLVwQiD58sXNILJE/d59d2cbaf87ce0d40e499f26f586ad77/pager.png?w=500' sizes='(min-width: 670px) 300px, 100vw'></picture></div><div class=text><h4>From issue 1</h4><h2>What happens when the pager goes off?</h2><p>To discover the state of incident response across the tech industry, we surveyed over thirty industry leaders (including Amazon, Dropbox, Facebook, Google, and Netflix) about their incident response processes.</p></div></a></div><p>Unfortunately, that on-call experience mirrors those of many other engineers at companies of all sizes, from startup shops to large enterprises. I am lucky enough to now work for PagerDuty, a company that not only helps to reduce these pain points, but is also a thought leader in digital operations and major incident response. At PagerDuty, a critical component of the incident response process is the learning and follow-up phase. It’s also one of my favorite parts of the process—the time when everyone gets together to reflect and have conversations about how to improve both the incident response process and the technical services and infrastructure.</p><p>Incident response also yields some essential documentation. One avenue for driving continuous improvement is through the post-mortem process. Post-mortems aren’t just meetings—they’re also documents that detail the Five Ws (who, what, where, when, and why) of an incident and help teams to garner actionable insights on how to make improvements. If done well, a post-mortem can be a powerful tool for both current and future teams.</p><p>A good post-mortem process is broken down into three major parts, the first of which will usually take up the bulk of your time:</p><ul><li><p>Writing a post-mortem.</p></li><li><p>Reviewing the post-mortem and publishing the post-mortem.</p></li><li><p>Tracking the post-mortem.</p></li></ul><p>Let’s go through each step in more detail.</p><h2>Writing the post-mortem</h2><p>The main goal of writing a post-mortem is to capture the timeline of events and the impact of an incident so that it can be presented in a subsequent review meeting. Fittingly enough, I’m a big fan of <a href=https://www.pagerduty.com/blog/better-incident-post-mortems/ target=_blank rel='noopener noreferrer'>PagerDuty’s own post-mortem tool</a>. Alternatively, a simple wiki template that’s easy to create and captures <a href=https://response.pagerduty.com/after/post_mortem_template/ target=_blank rel='noopener noreferrer'>all of the fields listed here</a> works. It’s also important to capture and save all post-mortems in a searchable place. (More on this later).</p><p>Some key highlights to include in the post-mortem are:</p><ul><li><p><strong>The timeline: </strong>This will constitute the majority of the post-mortem. Start by including important changes in incident status or impact to customers and any major actions taken by responders, engineers, or subject matter experts. Additionally, for each item, include a data source or metric (such as a DataDog graph, tweets showing customer impact, etc.).</p></li><li><p><strong>Analysis: </strong>A simple summary of what happened. This should capture the underlying cause of the incident, how many customers were affected, and the overall impact on customers (e.g., what functionality was degraded or affected).</p></li><li><p><strong>Action items: </strong>List the actions that were identified and undertaken during the incident, as well as any necessary follow-up tasks. These action items should be captured in the post-mortem so that they can be assigned later on.</p></li><li><p><strong>External messaging: </strong>Assuming this was a major incident, draft the external messaging to customers, recapping some of the details above.</p></li></ul><div class='Sidebar large'><blockquote><p>Having an easily searchable record of past incidents allows you to quickly look at similar cases and even reference specific graphs or data points.</p></blockquote></div><h2>Reviewing the post-mortem</h2><p>Once you’ve filled out the post-mortem template, send it out to all parties ahead of the post-mortem meeting. Key stakeholders to invite to the meeting include the Incident Commander (IC) and any ICs-in-training; technical service owners; key responders, engineers, or subject matter experts involved in the incident response; and, for major incidents, a customer liaison. Invite all members to leave comments or make edits to the report, especially to the timeline portion.</p><p>If everyone has had a chance to review and edit the post-mortem timeline ahead of time, the post-mortem review meeting itself should only take about 30 minutes. However, you may prefer an hourlong meeting for longer or larger incidents. Regardless of length, the post-mortem review meeting should focus on the following:</p><ul><li><p>Alignment on the timeline. Quickly recap and review the timeline and ensure that everyone is on the same page.</p></li><li><p>Discussion of how the problem could have been caught. Capture any new action items along the way.</p></li><li><p>Discussion of customer impact and the external messaging, if needed.</p></li><li><p>Review and assignment of action items, along with ETAs.</p></li></ul><h2>Publishing the post-mortem</h2><p>Once you’ve completed the post-mortem review meeting, there’s one final but important step you have to take: publishing the post-mortem. Distribute the post-mortem as an internal communication, typically via email, to all relevant stakeholders, describing the results and key learnings and providing a link to the full report.</p></div><div class='ArticleLayout flipped'><h2>Tracking post-mortems</h2><p>After some months of having a well-structured post-mortem process in place, you may find yourself with a list of post-mortem documents, ideally tracked in a wiki or another searchable tool. Why does this matter, and how does it help?</p><div class='Sidebar large'><blockquote><p>A pattern of similar or repeated incidents with the same underlying root cause can point to the need for larger architectural changes.</p></blockquote></div><p>There are many benefits to having a detailed, searchable collection of post-mortems:</p><ol><li><p><strong>A list of post-mortems serves as a major incident log that can be used to inform future incident response.</strong> The next time you are in the heat of a major incident, the information you need may not be at hand. Having an easily searchable record of past incidents allows you to quickly look at similar cases and even reference specific graphs or data points. No more digging for old information in new places.</p></li><li><p><strong>Post-mortems can help align the whole business by providing everyone access to the same information about an incident—a benefit no matter the size of the company.</strong> Once the post-mortem is published, the information within it can be used by many departments for a variety of purposes. For example, Sales can consult post-mortems when customers or prospects ask them about a past incident; having a log of these incidents will put the key messaging and details at the Sales team’s fingertips. Or, Finance can consult a post-mortem to evaluate the impact to the customer in case credits need to be issued for a service degradation. And so on.</p></li><li><p><strong>Post-mortems provide a business case for technical reinvestment.</strong> Having a rich post-mortem log allows engineering team leads to more easily inspect which parts of the technical architecture might need some reinvestment. A pattern of similar or repeated incidents with the same underlying root cause can point to the need for larger architectural changes. Post-mortems contain all of the data an engineering manager needs to help get buy-in and alignment from Product counterparts, as well as other teams that may need to spend time working on fixing issues in the longer term. Post-mortems are a great way to bring awareness to these issues and quantify them in business-speak.</p></li></ol><p>While it may seem like the creation of post-mortems as documentation takes a lot of time and investment, in reality the effort is quite minimal compared to the time and money that is lost when companies remain mired in major tech debt or disorganized incident response processes. I look back on my time spent putting out fires on call, and I think about how different things would have been if we had recognized the value of post-mortem documentation sooner. We could have saved many hours of lost sleep, fostered a culture of continuous learning, delivered better software, and saved customers a whole lot of pain.</p><p>If you find yourself in a similar situation, you don’t have to reinvent the wheel. Take advantage of all of the great work that has already done by people who have lived through this. It will really help you to <a href=https://response.pagerduty.com/after/post_mortem_process/ target=_blank rel='noopener noreferrer'>set yourself, your team, and your company up for success</a>.</p></div></article></div><div class='u-Container ArticleFooter'><div class=column><div class='u-Grid ContentBody small content'><div class=authors id=authors><div><div class=photo><figure style='background-image:url(https://images.ctfassets.net/3njn2qm7rrbs/5nvzr4d22l2JgBI9tZlvza/d13fa44f8ddcf9af74c172f2968451f4/Sweta_Ackerman.png?w=500)'></figure></div><div class=text><h4>About the author</h4><p><b>Sweta Ackerman</b> is an engineering leader at PagerDuty. In her free time, she enjoys spending time with her husband and watching and playing tennis.</p><p><a href=https://twitter.com/swetavajjhala target=_blank>@swetavajjhala</a></p></div></div><div><div class=photo><figure style='background-image:url(https://images.ctfassets.net/3njn2qm7rrbs/20gJNooeAQM6QvPo91haNz/5c4c721d1b244fce619522e56bceca53/content-strategy-2000-60e87769.jpeg?w=500)'></figure></div><div class=text><h4>Artwork by</h4><p><b>Ori Toor</b></p><p><a href=https://oritoor.com/ target=_blank>oritoor.com</a></p></div></div></div><div class=topics><div class=text><h4>Topics</h4><p><a href=/topics/guides/ >Guides &amp; Best Practices</a></p></div></div></div></div></div></div><div class=SubscribeBox><a id=newsletter class=anchor href=#newsletter></a><div class='ContentBody inverted store'><div class=u-Container><div class='u-Grid column'><div class=box style=background:#4c70b1><div class=text><h2>Buy the print edition</h2><p class=j-TextBalance>Visit the Increment Store to purchase print issues.</p><p><a class='t-Caps u-Arrow' href=https://store.increment.com/ >Store</a></p></div><a href=https://store.increment.com/ class=magazine><figure class=j-MaskedImage data-fill=/art/19/19-cutout-1000-7ffb5dba.png data-mask=/art/19/fill-cms-1000-83522f38.png></figure></a></div></div></div></div><div class='ContentBody email'><div class=u-Container><div class='u-Grid column'><div class='j-EmailForm box' style='box-shadow:0 -5px 0 #4c70b1'></div></div></div></div></div><div class=ContinueReading><div class=u-Container><div class=column><h2 class='t-Caps xlarge'>Continue Reading</h2><ul class='u-Grid articles'><li class='ArticleBlock footer' style=color:#53a88e><a class=j-Preload href=/development/a-guide-to-coding-accessible-developer-tools/ ><div class='IssueTitle tiny' style=color:#53a88e><div class='t-Caps meta'><span>3</span></div><h3 class='t-IssueTitle title'>Development</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Suz Hinton</span></h4><h3 class='t-TitleSans title'>A guide to coding accessible developer tools</h3></div><div class='t-BodySerif small intro'>When we talk about accessibility in the tech industry, most conversations are centered around the end user. But how accessible are the tools we code for other developers?</div></div></a></li><li class='ArticleBlock footer' style=color:#f5684d><a class=j-Preload href=/documentation/primer-on-documentation-content-strategy/ ><div class='IssueTitle tiny' style=color:#f5684d><div class='t-Caps meta'><span>6</span></div><h3 class='t-IssueTitle title'>Documentation</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Stephanie Blotner</span></h4><h3 class='t-TitleSans title'>A primer on documentation content strategy</h3></div><div class='t-BodySerif small intro'>Tips to help engineering teams produce high-quality documentation—with or without the support of designated technical&nbsp;writers.</div></div></a></li><li class='ArticleBlock footer' style=color:#f5684d><a class=j-Preload href=/documentation/why-investing-in-internal-docs-is-worth-it/ ><div class='IssueTitle tiny' style=color:#f5684d><div class='t-Caps meta'><span>6</span></div><h3 class='t-IssueTitle title'>Documentation</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Dave Nunez</span></h4><h3 class='t-TitleSans title'>Why it’s worth it to invest in internal docs</h3></div><div class='t-BodySerif small intro'>Good internal documentation leads to more stable and innovative development, and a better experience for users and developers alike. Here’s a look at some best practices, and how engineering orgs can make documentation a part of their&nbsp;culture.</div></div></a></li><li class='ArticleBlock footer' style=color:#f5684d><a class=j-Preload href=/documentation/what-a-deploy-bot-taught-glossier-about-documentation/ ><div class='IssueTitle tiny' style=color:#f5684d><div class='t-Caps meta'><span>6</span></div><h3 class='t-IssueTitle title'>Documentation</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Aaron Suggs</span></h4><h3 class='t-TitleSans title'>What a deploy bot taught us about documentation</h3></div><div class='t-BodySerif small intro'>How Glossier used a reader-centric approach to simplify their development team’s code deployment process and improve&nbsp;velocity.</div></div></a></li><li class='ArticleBlock footer' style=color:#f5684d><a class=j-Preload href=/documentation/documentation-in-an-agile-world/ ><div class='IssueTitle tiny' style=color:#f5684d><div class='t-Caps meta'><span>6</span></div><h3 class='t-IssueTitle title'>Documentation</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Poornima Apte</span></h4><h3 class='t-TitleSans title'>Documentation in an agile world</h3></div><div class='t-BodySerif small intro'>The long and short of the perks and pitfalls of using bug-tracking systems as central repositories for internal documentation.</div></div></a></li><li class='ArticleBlock footer' style=color:#f5684d><a class=j-Preload href=/documentation/transforming-spacys-docs/ ><div class='IssueTitle tiny' style=color:#f5684d><div class='t-Caps meta'><span>6</span></div><h3 class='t-IssueTitle title'>Documentation</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Ines Montani</span></h4><h3 class='t-TitleSans title'>The process: Transforming spaCy’s docs</h3></div><div class='t-BodySerif small intro'>Making your documentation work for users with vastly different needs is a challenge. Here’s how spaCy, an open-source library for natural language processing, did&nbsp;it.</div></div></a></li><li class='ArticleBlock footer' style=color:#4a5ad3><a class=j-Preload href=/open-source/beyond-maintenance/ ><div class='IssueTitle tiny' style=color:#4a5ad3><div class='t-Caps meta'><span>9</span></div><h3 class='t-IssueTitle title'>Open Source</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Safia Abdalla</span></h4><h3 class='t-TitleSans title'>Beyond maintenance</h3></div><div class='t-BodySerif small intro'>A consideration of how human-oriented investments are vital to a project’s long-term&nbsp;health.</div></div></a></li><li class='ArticleBlock footer' style=color:#863051><a class=j-Preload href=/reliability/open-source-software-resiliency/ ><div class='IssueTitle tiny' style=color:#863051><div class='t-Caps meta'><span>16</span></div><h3 class='t-IssueTitle title'>Reliability</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Safia Abdalla</span></h4><h3 class='t-TitleSans title'>Open-source excursions: Optimizing for operational resiliency</h3></div><div class='t-BodySerif small intro'>Documentation, automation, and a little sharing-is-caring can help OSS projects maintain their&nbsp;uptime.</div></div></a></li><li class='ArticleBlock footer' style=color:#ef766e><a class=j-Preload href=/on-call/crafting-sustainable-on-call-rotations/ ><div class='IssueTitle tiny' style=color:#ef766e><div class='t-Caps meta'><span>1</span></div><h3 class='t-IssueTitle title'>On-Call</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Ryn Daniels</span></h4><h3 class='t-TitleSans title'>Crafting sustainable on-call rotations</h3></div><div class='t-BodySerif small intro'>Ryn Daniels shares strategies that everyone can use to build better, kinder, and more sustainable on-call rotations.</div></div></a></li></ul><h3 class='t-Caps large'>Explore Topics</h3><ul class='t-BodySerif large topics'><li><a href=/topics/learn/ >Learn Something New</a></li><li><a href=/topics/scaling/ >Scaling &amp; Growth</a></li><li><a href=/topics/ask-an-expert/ >Ask an Expert</a></li><li><a href=/topics/interviews/ >Interviews &amp; Surveys</a></li><li><a href=/topics/guides/ >Guides &amp; Best Practices</a></li><li><a href=/topics/opinion/ >Essays &amp; Opinion</a></li><li><a href=/topics/culture/ >Workplace &amp; Culture</a></li></ul><h3 class='t-Caps large'>All Issues</h3><ul class=issues><li><a href=/planning/ ><div class='IssueTitle large' style=color:#4c70b1><div class='t-Caps meta'><span>Issue 19</span> <span>November 2021</span></div><h1 class='t-IssueTitle title'>Planning</h1></div></a></li><li><a href=/mobile/ ><div class='IssueTitle large' style=color:#439eab><div class='t-Caps meta'><span>Issue 18</span> <span>August 2021</span></div><h1 class='t-IssueTitle title'>Mobile</h1></div></a></li><li><a href=/containers/ ><div class='IssueTitle large' style=color:#443d79><div class='t-Caps meta'><span>Issue 17</span> <span>May 2021</span></div><h1 class='t-IssueTitle title'>Containers</h1></div></a></li><li><a href=/reliability/ ><div class='IssueTitle large' style=color:#863051><div class='t-Caps meta'><span>Issue 16</span> <span>February 2021</span></div><h1 class='t-IssueTitle title'>Reliability</h1></div></a></li><li><a href=/remote/ ><div class='IssueTitle large' style=color:#29386a><div class='t-Caps meta'><span>Issue 15</span> <span>November 2020</span></div><h1 class='t-IssueTitle title'>Remote</h1></div></a></li><li><a href=/apis/ ><div class='IssueTitle large' style=color:#00afbe><div class='t-Caps meta'><span>Issue 14</span> <span>August 2020</span></div><h1 class='t-IssueTitle title'>APIs</h1></div></a></li><li><a href=/frontend/ ><div class='IssueTitle large' style=color:#5ebe92><div class='t-Caps meta'><span>Issue 13</span> <span>May 2020</span></div><h1 class='t-IssueTitle title'>Frontend</h1></div></a></li><li><a href=/software-architecture/ ><div class='IssueTitle large' style=color:#40af9e><div class='t-Caps meta'><span>Issue 12</span> <span>February 2020</span></div><h1 class='t-IssueTitle title'>Software Architecture</h1></div></a></li><li><a href=/teams/ ><div class='IssueTitle large' style=color:#8e65bf><div class='t-Caps meta'><span>Issue 11</span> <span>November 2019</span></div><h1 class='t-IssueTitle title'>Teams</h1></div></a></li><li><a href=/testing/ ><div class='IssueTitle large' style=color:#e89e00><div class='t-Caps meta'><span>Issue 10</span> <span>August 2019</span></div><h1 class='t-IssueTitle title'>Testing</h1></div></a></li><li><a href=/open-source/ ><div class='IssueTitle large' style=color:#4a5ad3><div class='t-Caps meta'><span>Issue 9</span> <span>May 2019</span></div><h1 class='t-IssueTitle title'>Open Source</h1></div></a></li><li><a href=/internationalization/ ><div class='IssueTitle large' style=color:#c096ca><div class='t-Caps meta'><span>Issue 8</span> <span>February 2019</span></div><h1 class='t-IssueTitle title'>Internationalization</h1></div></a></li><li><a href=/security/ ><div class='IssueTitle large' style=color:#4dbac5><div class='t-Caps meta'><span>Issue 7</span> <span>October 2018</span></div><h1 class='t-IssueTitle title'>Security</h1></div></a></li><li><a href=/documentation/ ><div class='IssueTitle large' style=color:#f5684d><div class='t-Caps meta'><span>Issue 6</span> <span>August 2018</span></div><h1 class='t-IssueTitle title'>Documentation</h1></div></a></li><li><a href=/programming-languages/ ><div class='IssueTitle large' style=color:#d69336><div class='t-Caps meta'><span>Issue 5</span> <span>April 2018</span></div><h1 class='t-IssueTitle title'>Programming Languages</h1></div></a></li><li><a href=/energy-environment/ ><div class='IssueTitle large' style=color:#d6658e><div class='t-Caps meta'><span>Issue 4</span> <span>February 2018</span></div><h1 class='t-IssueTitle title'>Energy & Environment</h1></div></a></li><li><a href=/development/ ><div class='IssueTitle large' style=color:#53a88e><div class='t-Caps meta'><span>Issue 3</span> <span>October 2017</span></div><h1 class='t-IssueTitle title'>Development</h1></div></a></li><li><a href=/cloud/ ><div class='IssueTitle large' style=color:#707aed><div class='t-Caps meta'><span>Issue 2</span> <span>July 2017</span></div><h1 class='t-IssueTitle title'>Cloud</h1></div></a></li><li><a href=/on-call/ ><div class='IssueTitle large' style=color:#ef766e><div class='t-Caps meta'><span>Issue 1</span> <span>April 2017</span></div><h1 class='t-IssueTitle title'>On-Call</h1></div></a></li></ul></div></div></div><footer class=PageFooter><div class='u-Container ContentBody small'><svg style=display:none><symbol id=twitterIcon viewBox='0 0 32 32'><path d='M32.1 6c-1.2.5-2.5.9-3.8 1 1.4-.8 2.4-2.1 2.9-3.6-1.3.8-2.7 1.3-4.2 1.6a6.8 6.8 0 0 0-4.8-2c-3.6 0-6.6 3-6.6 6.6 0 .5.1 1 .2 1.5-5.5-.3-10.4-3-13.6-6.9-.6 1-.9 2.1-.9 3.3 0 2.3 1.2 4.3 2.9 5.5-1.1 0-2.1-.3-3-.8v.1c0 3.2 2.3 5.9 5.3 6.5-.6.2-1.1.2-1.7.2-.4 0-.8 0-1.2-.1.8 2.6 3.3 4.5 6.2 4.6-2.3 1.8-5.1 2.8-8.2 2.8-.5 0-1.1 0-1.6-.1C2.9 28 6.3 29 10 29c12.1 0 18.7-10 18.7-18.7v-.9c1.4-.9 2.5-2 3.4-3.4z' fill=currentColor /></symbol><symbol id=facebookIcon viewBox='0 0 32 32'><path d='M30.2 0H1.8C.8 0 0 .8 0 1.8v28.5c0 1 .8 1.8 1.8 1.8h15.3V19.6h-4.2v-4.8h4.2v-3.6c0-4.1 2.5-6.4 6.2-6.4 1.8 0 3.3.2 3.7.2v4.3h-2.6c-2 0-2.4 1-2.4 2.4v3.1h4.8l-.6 4.8H22V32h8.2c1 0 1.8-.8 1.8-1.8V1.8c0-1-.8-1.8-1.8-1.8z' fill=currentColor /></symbol><symbol id=rssIcon viewBox='0 0 32 32'><path d='M10.7 25.6c0 2.4-2 4.4-4.4 4.4S2 28 2 25.6s2-4.4 4.4-4.4 4.3 2 4.3 4.4zM6.1 2c-.6 0-1.3 0-2 .1-1.3.1-2.2 1.2-2.1 2.4.1 1.2 1.2 2.2 2.4 2.1.6 0 1.1-.1 1.6-.1 10.7 0 19.4 8.7 19.4 19.4 0 .5 0 1-.1 1.6-.1 1.2.8 2.3 2.1 2.4h.2c1.2 0 2.1-.9 2.2-2.1.1-.7.1-1.4.1-2C30 12.7 19.3 2 6.1 2zm-.7 9.6c-.4 0-.8 0-1.3.1-1.3.1-2.2 1.2-2.1 2.5.1 1.3 1.2 2.2 2.5 2.1h.9c5.7 0 10.4 4.7 10.4 10.4v.9c-.1 1.3.8 2.4 2.1 2.5h.2c1.2 0 2.2-.9 2.3-2.1 0-.4.1-.8.1-1.2-.1-8.4-6.9-15.2-15.1-15.2z' fill=currentColor /></symbol><symbol id=linkedInIcon viewBox='0 0 32 32'><path d='M29.6,0H2.4C1.1,0,0,1,0,2.3v27.4C0,31,1.1,32,2.4,32h27.3c1.3,0,2.4-1,2.4-2.3V2.3C32,1,30.9,0,29.6,0z M9.5,27.3H4.7V12 h4.8V27.3z M7.1,9.9c-1.5,0-2.8-1.2-2.8-2.8c0-1.5,1.2-2.8,2.8-2.8c1.5,0,2.8,1.2,2.8,2.8C9.9,8.7,8.6,9.9,7.1,9.9z M27.3,27.3 h-4.7v-7.4c0-1.8,0-4-2.5-4c-2.5,0-2.8,1.9-2.8,3.9v7.6h-4.7V12H17v2.1h0.1c0.6-1.2,2.2-2.5,4.5-2.5c4.8,0,5.7,3.2,5.7,7.3V27.3z' fill=currentColor /></symbol></svg><div class='column main'><section class=social><a href=https://twitter.com/incrementmag class=twitter><svg viewBox='0 0 32 32'><use xlink:href=#twitterIcon x=0 y=0></use></svg> <span>@incrementmag</span> </a><a href=https://facebook.com/incrementmag class=facebook><svg viewBox='0 0 32 32'><use xlink:href=#facebookIcon x=0 y=0></use></svg> <span>incrementmag</span> </a><a href=/feed.xml class=rss><svg viewBox='0 0 32 32'><use xlink:href=#rssIcon x=0 y=0></use></svg> <span>RSS Feed</span></a></section><section><h4>About</h4><p><em>Increment</em> is a print and digital magazine about how teams build and operate software systems at scale. <a href=/about/ >Learn more</a></p></section><section><h4>Work with us</h4><p>Interested in joining the team at Stripe? <a href=https://stripe.com/jobs>View job openings</a></p></section></div><p class='column copyright'><span>&copy; 2022 <em>Increment</em></span> <a href=https://stripe.com>Published by Stripe</a> <a href=https://stripe.com/privacy/media-policy>Privacy policy</a></p></div></footer></body></html>