Files
nexus/sreweekly/articles/259/04-increment-reliability-embrace-your-inner-incident-commander.html
2026-09-12 17:23:01 +08:00

19 lines
32 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!doctype html><html><head><meta charset=utf-8><title>Embrace Your Inner Incident Commander - Increment</title><meta name=description content='Incident commanders can help engineering teams resolve software outages quickly and efficiently. Read a primer on technical incident command for software development teams.'><link rel=canonical href=http://localhost:3000/reliability/technical-incident-command/ ><link rel=apple-touch-icon-precomposed href=/img/icon-571805a1.png><meta property=og:title content='Embrace your inner incident commander – Increment: Reliability'><meta property=og:url content=http://localhost:3000/reliability/technical-incident-command/ ><meta property=og:description content='The way we fight fires affects how quickly we can resolve outages. Appointing an incident commander can help—and you (yes, you) can be one.'><meta property=og:image content='https://images.ctfassets.net/3njn2qm7rrbs/37tBbLA33npvvnxMQYdyil/3ac7868939e1fe0b4f4f0d9dd0f9d370/cover-2000-0ca15751.jpeg?w=1000'><meta name=twitter:card content=summary_large_image><meta name=twitter:image content='https://images.ctfassets.net/3njn2qm7rrbs/37tBbLA33npvvnxMQYdyil/3ac7868939e1fe0b4f4f0d9dd0f9d370/cover-2000-0ca15751.jpeg?w=1000'><meta name=twitter:site content=@IncrementMag><meta name=twitter:title content='Embrace your inner incident commander – Increment: Reliability'><meta name=twitter:description content='The way we fight fires affects how quickly we can resolve outages. Appointing an incident commander can help—and you (yes, you) can be one.'><link rel=alternate type=application/rss+xml title=Increment href=/feed.xml><meta name=viewport content='width=device-width,initial-scale=1'><link rel=preload href=/fonts/baton-turbo/400-30a55d66.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=/fonts/baton-turbo/500-1603c0e8.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=/fonts/tiempos-text/400-c4810745.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=/fonts/tiempos-head/700-383ede62.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=stylesheet type=text/css href=/css/bundle-289d885f.css><link rel=stylesheet type=text/css href=/css/issues/16-c2b7b183.css><script>// Don't fade art if it loads ~instantly
setTimeout(()=>{document.documentElement.classList.add('fadeArt')},250);</script><script defer src=/js/defer-737fbd90.js></script><script>const INCREMENT_META={issueNumber:16,issueSlug:'reliability',articleSlug:'technical-incident-command'};</script></head><body class='Issue_reliability Article_technical-incident-command'><nav class=PageNav><div class=u-Container><div class=column><h1 class=logo><a href=/ ><img src=/img/logo-ae2c55d5.svg alt=Increment></a></h1><a class=out-now href=https://store.increment.com/ style=color:#595959><div class='IssueTitle tiny'><div class='t-Caps meta tiny'><span>NEW</span></div><h3 class='t-IssueTitle title'>Buy the print edition</h3></div></a><ul class=nav><li><a href=/issues/ ><span>Issues</span></a></li><li><a href=/topics/ ><span>Topics</span></a></li><li><a href=https://store.increment.com/ ><span>Store</span></a></li><li><a href=/about/ ><span>About</span></a></li></ul></div></div></nav><div class=ArticlePage itemscope itemtype=http://schema.org/Article><script type=application/ld+json>{
"@context": "http://schema.org",
"@type": "Article",
"headline": "Embrace your inner incident commander",
"image": " https://images.ctfassets.net/3njn2qm7rrbs/37tBbLA33npvvnxMQYdyil/3ac7868939e1fe0b4f4f0d9dd0f9d370/cover-2000-0ca15751.jpeg?w&#x3D;1000",
"datePublished": "Thu, 25 Feb 2021 19:00:00 GMT",
"dateModified": "Thu, 25 Feb 2021 19:00:00 GMT",
"publisher": {
"@type": "Organization",
"name": "Increment",
"logo": {
"@type": "ImageObject",
"url": "https://increment.com/img/logo.png"
}
},
"description": "The way we fight fires affects how quickly we can resolve outages. Appointing an incident commander can help—and you (yes, you) can be one.",
"mainEntityOfPage": "http://localhost:3000/reliability/technical-incident-command/"
}</script><header class='u-Container ArticleHeader'><div class='u-Container art'><div class=u-Art><div class=placeholder style=background:#f2f2f2></div><picture><source srcset='https://images.ctfassets.net/3njn2qm7rrbs/37tBbLA33npvvnxMQYdyil/3ac7868939e1fe0b4f4f0d9dd0f9d370/cover-2000-0ca15751.jpeg?w=2000 2000w, https://images.ctfassets.net/3njn2qm7rrbs/37tBbLA33npvvnxMQYdyil/3ac7868939e1fe0b4f4f0d9dd0f9d370/cover-2000-0ca15751.jpeg?w=1000 1000w, https://images.ctfassets.net/3njn2qm7rrbs/37tBbLA33npvvnxMQYdyil/3ac7868939e1fe0b4f4f0d9dd0f9d370/cover-2000-0ca15751.jpeg?w=500 500w, https://images.ctfassets.net/3njn2qm7rrbs/37tBbLA33npvvnxMQYdyil/3ac7868939e1fe0b4f4f0d9dd0f9d370/cover-2000-0ca15751.jpeg?w=100 100w' sizes=100vw><img class=j-SmoothLoad src='https://images.ctfassets.net/3njn2qm7rrbs/37tBbLA33npvvnxMQYdyil/3ac7868939e1fe0b4f4f0d9dd0f9d370/cover-2000-0ca15751.jpeg?w=1000' sizes=100vw alt='Embrace your inner incident commander'></picture></div></div><div class=column><div class=u-Grid><div class=main><h4 class='t-Byline byline large'><a href=#authors class=j-SmoothScroll itemprop=author><span>Tanya Reilly</span></a></h4><h1 class='t-TitleSerif large title' itemprop=name>Embrace your inner incident commander</h1><div class='t-BodySans large intro' itemprop=description>The way we fight fires affects how quickly we can resolve outages. Appointing an incident commander can help—and you (yes, you) can be&nbsp;one.</div></div><a class=issue href=/reliability/ ><span class='t-Caps tiny part-of'>Part of</span><div class='IssueTitle small' style=color:#863051><div class='t-Caps meta'><span>Issue 16</span> <span>February 2021</span></div><h2 class='t-IssueTitle title'>Reliability</h2></div></a></div></div></header><div class='u-Container ArticleContent'><article class='ContentBody column' itemprop=articleBody><div class=ArticleLayout><p>Your sales database just went down. Your source control is in a crash loop and a thousand software engineers are reading Twitter while they wait for it to come back online. You pushed a change that’s routing all of your North American traffic through Azerbaijan. Whatever just happened, you’ve got an incident on your hands. What happens now?</p><p>The beginning of an incident can be chaotic, especially when it involves multiple teams. If responders aren’t used to working together, they’re likely to talk over and past each other as they independently discover the same spiky metric or smoking log line. Suddenly the graphs show a spike followed by a dip, and the investigation takes a sharp turn—until the logs reveal someone has tried to restart the service. This well-intentioned but misguided attempt to help deleted a log another responder was running a debugging tool against—which means they’re going to have to restart their analysis.</p><p>A team member offers an insight that could shave 30 minutes off the outage, but they do so in a Slack thread nobody is paying attention to. (Later, their “I told you so” will not win them any friends.) Meanwhile, engineers are calling out facts using terminology only their team understands; scared of seeming incompetent, nobody asks for clarification. Stakeholders are asking increasingly strained questions about when the situation might be resolved. The engineers who best understand the system are too busy replying to focus on debugging.</p><p>And the incident is still going on.</p><p>If this sounds familiar, it’s because most growing organizations go through a phase of this kind of chaos. Incident response might only be a small part of any company’s reliability strategy, but it’s still important to commit to getting it right: Sloppy responses can prolong your outages, damage your credibility, demoralize your teams, and erode your customers’ trust. In the worst cases, the response can cause more disruption than the problem you’re setting out to solve.</p><p>Great incident response alone isn’t a panacea—you need to invest in resilient systems to reduce the number and severity of outages—but once an incident is underway, human behavior becomes a crucial component of those systems. The way responders react plays a big role in determining how quickly and smoothly the situation is resolved.</p></div><div class='ArticleLayout flipped'><h2>Enter incident command</h2><p>In 1970, Southern California had one of its worst wildfire seasons on record. Despite rapid response from the U.S. Forest Service, multiple fire departments, and other agencies, the damage was devastating. After the fires, the Forest Service worked with the other institutions to identify some of the weak links in their coordinated response. These included agencies planning and operating independently, a lack of shared terminology, hierarchies that made it difficult for teams to cooperate and communicate effectively, and siloed information and resources, which led to logistical messes where fire trucks from different departments <a href=http://www.emsics.com/history-of-ics/ target=_blank rel='noopener noreferrer'>passed each other on the way to fires that would have been closer to the other department</a>.</p><p>The fire protection agencies set out to “make a quantum jump” (<a href=http://www.emsics.com/history-of-ics/ target=_blank rel='noopener noreferrer'>in the words of the Forest Service’s 1973 “FIRESCOPE Program Charter”</a>) in their ability to coordinate and allocate resources during fires. Their scope later expanded to encompass all risks and hazards, and in 1974 the Incident Command System (ICS) was born as a simple but effective structure that could handle any kind of major incident.</p><p>While the ICS is a complex system that most of us in tech will never use in full—it includes defined locations, templates for assigning radio frequencies, frameworks for objections, and <a href=http://www.emsics.com/resources/organization-chart/ target=_blank rel='noopener noreferrer'>a hierarchical chain of command with 36 separate named roles</a>—many software organizations have adopted parts of it for managing their own incidents. In particular, they’ve found value in having a dedicated incident commander (IC—not to be confused with that other common IC, individual contributor), someone who takes charge of the situation instead of fighting the fire.</p></div><div class=ArticleLayout><h2>The incident commander’s role</h2><p>While tech outages (usually!) aren’t literal fires, they can also benefit from having a responder focused on leading the incident rather than actual debugging.</p><h3>Providing clarity</h3><p>By setting up a chain of command and laying down rules, the IC provides structure for the incident response. Everyone knows who to talk to for the most up-to-date information, as well as who gets the deciding vote when there are disagreements about how to move forward. When a situation is murky, the IC is expected to ask questions, clarify jargon, and make sure responders understand each other.</p><h3>Staying focused on the user</h3><p>Nominating an IC means there’s at least one person who won’t get distracted by the concerns of any particular team. While other responders may get wrapped up in solving technology problems, the IC should keep the bigger picture in mind, advocating for whichever solutions or mitigations reduce user impact and shorten the incident.</p><h3>Supplying status updates</h3><p>The IC tracks and documents the current customer impact, leads being explored, reminders to return to later, points of contact, decisions that still need to be made, and any other high-level concerns. They’re often responsible for making sure customers and stakeholders get regular updates about what’s going on, as well as fielding interruptions from people who aren’t involved in getting the systems back online. If this ends up being too time-consuming, they might delegate these responsibilities to a communications lead.</p><h3>Coordinating with others</h3><p>Responders can make a situation worse by not working together. The IC makes sure the various actions being taken aren’t in conflict. They have the authority to shut down what fire commanders refer to as “freelancers”—bystanders or firefighters from neighboring departments who endanger other responders by joining the fight without coordinating first.</p><h3>Reducing noise</h3><p>The IC can set the rules of engagement for communication channels, such as moving all debugging to a central channel or redirecting side conversations elsewhere. They can also designate a single operations lead for each team and ask all other members to send their updates to that lead, who then reports back to the IC. This reduces the noise in the channel and makes it less likely that important messages will be lost.</p><h3>Taking care of responders</h3><p>Most people aren’t good at collaborating or interpreting information when tired. The IC should be alert for fatigue, crankiness, or conflict and intervene when needed, including finding new folks to take over if responders (including themselves) have been running for too long.</p></div><div class='ArticleLayout flipped'><h2>Making it work</h2><p>For incident command to be effective, it needs to be a consistent process that’s well understood throughout the organization, with clear leadership support and encouragement. ICs should feel confident that if they step up, everyone else will understand what they’re doing and why. When introducing an incident command process to an organization, be clear about how you expect it to work.</p><h3>Decide which incidents merit an IC</h3><p>This will depend on the organization, but they might include things like user-visible outages, issues that affect more than one team or have many stakeholders, or issues that span an extended time frame and require consistent, coordinated communication.</p><h3>Be clear about who your ICs will be</h3><p>This might be a dedicated IC on-call rotation, a pool of trained volunteers, or an expectation that everyone above some level of seniority will be prepared to step in when needed.</p><h3>Make sure everyone knows how to work with an IC</h3><p>There should be a well-documented process for when and how to call in an incident commander. Everyone should be clear on the responsibilities of the role and the scope of the IC’s authority.</p></div><div class=ArticleLayout><h2>You’ve got to believe</h2><div class='Sidebar large'><blockquote><p>Someone stepping in to address a crisis and saying “I’m Batman” doesn’t help unless people have bought into the idea of Batman.</p></blockquote></div><p>Most of all, having an incident commander only works if everyone believes in the role. Someone stepping in to address a crisis and saying “I’m Batman” doesn’t help unless people have bought into the idea of Batman. Otherwise, it’s just one more person in the room (maybe in a ridiculous costume) adding to the noise. The IC needs enough authority to be able to move a team’s communication to another channel, delegate bystanders to go on a side quest to collect log output, or instruct “freelancers” to stop trying things. Their ability to coordinate the response only works if other responders are engaging with the process and with them.</p><p>Like all new processes, incident command will require some culture change, and perhaps a healthy amount of getting it wrong before you start getting it right. You can establish expectations through training, documentation, presentations at all-hands events, and emails to the company, while disaster simulations can help ensure everyone is well-practiced at calling in an IC and following their lead. Incidents should be visible to all, perhaps in a dedicated incident channel, and incident retrospectives should (blamelessly!) include specific failures in the process, such as information the IC couldn’t find or responders who didn’t check in.</p><h2>It’s your turn</h2><p>To be successful at incident command, you need two foundational skills: being comfortable telling people what to do, and being comfortable asking questions—even questions that might seem “obvious.” These skills tend to be found in more senior people, but anyone can take on the role if they’re willing to take charge. For those who aren’t quite ready, coordinating smaller incidents can help build that comfort. It’s good leadership experience—not to mention a great way to learn more about your systems end to end.</p><p>The most important aspect of the incident commander’s role is to be clear and consistent about what you need people to do, and to make sure they see others doing it. The more bystanders see someone say, “I’ll be the incident commander,” the more it becomes the cultural norm. Over time, responders will start expecting an incident commander to step up, and will call for one when an incident breaks out.</p><p>So step up. When you’re surrounded by sirens and flames and chaos, resist the urge to be just another person fighting the fire. Declare yourself the incident commander and take charge.</p></div></article></div><div class='u-Container ArticleFooter'><div class=column><div class='u-Grid ContentBody small content'><div class=authors id=authors><div><div class=photo><figure style='background-image:url(https://images.ctfassets.net/3njn2qm7rrbs/5aGUCgwpibtLsVjDqRgyq5/c155c1db288db63efa2faeabdf2eec82/Tanya_Reilly.jpeg?w=500)'></figure></div><div class=text><h4>About the author</h4><p><b>Tanya Reilly</b> is a principal engineer at Squarespace. She wishes she was on a train right now.</p><p><a href=https://twitter.com/whereistanya target=_blank>@whereistanya</a></p></div></div><div><div class=photo><figure style='background-image:url(https://images.ctfassets.net/3njn2qm7rrbs/37tBbLA33npvvnxMQYdyil/3ac7868939e1fe0b4f4f0d9dd0f9d370/cover-2000-0ca15751.jpeg?w=500)'></figure></div><div class=text><h4>Artwork by</h4><p><b>Sua Balac</b></p><p><a href=https://suabalac.com/ target=_blank>suabalac.com</a></p></div></div></div><div class=topics><div class=text><h4>Topics</h4><p><a href=/topics/culture/ >Workplace &amp; Culture</a></p><p><a href=/topics/opinion/ >Essays &amp; Opinion</a></p></div></div></div></div></div></div><div class=SubscribeBox><a id=newsletter class=anchor href=#newsletter></a><div class='ContentBody inverted store'><div class=u-Container><div class='u-Grid column'><div class=box style=background:#4c70b1><div class=text><h2>Buy the print edition</h2><p class=j-TextBalance>Visit the Increment Store to purchase print issues.</p><p><a class='t-Caps u-Arrow' href=https://store.increment.com/ >Store</a></p></div><a href=https://store.increment.com/ class=magazine><figure class=j-MaskedImage data-fill=/art/19/19-cutout-1000-7ffb5dba.png data-mask=/art/19/fill-cms-1000-83522f38.png></figure></a></div></div></div></div><div class='ContentBody email'><div class=u-Container><div class='u-Grid column'><div class='j-EmailForm box' style='box-shadow:0 -5px 0 #4c70b1'></div></div></div></div></div><div class=ContinueReading><div class=u-Container><div class=column><h2 class='t-Caps xlarge'>Continue Reading</h2><ul class='u-Grid articles'><li class='ArticleBlock footer' style=color:#863051><a class=j-Preload href=/reliability/failure-is-okay/ ><div class='IssueTitle tiny' style=color:#863051><div class='t-Caps meta'><span>16</span></div><h3 class='t-IssueTitle title'>Reliability</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Heidi Waterhouse</span></h4><h3 class='t-TitleSans title'>Everything is broken, and it’s okay</h3></div><div class='t-BodySerif small intro'>Accepting that imperfect things still work is fundamental to preventing failures from becoming catastrophes.</div></div></a></li><li class='ArticleBlock footer' style=color:#8e65bf><a class=j-Preload href=/teams/doing-a-little-bit-of-the-impossible/ ><div class='IssueTitle tiny' style=color:#8e65bf><div class='t-Caps meta'><span>11</span></div><h3 class='t-IssueTitle title'>Teams</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Anil Dash</span></h4><h3 class='t-TitleSans title'>Doing (a little bit of) the impossible</h3></div><div class='t-BodySerif small intro'>A commitment to trust, fairness, and inclusion is at the heart of a strong company—and healthy&nbsp;teams.</div></div></a></li><li class='ArticleBlock footer' style=color:#863051><a class=j-Preload href=/reliability/how-to-build-organizational-resilience/ ><div class='IssueTitle tiny' style=color:#863051><div class='t-Caps meta'><span>16</span></div><h3 class='t-IssueTitle title'>Reliability</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Ryn Daniels</span></h4><h3 class='t-TitleSans title'>How to build organizational resilience</h3></div><div class='t-BodySerif small intro'>By encoding resilience into an organization’s culture, engineering teams can be better equipped to tackle the unknown and unexpected.</div></div></a></li><li class='ArticleBlock footer' style=color:#863051><a class=j-Preload href=/reliability/brain-on-progress/ ><div class='IssueTitle tiny' style=color:#863051><div class='t-Caps meta'><span>16</span></div><h3 class='t-IssueTitle title'>Reliability</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Lara Hogan</span></h4><h3 class='t-TitleSans title'>Your brain on progress</h3></div><div class='t-BodySerif small intro'>Strategies for nurturing that feel-good sense of accomplishment when doing largely invisible&nbsp;work.</div></div></a></li><li class='ArticleBlock footer' style=color:#8e65bf><a class=j-Preload href=/teams/do-engineering-managers-need-to-be-technical/ ><div class='IssueTitle tiny' style=color:#8e65bf><div class='t-Caps meta'><span>11</span></div><h3 class='t-IssueTitle title'>Teams</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Will Larson</span></h4><h3 class='t-TitleSans title'>Do engineering managers need to be technical?</h3></div><div class='t-BodySerif small intro'>And what do we even mean when we say “technical”?</div></div></a></li><li class='ArticleBlock footer' style=color:#8e65bf><a class=j-Preload href=/teams/pay-fair/ ><div class='IssueTitle tiny' style=color:#8e65bf><div class='t-Caps meta'><span>11</span></div><h3 class='t-IssueTitle title'>Teams</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Lara Hogan</span></h4><h3 class='t-TitleSans title'>Pay fair</h3></div><div class='t-BodySerif small intro'>An introduction to correcting—and preventing—compensation&nbsp;inequity.</div></div></a></li><li class='ArticleBlock footer' style=color:#8e65bf><a class=j-Preload href=/teams/trust-the-process/ ><div class='IssueTitle tiny' style=color:#8e65bf><div class='t-Caps meta'><span>11</span></div><h3 class='t-IssueTitle title'>Teams</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Paul Ford</span></h4><h3 class='t-TitleSans title'>Trust the process</h3></div><div class='t-BodySerif small intro'>Though they run on a mixture of paper and lore, effective editorial organizations ship like clockwork. Can engineering teams learn from their enduring processes?</div></div></a></li><li class='ArticleBlock footer' style=color:#8e65bf><a class=j-Preload href=/teams/the-path-to-management/ ><div class='IssueTitle tiny' style=color:#8e65bf><div class='t-Caps meta'><span>11</span></div><h3 class='t-IssueTitle title'>Teams</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Kaya Thomas</span></h4><h3 class='t-TitleSans title'>The path to management</h3></div><div class='t-BodySerif small intro'>Insight for developers considering making a move from individual contributor to engineering&nbsp;manager.</div></div></a></li><li class='ArticleBlock footer' style=color:#8e65bf><a class=j-Preload href=/teams/the-epistemology-of-software-quality/ ><div class='IssueTitle tiny' style=color:#8e65bf><div class='t-Caps meta'><span>11</span></div><h3 class='t-IssueTitle title'>Teams</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Hillel Wayne</span></h4><h3 class='t-TitleSans title'>The epistemology of software quality</h3></div><div class='t-BodySerif small intro'>Studies show that human factors most influence the quality of our work. So why do we put so much stake in technical solutions?</div></div></a></li></ul><h3 class='t-Caps large'>Explore Topics</h3><ul class='t-BodySerif large topics'><li><a href=/topics/learn/ >Learn Something New</a></li><li><a href=/topics/scaling/ >Scaling &amp; Growth</a></li><li><a href=/topics/ask-an-expert/ >Ask an Expert</a></li><li><a href=/topics/interviews/ >Interviews &amp; Surveys</a></li><li><a href=/topics/guides/ >Guides &amp; Best Practices</a></li><li><a href=/topics/opinion/ >Essays &amp; Opinion</a></li><li><a href=/topics/culture/ >Workplace &amp; Culture</a></li></ul><h3 class='t-Caps large'>All Issues</h3><ul class=issues><li><a href=/planning/ ><div class='IssueTitle large' style=color:#4c70b1><div class='t-Caps meta'><span>Issue 19</span> <span>November 2021</span></div><h1 class='t-IssueTitle title'>Planning</h1></div></a></li><li><a href=/mobile/ ><div class='IssueTitle large' style=color:#439eab><div class='t-Caps meta'><span>Issue 18</span> <span>August 2021</span></div><h1 class='t-IssueTitle title'>Mobile</h1></div></a></li><li><a href=/containers/ ><div class='IssueTitle large' style=color:#443d79><div class='t-Caps meta'><span>Issue 17</span> <span>May 2021</span></div><h1 class='t-IssueTitle title'>Containers</h1></div></a></li><li><a href=/reliability/ ><div class='IssueTitle large' style=color:#863051><div class='t-Caps meta'><span>Issue 16</span> <span>February 2021</span></div><h1 class='t-IssueTitle title'>Reliability</h1></div></a></li><li><a href=/remote/ ><div class='IssueTitle large' style=color:#29386a><div class='t-Caps meta'><span>Issue 15</span> <span>November 2020</span></div><h1 class='t-IssueTitle title'>Remote</h1></div></a></li><li><a href=/apis/ ><div class='IssueTitle large' style=color:#00afbe><div class='t-Caps meta'><span>Issue 14</span> <span>August 2020</span></div><h1 class='t-IssueTitle title'>APIs</h1></div></a></li><li><a href=/frontend/ ><div class='IssueTitle large' style=color:#5ebe92><div class='t-Caps meta'><span>Issue 13</span> <span>May 2020</span></div><h1 class='t-IssueTitle title'>Frontend</h1></div></a></li><li><a href=/software-architecture/ ><div class='IssueTitle large' style=color:#40af9e><div class='t-Caps meta'><span>Issue 12</span> <span>February 2020</span></div><h1 class='t-IssueTitle title'>Software Architecture</h1></div></a></li><li><a href=/teams/ ><div class='IssueTitle large' style=color:#8e65bf><div class='t-Caps meta'><span>Issue 11</span> <span>November 2019</span></div><h1 class='t-IssueTitle title'>Teams</h1></div></a></li><li><a href=/testing/ ><div class='IssueTitle large' style=color:#e89e00><div class='t-Caps meta'><span>Issue 10</span> <span>August 2019</span></div><h1 class='t-IssueTitle title'>Testing</h1></div></a></li><li><a href=/open-source/ ><div class='IssueTitle large' style=color:#4a5ad3><div class='t-Caps meta'><span>Issue 9</span> <span>May 2019</span></div><h1 class='t-IssueTitle title'>Open Source</h1></div></a></li><li><a href=/internationalization/ ><div class='IssueTitle large' style=color:#c096ca><div class='t-Caps meta'><span>Issue 8</span> <span>February 2019</span></div><h1 class='t-IssueTitle title'>Internationalization</h1></div></a></li><li><a href=/security/ ><div class='IssueTitle large' style=color:#4dbac5><div class='t-Caps meta'><span>Issue 7</span> <span>October 2018</span></div><h1 class='t-IssueTitle title'>Security</h1></div></a></li><li><a href=/documentation/ ><div class='IssueTitle large' style=color:#f5684d><div class='t-Caps meta'><span>Issue 6</span> <span>August 2018</span></div><h1 class='t-IssueTitle title'>Documentation</h1></div></a></li><li><a href=/programming-languages/ ><div class='IssueTitle large' style=color:#d69336><div class='t-Caps meta'><span>Issue 5</span> <span>April 2018</span></div><h1 class='t-IssueTitle title'>Programming Languages</h1></div></a></li><li><a href=/energy-environment/ ><div class='IssueTitle large' style=color:#d6658e><div class='t-Caps meta'><span>Issue 4</span> <span>February 2018</span></div><h1 class='t-IssueTitle title'>Energy & Environment</h1></div></a></li><li><a href=/development/ ><div class='IssueTitle large' style=color:#53a88e><div class='t-Caps meta'><span>Issue 3</span> <span>October 2017</span></div><h1 class='t-IssueTitle title'>Development</h1></div></a></li><li><a href=/cloud/ ><div class='IssueTitle large' style=color:#707aed><div class='t-Caps meta'><span>Issue 2</span> <span>July 2017</span></div><h1 class='t-IssueTitle title'>Cloud</h1></div></a></li><li><a href=/on-call/ ><div class='IssueTitle large' style=color:#ef766e><div class='t-Caps meta'><span>Issue 1</span> <span>April 2017</span></div><h1 class='t-IssueTitle title'>On-Call</h1></div></a></li></ul></div></div></div><footer class=PageFooter><div class='u-Container ContentBody small'><svg style=display:none><symbol id=twitterIcon viewBox='0 0 32 32'><path d='M32.1 6c-1.2.5-2.5.9-3.8 1 1.4-.8 2.4-2.1 2.9-3.6-1.3.8-2.7 1.3-4.2 1.6a6.8 6.8 0 0 0-4.8-2c-3.6 0-6.6 3-6.6 6.6 0 .5.1 1 .2 1.5-5.5-.3-10.4-3-13.6-6.9-.6 1-.9 2.1-.9 3.3 0 2.3 1.2 4.3 2.9 5.5-1.1 0-2.1-.3-3-.8v.1c0 3.2 2.3 5.9 5.3 6.5-.6.2-1.1.2-1.7.2-.4 0-.8 0-1.2-.1.8 2.6 3.3 4.5 6.2 4.6-2.3 1.8-5.1 2.8-8.2 2.8-.5 0-1.1 0-1.6-.1C2.9 28 6.3 29 10 29c12.1 0 18.7-10 18.7-18.7v-.9c1.4-.9 2.5-2 3.4-3.4z' fill=currentColor /></symbol><symbol id=facebookIcon viewBox='0 0 32 32'><path d='M30.2 0H1.8C.8 0 0 .8 0 1.8v28.5c0 1 .8 1.8 1.8 1.8h15.3V19.6h-4.2v-4.8h4.2v-3.6c0-4.1 2.5-6.4 6.2-6.4 1.8 0 3.3.2 3.7.2v4.3h-2.6c-2 0-2.4 1-2.4 2.4v3.1h4.8l-.6 4.8H22V32h8.2c1 0 1.8-.8 1.8-1.8V1.8c0-1-.8-1.8-1.8-1.8z' fill=currentColor /></symbol><symbol id=rssIcon viewBox='0 0 32 32'><path d='M10.7 25.6c0 2.4-2 4.4-4.4 4.4S2 28 2 25.6s2-4.4 4.4-4.4 4.3 2 4.3 4.4zM6.1 2c-.6 0-1.3 0-2 .1-1.3.1-2.2 1.2-2.1 2.4.1 1.2 1.2 2.2 2.4 2.1.6 0 1.1-.1 1.6-.1 10.7 0 19.4 8.7 19.4 19.4 0 .5 0 1-.1 1.6-.1 1.2.8 2.3 2.1 2.4h.2c1.2 0 2.1-.9 2.2-2.1.1-.7.1-1.4.1-2C30 12.7 19.3 2 6.1 2zm-.7 9.6c-.4 0-.8 0-1.3.1-1.3.1-2.2 1.2-2.1 2.5.1 1.3 1.2 2.2 2.5 2.1h.9c5.7 0 10.4 4.7 10.4 10.4v.9c-.1 1.3.8 2.4 2.1 2.5h.2c1.2 0 2.2-.9 2.3-2.1 0-.4.1-.8.1-1.2-.1-8.4-6.9-15.2-15.1-15.2z' fill=currentColor /></symbol><symbol id=linkedInIcon viewBox='0 0 32 32'><path d='M29.6,0H2.4C1.1,0,0,1,0,2.3v27.4C0,31,1.1,32,2.4,32h27.3c1.3,0,2.4-1,2.4-2.3V2.3C32,1,30.9,0,29.6,0z M9.5,27.3H4.7V12 h4.8V27.3z M7.1,9.9c-1.5,0-2.8-1.2-2.8-2.8c0-1.5,1.2-2.8,2.8-2.8c1.5,0,2.8,1.2,2.8,2.8C9.9,8.7,8.6,9.9,7.1,9.9z M27.3,27.3 h-4.7v-7.4c0-1.8,0-4-2.5-4c-2.5,0-2.8,1.9-2.8,3.9v7.6h-4.7V12H17v2.1h0.1c0.6-1.2,2.2-2.5,4.5-2.5c4.8,0,5.7,3.2,5.7,7.3V27.3z' fill=currentColor /></symbol></svg><div class='column main'><section class=social><a href=https://twitter.com/incrementmag class=twitter><svg viewBox='0 0 32 32'><use xlink:href=#twitterIcon x=0 y=0></use></svg> <span>@incrementmag</span> </a><a href=https://facebook.com/incrementmag class=facebook><svg viewBox='0 0 32 32'><use xlink:href=#facebookIcon x=0 y=0></use></svg> <span>incrementmag</span> </a><a href=/feed.xml class=rss><svg viewBox='0 0 32 32'><use xlink:href=#rssIcon x=0 y=0></use></svg> <span>RSS Feed</span></a></section><section><h4>About</h4><p><em>Increment</em> is a print and digital magazine about how teams build and operate software systems at scale. <a href=/about/ >Learn more</a></p></section><section><h4>Work with us</h4><p>Interested in joining the team at Stripe? <a href=https://stripe.com/jobs>View job openings</a></p></section></div><p class='column copyright'><span>&copy; 2022 <em>Increment</em></span> <a href=https://stripe.com>Published by Stripe</a> <a href=https://stripe.com/privacy/media-policy>Privacy policy</a></p></div></footer></body></html>