19 lines
32 KiB
HTML
19 lines
32 KiB
HTML
<!doctype html><html><head><meta charset=utf-8><title>The benefits of transparency: Interview with Sytse “Sid” Sijbrandij, CEO of GitLab – Increment: On-Call</title><meta name=description content='We spoke with Sid about GitLab’s recent outage, its aftermath, and the impressive level of transparency GitLab offered the public during the outage.'><link rel=canonical href=http://localhost:3000/on-call/the-benefits-of-transparency/ ><link rel=apple-touch-icon-precomposed href=/img/icon-571805a1.png><meta property=og:title content='The benefits of transparency: Interview with Sytse “Sid” Sijbrandij, CEO of GitLab – Increment: On-Call'><meta property=og:url content=http://localhost:3000/on-call/the-benefits-of-transparency/ ><meta property=og:description content='We spoke with Sid about GitLab’s recent outage, its aftermath, and the impressive level of transparency GitLab offered the public during the outage.'><meta property=og:image content='https://images.ctfassets.net/3njn2qm7rrbs/41zK1dS4Q8FW5E1LgHV2Ta/83866f21aab2cf5deedcdb6f97de88af/cover.png?w=1000'><meta name=twitter:card content=summary_large_image><meta name=twitter:image content='https://images.ctfassets.net/3njn2qm7rrbs/41zK1dS4Q8FW5E1LgHV2Ta/83866f21aab2cf5deedcdb6f97de88af/cover.png?w=1000'><meta name=twitter:site content=@IncrementMag><meta name=twitter:title content='The benefits of transparency: Interview with Sytse “Sid” Sijbrandij, CEO of GitLab – Increment: On-Call'><meta name=twitter:description content='We spoke with Sid about GitLab’s recent outage, its aftermath, and the impressive level of transparency GitLab offered the public during the outage.'><link rel=alternate type=application/rss+xml title=Increment href=/feed.xml><meta name=viewport content='width=device-width,initial-scale=1'><link rel=preload href=/fonts/baton-turbo/400-30a55d66.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=/fonts/baton-turbo/500-1603c0e8.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=/fonts/tiempos-text/400-c4810745.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=/fonts/tiempos-head/700-383ede62.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=stylesheet type=text/css href=/css/bundle-289d885f.css><link rel=stylesheet type=text/css href=/css/issues/1-a2738f50.css><script>// Don't fade art if it loads ~instantly
|
||
setTimeout(()=>{document.documentElement.classList.add('fadeArt')},250);</script><script defer src=/js/defer-737fbd90.js></script><script>const INCREMENT_META={issueNumber:1,issueSlug:'on-call',articleSlug:'the-benefits-of-transparency'};</script></head><body class='Issue_on-call Article_the-benefits-of-transparency'><nav class=PageNav><div class=u-Container><div class=column><h1 class=logo><a href=/ ><img src=/img/logo-ae2c55d5.svg alt=Increment></a></h1><a class=out-now href=https://store.increment.com/ style=color:#595959><div class='IssueTitle tiny'><div class='t-Caps meta tiny'><span>NEW</span></div><h3 class='t-IssueTitle title'>Buy the print edition</h3></div></a><ul class=nav><li><a href=/issues/ ><span>Issues</span></a></li><li><a href=/topics/ ><span>Topics</span></a></li><li><a href=https://store.increment.com/ ><span>Store</span></a></li><li><a href=/about/ ><span>About</span></a></li></ul></div></div></nav><div class=ArticlePage itemscope itemtype=http://schema.org/Article><script type=application/ld+json>{
|
||
"@context": "http://schema.org",
|
||
"@type": "Article",
|
||
"headline": "The benefits of transparency: Interview with Sytse “Sid” Sijbrandij, CEO of GitLab",
|
||
"image": " https://images.ctfassets.net/3njn2qm7rrbs/41zK1dS4Q8FW5E1LgHV2Ta/83866f21aab2cf5deedcdb6f97de88af/cover.png?w=1000",
|
||
"datePublished": "Thu, 13 Apr 2017 09:00:00 GMT",
|
||
"dateModified": "Thu, 13 Apr 2017 09:00:00 GMT",
|
||
"publisher": {
|
||
"@type": "Organization",
|
||
"name": "Increment",
|
||
"logo": {
|
||
"@type": "ImageObject",
|
||
"url": "https://increment.com/img/logo.png"
|
||
}
|
||
},
|
||
"description": "We spoke with Sid about GitLab’s recent outage, its aftermath, and the impressive level of transparency GitLab offered the public during the outage.",
|
||
"mainEntityOfPage": "http://localhost:3000/on-call/the-benefits-of-transparency/"
|
||
}</script><header class='u-Container ArticleHeader'><div class=column><div class=u-Grid><div class=main><h4 class='t-Byline byline large'><a href=#authors class=j-SmoothScroll itemprop=author><span>Increment Staff</span></a></h4><h1 class='t-TitleSerif large title' itemprop=name>The benefits of transparency: Interview with Sytse “Sid” Sijbrandij, CEO of GitLab</h1><div class='t-BodySans large intro' itemprop=description>We spoke with Sid about GitLab’s recent outage, its aftermath, and the impressive level of transparency GitLab offered the public during the outage.</div></div><a class=issue href=/on-call/ ><span class='t-Caps tiny part-of'>Part of</span><div class='IssueTitle small' style=color:#ef766e><div class='t-Caps meta'><span>Issue 1</span> <span>April 2017</span></div><h2 class='t-IssueTitle title'>On-Call</h2></div></a></div></div></header><div class='u-Container ArticleContent'><article class='ContentBody column' itemprop=articleBody><div class=ArticleLayout><p>On January 31st, a production engineer at <a href=https://about.gitlab.com/ target=_blank rel='noopener noreferrer'>GitLab</a> was troubleshooting a spam problem with their PostgreSQL databases. Thinking that he was logged into the secondary database, the engineer deleted all of the <i>secondary</i> database’s data, only to discover (a little too late) that he had deleted all of the production data on the <i>primary</i> database instead. What followed was a strange and impressive series of events that culminated in a change to Gitlab’s on-call runbooks to ensure that every high-severity outage in the future would be livestreamed to the public on YouTube.</p><p>GitLab stores all of their production data for <a href=https://about.gitlab.com/ target=_blank rel='noopener noreferrer'>GitLab.com</a> on two PostgreSQL servers: a primary database server that is serving all production traffic and a secondary server that is standing by in case the primary server goes down. On the morning of January 31st, an engineer noticed that the load on the database servers had dramatically increased due to a problem with the way spam accounts were being detected and removed. Due to the heavy load, the automated replication from the primary server to the secondary server started to lag, eventually falling so far behind that it could no longer automatically replicate. The engineer debugging the spam problem noticed that replication was seriously delayed, and decided to delete the data on the secondary server before forcing a manual replication. Unfortunately, the engineer was logged into the primary (not the secondary) server at the time, and accidentally wiped the primary database—deleting all production data. Once the engineer realized that the primary database had been wiped, he contacted the other production engineers who were on-call, and it was all-hands-on-deck at GitLab until the outage was resolved.</p><p>This outage stands out as one of the most interesting outages that the industry has seen in the past year, primarily because of the way GitLab decided to share the outage with the public. The industry standard of outage communication is to keep the details of outages internal and very quiet until the issue is resolved, communicating with external clients and users only if it is absolutely necessary. GitLab, however, took the opposite approach, giving the public full outage transparency: the document tracking the incident was immediately made public, debugging the outage was livestreamed on <a href=https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg target=_blank rel='noopener noreferrer'>YouTube</a>, status updates were shared via <a href=https://twitter.com/gitlabstatus target=_blank rel='noopener noreferrer'>GitLab’s Twitter account</a>, all <a href=https://gitlab.com/gitlab-com/runbooks target=_blank rel='noopener noreferrer'>on-call runbooks</a> were made publicly available, and the <a href=https://about.gitlab.com/2017/02/10/postmortem-of-database-outage-of-january-31/ target=_blank rel='noopener noreferrer'>postmortem</a> they published contained all of the details of the outage and its follow-up tasks.</p><p><i>Increment</i> spoke with Sid Sijbrandij, GitLab’s CEO, about the outage, its aftermath, and the impressive level of transparency that GitLab offered to its users both during and after the outage. We have edited the interview slightly for brevity and clarity.</p></div><div class='ArticleLayout interview'><p><strong><i>Increment: Let’s start with what things looked like before the outage. What did Gitlab’s on-call and incident response process look like? Was there a dedicated incident response or operations team that was trained and prepared to handle these kinds of outages?</i></strong></p><blockquote><p><b><i>Sid:</i></b> We have a team of people on-call—mostly production engineers—who have escalation policies and runbooks to deal with these incidents. The production engineers are the ones getting paged, and they get paged either because some of the monitoring is going off or because someone uses ChatOps to contact them.</p></blockquote><p><strong><i>The engineers debugging the database issue on the secondary database—were they following a runbook or incident response process that already existed?</i></strong></p><blockquote><p>There wasn’t a runbook, because [the debugging had not yet risen to the level of an outage]. They weren’t working on a production outage at the time.</p></blockquote><p><strong><i>So they were working to debug whatever was happening with the secondary database, then, correct?</i></strong></p><blockquote><p>Yes, exactly.</p></blockquote><p><strong><i>When did it turn into an outage, and how did the response then change?</i></strong></p><blockquote><p>When the alarms went off. The production engineer who was working on the database was not the person on-call, and as soon as he realized that he had made a mistake, he went into the chat rooms and said “look people, I’ve made a big mistake, and we have an incident.” It wasn’t even the monitoring that set it off, although the monitoring went off several minutes after that and the person on-call was paged.</p></blockquote><p><strong><i>We know from the </i></strong><a href=https://about.gitlab.com/2017/02/10/postmortem-of-database-outage-of-january-31/ target=_blank rel='noopener noreferrer'><i><strong>postmortem</strong></i></a><strong><i> what happened after that, and how the outage was debugged, mitigated, and resolved. In the postmortem, there’s also a list of follow-up tasks, all of which are pretty technical in nature: making sure there are appropriate database snapshots, improving runbooks, and the like. One thing we don’t see on this list is a set of organizational changes you might be implementing to prevent this from happening in the future. Are you implementing any organizational changes as a result of this outage?</i></strong></p><blockquote><p>What happened with the transparency around the issue happened organically, and we think that was a very <i>good</i> part of how we followed up, so we have made sure that it’s an integral part of all the runbooks. That includes escalating to marketing when there’s a major incident, it includes sharing a google doc where we are making changes, and it includes doing a live broadcast. Now, all of that happened organically [this time], but it was a success.</p><p>I think that what was less of a success was that some people thought that everyone using GitLab was affected. The issue had an effect on the users of <a href=https://about.gitlab.com/ target=_blank rel='noopener noreferrer'>gitlab.com</a>, but most of GitLab’s customers are using our [self-hosted] Community or Enterprise editions, which luckily were not affected. [To avoid this confusion in the future], we will be pinging the marketing team so that they can better describe what the impact is, so that people actually know what the impact is too. There’s a <a href=https://gitlab.com/gitlab-com/runbooks/merge_requests/194/diffs target=_blank rel='noopener noreferrer'>merge request</a> that explains our new communications strategy, and you’ll see that we’ve split it up into minor and major outages. We have a runbook now for escalation to marketing, and getting the livestream online. People are interested in that, so they can see what we’ve changed. The next time, this doesn’t have to be an organic thing: we’ll be transparent by default.</p></blockquote><p><strong><i>It’s remarkable that the transparency happened organically. What were some of the benefits that you saw come out of this?</i></strong></p><blockquote><p>The benefit was that developers were empathetic to what happened. If you can understand <i>why</i> something happened, then you are able to empathize. They saw [why it happened], they saw how we felt about it, they saw what we were putting in place to prevent it. It was obviously unacceptable what had happened, and it was a good, good reminder for everyone [else in the industry] not to make the same mistakes. I think people appreciated that we contributed our own horrible experience, and that it has served as, well, as a warning not only to ourselves but to others. I think everyone [who was working on the outage] felt good about contributing to the body of knowledge.</p></blockquote><p><strong><i>Did publicly debugging help resolve the outage more quickly than you think it would have been resolved otherwise? And did the people who were following the debugging and watching the livestream help resolve the outage?</i></strong></p><blockquote><p>I don’t think [it helped us resolve the issue more quickly], but I’m not 100% sure. I am for sure thankful to everyone who tried to help us, but I don’t think there was an increase in speed. There <i>was</i> a Postgres expert who joined the calls, and that really helped. I don’t think there was a big speed increase because of it, but we’re not ruling that out for next time. What happened this time may not be the case next time: hopefully next time the mistakes are more complex, and there’s more knowledge needed to get there. On the other hand, if it’s more complex, you might need more knowledge up front about what the situation looks like. But [transparency] won’t slow it down and I think it will, in some cases, speed it up.</p></blockquote><p><strong><i>Do you think that this level of outage transparency is something that could help others in the industry?</i></strong></p><blockquote><p>I think that there is a lot of generic advice [about resolving incidents] out there, and what people really crave are specifics. We want to add to that body of knowledge [by providing transparency]. Specifics resonate with people in a way that generic things don’t do. So yes, I think it will be very helpful, but we’re not telling other people what to do, we’re just trying to do our own thing.</p><p>For us, being transparent has brought a lot of benefits, and not as many downsides as would be expected. We would do it even if it was hard, but it hasn’t been hard. Now, maybe that’s because it’s ingrained in the whole [GitLab] organization, but we find that most of the time, people appreciate a look inside, and that can compensate for the fact that when you’re getting a deeper look into something, you’ll see more ugly things—something that is especially true for developers, and so if developers are your audience, transparency gets easier.</p></blockquote><p><strong><i>Is there anything else you’d like to add?</i></strong></p><blockquote><p>I want to thank everyone for all of the hugops we received. Lots of people sent us kind messages over Twitter, and there were even some deliveries to our office. I think that’s really great. I think people were very accommodating, and we are very determined not to let this happen again. I think people will forgive us our first mistake as long as we don’t repeat our mistakes and we [continue to be transparent].</p></blockquote></div><div class=ArticleLayout><h3>Editor's note</h3><p>During our conversation, Sid told <i>Increment</i> that the engineer who accidentally deleted the production database (and who still works at GitLab) included his full name in the public documentation tracking the issue. Sid recommended that he remove his name, and the engineer eventually agreed with Sid, but not until after he had argued that publishing his name was the most transparent thing to do. We think it speaks volumes about GitLab's culture of transparency that engineers involved in outages feel safe about identifying themselves to the public: it speaks of an impressive, strong, and encouraging engineering culture at a company that recognizes that outages aren't the result of individual mistakes but the consequence of failures of process and procedure.<br></p></div></article></div><div class='u-Container ArticleFooter'><div class=column><div class='u-Grid ContentBody small content'><div class=authors id=authors></div><div class=topics><div class=text><h4>Topics</h4><p><a href=/topics/interviews/ >Interviews & Surveys</a></p></div></div></div></div></div></div><div class=SubscribeBox><a id=newsletter class=anchor href=#newsletter></a><div class='ContentBody inverted store'><div class=u-Container><div class='u-Grid column'><div class=box style=background:#4c70b1><div class=text><h2>Buy the print edition</h2><p class=j-TextBalance>Visit the Increment Store to purchase print issues.</p><p><a class='t-Caps u-Arrow' href=https://store.increment.com/ >Store</a></p></div><a href=https://store.increment.com/ class=magazine><figure class=j-MaskedImage data-fill=/art/19/19-cutout-1000-7ffb5dba.png data-mask=/art/19/fill-cms-1000-83522f38.png></figure></a></div></div></div></div><div class='ContentBody email'><div class=u-Container><div class='u-Grid column'><div class='j-EmailForm box' style='box-shadow:0 -5px 0 #4c70b1'></div></div></div></div></div><div class=ContinueReading><div class=u-Container><div class=column><h2 class='t-Caps xlarge'>Continue Reading</h2><ul class='u-Grid articles'><li class='ArticleBlock footer' style=color:#ef766e><a class=j-Preload href=/on-call/when-the-pager-goes-off/ ><div class='IssueTitle tiny' style=color:#ef766e><div class='t-Caps meta'><span>1</span></div><h3 class='t-IssueTitle title'>On-Call</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Increment Staff</span></h4><h3 class='t-TitleSans title'>What happens when the pager goes off?</h3></div><div class='t-BodySerif small intro'>To discover the state of incident response across the tech industry, we surveyed over thirty industry leaders (including Amazon, Dropbox, Facebook, Google, and Netflix) about their incident response processes.</div></div></a></li><li class='ArticleBlock footer' style=color:#ef766e><a class=j-Preload href=/on-call/who-owns-on-call/ ><div class='IssueTitle tiny' style=color:#ef766e><div class='t-Caps meta'><span>1</span></div><h3 class='t-IssueTitle title'>On-Call</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Increment Staff</span></h4><h3 class='t-TitleSans title'>Who owns on-call?</h3></div><div class='t-BodySerif small intro'>Industry leaders like Google, Amazon, Dropbox, Spotify, and Netflix are putting developers on-call for their software. We spoke to these companies to find out why.</div></div></a></li><li class='ArticleBlock footer' style=color:#ef766e><a class=j-Preload href=/on-call/ask-an-expert/ ><div class='IssueTitle tiny' style=color:#ef766e><div class='t-Caps meta'><span>1</span></div><h3 class='t-IssueTitle title'>On-Call</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Increment Staff</span></h4><h3 class='t-TitleSans title'>Ask an expert: How should startups approach on-call and incident response?</h3></div><div class='t-BodySerif small intro'>We asked several industry experts if they had any advice for small companies who are just starting to set up their on-call and incident response processes, and here’s what they said.</div></div></a></li><li class='ArticleBlock footer' style=color:#ef766e><a class=j-Preload href=/on-call/on-call-at-any-size/ ><div class='IssueTitle tiny' style=color:#ef766e><div class='t-Caps meta'><span>1</span></div><h3 class='t-IssueTitle title'>On-Call</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Increment Staff</span></h4><h3 class='t-TitleSans title'>On-call at any size</h3></div><div class='t-BodySerif small intro'>We take a close look at how to make on-call work at any scale, sharing industry best practices that apply to companies at any size, from tiny startups in garages to companies the size of Amazon, Facebook, and Google.</div></div></a></li><li class='ArticleBlock footer' style=color:#707aed><a class=j-Preload href=/cloud/interview-with-ben-uretsky-julia-austin-digitalocean/ ><div class='IssueTitle tiny' style=color:#707aed><div class='t-Caps meta'><span>2</span></div><h3 class='t-IssueTitle title'>Cloud</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Increment Staff</span></h4><h3 class='t-TitleSans title'>An interview with Ben Uretsky and Julia Austin, CEO and CTO of DigitalOcean</h3></div><div class='t-BodySerif small intro'>How DigitalOcean got started, how they managed to position it in the market, how they overcame the technical and economic challenges they faced along the way, and what they’re planning for the future.</div></div></a></li><li class='ArticleBlock footer' style=color:#53a88e><a class=j-Preload href=/development/what-its-like-to-be-a-developer-at/ ><div class='IssueTitle tiny' style=color:#53a88e><div class='t-Caps meta'><span>3</span></div><h3 class='t-IssueTitle title'>Development</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Increment Staff</span></h4><h3 class='t-TitleSans title'>What it’s like to be a developer at …</h3></div><div class='t-BodySerif small intro'>From popular tools to code review, deployment to daily life, here’s a look at the developer experience at Slack, Lyft, DigitalOcean, and more.</div></div></a></li><li class='ArticleBlock footer' style=color:#e89e00><a class=j-Preload href=/testing/qa-q-and-a/ ><div class='IssueTitle tiny' style=color:#e89e00><div class='t-Caps meta'><span>10</span></div><h3 class='t-IssueTitle title'>Testing</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Increment Staff</span></h4><h3 class='t-TitleSans title'>The QA Q&A</h3></div><div class='t-BodySerif small intro'>NPR test engineer Ijeoma Ezeonyebuchi discusses the limits of automation and the unique challenges of mobile testing.</div></div></a></li><li class='ArticleBlock footer' style=color:#d69336><a class=j-Preload href=/programming-languages/six-questions-on-programming-languages/ ><div class='IssueTitle tiny' style=color:#d69336><div class='t-Caps meta'><span>5</span></div><h3 class='t-IssueTitle title'>Programming Languages</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Increment Staff</span></h4><h3 class='t-TitleSans title'>Six questions on programming languages</h3></div><div class='t-BodySerif small intro'>Engineers at Fastly, Glossier, Optimizely, and more share which languages they use (and love!), how language knowledge factors into hiring, and where they see their current languages heading.</div></div></a></li><li class='ArticleBlock footer' style=color:#4a5ad3><a class=j-Preload href=/open-source/open-source-at-scale/ ><div class='IssueTitle tiny' style=color:#4a5ad3><div class='t-Caps meta'><span>9</span></div><h3 class='t-IssueTitle title'>Open Source</h3></div><div class=text><div class=head><h4 class='t-Byline byline'><span>Increment Staff</span></h4><h3 class='t-TitleSans title'>Open source at scale</h3></div><div class='t-BodySerif small intro'>Technical leaders at Microsoft, Kickstarter, DigitalOcean, and Red Hat answer questions about when and why to opt for OSS, how open source has influenced their organizations, and its future role in corporations.</div></div></a></li></ul><h3 class='t-Caps large'>Explore Topics</h3><ul class='t-BodySerif large topics'><li><a href=/topics/learn/ >Learn Something New</a></li><li><a href=/topics/scaling/ >Scaling & Growth</a></li><li><a href=/topics/ask-an-expert/ >Ask an Expert</a></li><li><a href=/topics/interviews/ >Interviews & Surveys</a></li><li><a href=/topics/guides/ >Guides & Best Practices</a></li><li><a href=/topics/opinion/ >Essays & Opinion</a></li><li><a href=/topics/culture/ >Workplace & Culture</a></li></ul><h3 class='t-Caps large'>All Issues</h3><ul class=issues><li><a href=/planning/ ><div class='IssueTitle large' style=color:#4c70b1><div class='t-Caps meta'><span>Issue 19</span> <span>November 2021</span></div><h1 class='t-IssueTitle title'>Planning</h1></div></a></li><li><a href=/mobile/ ><div class='IssueTitle large' style=color:#439eab><div class='t-Caps meta'><span>Issue 18</span> <span>August 2021</span></div><h1 class='t-IssueTitle title'>Mobile</h1></div></a></li><li><a href=/containers/ ><div class='IssueTitle large' style=color:#443d79><div class='t-Caps meta'><span>Issue 17</span> <span>May 2021</span></div><h1 class='t-IssueTitle title'>Containers</h1></div></a></li><li><a href=/reliability/ ><div class='IssueTitle large' style=color:#863051><div class='t-Caps meta'><span>Issue 16</span> <span>February 2021</span></div><h1 class='t-IssueTitle title'>Reliability</h1></div></a></li><li><a href=/remote/ ><div class='IssueTitle large' style=color:#29386a><div class='t-Caps meta'><span>Issue 15</span> <span>November 2020</span></div><h1 class='t-IssueTitle title'>Remote</h1></div></a></li><li><a href=/apis/ ><div class='IssueTitle large' style=color:#00afbe><div class='t-Caps meta'><span>Issue 14</span> <span>August 2020</span></div><h1 class='t-IssueTitle title'>APIs</h1></div></a></li><li><a href=/frontend/ ><div class='IssueTitle large' style=color:#5ebe92><div class='t-Caps meta'><span>Issue 13</span> <span>May 2020</span></div><h1 class='t-IssueTitle title'>Frontend</h1></div></a></li><li><a href=/software-architecture/ ><div class='IssueTitle large' style=color:#40af9e><div class='t-Caps meta'><span>Issue 12</span> <span>February 2020</span></div><h1 class='t-IssueTitle title'>Software Architecture</h1></div></a></li><li><a href=/teams/ ><div class='IssueTitle large' style=color:#8e65bf><div class='t-Caps meta'><span>Issue 11</span> <span>November 2019</span></div><h1 class='t-IssueTitle title'>Teams</h1></div></a></li><li><a href=/testing/ ><div class='IssueTitle large' style=color:#e89e00><div class='t-Caps meta'><span>Issue 10</span> <span>August 2019</span></div><h1 class='t-IssueTitle title'>Testing</h1></div></a></li><li><a href=/open-source/ ><div class='IssueTitle large' style=color:#4a5ad3><div class='t-Caps meta'><span>Issue 9</span> <span>May 2019</span></div><h1 class='t-IssueTitle title'>Open Source</h1></div></a></li><li><a href=/internationalization/ ><div class='IssueTitle large' style=color:#c096ca><div class='t-Caps meta'><span>Issue 8</span> <span>February 2019</span></div><h1 class='t-IssueTitle title'>Internationalization</h1></div></a></li><li><a href=/security/ ><div class='IssueTitle large' style=color:#4dbac5><div class='t-Caps meta'><span>Issue 7</span> <span>October 2018</span></div><h1 class='t-IssueTitle title'>Security</h1></div></a></li><li><a href=/documentation/ ><div class='IssueTitle large' style=color:#f5684d><div class='t-Caps meta'><span>Issue 6</span> <span>August 2018</span></div><h1 class='t-IssueTitle title'>Documentation</h1></div></a></li><li><a href=/programming-languages/ ><div class='IssueTitle large' style=color:#d69336><div class='t-Caps meta'><span>Issue 5</span> <span>April 2018</span></div><h1 class='t-IssueTitle title'>Programming Languages</h1></div></a></li><li><a href=/energy-environment/ ><div class='IssueTitle large' style=color:#d6658e><div class='t-Caps meta'><span>Issue 4</span> <span>February 2018</span></div><h1 class='t-IssueTitle title'>Energy & Environment</h1></div></a></li><li><a href=/development/ ><div class='IssueTitle large' style=color:#53a88e><div class='t-Caps meta'><span>Issue 3</span> <span>October 2017</span></div><h1 class='t-IssueTitle title'>Development</h1></div></a></li><li><a href=/cloud/ ><div class='IssueTitle large' style=color:#707aed><div class='t-Caps meta'><span>Issue 2</span> <span>July 2017</span></div><h1 class='t-IssueTitle title'>Cloud</h1></div></a></li><li><a href=/on-call/ ><div class='IssueTitle large' style=color:#ef766e><div class='t-Caps meta'><span>Issue 1</span> <span>April 2017</span></div><h1 class='t-IssueTitle title'>On-Call</h1></div></a></li></ul></div></div></div><footer class=PageFooter><div class='u-Container ContentBody small'><svg style=display:none><symbol id=twitterIcon viewBox='0 0 32 32'><path d='M32.1 6c-1.2.5-2.5.9-3.8 1 1.4-.8 2.4-2.1 2.9-3.6-1.3.8-2.7 1.3-4.2 1.6a6.8 6.8 0 0 0-4.8-2c-3.6 0-6.6 3-6.6 6.6 0 .5.1 1 .2 1.5-5.5-.3-10.4-3-13.6-6.9-.6 1-.9 2.1-.9 3.3 0 2.3 1.2 4.3 2.9 5.5-1.1 0-2.1-.3-3-.8v.1c0 3.2 2.3 5.9 5.3 6.5-.6.2-1.1.2-1.7.2-.4 0-.8 0-1.2-.1.8 2.6 3.3 4.5 6.2 4.6-2.3 1.8-5.1 2.8-8.2 2.8-.5 0-1.1 0-1.6-.1C2.9 28 6.3 29 10 29c12.1 0 18.7-10 18.7-18.7v-.9c1.4-.9 2.5-2 3.4-3.4z' fill=currentColor /></symbol><symbol id=facebookIcon viewBox='0 0 32 32'><path d='M30.2 0H1.8C.8 0 0 .8 0 1.8v28.5c0 1 .8 1.8 1.8 1.8h15.3V19.6h-4.2v-4.8h4.2v-3.6c0-4.1 2.5-6.4 6.2-6.4 1.8 0 3.3.2 3.7.2v4.3h-2.6c-2 0-2.4 1-2.4 2.4v3.1h4.8l-.6 4.8H22V32h8.2c1 0 1.8-.8 1.8-1.8V1.8c0-1-.8-1.8-1.8-1.8z' fill=currentColor /></symbol><symbol id=rssIcon viewBox='0 0 32 32'><path d='M10.7 25.6c0 2.4-2 4.4-4.4 4.4S2 28 2 25.6s2-4.4 4.4-4.4 4.3 2 4.3 4.4zM6.1 2c-.6 0-1.3 0-2 .1-1.3.1-2.2 1.2-2.1 2.4.1 1.2 1.2 2.2 2.4 2.1.6 0 1.1-.1 1.6-.1 10.7 0 19.4 8.7 19.4 19.4 0 .5 0 1-.1 1.6-.1 1.2.8 2.3 2.1 2.4h.2c1.2 0 2.1-.9 2.2-2.1.1-.7.1-1.4.1-2C30 12.7 19.3 2 6.1 2zm-.7 9.6c-.4 0-.8 0-1.3.1-1.3.1-2.2 1.2-2.1 2.5.1 1.3 1.2 2.2 2.5 2.1h.9c5.7 0 10.4 4.7 10.4 10.4v.9c-.1 1.3.8 2.4 2.1 2.5h.2c1.2 0 2.2-.9 2.3-2.1 0-.4.1-.8.1-1.2-.1-8.4-6.9-15.2-15.1-15.2z' fill=currentColor /></symbol><symbol id=linkedInIcon viewBox='0 0 32 32'><path d='M29.6,0H2.4C1.1,0,0,1,0,2.3v27.4C0,31,1.1,32,2.4,32h27.3c1.3,0,2.4-1,2.4-2.3V2.3C32,1,30.9,0,29.6,0z M9.5,27.3H4.7V12 h4.8V27.3z M7.1,9.9c-1.5,0-2.8-1.2-2.8-2.8c0-1.5,1.2-2.8,2.8-2.8c1.5,0,2.8,1.2,2.8,2.8C9.9,8.7,8.6,9.9,7.1,9.9z M27.3,27.3 h-4.7v-7.4c0-1.8,0-4-2.5-4c-2.5,0-2.8,1.9-2.8,3.9v7.6h-4.7V12H17v2.1h0.1c0.6-1.2,2.2-2.5,4.5-2.5c4.8,0,5.7,3.2,5.7,7.3V27.3z' fill=currentColor /></symbol></svg><div class='column main'><section class=social><a href=https://twitter.com/incrementmag class=twitter><svg viewBox='0 0 32 32'><use xlink:href=#twitterIcon x=0 y=0></use></svg> <span>@incrementmag</span> </a><a href=https://facebook.com/incrementmag class=facebook><svg viewBox='0 0 32 32'><use xlink:href=#facebookIcon x=0 y=0></use></svg> <span>incrementmag</span> </a><a href=/feed.xml class=rss><svg viewBox='0 0 32 32'><use xlink:href=#rssIcon x=0 y=0></use></svg> <span>RSS Feed</span></a></section><section><h4>About</h4><p><em>Increment</em> is a print and digital magazine about how teams build and operate software systems at scale. <a href=/about/ >Learn more</a></p></section><section><h4>Work with us</h4><p>Interested in joining the team at Stripe? <a href=https://stripe.com/jobs>View job openings</a></p></section></div><p class='column copyright'><span>© 2022 <em>Increment</em></span> <a href=https://stripe.com>Published by Stripe</a> <a href=https://stripe.com/privacy/media-policy>Privacy policy</a></p></div></footer></body></html> |