Files
nexus/sreweekly/articles/323/02-failing-forward-how-we-grow-from-incidents.html
2026-09-12 17:23:01 +08:00

35 lines
90 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=yes"/><link rel="preload" as="image" href="/images/logo.png"/><link rel="preload" as="image" href="/images/linked-in.svg"/><link rel="preload" as="image" href="/images/x.svg"/><link rel="preload" as="image" href="/images/yt.svg"/><link rel="preload" as="image" href="/images/insta.svg"/><link rel="preload" as="image" href="/images/f.svg"/><link rel="preload" as="image" href="/images/feed.svg"/><link rel="preload" as="image" href="https://images.ctfassets.net/p762jor363g1/b3b1dafeaa26a45930324439bbcc90a0/b0bcd46dc2bee14f25026c2f8c3f1ec4/EN164_1200_x_630.png"/><link rel="preload" as="image" href="/images/x-blue.svg"/><link rel="preload" as="image" href="/images/f-blue.svg"/><link rel="stylesheet" href="/_next/static/chunks/0sxc27dsa.k6-.css" data-precedence="next"/><link rel="preload" as="script" fetchPriority="low" href="/_next/static/chunks/0zaia6i55l01..js"/><script src="/_next/static/chunks/0pqt~8bl3ukh4.js" async=""></script><script src="/_next/static/chunks/0m_p1bxtorv5i.js" async=""></script><script src="/_next/static/chunks/02g3221oh~3le.js" async=""></script><script src="/_next/static/chunks/turbopack-0~dymj8._h2gc.js" async=""></script><script src="/_next/static/chunks/0a_f9~n73ra-c.js" async=""></script><script src="/_next/static/chunks/0d3shmwh5_nmn.js" async=""></script><script src="/_next/static/chunks/08c8uq~xsq31o.js" async=""></script><script src="/_next/static/chunks/098votwdj80t9.js" async=""></script><script src="/_next/static/chunks/0wni0lk_-haja.js" async=""></script><link rel="preload" as="image" href="/images/linked-in-blue.svg"/><link rel="preload" as="image" href="/images/copy.svg"/><link rel="preload" as="image" href="https://images.ctfassets.net/p762jor363g1/69pB35ICFY1sL97klnboyf/aed20373e05048d11cdd12de739b4524/image1.png"/><link rel="preload" as="image" href="https://images.ctfassets.net/p762jor363g1/7l2eUSuEW1R60JzrWTVCTa/c29b1e8011108d7c79a3b4d4d6f6237f/IMG_5977.png"/><link rel="preload" as="image" href="https://images.ctfassets.net/p762jor363g1/2SQWwDBa8t9F9P8s1IkaeV/645549d40f361b6fe56be1575e78cb5b/cwc01-header-a.png"/><link rel="preload" as="image" href="https://images.ctfassets.net/p762jor363g1/1liN9CCW7JU64GI5r03T6N/95c8d19c1287e6209831f31f8dd89158/3__1_.png"/><title>Failing Forward — How We Grow from Incidents | Spotify Engineering</title><meta name="description" content="This past year, a few of us who focus on the reliability of (S4A) spent some time taking a look at the collection of incidents directly affecting S4A in 2021. We made some hypotheses, and then went to work analyzing each one to build a set of quantitative and qualitative metrics. We learned a few things, and we hope those insights will help you as well."/><meta name="application-name" content="Spotify Engineering"/><meta name="author" content="Clint Byrum, Staff Engineer"/><meta name="robots" content="index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1"/><meta name="twitter:data1" content="Clint Byrum, Staff Engineer"/><meta name="twitter:data2" content="5 minutes"/><meta name="twitter:label1" content="Written by"/><meta name="twitter:label2" content="Est. reading time"/><meta property="og:title" content="Failing Forward — How We Grow from Incidents | Spotify Engineering"/><meta property="og:description" content="This past year, a few of us who focus on the reliability of (S4A) spent some time taking a look at the collection of incidents directly affecting S4A in 2021. We made some hypotheses, and then went to work analyzing each one to build a set of quantitative and qualitative metrics. We learned a few things, and we hope those insights will help you as well."/><meta property="og:site_name" content="Spotify Engineering"/><meta property="og:image" content="https://images.ctfassets.net/p762jor363g1/3f66be7a710f26cc501d8bbe613f6a84/caaf6d58d3ff816878781e3464ceb5f8/EN164_1200_x_630.png___LOGO"/><meta property="og:image:type" content="image/jpeg"/><meta property="og:image:width" content="1200"/><meta property="og:image:height" content="630"/><meta property="og:image:alt" content="featured image"/><meta property="og:type" content="article"/><meta name="twitter:card" content="summary_large_image"/><meta name="twitter:creator" content="Clint Byrum, Staff Engineer"/><meta name="twitter:title" content="Failing Forward — How We Grow from Incidents | Spotify Engineering"/><meta name="twitter:description" content="Failing Forward — How We Grow from Incidents | Spotify Engineering"/><meta name="twitter:image" content="https://images.ctfassets.net/p762jor363g1/3f66be7a710f26cc501d8bbe613f6a84/caaf6d58d3ff816878781e3464ceb5f8/EN164_1200_x_630.png___LOGO"/><meta name="twitter:image:alt" content="featured image"/><meta name="twitter:image:type" content="image/jpeg"/><meta name="twitter:image:width" content="1200"/><meta name="twitter:image:height" content="630"/><link rel="icon" href="/favicon.ico?favicon.0cd2g1ew_zd4v.ico" sizes="250x250" type="image/x-icon"/><script src="/_next/static/chunks/03~yq9q893hmn.js" noModule=""></script></head><body><div hidden=""><!--$--><!--/$--></div><div class="page-wrapper show-notification"><div class="content-wrapper "><!--$?--><template id="B:0"></template><!--/$--><div class="single-post"><div class="container"><h1 class="single-post__title">Failing Forward — How We Grow from Incidents</h1></div><div class="container"><div class="single-post__content"><div class="single-post__featured-image"><img alt="Feature Image" width="740" height="404" decoding="async" data-nimg="1" style="color:transparent" src="https://images.ctfassets.net/p762jor363g1/b3b1dafeaa26a45930324439bbcc90a0/b0bcd46dc2bee14f25026c2f8c3f1ec4/EN164_1200_x_630.png"/></div><div class="html-content"><p >This past year, a few of us who focus on the reliability of <a href="https://artists.spotify.com/" target="_blank" rel="noopener noreferrer">Spotify for Artists</a> (S4A) spent some time taking a look at the collection of incidents directly affecting S4A in 2021. We made some hypotheses, and then went to work analyzing each one to build a set of quantitative and qualitative metrics. We learned a few things, and we hope those insights will help you as well.</p><h2><b>We trade productivity now for productivity later</b></h2><p >Every time something doesn’t go according to plan, there are costs, and benefits. Our minds typically jump to the kinds of negative costs we incur when users are impacted, but there is another cost — productivity.</p><p >We found that most incidents have a moderate to high productivity impact. This means that when an incident happens, 55% of the time at least one responder spends the better part of a day addressing the problem. Quite often, 23% of the time in fact, the blast radius for productivity impact is even bigger.</p>
<figure class="figure-image">
<img
src="//images.ctfassets.net/p762jor363g1/attachment_2742a30e0a5c166162f9e7f62725dc8d/09fa6ee14b9590486076ffc8ae67ebc4/attachment_2742a30e0a5c166162f9e7f62725dc8d.png"
style="width: 700px; max-width: 100%; height: auto; "
class="blog-image"
alt="attachment_2742a30e0a5c166162f9e7f62725dc8d"
/>
</figure>
<p >Now, you might say, “Oh wow, you waste a lot of time fixing stuff!” But, as mentioned earlier, this cost has a benefit, when properly managed. When we take the time to run an incident review (we used to call this a postmortem), we have an opportunity to investigate and internalize how and where the system broke down. While we can read code and docs and hypothesize where the issue stemmed from, analyzing a failure is an opportunity to correct our understanding and bring it closer to reality. Leveling up our engineers in this way leads to better outcomes. This is how we buy productivity later, by sacrificing some of it today.</p><h2><b>Most incidents are (technically) preventable</b></h2><p >For every incident, we looked for signs that the event could have been prevented. We built a rubric based on past evaluations of incidents, with a scale going from 1, which is “almost impossible to prevent”, up to 5, which is “we saw this one coming and let it happen”. Thankfully, we didn’t have any 5s!</p><p class="image-paragraph">
<img
src="//images.ctfassets.net/p762jor363g1/attachment_8aa473d9ce8437b87aef09275b7c1be2/ecdd23bf269846478f44e2665bf0d6c3/attachment_8aa473d9ce8437b87aef09275b7c1be2.png"
alt="attachment_8aa473d9ce8437b87aef09275b7c1be2"
class="inline-image"
style="width: 700px; height: auto;"
/>
<img
src="//images.ctfassets.net/p762jor363g1/attachment_217b36dcb3390a5f3ac6ac1b538c9859/344a2b6e49572e431bceb571ad0722b6/attachment_217b36dcb3390a5f3ac6ac1b538c9859.png"
alt="attachment_217b36dcb3390a5f3ac6ac1b538c9859"
class="inline-image"
style="width: 700px; height: auto;"
/>
</p><p >As seen above in our scoring rubric, 2 and 1 are generally unclear or outside our control and thus wouldn’t really count as highly preventable. However, starting at 3, these are localized failures that can be prevented and/or mitigated by taking known and well-understood preventative actions.</p><p >This isn’t to say that prevention is free or easy. But, by defining a service level objective (SLO), we are setting an error budget — if we can stay within our error budget without a preventative measure, we should. That just means we can spend our time working on improving other aspects of the user experience and, ultimately, focusing on our core mission.</p><h2><b>Humans don’t like paperwork</b></h2><p >For the last few years, the DevOps Research and Assessment, or <a href="https://cloud.google.com/devops/state-of-devops" target="_blank" rel="noopener noreferrer">DORA</a>, has produced an annual “State of DevOps” report. They survey thousands of industry professionals and determine what makes engineering efforts tick.</p><p >Two of the “golden metrics” they have identified are “Mean Time To Recover” and “Change Fail Rate”. These two metrics can mean different things to different orgs. It can be tempting to see them as the end of a conversation about reliability, but they are more like a map, helping us understand where we are, and where we want to go.</p><p >For our 2021 incident study, we carefully looked at every S4A-related incident timeline as recorded, and then the supporting evidence in order to establish a clear picture of each incident’s Time to Recover (TTR).</p><p >We found that about 50% of the time, no attempt was made to record a start and end time. And when people did make an attempt at recording their time to recover, about 81% of the time we had to adjust the time entered by more than 5 minutes to reflect the true number.</p><p >This isn’t a failing of operators, but a failing of the system. We asked folks to fill out some paperwork, but we didn’t pair it with an incentive model, and predictably, they opted to spend their time on things with greater incentives.</p><h2><b>Having synthetic tests reduces time to recover</b></h2><p >After this work involving timelines, we got some signals on time to recover. One such signal was that TTR was all over the place and genuinely hard to correlate with any single aspect of our systems.</p><p >However, we got a hit. One of the more exciting things we learned through our incident study was that synthetic testing works. We spent a fair amount of time grading whether or not a synthetic test would have plausibly detected outages, and then looked at the TTR for those that were in fact detected by synthetic tests, versus those that were not because they were not covered by a synthetic test.</p><p >The results were even more striking than we thought. We found that incidents involving coverable features that did have a synthetic test saw a recovery time that was generally <b>10 times faster.</b> No really, read it again!</p><p >This may seem obvious, but we never want to discount the power of data to drive decisions. This isn’t just a curiosity. We’ve adjusted our priorities to put a greater emphasis on synthetic testing, as we think it’s pretty important to get things back up and running as quickly as possible.</p><h2><b>So remember…</b></h2><p >If you take something away from this post, I hope it’s an appreciation for the power of celebrating incidents as learning opportunities. Whether studying incidents as a collection, the way we did, or taking time to learn from them in your own context, there’s simply no replacement for this real-world source of data. It may seem tedious to interrupt your flow to investigate the failure. However, we think it’s vital to do so if you want to grow and sustain your service’s uptime.</p><p >If this tale has captured your imagination, and you are interested in working on a data-driven, reliability-focused team, <a href="https://jobs.lever.co/spotify/bfce3ecc-18f0-4d6f-96ea-4df3476da21c" target="_blank" rel="noopener noreferrer">come join us</a>!</p></div><div>Tags:<!-- --> <a href="/tag/backend">backend</a>, <a href="/tag/engineering-leadership">engineering leadership</a></div><div class="social"><p>SHARE THIS ARTICLE</p><ul class="social__links"><li><a href="https://twitter.com/intent/tweet?url=https%3A%2F%2Fengineering.atspotify.com%2F2022%2F05%2Ffailing-forward-how-we-grow-from-incidents&amp;text=Failing%20Forward%20%E2%80%94%20How%20We%20Grow%20from%20Incidents" target="_blank"><img alt="x" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/x-blue.svg"/></a></li><li><a href="https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fengineering.atspotify.com%2F2022%2F05%2Ffailing-forward-how-we-grow-from-incidents" target="_blank" rel="noopener noreferrer"><img alt="fb" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/f-blue.svg"/></a></li><li><a href="https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fengineering.atspotify.com%2F2022%2F05%2Ffailing-forward-how-we-grow-from-incidents" target="_blank" rel="noopener noreferrer"><img alt="linkedin" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/linked-in-blue.svg"/></a></li><li><a><img alt="copy" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/copy.svg"/></a></li></ul></div></div><aside class="sidebar"><div class="date">May 12, 2022</div><div class="published">Published by Clint Byrum, Staff Engineer</div><div class="tag-wrapper"><a class="tag fill" href="/category/developer-tools">Developer Tools</a><a class="tag fill" href="/category/infrastructure">Infrastructure</a><a class="tag fill" href="/category/people">People</a><a class="tag fill" href="/category/platform">Platform</a></div><div class="social"><p>SHARE THIS ARTICLE</p><ul class="social__links"><li><a href="https://twitter.com/intent/tweet?url=https%3A%2F%2Fengineering.atspotify.com%2F2022%2F05%2Ffailing-forward-how-we-grow-from-incidents&amp;text=Failing%20Forward%20%E2%80%94%20How%20We%20Grow%20from%20Incidents" target="_blank"><img alt="x" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/x-blue.svg"/></a></li><li><a href="https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fengineering.atspotify.com%2F2022%2F05%2Ffailing-forward-how-we-grow-from-incidents" target="_blank" rel="noopener noreferrer"><img alt="fb" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/f-blue.svg"/></a></li><li><a href="https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fengineering.atspotify.com%2F2022%2F05%2Ffailing-forward-how-we-grow-from-incidents" target="_blank" rel="noopener noreferrer"><img alt="linkedin" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/linked-in-blue.svg"/></a></li><li><a><img alt="copy" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/copy.svg"/></a></li></ul></div></aside></div></div><div class="related-posts"><div class="container"><h3>Related articles</h3></div><div class="container"><div class="cards"><div class="post-card related"><div class="post-card__image"><a href="/2026/7/indexing-the-data-lake-for-online-point-queries"><img alt="sticky" width="400" height="197" decoding="async" data-nimg="1" style="color:transparent" src="https://images.ctfassets.net/p762jor363g1/69pB35ICFY1sL97klnboyf/aed20373e05048d11cdd12de739b4524/image1.png"/></a></div><div class="post-card__content"><div class="top-content"><div class="post-card__date">Jul 27, 2026</div><a href="/2026/7/indexing-the-data-lake-for-online-point-queries"><h4 class="post-card__title">Indexing the Data Lake for Online Point Queries</h4></a><div class="post-card__description">Companies like Spotify need vast quantities of data accessible at low latency for online services and,...</div><div class="post-card__published"></div></div><div class="post-card__tags"></div></div></div><div class="post-card related"><div class="post-card__image"><a href="/2026/6/encoding-your-domain-expert-the-context-layer-behind-spotifys-data-assistant"><img alt="sticky" width="400" height="197" decoding="async" data-nimg="1" style="color:transparent" src="https://images.ctfassets.net/p762jor363g1/7l2eUSuEW1R60JzrWTVCTa/c29b1e8011108d7c79a3b4d4d6f6237f/IMG_5977.png"/></a></div><div class="post-card__content"><div class="top-content"><div class="post-card__date">Jun 10, 2026</div><a href="/2026/6/encoding-your-domain-expert-the-context-layer-behind-spotifys-data-assistant"><h4 class="post-card__title">Encoding Your Domain Expert: The Context Layer Behind Spotify&#x27;s Data Assistant</h4></a><div class="post-card__description">At Spotify, data problems used to follow a specific pattern. You&#x27;d look for the relevant dashboard, there...</div><div class="post-card__published"></div></div><div class="post-card__tags"></div></div></div><div class="post-card related"><div class="post-card__image"><a href="/2026/6/code-with-claude-coding-is-no-longer-the-constraint"><img alt="sticky" width="400" height="197" decoding="async" data-nimg="1" style="color:transparent" src="https://images.ctfassets.net/p762jor363g1/2SQWwDBa8t9F9P8s1IkaeV/645549d40f361b6fe56be1575e78cb5b/cwc01-header-a.png"/></a></div><div class="post-card__content"><div class="top-content"><div class="post-card__date">Jun 3, 2026</div><a href="/2026/6/code-with-claude-coding-is-no-longer-the-constraint"><h4 class="post-card__title">Coding Is No Longer the Constraint: Scaling Developer Experience to Teams and Agents at Spotify</h4></a><div class="post-card__description">At Code with Claude, Spotify’s chief architect shared how we make both teams and AI agents more effective.</div><div class="post-card__published"></div></div><div class="post-card__tags"></div></div></div><div class="post-card related"><div class="post-card__image"><a href="/2026/5/better-experiments-with-llm-evals-a-funnel-not-a-fork"><img alt="sticky" width="400" height="197" decoding="async" data-nimg="1" style="color:transparent" src="https://images.ctfassets.net/p762jor363g1/1liN9CCW7JU64GI5r03T6N/95c8d19c1287e6209831f31f8dd89158/3__1_.png"/></a></div><div class="post-card__content"><div class="top-content"><div class="post-card__date">May 18, 2026</div><a href="/2026/5/better-experiments-with-llm-evals-a-funnel-not-a-fork"><h4 class="post-card__title">Better Experiments with LLM Evals — A funnel, not a fork</h4></a><div class="post-card__description">TL;DR  LLM evals, automated judges that assess relevance, coherence, and quality at scale, are a powerful new...</div><div class="post-card__published"></div></div><div class="post-card__tags"></div></div></div></div></div></div><!--$--><!--/$--></div><footer class="footer"><div class="container"><section class="subscription"><div class="subscription__left"><h3>Want to stay in the loop?</h3><p>Subscribe to our newsletter to never miss an update!</p><small>By clicking sign up you’ll receive occasional emails from Spotify. You always have the choice to adjust your interest settings or unsubscribe.</small></div><div class="subscription__right"><div id="mc_embed_signup"><form id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" action="https://spotify.us20.list-manage.com/subscribe/post?u=99aa88d3fba458716d0cd1299&amp;id=28cc680980" method="post" target="_blank"><div id="mc_embed_signup_scroll"><div class="mc-field-group"><input type="email" class="required email" id="mce-EMAIL" placeholder="Enter your best email" name="EMAIL"/><input type="submit" id="mc-embedded-subscribe" class="button button--submit" name="subscribe" value="Subscribe"/></div><div id="mce-responses" class="clear"><div class="response" id="mce-error-response" style="display:none"></div><div class="response" id="mce-success-response" style="display:none"></div></div><div style="left:-5000px;position:absolute" aria-hidden="true"><input type="text" name="b_99aa88d3fba458716d0cd1299_28cc680980"/></div></div></form></div></div></section><section class="links"><div class="links__left"><div class="logo"><a href="/"><img alt="feed" loading="lazy" width="167" height="50" decoding="async" data-nimg="1" style="color:transparent" src="/images/footer-logo.svg"/></a></div><div class="nav-wrapper"><nav class="nav"><ul><li><a target="_blank" rel="noopener noreferrer" href="https://spotify.com/">Spotify.com</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.lifeatspotify.com/">Spotify Jobs</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://newsroom.spotify.com/">Newsroom</a></li></ul><ul><li><a target="_blank" rel="noopener noreferrer" href="https://research.atspotify.com/">Spotify R&amp;D Research</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://backstage.spotify.com/">Spotify for Backstage</a></li></ul></nav></div></div><div class="links__right"><ul class="social"><li><a target="_blank" rel="noopener noreferrer" href="https://www.linkedin.com/showcase/spotify-r&amp;d"><img alt="Linked In" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/linked-in.svg"/></a></li><li><a target="_blank" rel="noopener noreferrer" href="https://twitter.com/spotifyeng"><img alt="X" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/x.svg"/></a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.youtube.com/@SpotifyRnD"><img alt="Youtube" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/yt.svg"/></a></li><li><a target="_blank" rel="noopener noreferrer" href="http://www.instagram.com/lifeatspotify"><img alt="Instagram" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/insta.svg"/></a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.facebook.com/spotify"><img alt="facebook" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/f.svg"/></a></li><li><a target="_blank" rel="noopener noreferrer" href="/feed"><img alt="feed" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/feed.svg"/></a></li></ul></div></section><section class="copy"><nav class="copy__nav"><ul><li><a target="_blank" rel="noopener noreferrer" href="https://www.spotify.com/us/legal/end-user-agreement/">Legal</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.spotify.com/us/legal/privacy-policy/">Privacy</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.spotify.com/legal/cookies-policy/">Cookies</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.spotify.com/us/legal/privacy-policy/">About Ads</a></li></ul></nav><span>© <!-- -->2026<!-- --> Spotify AB</span></section></div></footer></div><script>requestAnimationFrame(function(){$RT=performance.now()});</script><script src="/_next/static/chunks/0zaia6i55l01..js" id="_R_" async=""></script><div hidden id="S:0"><div class="notification"><div class="content"></div></div><header class="header "><section class="container"><div class="container-left"><a href="/"><img alt="Engineering logo" width="90" height="36" decoding="async" data-nimg="1" style="color:transparent" src="/images/logo.png"/></a><nav class="nav "><ul><li class=""><a href="/">Blog</a></li><li class=""><a href="/podcasts">Podcasts</a></li><li class=""><a href="/opensource">Open Source</a></li><li class=""><a href="/jobs">Jobs</a></li><li class=""><a href="/about">About</a></li></ul></nav></div><div class="container-right"><div class="search "><div class="search__input"><input type="text" id="search-input" placeholder="Search" value=""/></div><div class="search__icon"><img alt="Search" id="search-icon" loading="lazy" width="24" height="24" decoding="async" data-nimg="1" class="mag" style="color:transparent" src="/images/icon-search.svg"/><img alt="Search" loading="lazy" width="24" height="24" decoding="async" data-nimg="1" class="close" style="color:transparent" src="/images/icon-close.svg"/></div><div class="menu-button "><span></span><span></span><span></span></div></div></div></section></header></div><script>$RB=[];$RV=function(a){$RT=performance.now();for(var b=0;b<a.length;b+=2){var c=a[b],e=a[b+1];null!==e.parentNode&&e.parentNode.removeChild(e);var f=c.parentNode;if(f){var g=c.previousSibling,h=0;do{if(c&&8===c.nodeType){var d=c.data;if("/$"===d||"/&"===d)if(0===h)break;else h--;else"$"!==d&&"$?"!==d&&"$~"!==d&&"$!"!==d&&"&"!==d||h++}d=c.nextSibling;f.removeChild(c);c=d}while(c);for(;e.firstChild;)f.insertBefore(e.firstChild,c);g.data="$";g._reactRetry&&requestAnimationFrame(g._reactRetry)}}a.length=0};
$RC=function(a,b){if(b=document.getElementById(b))(a=document.getElementById(a))?(a.previousSibling.data="$~",$RB.push(a,b),2===$RB.length&&("number"!==typeof $RT?requestAnimationFrame($RV.bind(null,$RB)):(a=performance.now(),setTimeout($RV.bind(null,$RB),2300>a&&2E3<a?2300-a:$RT+300-a)))):b.parentNode.removeChild(b)};$RC("B:0","S:0")</script><script>(self.__next_f=self.__next_f||[]).push([0])</script><script>self.__next_f.push([1,"1:\"$Sreact.fragment\"\n3:I[39756,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"default\"]\n4:I[37457,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"default\"]\n6:I[97367,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"OutletBoundary\"]\n7:\"$Sreact.suspense\"\na:I[97367,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"ViewportBoundary\"]\nc:I[97367,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"MetadataBoundary\"]\ne:I[68027,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"default\",1]\n:HL[\"/_next/static/chunks/0sxc27dsa.k6-.css\",\"style\"]\n"])</script><script>self.__next_f.push([1,"0:{\"P\":null,\"c\":[\"\",\"2022\",\"05\",\"failing-forward-how-we-grow-from-incidents\"],\"q\":\"\",\"i\":false,\"f\":[[[\"\",{\"children\":[[\"year\",\"2022\",\"d\",null],{\"children\":[[\"month\",\"05\",\"d\",null],{\"children\":[[\"slug\",\"failing-forward-how-we-grow-from-incidents\",\"d\",null],{\"children\":[\"__PAGE__\",{}]}]}]}]},\"$undefined\",\"$undefined\",16],[[\"$\",\"$1\",\"c\",{\"children\":[[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/chunks/0sxc27dsa.k6-.css\",\"precedence\":\"next\",\"crossOrigin\":\"$undefined\",\"nonce\":\"$undefined\"}],[\"$\",\"script\",\"script-0\",{\"src\":\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"async\":true,\"nonce\":\"$undefined\"}],[\"$\",\"script\",\"script-1\",{\"src\":\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"async\":true,\"nonce\":\"$undefined\"}],[\"$\",\"script\",\"script-2\",{\"src\":\"/_next/static/chunks/08c8uq~xsq31o.js\",\"async\":true,\"nonce\":\"$undefined\"}]],\"$L2\"]}],{\"children\":[[\"$\",\"$1\",\"c\",{\"children\":[null,[\"$\",\"$L3\",null,{\"parallelRouterKey\":\"children\",\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$L4\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":\"$undefined\",\"forbidden\":\"$undefined\",\"unauthorized\":\"$undefined\"}]]}],{\"children\":[[\"$\",\"$1\",\"c\",{\"children\":[null,[\"$\",\"$L3\",null,{\"parallelRouterKey\":\"children\",\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$L4\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":\"$undefined\",\"forbidden\":\"$undefined\",\"unauthorized\":\"$undefined\"}]]}],{\"children\":[[\"$\",\"$1\",\"c\",{\"children\":[null,[\"$\",\"$L3\",null,{\"parallelRouterKey\":\"children\",\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$L4\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":\"$undefined\",\"forbidden\":\"$undefined\",\"unauthorized\":\"$undefined\"}]]}],{\"children\":[[\"$\",\"$1\",\"c\",{\"children\":[\"$L5\",[[\"$\",\"script\",\"script-0\",{\"src\":\"/_next/static/chunks/0wni0lk_-haja.js\",\"async\":true,\"nonce\":\"$undefined\"}]],[\"$\",\"$L6\",null,{\"children\":[\"$\",\"$7\",null,{\"name\":\"Next.MetadataOutlet\",\"children\":\"$@8\"}]}]]}],{},null,false,null]},null,false,\"$@9\"]},null,false,\"$@9\"]},null,false,\"$@9\"]},null,false,null],[\"$\",\"$1\",\"h\",{\"children\":[null,[\"$\",\"$La\",null,{\"children\":\"$Lb\"}],[\"$\",\"div\",null,{\"hidden\":true,\"children\":[\"$\",\"$Lc\",null,{\"children\":[\"$\",\"$7\",null,{\"name\":\"Next.Metadata\",\"children\":\"$Ld\"}]}]}],null]}],false]],\"m\":\"$undefined\",\"G\":[\"$e\",[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/chunks/0sxc27dsa.k6-.css\",\"precedence\":\"next\",\"crossOrigin\":\"$undefined\",\"nonce\":\"$undefined\"}]]],\"S\":false,\"h\":null,\"s\":\"$undefined\",\"l\":\"$undefined\",\"p\":\"$undefined\",\"d\":\"$undefined\",\"b\":\"z1ONPGug6r-HUxPkOhj0S\"}\n"])</script><script>self.__next_f.push([1,"f:[]\n9:\"$Wf\"\nb:[[\"$\",\"meta\",\"0\",{\"charSet\":\"utf-8\"}],[\"$\",\"meta\",\"1\",{\"name\":\"viewport\",\"content\":\"width=device-width, initial-scale=1, maximum-scale=1, user-scalable=yes\"}]]\n"])</script><script>self.__next_f.push([1,"10:I[79520,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"\"]\n11:I[1013,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"HeaderContextProvider\"]\n12:I[32666,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"PostProvider\"]\n13:I[28044,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"UiTextProvider\"]\n14:I[70292,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"Initializer\"]\n15:I[97341,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"default\"]\n16:I[36676,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"default\"]\n17:I[58298,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\",\"/_next/static/chunks/098votwdj80t9.js\"],\"default\"]\n18:I[22016,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\",\"/_next/static/chunks/0wni0lk_-haja.js\"],\"\"]\n"])</script><script>self.__next_f.push([1,"2:[\"$\",\"html\",null,{\"lang\":\"en\",\"children\":[[\"$\",\"head\",null,{\"children\":[\"$\",\"$L10\",null,{\"id\":\"gtm\",\"dangerouslySetInnerHTML\":{\"__html\":\"(function(w, d, s, l, i) {\\n w[l] = w[l] || [];\\n w[l].push({\\n 'gtm.start': new Date().getTime(),\\n event: 'gtm.js'\\n });\\n var f = d.getElementsByTagName(s)[0],\\n j = d.createElement(s),\\n dl = l != 'dataLayer' ? '\u0026l=' + l : '';\\n j.async = true;\\n j.src =\\n 'https://www.googletagmanager.com/gtm.js?id=' + i + dl;\\n f.parentNode.insertBefore(j, f);\\n })(window, document, 'script', 'dataLayer', 'GTM-W5NPJP7');\"}}]}],[\"$\",\"body\",null,{\"children\":[\"$\",\"div\",null,{\"className\":\"page-wrapper show-notification\",\"children\":[[\"$\",\"$L11\",null,{\"children\":[\"$\",\"$L12\",null,{\"children\":[\"$\",\"$L13\",null,{\"defaultUiText\":{\"homepageBannerText\":{\"data\":{},\"content\":[{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"Read about the magic behind the music \u0026 more. Welcome to our official technology blog.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"}],\"nodeType\":\"document\"},\"podcastTagline\":{\"data\":{},\"content\":[{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"Two original shows from Spotify R\u0026D.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"Listen and fill your brain with great tech stories.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"}],\"nodeType\":\"document\"},\"opensourceTagline\":{\"data\":{},\"content\":[{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"There’s never been a better time to take part in open source at Spotify. We don’t just want to contribute back to the community — we want to lead the way with projects we truly believe in. Jump into one of the projects below and join us.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"}],\"nodeType\":\"document\"},\"jobsTagline\":{\"nodeType\":\"document\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Complex problems, exceptional colleagues, and a direct impact on the joy of \",\"marks\":[],\"data\":{}},{\"nodeType\":\"text\",\"value\":\"713\",\"marks\":[],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" million people. Our beliefs drive our success.\",\"marks\":[],\"data\":{}}]}]},\"aboutTagline\":{\"data\":{},\"content\":[{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"Spotify Engineering is a diverse community of engineers, developers, researchers and more. Together, we build the infrastructure, features, and experiences you love – and help to shape the future of audio.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"}],\"nodeType\":\"document\"}},\"children\":[\"$\",\"$L14\",null,{\"children\":[[\"$\",\"$7\",null,{\"children\":[[\"$\",\"$L15\",null,{}],[\"$\",\"$L16\",null,{}]]}],[\"$\",\"$L3\",null,{\"parallelRouterKey\":\"children\",\"error\":\"$17\",\"errorStyles\":[],\"errorScripts\":[[\"$\",\"script\",\"script-0\",{\"src\":\"/_next/static/chunks/098votwdj80t9.js\",\"async\":true}]],\"template\":[\"$\",\"$L4\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":[[\"$\",\"div\",null,{\"className\":\"not-found\",\"children\":[\"$\",\"div\",null,{\"className\":\"container\",\"children\":[[\"$\",\"h1\",null,{\"children\":\"404\"}],[\"$\",\"div\",null,{\"className\":\"content\",\"children\":[[\"$\",\"h3\",null,{\"children\":\"OOOOPS... Something went wrong\"}],[\"$\",\"p\",null,{\"children\":\"We couldn't find the page you're looking for.\"}]]}],[\"$\",\"$L18\",null,{\"href\":\"/\",\"className\":\"button\",\"children\":\"Go back home\"}]]}]}],[]],\"forbidden\":\"$undefined\",\"unauthorized\":\"$undefined\"}]]}]}]}]}],[\"$\",\"footer\",null,{\"className\":\"footer\",\"children\":[\"$\",\"div\",null,{\"className\":\"container\",\"children\":[[\"$\",\"section\",null,{\"className\":\"subscription\",\"children\":[[\"$\",\"div\",null,{\"className\":\"subscription__left\",\"children\":[[\"$\",\"h3\",null,{\"children\":\"Want to stay in the loop?\"}],[\"$\",\"p\",null,{\"children\":\"Subscribe to our newsletter to never miss an update!\"}],[\"$\",\"small\",null,{\"children\":\"By clicking sign up you’ll receive occasional emails from Spotify. You always have the choice to adjust your interest settings or unsubscribe.\"}]]}],[\"$\",\"div\",null,{\"className\":\"subscription__right\",\"children\":[\"$\",\"div\",null,{\"id\":\"mc_embed_signup\",\"children\":[\"$\",\"form\",null,{\"action\":\"https://spotify.us20.list-manage.com/subscribe/post?u=99aa88d3fba458716d0cd1299\u0026id=28cc680980\",\"method\":\"post\",\"id\":\"mc-embedded-subscribe-form\",\"name\":\"mc-embedded-subscribe-form\",\"className\":\"validate\",\"target\":\"_blank\",\"children\":\"$L19\"}]}]}]]}],\"$L1a\",\"$L1b\"]}]}]]}]}]]}]\n"])</script><script>self.__next_f.push([1,"1c:I[5500,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\",\"/_next/static/chunks/0wni0lk_-haja.js\"],\"Image\"]\n19:[\"$\",\"div\",null,{\"id\":\"mc_embed_signup_scroll\",\"children\":[[\"$\",\"div\",null,{\"className\":\"mc-field-group\",\"children\":[[\"$\",\"input\",null,{\"type\":\"email\",\"name\":\"EMAIL\",\"className\":\"required email\",\"id\":\"mce-EMAIL\",\"placeholder\":\"Enter your best email\"}],[\"$\",\"input\",null,{\"type\":\"submit\",\"name\":\"subscribe\",\"value\":\"Subscribe\",\"id\":\"mc-embedded-subscribe\",\"className\":\"button button--submit\"}]]}],[\"$\",\"div\",null,{\"id\":\"mce-responses\",\"className\":\"clear\",\"children\":[[\"$\",\"div\",null,{\"className\":\"response\",\"id\":\"mce-error-response\",\"style\":{\"display\":\"none\"}}],[\"$\",\"div\",null,{\"className\":\"response\",\"id\":\"mce-success-response\",\"style\":{\"display\":\"none\"}}]]}],[\"$\",\"div\",null,{\"style\":{\"left\":\"-5000px\",\"position\":\"absolute\"},\"aria-hidden\":\"true\",\"children\":[\"$\",\"input\",null,{\"type\":\"text\",\"name\":\"b_99aa88d3fba458716d0cd1299_28cc680980\"}]}]]}]\n"])</script><script>self.__next_f.push([1,"1a:[\"$\",\"section\",null,{\"className\":\"links\",\"children\":[[\"$\",\"div\",null,{\"className\":\"links__left\",\"children\":[[\"$\",\"div\",null,{\"className\":\"logo\",\"children\":[\"$\",\"$L18\",null,{\"href\":\"/\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/footer-logo.svg\",\"alt\":\"feed\",\"width\":167,\"height\":50}]}]}],[\"$\",\"div\",null,{\"className\":\"nav-wrapper\",\"children\":[\"$\",\"nav\",null,{\"className\":\"nav\",\"children\":[[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://spotify.com/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Spotify.com\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.lifeatspotify.com/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Spotify Jobs\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://newsroom.spotify.com/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Newsroom\"}]}]]}],[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://research.atspotify.com/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Spotify R\u0026D Research\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://backstage.spotify.com/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Spotify for Backstage\"}]}]]}]]}]}]]}],[\"$\",\"div\",null,{\"className\":\"links__right\",\"children\":[\"$\",\"ul\",null,{\"className\":\"social\",\"children\":[[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.linkedin.com/showcase/spotify-r\u0026d\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/linked-in.svg\",\"alt\":\"Linked In\",\"width\":24,\"height\":24,\"priority\":true}]}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://twitter.com/spotifyeng\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/x.svg\",\"alt\":\"X\",\"width\":24,\"height\":24,\"priority\":true}]}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.youtube.com/@SpotifyRnD\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/yt.svg\",\"alt\":\"Youtube\",\"width\":24,\"height\":24,\"priority\":true}]}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"http://www.instagram.com/lifeatspotify\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/insta.svg\",\"alt\":\"Instagram\",\"width\":24,\"height\":24,\"priority\":true}]}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.facebook.com/spotify\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/f.svg\",\"alt\":\"facebook\",\"width\":24,\"height\":24,\"priority\":true}]}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"/feed\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/feed.svg\",\"alt\":\"feed\",\"width\":24,\"height\":24,\"priority\":true}]}]}]]}]}]]}]\n"])</script><script>self.__next_f.push([1,"1b:[\"$\",\"section\",null,{\"className\":\"copy\",\"children\":[[\"$\",\"nav\",null,{\"className\":\"copy__nav\",\"children\":[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.spotify.com/us/legal/end-user-agreement/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Legal\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.spotify.com/us/legal/privacy-policy/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Privacy\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.spotify.com/legal/cookies-policy/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Cookies\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.spotify.com/us/legal/privacy-policy/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"About Ads\"}]}]]}]}],[\"$\",\"span\",null,{\"children\":[\"© \",2026,\" Spotify AB\"]}]]}]\n"])</script><script>self.__next_f.push([1,"1d:I[27201,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"IconMark\"]\n8:null\n"])</script><script>self.__next_f.push([1,"d:[[\"$\",\"title\",\"0\",{\"children\":\"Failing Forward — How We Grow from Incidents | Spotify Engineering\"}],[\"$\",\"meta\",\"1\",{\"name\":\"description\",\"content\":\"This past year, a few of us who focus on the reliability of (S4A) spent some time taking a look at the collection of incidents directly affecting S4A in 2021. We made some hypotheses, and then went to work analyzing each one to build a set of quantitative and qualitative metrics. We learned a few things, and we hope those insights will help you as well.\"}],[\"$\",\"meta\",\"2\",{\"name\":\"application-name\",\"content\":\"Spotify Engineering\"}],[\"$\",\"meta\",\"3\",{\"name\":\"author\",\"content\":\"Clint Byrum, Staff Engineer\"}],[\"$\",\"meta\",\"4\",{\"name\":\"robots\",\"content\":\"index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1\"}],[\"$\",\"meta\",\"5\",{\"name\":\"twitter:data1\",\"content\":\"Clint Byrum, Staff Engineer\"}],[\"$\",\"meta\",\"6\",{\"name\":\"twitter:data2\",\"content\":\"5 minutes\"}],[\"$\",\"meta\",\"7\",{\"name\":\"twitter:label1\",\"content\":\"Written by\"}],[\"$\",\"meta\",\"8\",{\"name\":\"twitter:label2\",\"content\":\"Est. reading time\"}],[\"$\",\"meta\",\"9\",{\"property\":\"og:title\",\"content\":\"Failing Forward — How We Grow from Incidents | Spotify Engineering\"}],[\"$\",\"meta\",\"10\",{\"property\":\"og:description\",\"content\":\"This past year, a few of us who focus on the reliability of (S4A) spent some time taking a look at the collection of incidents directly affecting S4A in 2021. We made some hypotheses, and then went to work analyzing each one to build a set of quantitative and qualitative metrics. We learned a few things, and we hope those insights will help you as well.\"}],[\"$\",\"meta\",\"11\",{\"property\":\"og:site_name\",\"content\":\"Spotify Engineering\"}],[\"$\",\"meta\",\"12\",{\"property\":\"og:image\",\"content\":\"https://images.ctfassets.net/p762jor363g1/3f66be7a710f26cc501d8bbe613f6a84/caaf6d58d3ff816878781e3464ceb5f8/EN164_1200_x_630.png___LOGO\"}],[\"$\",\"meta\",\"13\",{\"property\":\"og:image:type\",\"content\":\"image/jpeg\"}],[\"$\",\"meta\",\"14\",{\"property\":\"og:image:width\",\"content\":\"1200\"}],[\"$\",\"meta\",\"15\",{\"property\":\"og:image:height\",\"content\":\"630\"}],[\"$\",\"meta\",\"16\",{\"property\":\"og:image:alt\",\"content\":\"featured image\"}],[\"$\",\"meta\",\"17\",{\"property\":\"og:type\",\"content\":\"article\"}],[\"$\",\"meta\",\"18\",{\"name\":\"twitter:card\",\"content\":\"summary_large_image\"}],[\"$\",\"meta\",\"19\",{\"name\":\"twitter:creator\",\"content\":\"Clint Byrum, Staff Engineer\"}],[\"$\",\"meta\",\"20\",{\"name\":\"twitter:title\",\"content\":\"Failing Forward — How We Grow from Incidents | Spotify Engineering\"}],[\"$\",\"meta\",\"21\",{\"name\":\"twitter:description\",\"content\":\"Failing Forward — How We Grow from Incidents | Spotify Engineering\"}],[\"$\",\"meta\",\"22\",{\"name\":\"twitter:image\",\"content\":\"https://images.ctfassets.net/p762jor363g1/3f66be7a710f26cc501d8bbe613f6a84/caaf6d58d3ff816878781e3464ceb5f8/EN164_1200_x_630.png___LOGO\"}],[\"$\",\"meta\",\"23\",{\"name\":\"twitter:image:alt\",\"content\":\"featured image\"}],[\"$\",\"meta\",\"24\",{\"name\":\"twitter:image:type\",\"content\":\"image/jpeg\"}],[\"$\",\"meta\",\"25\",{\"name\":\"twitter:image:width\",\"content\":\"1200\"}],[\"$\",\"meta\",\"26\",{\"name\":\"twitter:image:height\",\"content\":\"630\"}],[\"$\",\"link\",\"27\",{\"rel\":\"icon\",\"href\":\"/favicon.ico?favicon.0cd2g1ew_zd4v.ico\",\"sizes\":\"250x250\",\"type\":\"image/x-icon\"}],[\"$\",\"$L1d\",\"28\",{}]]\n"])</script><script>self.__next_f.push([1,"1e:T1d3d,"])</script><script>self.__next_f.push([1,"\u003cp \u003eThis past year, a few of us who focus on the reliability of \u003ca href=\"https://artists.spotify.com/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eSpotify for Artists\u003c/a\u003e (S4A) spent some time taking a look at the collection of incidents directly affecting S4A in 2021. We made some hypotheses, and then went to work analyzing each one to build a set of quantitative and qualitative metrics. We learned a few things, and we hope those insights will help you as well.\u003c/p\u003e\u003ch2\u003e\u003cb\u003eWe trade productivity now for productivity later\u003c/b\u003e\u003c/h2\u003e\u003cp \u003eEvery time something doesn’t go according to plan, there are costs, and benefits. Our minds typically jump to the kinds of negative costs we incur when users are impacted, but there is another cost — productivity.\u003c/p\u003e\u003cp \u003eWe found that most incidents have a moderate to high productivity impact. This means that when an incident happens, 55% of the time at least one responder spends the better part of a day addressing the problem. Quite often, 23% of the time in fact, the blast radius for productivity impact is even bigger.\u003c/p\u003e\n \u003cfigure class=\"figure-image\"\u003e\n \n \u003cimg \n src=\"//images.ctfassets.net/p762jor363g1/attachment_2742a30e0a5c166162f9e7f62725dc8d/09fa6ee14b9590486076ffc8ae67ebc4/attachment_2742a30e0a5c166162f9e7f62725dc8d.png\" \n style=\"width: 700px; max-width: 100%; height: auto; \"\n class=\"blog-image\"\n alt=\"attachment_2742a30e0a5c166162f9e7f62725dc8d\"\n /\u003e\n \n \n \u003c/figure\u003e\n \u003cp \u003eNow, you might say, “Oh wow, you waste a lot of time fixing stuff!” But, as mentioned earlier, this cost has a benefit, when properly managed. When we take the time to run an incident review (we used to call this a postmortem), we have an opportunity to investigate and internalize how and where the system broke down. While we can read code and docs and hypothesize where the issue stemmed from, analyzing a failure is an opportunity to correct our understanding and bring it closer to reality. Leveling up our engineers in this way leads to better outcomes. This is how we buy productivity later, by sacrificing some of it today.\u003c/p\u003e\u003ch2\u003e\u003cb\u003eMost incidents are (technically) preventable\u003c/b\u003e\u003c/h2\u003e\u003cp \u003eFor every incident, we looked for signs that the event could have been prevented. We built a rubric based on past evaluations of incidents, with a scale going from 1, which is “almost impossible to prevent”, up to 5, which is “we saw this one coming and let it happen”. Thankfully, we didn’t have any 5s!\u003c/p\u003e\u003cp class=\"image-paragraph\"\u003e\n \n \u003cimg \n src=\"//images.ctfassets.net/p762jor363g1/attachment_8aa473d9ce8437b87aef09275b7c1be2/ecdd23bf269846478f44e2665bf0d6c3/attachment_8aa473d9ce8437b87aef09275b7c1be2.png\" \n alt=\"attachment_8aa473d9ce8437b87aef09275b7c1be2\"\n class=\"inline-image\"\n style=\"width: 700px; height: auto;\"\n /\u003e\n \n \n \n\n \n \u003cimg \n src=\"//images.ctfassets.net/p762jor363g1/attachment_217b36dcb3390a5f3ac6ac1b538c9859/344a2b6e49572e431bceb571ad0722b6/attachment_217b36dcb3390a5f3ac6ac1b538c9859.png\" \n alt=\"attachment_217b36dcb3390a5f3ac6ac1b538c9859\"\n class=\"inline-image\"\n style=\"width: 700px; height: auto;\"\n /\u003e\n \n \n \u003c/p\u003e\u003cp \u003eAs seen above in our scoring rubric, 2 and 1 are generally unclear or outside our control and thus wouldn’t really count as highly preventable. However, starting at 3, these are localized failures that can be prevented and/or mitigated by taking known and well-understood preventative actions.\u003c/p\u003e\u003cp \u003eThis isn’t to say that prevention is free or easy. But, by defining a service level objective (SLO), we are setting an error budget — if we can stay within our error budget without a preventative measure, we should. That just means we can spend our time working on improving other aspects of the user experience and, ultimately, focusing on our core mission.\u003c/p\u003e\u003ch2\u003e\u003cb\u003eHumans don’t like paperwork\u003c/b\u003e\u003c/h2\u003e\u003cp \u003eFor the last few years, the DevOps Research and Assessment, or \u003ca href=\"https://cloud.google.com/devops/state-of-devops\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eDORA\u003c/a\u003e, has produced an annual “State of DevOps” report. They survey thousands of industry professionals and determine what makes engineering efforts tick.\u003c/p\u003e\u003cp \u003eTwo of the “golden metrics” they have identified are “Mean Time To Recover” and “Change Fail Rate”. These two metrics can mean different things to different orgs. It can be tempting to see them as the end of a conversation about reliability, but they are more like a map, helping us understand where we are, and where we want to go.\u003c/p\u003e\u003cp \u003eFor our 2021 incident study, we carefully looked at every S4A-related incident timeline as recorded, and then the supporting evidence in order to establish a clear picture of each incident’s Time to Recover (TTR).\u003c/p\u003e\u003cp \u003eWe found that about 50% of the time, no attempt was made to record a start and end time. And when people did make an attempt at recording their time to recover, about 81% of the time we had to adjust the time entered by more than 5 minutes to reflect the true number.\u003c/p\u003e\u003cp \u003eThis isn’t a failing of operators, but a failing of the system. We asked folks to fill out some paperwork, but we didn’t pair it with an incentive model, and predictably, they opted to spend their time on things with greater incentives.\u003c/p\u003e\u003ch2\u003e\u003cb\u003eHaving synthetic tests reduces time to recover\u003c/b\u003e\u003c/h2\u003e\u003cp \u003eAfter this work involving timelines, we got some signals on time to recover. One such signal was that TTR was all over the place and genuinely hard to correlate with any single aspect of our systems.\u003c/p\u003e\u003cp \u003eHowever, we got a hit. One of the more exciting things we learned through our incident study was that synthetic testing works. We spent a fair amount of time grading whether or not a synthetic test would have plausibly detected outages, and then looked at the TTR for those that were in fact detected by synthetic tests, versus those that were not because they were not covered by a synthetic test.\u003c/p\u003e\u003cp \u003eThe results were even more striking than we thought. We found that incidents involving coverable features that did have a synthetic test saw a recovery time that was generally \u003cb\u003e10 times faster.\u003c/b\u003e No really, read it again!\u003c/p\u003e\u003cp \u003eThis may seem obvious, but we never want to discount the power of data to drive decisions. This isn’t just a curiosity. We’ve adjusted our priorities to put a greater emphasis on synthetic testing, as we think it’s pretty important to get things back up and running as quickly as possible.\u003c/p\u003e\u003ch2\u003e\u003cb\u003eSo remember…\u003c/b\u003e\u003c/h2\u003e\u003cp \u003eIf you take something away from this post, I hope it’s an appreciation for the power of celebrating incidents as learning opportunities. Whether studying incidents as a collection, the way we did, or taking time to learn from them in your own context, there’s simply no replacement for this real-world source of data. It may seem tedious to interrupt your flow to investigate the failure. However, we think it’s vital to do so if you want to grow and sustain your service’s uptime.\u003c/p\u003e\u003cp \u003eIf this tale has captured your imagination, and you are interested in working on a data-driven, reliability-focused team, \u003ca href=\"https://jobs.lever.co/spotify/bfce3ecc-18f0-4d6f-96ea-4df3476da21c\" target=\"_blank\" rel=\"noopener noreferrer\"\u003ecome join us\u003c/a\u003e!\u003c/p\u003e"])</script><script>self.__next_f.push([1,"5:[[\"$\",\"div\",null,{\"className\":\"single-post\",\"children\":[[\"$\",\"div\",null,{\"className\":\"container\",\"children\":[\"$\",\"h1\",null,{\"className\":\"single-post__title\",\"children\":\"Failing Forward — How We Grow from Incidents\"}]}],[\"$\",\"div\",null,{\"className\":\"container\",\"children\":[[\"$\",\"div\",null,{\"className\":\"single-post__content\",\"children\":[[\"$\",\"div\",null,{\"className\":\"single-post__featured-image\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"https://images.ctfassets.net/p762jor363g1/b3b1dafeaa26a45930324439bbcc90a0/b0bcd46dc2bee14f25026c2f8c3f1ec4/EN164_1200_x_630.png\",\"alt\":\"Feature Image\",\"width\":740,\"height\":404,\"priority\":true}]}],[\"$\",\"div\",null,{\"className\":\"html-content\",\"dangerouslySetInnerHTML\":{\"__html\":\"$1e\"}}],\"$L1f\",\"$L20\"]}],\"$L21\"]}]]}],\"$L22\"]\n"])</script><script>self.__next_f.push([1,"23:I[57581,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\",\"/_next/static/chunks/0wni0lk_-haja.js\"],\"PostTags\"]\n24:I[58931,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\",\"/_next/static/chunks/0wni0lk_-haja.js\"],\"SocialShare\"]\n25:I[40367,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\",\"/_next/static/chunks/0wni0lk_-haja.js\"],\"default\"]\n1f:[\"$\",\"$L23\",null,{\"tags\":[{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"blogtag_backend\",\"type\":\"Entry\",\"createdAt\":\"2025-03-13T13:23:41.774Z\",\"updatedAt\":\"2025-03-13T13:23:41.774Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":2,\"revision\":1,\"contentType\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"ContentType\",\"id\":\"blogTag\"}},\"locale\":\"en-US\"},\"fields\":{\"name\":\"backend\",\"slug\":\"backend\"}},{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"blogtag_engineering-leadership\",\"type\":\"Entry\",\"createdAt\":\"2025-03-13T13:24:54.486Z\",\"updatedAt\":\"2025-03-13T13:24:54.486Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":2,\"revision\":1,\"contentType\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"ContentType\",\"id\":\"blogTag\"}},\"locale\":\"en-US\"},\"fields\":{\"name\":\"engineering leadership\",\"slug\":\"engineering-leadership\"}}]}]\n20:[\"$\",\"$L24\",null,{\"heading\":\"SHARE THIS ARTICLE\",\"title\":\"Failing Forward — How We Grow from Incidents\"}]\n26:T463,\u003cimg alt=\"\" class=\"wp-image-5309\" decoding=\"async\" height=\"344\" loading=\"lazy\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" src=\"https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-700x344.png\" srcset=\"https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-700x344.png 700w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-250x123.png 250w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-768x378.png 768w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-1536x755.png 1536w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-2048x1007.png 2048w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-120x59.png 120w\" width=\"700\"/\u003e27:T463,\u003cimg alt=\"\" class=\"wp-image-5311\" decoding=\"async\" height=\"344\" loading=\"lazy\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" src=\"https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-700x344.png\" srcset=\"https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-700x344.png 700w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-250x123.png 250w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-768x378.png 768w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-1536x755.png 1536w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-2048x1007.png 2048w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-120x59.png 120w\" width=\"700\"/\u003e28:T455,\u003cimg alt=\"\" class=\"wp-image-5312\" decoding=\"async\" height=\"344\" loading=\"lazy\" sizes="])</script><script>self.__next_f.push([1,"\"auto, (max-width: 700px) 100vw, 700px\" src=\"https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-700x344.png\" srcset=\"https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-700x344.png 700w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-250x123.png 250w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-768x378.png 768w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-1536x755.png 1536w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-2048x1007.png 2048w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-120x59.png 120w\" width=\"700\"/\u003e"])</script><script>self.__next_f.push([1,"21:[\"$\",\"$L25\",null,{\"blogPost\":{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"blogpost_f11626d6acc98fad985bdfe40fe37ec4\",\"type\":\"Entry\",\"createdAt\":\"2025-03-13T22:28:35.009Z\",\"updatedAt\":\"2025-04-24T15:37:50.854Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":10,\"revision\":4,\"contentType\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"ContentType\",\"id\":\"blogPost\"}},\"locale\":\"en-US\"},\"fields\":{\"title\":\"Failing Forward — How We Grow from Incidents\",\"slug\":\"failing-forward-how-we-grow-from-incidents\",\"content\":{\"data\":{},\"content\":[{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"This past year, a few of us who focus on the reliability of \",\"nodeType\":\"text\"},{\"data\":{\"uri\":\"https://artists.spotify.com/\"},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"Spotify for Artists\",\"nodeType\":\"text\"}],\"nodeType\":\"hyperlink\"},{\"data\":{},\"marks\":[],\"value\":\" (S4A) spent some time taking a look at the collection of incidents directly affecting S4A in 2021. We made some hypotheses, and then went to work analyzing each one to build a set of quantitative and qualitative metrics. We learned a few things, and we hope those insights will help you as well.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[{\"type\":\"bold\"}],\"value\":\"We trade productivity now for productivity later\",\"nodeType\":\"text\"}],\"nodeType\":\"heading-2\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"Every time something doesn’t go according to plan, there are costs, and benefits. Our minds typically jump to the kinds of negative costs we incur when users are impacted, but there is another cost — productivity.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"We found that most incidents have a moderate to high productivity impact. This means that when an incident happens, 55% of the time at least one responder spends the better part of a day addressing the problem. Quite often, 23% of the time in fact, the blast radius for productivity impact is even bigger.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{\"target\":{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"assetwrapper_d4b27e571c98e306df5f615bf914131d\",\"type\":\"Entry\",\"createdAt\":\"2025-03-13T14:07:45.808Z\",\"updatedAt\":\"2025-03-13T14:07:45.808Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":2,\"revision\":1,\"contentType\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"ContentType\",\"id\":\"assetWrapper\"}},\"locale\":\"en-US\"},\"fields\":{\"title\":\"attachment_2742a30e0a5c166162f9e7f62725dc8d\",\"asset\":{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"attachment_2742a30e0a5c166162f9e7f62725dc8d\",\"type\":\"Asset\",\"createdAt\":\"2025-03-13T13:50:32.694Z\",\"updatedAt\":\"2025-03-13T13:50:32.694Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":3,\"revision\":1,\"locale\":\"en-US\"},\"fields\":{\"title\":\"attachment_2742a30e0a5c166162f9e7f62725dc8d\",\"file\":{\"url\":\"//images.ctfassets.net/p762jor363g1/attachment_2742a30e0a5c166162f9e7f62725dc8d/09fa6ee14b9590486076ffc8ae67ebc4/attachment_2742a30e0a5c166162f9e7f62725dc8d.png\",\"details\":{\"size\":219367,\"image\":{\"width\":2048,\"height\":1007}},\"fileName\":\"attachment_2742a30e0a5c166162f9e7f62725dc8d.png\",\"contentType\":\"image/png\"}}},\"caption\":null,\"alt\":null,\"width\":\"700\",\"height\":\"344\",\"type\":\"image\",\"alignment\":\"left\",\"raw_html\":\"$26\",\"html_attributes\":{\"loading\":\"lazy\",\"decoding\":\"async\",\"width\":\"700\",\"height\":\"344\",\"src\":\"https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-700x344.png\",\"alt\":\"\",\"class\":[\"wp-image-5309\"],\"srcset\":\"https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-700x344.png 700w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-250x123.png 250w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-768x378.png 768w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-1536x755.png 1536w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-2048x1007.png 2048w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-1-120x59.png 120w\",\"sizes\":\"auto, (max-width: 700px) 100vw, 700px\"}}}},\"content\":[],\"nodeType\":\"embedded-entry-block\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"Now, you might say, “Oh wow, you waste a lot of time fixing stuff!” But, as mentioned earlier, this cost has a benefit, when properly managed. When we take the time to run an incident review (we used to call this a postmortem), we have an opportunity to investigate and internalize how and where the system broke down. While we can read code and docs and hypothesize where the issue stemmed from, analyzing a failure is an opportunity to correct our understanding and bring it closer to reality. Leveling up our engineers in this way leads to better outcomes. This is how we buy productivity later, by sacrificing some of it today.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[{\"type\":\"bold\"}],\"value\":\"Most incidents are (technically) preventable\",\"nodeType\":\"text\"}],\"nodeType\":\"heading-2\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"For every incident, we looked for signs that the event could have been prevented. We built a rubric based on past evaluations of incidents, with a scale going from 1, which is “almost impossible to prevent”, up to 5, which is “we saw this one coming and let it happen”. Thankfully, we didn’t have any 5s!\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{\"target\":{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"assetwrapper_bf836d4b7bc7c440943b203eab40319c\",\"type\":\"Entry\",\"createdAt\":\"2025-03-13T14:08:08.028Z\",\"updatedAt\":\"2025-03-13T14:08:08.028Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":2,\"revision\":1,\"contentType\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"ContentType\",\"id\":\"assetWrapper\"}},\"locale\":\"en-US\"},\"fields\":{\"title\":\"attachment_8aa473d9ce8437b87aef09275b7c1be2\",\"asset\":{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"attachment_8aa473d9ce8437b87aef09275b7c1be2\",\"type\":\"Asset\",\"createdAt\":\"2025-03-13T13:50:33.506Z\",\"updatedAt\":\"2025-03-13T13:50:33.506Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":3,\"revision\":1,\"locale\":\"en-US\"},\"fields\":{\"title\":\"attachment_8aa473d9ce8437b87aef09275b7c1be2\",\"file\":{\"url\":\"//images.ctfassets.net/p762jor363g1/attachment_8aa473d9ce8437b87aef09275b7c1be2/ecdd23bf269846478f44e2665bf0d6c3/attachment_8aa473d9ce8437b87aef09275b7c1be2.png\",\"details\":{\"size\":193925,\"image\":{\"width\":2048,\"height\":1007}},\"fileName\":\"attachment_8aa473d9ce8437b87aef09275b7c1be2.png\",\"contentType\":\"image/png\"}}},\"caption\":null,\"alt\":null,\"width\":\"700\",\"height\":\"344\",\"type\":\"image\",\"alignment\":\"left\",\"raw_html\":\"$27\",\"html_attributes\":{\"loading\":\"lazy\",\"decoding\":\"async\",\"width\":\"700\",\"height\":\"344\",\"src\":\"https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-700x344.png\",\"alt\":\"\",\"class\":[\"wp-image-5311\"],\"srcset\":\"https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-700x344.png 700w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-250x123.png 250w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-768x378.png 768w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-1536x755.png 1536w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-2048x1007.png 2048w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4_Graph-2-120x59.png 120w\",\"sizes\":\"auto, (max-width: 700px) 100vw, 700px\"}}}},\"content\":[],\"nodeType\":\"embedded-entry-inline\"},{\"data\":{},\"marks\":[],\"value\":\"\\n\",\"nodeType\":\"text\"},{\"data\":{\"target\":{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"assetwrapper_97e58553dcd4ad2766794686d985dd29\",\"type\":\"Entry\",\"createdAt\":\"2025-03-13T14:08:11.013Z\",\"updatedAt\":\"2025-03-13T14:08:11.013Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":2,\"revision\":1,\"contentType\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"ContentType\",\"id\":\"assetWrapper\"}},\"locale\":\"en-US\"},\"fields\":{\"title\":\"attachment_217b36dcb3390a5f3ac6ac1b538c9859\",\"asset\":{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"attachment_217b36dcb3390a5f3ac6ac1b538c9859\",\"type\":\"Asset\",\"createdAt\":\"2025-03-13T13:50:34.359Z\",\"updatedAt\":\"2025-03-13T13:50:34.359Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":3,\"revision\":1,\"locale\":\"en-US\"},\"fields\":{\"title\":\"attachment_217b36dcb3390a5f3ac6ac1b538c9859\",\"file\":{\"url\":\"//images.ctfassets.net/p762jor363g1/attachment_217b36dcb3390a5f3ac6ac1b538c9859/344a2b6e49572e431bceb571ad0722b6/attachment_217b36dcb3390a5f3ac6ac1b538c9859.png\",\"details\":{\"size\":330250,\"image\":{\"width\":2048,\"height\":1007}},\"fileName\":\"attachment_217b36dcb3390a5f3ac6ac1b538c9859.png\",\"contentType\":\"image/png\"}}},\"caption\":null,\"alt\":null,\"width\":\"700\",\"height\":\"344\",\"type\":\"image\",\"alignment\":\"left\",\"raw_html\":\"$28\",\"html_attributes\":{\"loading\":\"lazy\",\"decoding\":\"async\",\"width\":\"700\",\"height\":\"344\",\"src\":\"https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-700x344.png\",\"alt\":\"\",\"class\":[\"wp-image-5312\"],\"srcset\":\"https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-700x344.png 700w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-250x123.png 250w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-768x378.png 768w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-1536x755.png 1536w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-2048x1007.png 2048w, https://storage.googleapis.com/production-eng/1/2022/05/N164-Failing-Forward-How-We-Grow-from-Incidents_final-v4-Table-120x59.png 120w\",\"sizes\":\"auto, (max-width: 700px) 100vw, 700px\"}}}},\"content\":[],\"nodeType\":\"embedded-entry-inline\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"As seen above in our scoring rubric, 2 and 1 are generally unclear or outside our control and thus wouldn’t really count as highly preventable. However, starting at 3, these are localized failures that can be prevented and/or mitigated by taking known and well-understood preventative actions.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"This isn’t to say that prevention is free or easy. But, by defining a service level objective (SLO), we are setting an error budget — if we can stay within our error budget without a preventative measure, we should. That just means we can spend our time working on improving other aspects of the user experience and, ultimately, focusing on our core mission.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[{\"type\":\"bold\"}],\"value\":\"Humans don’t like paperwork\",\"nodeType\":\"text\"}],\"nodeType\":\"heading-2\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"For the last few years, the DevOps Research and Assessment, or \",\"nodeType\":\"text\"},{\"data\":{\"uri\":\"https://cloud.google.com/devops/state-of-devops\"},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"DORA\",\"nodeType\":\"text\"}],\"nodeType\":\"hyperlink\"},{\"data\":{},\"marks\":[],\"value\":\", has produced an annual “State of DevOps” report. They survey thousands of industry professionals and determine what makes engineering efforts tick.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"Two of the “golden metrics” they have identified are “Mean Time To Recover” and “Change Fail Rate”. These two metrics can mean different things to different orgs. It can be tempting to see them as the end of a conversation about reliability, but they are more like a map, helping us understand where we are, and where we want to go.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"For our 2021 incident study, we carefully looked at every S4A-related incident timeline as recorded, and then the supporting evidence in order to establish a clear picture of each incident’s Time to Recover (TTR).\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"We found that about 50% of the time, no attempt was made to record a start and end time. And when people did make an attempt at recording their time to recover, about 81% of the time we had to adjust the time entered by more than 5 minutes to reflect the true number.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"This isn’t a failing of operators, but a failing of the system. We asked folks to fill out some paperwork, but we didn’t pair it with an incentive model, and predictably, they opted to spend their time on things with greater incentives.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[{\"type\":\"bold\"}],\"value\":\"Having synthetic tests reduces time to recover\",\"nodeType\":\"text\"}],\"nodeType\":\"heading-2\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"After this work involving timelines, we got some signals on time to recover. One such signal was that TTR was all over the place and genuinely hard to correlate with any single aspect of our systems.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"However, we got a hit. One of the more exciting things we learned through our incident study was that synthetic testing works. We spent a fair amount of time grading whether or not a synthetic test would have plausibly detected outages, and then looked at the TTR for those that were in fact detected by synthetic tests, versus those that were not because they were not covered by a synthetic test.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"The results were even more striking than we thought. We found that incidents involving coverable features that did have a synthetic test saw a recovery time that was generally \",\"nodeType\":\"text\"},{\"data\":{},\"marks\":[{\"type\":\"bold\"}],\"value\":\"10 times faster.\",\"nodeType\":\"text\"},{\"data\":{},\"marks\":[],\"value\":\" No really, read it again!\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"This may seem obvious, but we never want to discount the power of data to drive decisions. This isn’t just a curiosity. We’ve adjusted our priorities to put a greater emphasis on synthetic testing, as we think it’s pretty important to get things back up and running as quickly as possible.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[{\"type\":\"bold\"}],\"value\":\"So remember…\",\"nodeType\":\"text\"}],\"nodeType\":\"heading-2\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"If you take something away from this post, I hope it’s an appreciation for the power of celebrating incidents as learning opportunities. Whether studying incidents as a collection, the way we did, or taking time to learn from them in your own context, there’s simply no replacement for this real-world source of data. It may seem tedious to interrupt your flow to investigate the failure. However, we think it’s vital to do so if you want to grow and sustain your service’s uptime.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"If this tale has captured your imagination, and you are interested in working on a data-driven, reliability-focused team, \",\"nodeType\":\"text\"},{\"data\":{\"uri\":\"https://jobs.lever.co/spotify/bfce3ecc-18f0-4d6f-96ea-4df3476da21c\"},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"come join us\",\"nodeType\":\"text\"}],\"nodeType\":\"hyperlink\"},{\"data\":{},\"marks\":[],\"value\":\"!\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"}],\"nodeType\":\"document\"},\"categories\":[{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"blogcategory_developer-tools\",\"type\":\"Entry\",\"createdAt\":\"2025-03-13T13:22:29.266Z\",\"updatedAt\":\"2025-05-26T09:53:09.803Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":4,\"revision\":2,\"contentType\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"ContentType\",\"id\":\"blogCategory\"}},\"locale\":\"en-US\"},\"fields\":{\"name\":\"Developer Tools\",\"slug\":\"developer-tools\",\"has_blogs\":true}},{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"blogcategory_infrastructure\",\"type\":\"Entry\",\"createdAt\":\"2025-03-13T13:22:31.760Z\",\"updatedAt\":\"2025-05-26T09:53:14.772Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":4,\"revision\":2,\"contentType\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"ContentType\",\"id\":\"blogCategory\"}},\"locale\":\"en-US\"},\"fields\":{\"name\":\"Infrastructure\",\"slug\":\"infrastructure\",\"has_blogs\":true}},{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"blogcategory_people\",\"type\":\"Entry\",\"createdAt\":\"2025-03-13T13:22:47.242Z\",\"updatedAt\":\"2025-05-26T09:54:46.114Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":4,\"revision\":2,\"contentType\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"ContentType\",\"id\":\"blogCategory\"}},\"locale\":\"en-US\"},\"fields\":{\"name\":\"People\",\"slug\":\"people\",\"has_blogs\":true}},{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"blogcategory_platform\",\"type\":\"Entry\",\"createdAt\":\"2025-03-13T13:22:49.789Z\",\"updatedAt\":\"2025-05-26T09:54:54.477Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":4,\"revision\":2,\"contentType\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"ContentType\",\"id\":\"blogCategory\"}},\"locale\":\"en-US\"},\"fields\":{\"name\":\"Platform\",\"slug\":\"platform\",\"has_blogs\":true}}],\"tags\":\"$1f:props:tags\",\"featured_media\":{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"b3b1dafeaa26a45930324439bbcc90a0\",\"type\":\"Asset\",\"createdAt\":\"2025-04-04T12:03:11.546Z\",\"updatedAt\":\"2025-04-04T12:03:11.546Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":2,\"revision\":1,\"locale\":\"en-US\"},\"fields\":{\"title\":\"EN164 1200 x 630.png\",\"file\":{\"url\":\"//images.ctfassets.net/p762jor363g1/b3b1dafeaa26a45930324439bbcc90a0/b0bcd46dc2bee14f25026c2f8c3f1ec4/EN164_1200_x_630.png\",\"details\":{\"size\":145908,\"image\":{\"width\":1200,\"height\":630}},\"fileName\":\"EN164 1200 x 630.png\",\"contentType\":\"image/png\"}}},\"social_media\":{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"3f66be7a710f26cc501d8bbe613f6a84\",\"type\":\"Asset\",\"createdAt\":\"2025-04-04T12:03:11.544Z\",\"updatedAt\":\"2025-04-04T12:03:11.544Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":2,\"revision\":1,\"locale\":\"en-US\"},\"fields\":{\"title\":\"EN164 1200 x 630.png + LOGO\",\"file\":{\"url\":\"//images.ctfassets.net/p762jor363g1/3f66be7a710f26cc501d8bbe613f6a84/caaf6d58d3ff816878781e3464ceb5f8/EN164_1200_x_630.png___LOGO\",\"details\":{\"size\":149234,\"image\":{\"width\":1200,\"height\":630}},\"fileName\":\"EN164 1200 x 630.png + LOGO\",\"contentType\":\"image/png\"}}},\"is_sticky\":false,\"is_old\":true,\"created_at\":\"2022-05-12T13:21:30+00:00\",\"updated_at\":\"2023-03-29T12:08:52+00:00\",\"author\":\"Clint Byrum, Staff Engineer\",\"user_info\":{\"data\":{},\"content\":[],\"nodeType\":\"document\"}}}}]\n"])</script><script>self.__next_f.push([1,"22:[\"$\",\"div\",null,{\"className\":\"related-posts\",\"children\":[[\"$\",\"div\",null,{\"className\":\"container\",\"children\":[\"$\",\"h3\",null,{\"children\":\"Related articles\"}]}],[\"$\",\"div\",null,{\"className\":\"container\",\"children\":[\"$\",\"div\",null,{\"className\":\"cards\",\"children\":[[\"$\",\"div\",\"2BHiaulFaaun4MVkEBakNn\",{\"className\":\"post-card related\",\"children\":[[\"$\",\"div\",null,{\"className\":\"post-card__image\",\"children\":[\"$\",\"$L18\",null,{\"href\":\"/2026/7/indexing-the-data-lake-for-online-point-queries\",\"onClick\":\"$undefined\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"https://images.ctfassets.net/p762jor363g1/69pB35ICFY1sL97klnboyf/aed20373e05048d11cdd12de739b4524/image1.png\",\"alt\":\"sticky\",\"width\":400,\"height\":197,\"quality\":100,\"priority\":true}]}]}],[\"$\",\"div\",null,{\"className\":\"post-card__content\",\"children\":[[\"$\",\"div\",null,{\"className\":\"top-content\",\"children\":[[\"$\",\"div\",null,{\"className\":\"post-card__date\",\"children\":\"Jul 27, 2026\"}],[\"$\",\"$L18\",null,{\"href\":\"/2026/7/indexing-the-data-lake-for-online-point-queries\",\"onClick\":\"$undefined\",\"children\":[\"$\",\"h4\",null,{\"className\":\"post-card__title\",\"children\":\"Indexing the Data Lake for Online Point Queries\"}]}],[\"$\",\"div\",null,{\"className\":\"post-card__description\",\"children\":\"Companies like Spotify need vast quantities of data accessible at low latency for online services and,...\"}],[\"$\",\"div\",null,{\"className\":\"post-card__published\",\"children\":\"$undefined\"}]]}],[\"$\",\"div\",null,{\"className\":\"post-card__tags\",\"children\":\"$undefined\"}]]}]]}],[\"$\",\"div\",\"6OwIaRIOSFOCZSHMjmLIxv\",{\"className\":\"post-card related\",\"children\":[[\"$\",\"div\",null,{\"className\":\"post-card__image\",\"children\":[\"$\",\"$L18\",null,{\"href\":\"/2026/6/encoding-your-domain-expert-the-context-layer-behind-spotifys-data-assistant\",\"onClick\":\"$undefined\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"https://images.ctfassets.net/p762jor363g1/7l2eUSuEW1R60JzrWTVCTa/c29b1e8011108d7c79a3b4d4d6f6237f/IMG_5977.png\",\"alt\":\"sticky\",\"width\":400,\"height\":197,\"quality\":100,\"priority\":true}]}]}],[\"$\",\"div\",null,{\"className\":\"post-card__content\",\"children\":[[\"$\",\"div\",null,{\"className\":\"top-content\",\"children\":[[\"$\",\"div\",null,{\"className\":\"post-card__date\",\"children\":\"Jun 10, 2026\"}],[\"$\",\"$L18\",null,{\"href\":\"/2026/6/encoding-your-domain-expert-the-context-layer-behind-spotifys-data-assistant\",\"onClick\":\"$undefined\",\"children\":[\"$\",\"h4\",null,{\"className\":\"post-card__title\",\"children\":\"Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant\"}]}],[\"$\",\"div\",null,{\"className\":\"post-card__description\",\"children\":\"At Spotify, data problems used to follow a specific pattern. You'd look for the relevant dashboard, there...\"}],[\"$\",\"div\",null,{\"className\":\"post-card__published\",\"children\":\"$undefined\"}]]}],[\"$\",\"div\",null,{\"className\":\"post-card__tags\",\"children\":\"$undefined\"}]]}]]}],[\"$\",\"div\",\"2d34o9wKbvUemRP62vcSEJ\",{\"className\":\"post-card related\",\"children\":[[\"$\",\"div\",null,{\"className\":\"post-card__image\",\"children\":[\"$\",\"$L18\",null,{\"href\":\"/2026/6/code-with-claude-coding-is-no-longer-the-constraint\",\"onClick\":\"$undefined\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"https://images.ctfassets.net/p762jor363g1/2SQWwDBa8t9F9P8s1IkaeV/645549d40f361b6fe56be1575e78cb5b/cwc01-header-a.png\",\"alt\":\"sticky\",\"width\":400,\"height\":197,\"quality\":100,\"priority\":true}]}]}],[\"$\",\"div\",null,{\"className\":\"post-card__content\",\"children\":[[\"$\",\"div\",null,{\"className\":\"top-content\",\"children\":[[\"$\",\"div\",null,{\"className\":\"post-card__date\",\"children\":\"Jun 3, 2026\"}],[\"$\",\"$L18\",null,{\"href\":\"/2026/6/code-with-claude-coding-is-no-longer-the-constraint\",\"onClick\":\"$undefined\",\"children\":[\"$\",\"h4\",null,{\"className\":\"post-card__title\",\"children\":\"Coding Is No Longer the Constraint: Scaling Developer Experience to Teams and Agents at Spotify\"}]}],[\"$\",\"div\",null,{\"className\":\"post-card__description\",\"children\":\"At Code with Claude, Spotify’s chief architect shared how we make both teams and AI agents more effective.\"}],[\"$\",\"div\",null,{\"className\":\"post-card__published\",\"children\":\"$undefined\"}]]}],[\"$\",\"div\",null,{\"className\":\"post-card__tags\",\"children\":\"$undefined\"}]]}]]}],[\"$\",\"div\",\"1XeG8OFPaFVQT83ShkXhoz\",{\"className\":\"post-card related\",\"children\":[[\"$\",\"div\",null,{\"className\":\"post-card__image\",\"children\":[\"$\",\"$L18\",null,{\"href\":\"/2026/5/better-experiments-with-llm-evals-a-funnel-not-a-fork\",\"onClick\":\"$undefined\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"https://images.ctfassets.net/p762jor363g1/1liN9CCW7JU64GI5r03T6N/95c8d19c1287e6209831f31f8dd89158/3__1_.png\",\"alt\":\"sticky\",\"width\":400,\"height\":197,\"quality\":100,\"priority\":true}]}]}],\"$L29\"]}]]}]}]]}]\n"])</script><script>self.__next_f.push([1,"29:[\"$\",\"div\",null,{\"className\":\"post-card__content\",\"children\":[[\"$\",\"div\",null,{\"className\":\"top-content\",\"children\":[[\"$\",\"div\",null,{\"className\":\"post-card__date\",\"children\":\"May 18, 2026\"}],[\"$\",\"$L18\",null,{\"href\":\"/2026/5/better-experiments-with-llm-evals-a-funnel-not-a-fork\",\"onClick\":\"$undefined\",\"children\":[\"$\",\"h4\",null,{\"className\":\"post-card__title\",\"children\":\"Better Experiments with LLM Evals — A funnel, not a fork\"}]}],[\"$\",\"div\",null,{\"className\":\"post-card__description\",\"children\":\"TL;DR  LLM evals, automated judges that assess relevance, coherence, and quality at scale, are a powerful new...\"}],[\"$\",\"div\",null,{\"className\":\"post-card__published\",\"children\":\"$undefined\"}]]}],[\"$\",\"div\",null,{\"className\":\"post-card__tags\",\"children\":\"$undefined\"}]]}]\n"])</script></body></html>