Files
nexus/sreweekly/articles/528/01-content-ingestion-podcast-video-incident-report.html

12 lines
67 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=yes"/><link rel="preload" as="image" href="/images/logo.png"/><link rel="preload" as="image" href="/images/linked-in.svg"/><link rel="preload" as="image" href="/images/x.svg"/><link rel="preload" as="image" href="/images/yt.svg"/><link rel="preload" as="image" href="/images/insta.svg"/><link rel="preload" as="image" href="/images/f.svg"/><link rel="preload" as="image" href="/images/feed.svg"/><link rel="preload" as="image" href="https://images.ctfassets.net/p762jor363g1/3AiqgaI6uFPcF6zc7lYX5t/bbed1b145db94423441163d9f149a02e/incident_report.png"/><link rel="preload" as="image" href="/images/x-blue.svg"/><link rel="preload" as="image" href="/images/f-blue.svg"/><link rel="stylesheet" href="/_next/static/chunks/0sxc27dsa.k6-.css" data-precedence="next"/><link rel="preload" as="script" fetchPriority="low" href="/_next/static/chunks/0zaia6i55l01..js"/><script src="/_next/static/chunks/0pqt~8bl3ukh4.js" async=""></script><script src="/_next/static/chunks/0m_p1bxtorv5i.js" async=""></script><script src="/_next/static/chunks/02g3221oh~3le.js" async=""></script><script src="/_next/static/chunks/turbopack-0~dymj8._h2gc.js" async=""></script><script src="/_next/static/chunks/0a_f9~n73ra-c.js" async=""></script><script src="/_next/static/chunks/0d3shmwh5_nmn.js" async=""></script><script src="/_next/static/chunks/08c8uq~xsq31o.js" async=""></script><script src="/_next/static/chunks/098votwdj80t9.js" async=""></script><script src="/_next/static/chunks/0wni0lk_-haja.js" async=""></script><link rel="preload" as="image" href="/images/linked-in-blue.svg"/><link rel="preload" as="image" href="/images/copy.svg"/><title>Content Ingestion &amp; Podcast Video Incident Report | Spotify Engineering</title><meta name="description" content="Over the past two months, podcast creators have experienced a series of reliability issues on Spotify. This report covers the most significant of these, the June 24 publishing delay, in full detail and describes the broader reliability program now underway across our publishing pipeline."/><meta name="application-name" content="Spotify Engineering"/><meta name="author" content="Jim Whitehead (Director, Engineering), Ulrik Mikaelsson (Staff Engineer), John Lagomarsino (Product Manager II), and Saunak Jai Chakrabarti (Senior Director, Engineering)"/><meta name="robots" content="index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1"/><meta name="twitter:data1" content="Jim Whitehead (Director, Engineering), Ulrik Mikaelsson (Staff Engineer), John Lagomarsino (Product Manager II), and Saunak Jai Chakrabarti (Senior Director, Engineering)"/><meta name="twitter:data2" content="2 minutes"/><meta name="twitter:label1" content="Written by"/><meta name="twitter:label2" content="Est. reading time"/><meta property="og:title" content="Content Ingestion &amp; Podcast Video Incident Report | Spotify Engineering"/><meta property="og:description" content="Over the past two months, podcast creators have experienced a series of reliability issues on Spotify. This report covers the most significant of these, the June 24 publishing delay, in full detail and describes the broader reliability program now underway across our publishing pipeline."/><meta property="og:site_name" content="Spotify Engineering"/><meta property="og:image" content="https://images.ctfassets.net/p762jor363g1/3AiqgaI6uFPcF6zc7lYX5t/bbed1b145db94423441163d9f149a02e/incident_report.png"/><meta property="og:image:type" content="image/jpeg"/><meta property="og:image:width" content="1200"/><meta property="og:image:height" content="630"/><meta property="og:image:alt" content="featured image"/><meta property="og:type" content="article"/><meta name="twitter:card" content="summary_large_image"/><meta name="twitter:creator" content="Jim Whitehead (Director, Engineering), Ulrik Mikaelsson (Staff Engineer), John Lagomarsino (Product Manager II), and Saunak Jai Chakrabarti (Senior Director, Engineering)"/><meta name="twitter:title" content="Content Ingestion &amp; Podcast Video Incident Report | Spotify Engineering"/><meta name="twitter:description" content="Content Ingestion &amp; Podcast Video Incident Report | Spotify Engineering"/><meta name="twitter:image" content="https://images.ctfassets.net/p762jor363g1/3AiqgaI6uFPcF6zc7lYX5t/bbed1b145db94423441163d9f149a02e/incident_report.png"/><meta name="twitter:image:alt" content="featured image"/><meta name="twitter:image:type" content="image/jpeg"/><meta name="twitter:image:width" content="1200"/><meta name="twitter:image:height" content="630"/><link rel="icon" href="/favicon.ico?favicon.0cd2g1ew_zd4v.ico" sizes="250x250" type="image/x-icon"/><script src="/_next/static/chunks/03~yq9q893hmn.js" noModule=""></script></head><body><div hidden=""><!--$--><!--/$--></div><div class="page-wrapper show-notification"><div class="content-wrapper "><!--$?--><template id="B:0"></template><!--/$--><div class="single-post"><div class="container"><h1 class="single-post__title">Content Ingestion &amp; Podcast Video Incident Report </h1></div><div class="container"><div class="single-post__content"><div class="single-post__featured-image"><img alt="Feature Image" width="740" height="404" decoding="async" data-nimg="1" style="color:transparent" src="https://images.ctfassets.net/p762jor363g1/3AiqgaI6uFPcF6zc7lYX5t/bbed1b145db94423441163d9f149a02e/incident_report.png"/></div><div class="html-content"><p >Over the past two months, podcast creators have experienced a series of reliability issues on Spotify. This report covers the most significant of these, the June 24 publishing delay, in full detail and describes the broader reliability program now underway across our publishing pipeline.</p><h2>What happened?</h2><p >When a podcast creator publishes a new episode, the audio and video content goes through a series of processing steps before it becomes available to Spotify users. These steps include transcoding (converting media into the formats our apps need) and content analysis.</p><p >On June 24, our video transcoding infrastructure reached maximum capacity. This created a backlog that delayed the publication of video podcast episodes for several hours. Creators reported that their episodes were not appearing on Spotify as expected. A queue built up and video podcast episodes that would normally be published within minutes were delayed for hours. Understandably, some creators re-uploaded episodes that had not appeared, which added further load. That is on us, not them: the system should have confirmed their upload was received and queued, and it did not.</p>
<figure class="figure-image">
<img
src="//images.ctfassets.net/p762jor363g1/7MLYmEAuLL5pj8xOtqHT6I/bf82c56512ed295e1ddea673040e9f1a/BlogPostGraph.png"
style="width: 100%; max-width: 100%; height: auto;"
class="blog-image"
alt="BlogPostGraph"
/>
</figure>
<p >The image above shows the build-up and subsequent emptying of the medium-priority (used for new episodes - shown in blue) and low-priority (used for updates to older episodes - shown in yellow) queues.</p><p >Four factors converged to create this situation:</p><ul><li><p ><b>Our transcoding infrastructure was running with insufficient headroom to handle large spikes of content delivery</b>. During typical periods of content submission our transcoding systems were able to scale and support both low-priority transcoding as well as the high-priority publication of new content. However, there wasn’t sufficient headroom to scale and support large spikes caused by the bulk delivery of new content.</p></li></ul><ul><li><p ><b>A scheduled batch processing job was running.</b> On occasion, we need to re-process existing episodes to be compatible with changes or additions to Spotify’s playback systems. During the June 24 disruption, a routine batch job processing existing content was consuming additional capacity alongside the regular processing for new episodes. While this job appeared fine earlier in the day, it became problematic when combined with increased content submissions.</p></li></ul><ul><li><p ><b>Recent improvements increased per-item processing cost.</b> We had recently changed our video transcoding to deliver better quality at lower bitrates. That change increased the time and processing power each episode requires, and we did not fully account for that added demand in our capacity planning.</p></li></ul><ul><li><p ><b>A software bug was underutilizing available compute resources.</b> Following a recent infrastructure migration to more powerful hardware, a bug in our resource scheduling caused our systems to underuse available processing capacity, reducing throughput by about 10%.</p></li></ul><p >When we identified the issue, we stopped the batch job, deployed a fix for the resource scheduling bug, and added additional processing capacity overnight. By the following morning, all backlogs had cleared and publishing was operating normally. We subsequently added further capacity to provide the headroom that had been missing.</p><h2>Timeline (UTC)</h2><ul><li><p ><b>13:30</b> — Early alerts fire in our internal monitoring. Not immediately recognized as a broader capacity issue.</p></li><li><p ><b>15:00</b> — Video podcast delivery spike pushes transcoding close to maximum capacity.</p></li><li><p ><b>16:35</b> — Batch processing job stopped to free capacity.</p></li><li><p ><b>17:31</b> — First Creator report of an issue impacting podcast video publishing received.</p></li><li><p ><b>17:34</b> — Automated alerts confirm queue backlog exceeding thresholds. Incident response begins.</p></li><li><p ><b>19:00</b> — Creator reports escalated to incident team.</p></li><li><p ><b>20:49</b> — Software fix deployed to improve resource utilization.</p></li><li><p ><b>00:14 (Jun 25)</b> — Additional processing cluster brought online.</p></li><li><p ><b>01:02</b> — All queues cleared.</p></li><li><p ><b>07:30</b> — Full confirmation: all publishing pipelines operating normally.</p></li></ul><p >One thing this timeline makes plain: roughly four hours passed between the first alerts and formal incident response. We want to do better. Engineers investigating the early alerts stopped the batch job at 16:35, but we did not recognize the full scope of the capacity problem until queues breached thresholds at 17:34. The monitoring improvements described below exist to close exactly that gap.</p><h2>Where do we go from here?</h2><p >We have already taken several steps to address this specific incident:</p><ul><li><p >Increased our transcoding capacity by approximately 67%, providing significantly more headroom for traffic spikes and batch operations.</p></li><li><p >Fixed the resource scheduling bug that was leaving under-utilized compute capacity.</p></li><li><p >Improved our monitoring to alert earlier when capacity is approaching limits.</p></li></ul><p >Beyond this specific incident, we are investing in broader improvements to the reliability of our podcast publishing pipeline. We&#39;ve formed a dedicated cross-team effort focused on:</p><ul><li><p >Building better capacity planning that accounts for not just steady-state traffic, but also burst capacity and incident recovery.</p></li><li><p >Improving prioritization across our publishing systems so that real-time content from creators is always processed ahead of background operations.</p></li><li><p >Extending rate limiting and backpressure mechanisms throughout the pipeline to handle unexpected load gracefully.</p></li><li><p >During this incident, many creators learned something was wrong from their audiences before they heard anything from us. We are improving our processes and technical capabilities so creators get notified as soon as possible when things aren’t working. </p></li></ul><p >When a creator hits publish, their audience is waiting, and hours matter. We fell short repeatedly this summer, and we know a report like this only counts if the next incident is handled better than the last. The work above is how we intend to earn that trust back.</p></div><div class="social"><p>SHARE THIS ARTICLE</p><ul class="social__links"><li><a href="https://twitter.com/intent/tweet?url=https%3A%2F%2Fengineering.atspotify.com%2F2026%2F7%2Fcontent-ingestion-and-podcast-video-incident-report&amp;text=Content%20Ingestion%20%26%20Podcast%20Video%20Incident%20Report%20" target="_blank"><img alt="x" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/x-blue.svg"/></a></li><li><a href="https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fengineering.atspotify.com%2F2026%2F7%2Fcontent-ingestion-and-podcast-video-incident-report" target="_blank" rel="noopener noreferrer"><img alt="fb" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/f-blue.svg"/></a></li><li><a href="https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fengineering.atspotify.com%2F2026%2F7%2Fcontent-ingestion-and-podcast-video-incident-report" target="_blank" rel="noopener noreferrer"><img alt="linkedin" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/linked-in-blue.svg"/></a></li><li><a><img alt="copy" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/copy.svg"/></a></li></ul></div></div><aside class="sidebar"><div class="date">Jul 20, 2026</div><div class="published">Published by Jim Whitehead (Director, Engineering), Ulrik Mikaelsson (Staff Engineer), John Lagomarsino (Product Manager II), and Saunak Jai Chakrabarti (Senior Director, Engineering)</div><div class="tag-wrapper"></div><div class="social"><p>SHARE THIS ARTICLE</p><ul class="social__links"><li><a href="https://twitter.com/intent/tweet?url=https%3A%2F%2Fengineering.atspotify.com%2F2026%2F7%2Fcontent-ingestion-and-podcast-video-incident-report&amp;text=Content%20Ingestion%20%26%20Podcast%20Video%20Incident%20Report%20" target="_blank"><img alt="x" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/x-blue.svg"/></a></li><li><a href="https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fengineering.atspotify.com%2F2026%2F7%2Fcontent-ingestion-and-podcast-video-incident-report" target="_blank" rel="noopener noreferrer"><img alt="fb" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/f-blue.svg"/></a></li><li><a href="https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fengineering.atspotify.com%2F2026%2F7%2Fcontent-ingestion-and-podcast-video-incident-report" target="_blank" rel="noopener noreferrer"><img alt="linkedin" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/linked-in-blue.svg"/></a></li><li><a><img alt="copy" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/copy.svg"/></a></li></ul></div></aside></div></div><!--$--><!--/$--></div><footer class="footer"><div class="container"><section class="subscription"><div class="subscription__left"><h3>Want to stay in the loop?</h3><p>Subscribe to our newsletter to never miss an update!</p><small>By clicking sign up you’ll receive occasional emails from Spotify. You always have the choice to adjust your interest settings or unsubscribe.</small></div><div class="subscription__right"><div id="mc_embed_signup"><form id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" action="https://spotify.us20.list-manage.com/subscribe/post?u=99aa88d3fba458716d0cd1299&amp;id=28cc680980" method="post" target="_blank"><div id="mc_embed_signup_scroll"><div class="mc-field-group"><input type="email" class="required email" id="mce-EMAIL" placeholder="Enter your best email" name="EMAIL"/><input type="submit" id="mc-embedded-subscribe" class="button button--submit" name="subscribe" value="Subscribe"/></div><div id="mce-responses" class="clear"><div class="response" id="mce-error-response" style="display:none"></div><div class="response" id="mce-success-response" style="display:none"></div></div><div style="left:-5000px;position:absolute" aria-hidden="true"><input type="text" name="b_99aa88d3fba458716d0cd1299_28cc680980"/></div></div></form></div></div></section><section class="links"><div class="links__left"><div class="logo"><a href="/"><img alt="feed" loading="lazy" width="167" height="50" decoding="async" data-nimg="1" style="color:transparent" src="/images/footer-logo.svg"/></a></div><div class="nav-wrapper"><nav class="nav"><ul><li><a target="_blank" rel="noopener noreferrer" href="https://spotify.com/">Spotify.com</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.lifeatspotify.com/">Spotify Jobs</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://newsroom.spotify.com/">Newsroom</a></li></ul><ul><li><a target="_blank" rel="noopener noreferrer" href="https://research.atspotify.com/">Spotify R&amp;D Research</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://backstage.spotify.com/">Spotify for Backstage</a></li></ul></nav></div></div><div class="links__right"><ul class="social"><li><a target="_blank" rel="noopener noreferrer" href="https://www.linkedin.com/showcase/spotify-r&amp;d"><img alt="Linked In" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/linked-in.svg"/></a></li><li><a target="_blank" rel="noopener noreferrer" href="https://twitter.com/spotifyeng"><img alt="X" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/x.svg"/></a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.youtube.com/@SpotifyRnD"><img alt="Youtube" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/yt.svg"/></a></li><li><a target="_blank" rel="noopener noreferrer" href="http://www.instagram.com/lifeatspotify"><img alt="Instagram" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/insta.svg"/></a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.facebook.com/spotify"><img alt="facebook" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/f.svg"/></a></li><li><a target="_blank" rel="noopener noreferrer" href="/feed"><img alt="feed" width="24" height="24" decoding="async" data-nimg="1" style="color:transparent" src="/images/feed.svg"/></a></li></ul></div></section><section class="copy"><nav class="copy__nav"><ul><li><a target="_blank" rel="noopener noreferrer" href="https://www.spotify.com/us/legal/end-user-agreement/">Legal</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.spotify.com/us/legal/privacy-policy/">Privacy</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.spotify.com/legal/cookies-policy/">Cookies</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.spotify.com/us/legal/privacy-policy/">About Ads</a></li></ul></nav><span>© <!-- -->2026<!-- --> Spotify AB</span></section></div></footer></div><script>requestAnimationFrame(function(){$RT=performance.now()});</script><script src="/_next/static/chunks/0zaia6i55l01..js" id="_R_" async=""></script><div hidden id="S:0"><div class="notification"><div class="content"></div></div><header class="header "><section class="container"><div class="container-left"><a href="/"><img alt="Engineering logo" width="90" height="36" decoding="async" data-nimg="1" style="color:transparent" src="/images/logo.png"/></a><nav class="nav "><ul><li class=""><a href="/">Blog</a></li><li class=""><a href="/podcasts">Podcasts</a></li><li class=""><a href="/opensource">Open Source</a></li><li class=""><a href="/jobs">Jobs</a></li><li class=""><a href="/about">About</a></li></ul></nav></div><div class="container-right"><div class="search "><div class="search__input"><input type="text" id="search-input" placeholder="Search" value=""/></div><div class="search__icon"><img alt="Search" id="search-icon" loading="lazy" width="24" height="24" decoding="async" data-nimg="1" class="mag" style="color:transparent" src="/images/icon-search.svg"/><img alt="Search" loading="lazy" width="24" height="24" decoding="async" data-nimg="1" class="close" style="color:transparent" src="/images/icon-close.svg"/></div><div class="menu-button "><span></span><span></span><span></span></div></div></div></section></header></div><script>$RB=[];$RV=function(a){$RT=performance.now();for(var b=0;b<a.length;b+=2){var c=a[b],e=a[b+1];null!==e.parentNode&&e.parentNode.removeChild(e);var f=c.parentNode;if(f){var g=c.previousSibling,h=0;do{if(c&&8===c.nodeType){var d=c.data;if("/$"===d||"/&"===d)if(0===h)break;else h--;else"$"!==d&&"$?"!==d&&"$~"!==d&&"$!"!==d&&"&"!==d||h++}d=c.nextSibling;f.removeChild(c);c=d}while(c);for(;e.firstChild;)f.insertBefore(e.firstChild,c);g.data="$";g._reactRetry&&requestAnimationFrame(g._reactRetry)}}a.length=0};
$RC=function(a,b){if(b=document.getElementById(b))(a=document.getElementById(a))?(a.previousSibling.data="$~",$RB.push(a,b),2===$RB.length&&("number"!==typeof $RT?requestAnimationFrame($RV.bind(null,$RB)):(a=performance.now(),setTimeout($RV.bind(null,$RB),2300>a&&2E3<a?2300-a:$RT+300-a)))):b.parentNode.removeChild(b)};$RC("B:0","S:0")</script><script>(self.__next_f=self.__next_f||[]).push([0])</script><script>self.__next_f.push([1,"1:\"$Sreact.fragment\"\n3:I[39756,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"default\"]\n4:I[37457,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"default\"]\n6:I[97367,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"OutletBoundary\"]\n7:\"$Sreact.suspense\"\na:I[97367,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"ViewportBoundary\"]\nc:I[97367,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"MetadataBoundary\"]\ne:I[68027,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"default\",1]\n:HL[\"/_next/static/chunks/0sxc27dsa.k6-.css\",\"style\"]\n"])</script><script>self.__next_f.push([1,"0:{\"P\":null,\"c\":[\"\",\"2026\",\"7\",\"content-ingestion-and-podcast-video-incident-report\"],\"q\":\"\",\"i\":false,\"f\":[[[\"\",{\"children\":[[\"year\",\"2026\",\"d\",null],{\"children\":[[\"month\",\"7\",\"d\",null],{\"children\":[[\"slug\",\"content-ingestion-and-podcast-video-incident-report\",\"d\",null],{\"children\":[\"__PAGE__\",{}]}]}]}]},\"$undefined\",\"$undefined\",16],[[\"$\",\"$1\",\"c\",{\"children\":[[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/chunks/0sxc27dsa.k6-.css\",\"precedence\":\"next\",\"crossOrigin\":\"$undefined\",\"nonce\":\"$undefined\"}],[\"$\",\"script\",\"script-0\",{\"src\":\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"async\":true,\"nonce\":\"$undefined\"}],[\"$\",\"script\",\"script-1\",{\"src\":\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"async\":true,\"nonce\":\"$undefined\"}],[\"$\",\"script\",\"script-2\",{\"src\":\"/_next/static/chunks/08c8uq~xsq31o.js\",\"async\":true,\"nonce\":\"$undefined\"}]],\"$L2\"]}],{\"children\":[[\"$\",\"$1\",\"c\",{\"children\":[null,[\"$\",\"$L3\",null,{\"parallelRouterKey\":\"children\",\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$L4\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":\"$undefined\",\"forbidden\":\"$undefined\",\"unauthorized\":\"$undefined\"}]]}],{\"children\":[[\"$\",\"$1\",\"c\",{\"children\":[null,[\"$\",\"$L3\",null,{\"parallelRouterKey\":\"children\",\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$L4\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":\"$undefined\",\"forbidden\":\"$undefined\",\"unauthorized\":\"$undefined\"}]]}],{\"children\":[[\"$\",\"$1\",\"c\",{\"children\":[null,[\"$\",\"$L3\",null,{\"parallelRouterKey\":\"children\",\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$L4\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":\"$undefined\",\"forbidden\":\"$undefined\",\"unauthorized\":\"$undefined\"}]]}],{\"children\":[[\"$\",\"$1\",\"c\",{\"children\":[\"$L5\",[[\"$\",\"script\",\"script-0\",{\"src\":\"/_next/static/chunks/0wni0lk_-haja.js\",\"async\":true,\"nonce\":\"$undefined\"}]],[\"$\",\"$L6\",null,{\"children\":[\"$\",\"$7\",null,{\"name\":\"Next.MetadataOutlet\",\"children\":\"$@8\"}]}]]}],{},null,false,null]},null,false,\"$@9\"]},null,false,\"$@9\"]},null,false,\"$@9\"]},null,false,null],[\"$\",\"$1\",\"h\",{\"children\":[null,[\"$\",\"$La\",null,{\"children\":\"$Lb\"}],[\"$\",\"div\",null,{\"hidden\":true,\"children\":[\"$\",\"$Lc\",null,{\"children\":[\"$\",\"$7\",null,{\"name\":\"Next.Metadata\",\"children\":\"$Ld\"}]}]}],null]}],false]],\"m\":\"$undefined\",\"G\":[\"$e\",[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/chunks/0sxc27dsa.k6-.css\",\"precedence\":\"next\",\"crossOrigin\":\"$undefined\",\"nonce\":\"$undefined\"}]]],\"S\":false,\"h\":null,\"s\":\"$undefined\",\"l\":\"$undefined\",\"p\":\"$undefined\",\"d\":\"$undefined\",\"b\":\"z1ONPGug6r-HUxPkOhj0S\"}\n"])</script><script>self.__next_f.push([1,"f:[]\n9:\"$Wf\"\nb:[[\"$\",\"meta\",\"0\",{\"charSet\":\"utf-8\"}],[\"$\",\"meta\",\"1\",{\"name\":\"viewport\",\"content\":\"width=device-width, initial-scale=1, maximum-scale=1, user-scalable=yes\"}]]\n"])</script><script>self.__next_f.push([1,"10:I[79520,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"\"]\n11:I[1013,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"HeaderContextProvider\"]\n12:I[32666,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"PostProvider\"]\n13:I[28044,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"UiTextProvider\"]\n14:I[70292,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"Initializer\"]\n15:I[97341,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"default\"]\n16:I[36676,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"default\"]\n17:I[58298,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\",\"/_next/static/chunks/098votwdj80t9.js\"],\"default\"]\n18:I[22016,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\",\"/_next/static/chunks/0wni0lk_-haja.js\"],\"\"]\n"])</script><script>self.__next_f.push([1,"2:[\"$\",\"html\",null,{\"lang\":\"en\",\"children\":[[\"$\",\"head\",null,{\"children\":[\"$\",\"$L10\",null,{\"id\":\"gtm\",\"dangerouslySetInnerHTML\":{\"__html\":\"(function(w, d, s, l, i) {\\n w[l] = w[l] || [];\\n w[l].push({\\n 'gtm.start': new Date().getTime(),\\n event: 'gtm.js'\\n });\\n var f = d.getElementsByTagName(s)[0],\\n j = d.createElement(s),\\n dl = l != 'dataLayer' ? '\u0026l=' + l : '';\\n j.async = true;\\n j.src =\\n 'https://www.googletagmanager.com/gtm.js?id=' + i + dl;\\n f.parentNode.insertBefore(j, f);\\n })(window, document, 'script', 'dataLayer', 'GTM-W5NPJP7');\"}}]}],[\"$\",\"body\",null,{\"children\":[\"$\",\"div\",null,{\"className\":\"page-wrapper show-notification\",\"children\":[[\"$\",\"$L11\",null,{\"children\":[\"$\",\"$L12\",null,{\"children\":[\"$\",\"$L13\",null,{\"defaultUiText\":{\"homepageBannerText\":{\"data\":{},\"content\":[{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"Read about the magic behind the music \u0026 more. Welcome to our official technology blog.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"}],\"nodeType\":\"document\"},\"podcastTagline\":{\"data\":{},\"content\":[{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"Two original shows from Spotify R\u0026D.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"Listen and fill your brain with great tech stories.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"},{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"}],\"nodeType\":\"document\"},\"opensourceTagline\":{\"data\":{},\"content\":[{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"There’s never been a better time to take part in open source at Spotify. We don’t just want to contribute back to the community — we want to lead the way with projects we truly believe in. Jump into one of the projects below and join us.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"}],\"nodeType\":\"document\"},\"jobsTagline\":{\"nodeType\":\"document\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Complex problems, exceptional colleagues, and a direct impact on the joy of \",\"marks\":[],\"data\":{}},{\"nodeType\":\"text\",\"value\":\"713\",\"marks\":[],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" million people. Our beliefs drive our success.\",\"marks\":[],\"data\":{}}]}]},\"aboutTagline\":{\"data\":{},\"content\":[{\"data\":{},\"content\":[{\"data\":{},\"marks\":[],\"value\":\"Spotify Engineering is a diverse community of engineers, developers, researchers and more. Together, we build the infrastructure, features, and experiences you love – and help to shape the future of audio.\",\"nodeType\":\"text\"}],\"nodeType\":\"paragraph\"}],\"nodeType\":\"document\"}},\"children\":[\"$\",\"$L14\",null,{\"children\":[[\"$\",\"$7\",null,{\"children\":[[\"$\",\"$L15\",null,{}],[\"$\",\"$L16\",null,{}]]}],[\"$\",\"$L3\",null,{\"parallelRouterKey\":\"children\",\"error\":\"$17\",\"errorStyles\":[],\"errorScripts\":[[\"$\",\"script\",\"script-0\",{\"src\":\"/_next/static/chunks/098votwdj80t9.js\",\"async\":true}]],\"template\":[\"$\",\"$L4\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":[[\"$\",\"div\",null,{\"className\":\"not-found\",\"children\":[\"$\",\"div\",null,{\"className\":\"container\",\"children\":[[\"$\",\"h1\",null,{\"children\":\"404\"}],[\"$\",\"div\",null,{\"className\":\"content\",\"children\":[[\"$\",\"h3\",null,{\"children\":\"OOOOPS... Something went wrong\"}],[\"$\",\"p\",null,{\"children\":\"We couldn't find the page you're looking for.\"}]]}],[\"$\",\"$L18\",null,{\"href\":\"/\",\"className\":\"button\",\"children\":\"Go back home\"}]]}]}],[]],\"forbidden\":\"$undefined\",\"unauthorized\":\"$undefined\"}]]}]}]}]}],[\"$\",\"footer\",null,{\"className\":\"footer\",\"children\":[\"$\",\"div\",null,{\"className\":\"container\",\"children\":[[\"$\",\"section\",null,{\"className\":\"subscription\",\"children\":[[\"$\",\"div\",null,{\"className\":\"subscription__left\",\"children\":[[\"$\",\"h3\",null,{\"children\":\"Want to stay in the loop?\"}],[\"$\",\"p\",null,{\"children\":\"Subscribe to our newsletter to never miss an update!\"}],[\"$\",\"small\",null,{\"children\":\"By clicking sign up you’ll receive occasional emails from Spotify. You always have the choice to adjust your interest settings or unsubscribe.\"}]]}],[\"$\",\"div\",null,{\"className\":\"subscription__right\",\"children\":[\"$\",\"div\",null,{\"id\":\"mc_embed_signup\",\"children\":[\"$\",\"form\",null,{\"action\":\"https://spotify.us20.list-manage.com/subscribe/post?u=99aa88d3fba458716d0cd1299\u0026id=28cc680980\",\"method\":\"post\",\"id\":\"mc-embedded-subscribe-form\",\"name\":\"mc-embedded-subscribe-form\",\"className\":\"validate\",\"target\":\"_blank\",\"children\":\"$L19\"}]}]}]]}],\"$L1a\",\"$L1b\"]}]}]]}]}]]}]\n"])</script><script>self.__next_f.push([1,"1c:I[5500,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\",\"/_next/static/chunks/0wni0lk_-haja.js\"],\"Image\"]\n19:[\"$\",\"div\",null,{\"id\":\"mc_embed_signup_scroll\",\"children\":[[\"$\",\"div\",null,{\"className\":\"mc-field-group\",\"children\":[[\"$\",\"input\",null,{\"type\":\"email\",\"name\":\"EMAIL\",\"className\":\"required email\",\"id\":\"mce-EMAIL\",\"placeholder\":\"Enter your best email\"}],[\"$\",\"input\",null,{\"type\":\"submit\",\"name\":\"subscribe\",\"value\":\"Subscribe\",\"id\":\"mc-embedded-subscribe\",\"className\":\"button button--submit\"}]]}],[\"$\",\"div\",null,{\"id\":\"mce-responses\",\"className\":\"clear\",\"children\":[[\"$\",\"div\",null,{\"className\":\"response\",\"id\":\"mce-error-response\",\"style\":{\"display\":\"none\"}}],[\"$\",\"div\",null,{\"className\":\"response\",\"id\":\"mce-success-response\",\"style\":{\"display\":\"none\"}}]]}],[\"$\",\"div\",null,{\"style\":{\"left\":\"-5000px\",\"position\":\"absolute\"},\"aria-hidden\":\"true\",\"children\":[\"$\",\"input\",null,{\"type\":\"text\",\"name\":\"b_99aa88d3fba458716d0cd1299_28cc680980\"}]}]]}]\n"])</script><script>self.__next_f.push([1,"1a:[\"$\",\"section\",null,{\"className\":\"links\",\"children\":[[\"$\",\"div\",null,{\"className\":\"links__left\",\"children\":[[\"$\",\"div\",null,{\"className\":\"logo\",\"children\":[\"$\",\"$L18\",null,{\"href\":\"/\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/footer-logo.svg\",\"alt\":\"feed\",\"width\":167,\"height\":50}]}]}],[\"$\",\"div\",null,{\"className\":\"nav-wrapper\",\"children\":[\"$\",\"nav\",null,{\"className\":\"nav\",\"children\":[[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://spotify.com/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Spotify.com\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.lifeatspotify.com/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Spotify Jobs\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://newsroom.spotify.com/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Newsroom\"}]}]]}],[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://research.atspotify.com/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Spotify R\u0026D Research\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://backstage.spotify.com/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Spotify for Backstage\"}]}]]}]]}]}]]}],[\"$\",\"div\",null,{\"className\":\"links__right\",\"children\":[\"$\",\"ul\",null,{\"className\":\"social\",\"children\":[[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.linkedin.com/showcase/spotify-r\u0026d\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/linked-in.svg\",\"alt\":\"Linked In\",\"width\":24,\"height\":24,\"priority\":true}]}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://twitter.com/spotifyeng\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/x.svg\",\"alt\":\"X\",\"width\":24,\"height\":24,\"priority\":true}]}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.youtube.com/@SpotifyRnD\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/yt.svg\",\"alt\":\"Youtube\",\"width\":24,\"height\":24,\"priority\":true}]}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"http://www.instagram.com/lifeatspotify\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/insta.svg\",\"alt\":\"Instagram\",\"width\":24,\"height\":24,\"priority\":true}]}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.facebook.com/spotify\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/f.svg\",\"alt\":\"facebook\",\"width\":24,\"height\":24,\"priority\":true}]}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"/feed\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"/images/feed.svg\",\"alt\":\"feed\",\"width\":24,\"height\":24,\"priority\":true}]}]}]]}]}]]}]\n"])</script><script>self.__next_f.push([1,"1b:[\"$\",\"section\",null,{\"className\":\"copy\",\"children\":[[\"$\",\"nav\",null,{\"className\":\"copy__nav\",\"children\":[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.spotify.com/us/legal/end-user-agreement/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Legal\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.spotify.com/us/legal/privacy-policy/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Privacy\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.spotify.com/legal/cookies-policy/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"Cookies\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"$L18\",null,{\"href\":\"https://www.spotify.com/us/legal/privacy-policy/\",\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"children\":\"About Ads\"}]}]]}]}],[\"$\",\"span\",null,{\"children\":[\"© \",2026,\" Spotify AB\"]}]]}]\n"])</script><script>self.__next_f.push([1,"1d:I[27201,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\"],\"IconMark\"]\n8:null\n"])</script><script>self.__next_f.push([1,"d:[[\"$\",\"title\",\"0\",{\"children\":\"Content Ingestion \u0026 Podcast Video Incident Report | Spotify Engineering\"}],[\"$\",\"meta\",\"1\",{\"name\":\"description\",\"content\":\"Over the past two months, podcast creators have experienced a series of reliability issues on Spotify. This report covers the most significant of these, the June 24 publishing delay, in full detail and describes the broader reliability program now underway across our publishing pipeline.\"}],[\"$\",\"meta\",\"2\",{\"name\":\"application-name\",\"content\":\"Spotify Engineering\"}],[\"$\",\"meta\",\"3\",{\"name\":\"author\",\"content\":\"Jim Whitehead (Director, Engineering), Ulrik Mikaelsson (Staff Engineer), John Lagomarsino (Product Manager II), and Saunak Jai Chakrabarti (Senior Director, Engineering)\"}],[\"$\",\"meta\",\"4\",{\"name\":\"robots\",\"content\":\"index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1\"}],[\"$\",\"meta\",\"5\",{\"name\":\"twitter:data1\",\"content\":\"Jim Whitehead (Director, Engineering), Ulrik Mikaelsson (Staff Engineer), John Lagomarsino (Product Manager II), and Saunak Jai Chakrabarti (Senior Director, Engineering)\"}],[\"$\",\"meta\",\"6\",{\"name\":\"twitter:data2\",\"content\":\"2 minutes\"}],[\"$\",\"meta\",\"7\",{\"name\":\"twitter:label1\",\"content\":\"Written by\"}],[\"$\",\"meta\",\"8\",{\"name\":\"twitter:label2\",\"content\":\"Est. reading time\"}],[\"$\",\"meta\",\"9\",{\"property\":\"og:title\",\"content\":\"Content Ingestion \u0026 Podcast Video Incident Report | Spotify Engineering\"}],[\"$\",\"meta\",\"10\",{\"property\":\"og:description\",\"content\":\"Over the past two months, podcast creators have experienced a series of reliability issues on Spotify. This report covers the most significant of these, the June 24 publishing delay, in full detail and describes the broader reliability program now underway across our publishing pipeline.\"}],[\"$\",\"meta\",\"11\",{\"property\":\"og:site_name\",\"content\":\"Spotify Engineering\"}],[\"$\",\"meta\",\"12\",{\"property\":\"og:image\",\"content\":\"https://images.ctfassets.net/p762jor363g1/3AiqgaI6uFPcF6zc7lYX5t/bbed1b145db94423441163d9f149a02e/incident_report.png\"}],[\"$\",\"meta\",\"13\",{\"property\":\"og:image:type\",\"content\":\"image/jpeg\"}],[\"$\",\"meta\",\"14\",{\"property\":\"og:image:width\",\"content\":\"1200\"}],[\"$\",\"meta\",\"15\",{\"property\":\"og:image:height\",\"content\":\"630\"}],[\"$\",\"meta\",\"16\",{\"property\":\"og:image:alt\",\"content\":\"featured image\"}],[\"$\",\"meta\",\"17\",{\"property\":\"og:type\",\"content\":\"article\"}],[\"$\",\"meta\",\"18\",{\"name\":\"twitter:card\",\"content\":\"summary_large_image\"}],[\"$\",\"meta\",\"19\",{\"name\":\"twitter:creator\",\"content\":\"Jim Whitehead (Director, Engineering), Ulrik Mikaelsson (Staff Engineer), John Lagomarsino (Product Manager II), and Saunak Jai Chakrabarti (Senior Director, Engineering)\"}],[\"$\",\"meta\",\"20\",{\"name\":\"twitter:title\",\"content\":\"Content Ingestion \u0026 Podcast Video Incident Report | Spotify Engineering\"}],[\"$\",\"meta\",\"21\",{\"name\":\"twitter:description\",\"content\":\"Content Ingestion \u0026 Podcast Video Incident Report | Spotify Engineering\"}],[\"$\",\"meta\",\"22\",{\"name\":\"twitter:image\",\"content\":\"https://images.ctfassets.net/p762jor363g1/3AiqgaI6uFPcF6zc7lYX5t/bbed1b145db94423441163d9f149a02e/incident_report.png\"}],[\"$\",\"meta\",\"23\",{\"name\":\"twitter:image:alt\",\"content\":\"featured image\"}],[\"$\",\"meta\",\"24\",{\"name\":\"twitter:image:type\",\"content\":\"image/jpeg\"}],[\"$\",\"meta\",\"25\",{\"name\":\"twitter:image:width\",\"content\":\"1200\"}],[\"$\",\"meta\",\"26\",{\"name\":\"twitter:image:height\",\"content\":\"630\"}],[\"$\",\"link\",\"27\",{\"rel\":\"icon\",\"href\":\"/favicon.ico?favicon.0cd2g1ew_zd4v.ico\",\"sizes\":\"250x250\",\"type\":\"image/x-icon\"}],[\"$\",\"$L1d\",\"28\",{}]]\n"])</script><script>self.__next_f.push([1,"1e:T1a5a,"])</script><script>self.__next_f.push([1,"\u003cp \u003eOver the past two months, podcast creators have experienced a series of reliability issues on Spotify. This report covers the most significant of these, the June 24 publishing delay, in full detail and describes the broader reliability program now underway across our publishing pipeline.\u003c/p\u003e\u003ch2\u003eWhat happened?\u003c/h2\u003e\u003cp \u003eWhen a podcast creator publishes a new episode, the audio and video content goes through a series of processing steps before it becomes available to Spotify users. These steps include transcoding (converting media into the formats our apps need) and content analysis.\u003c/p\u003e\u003cp \u003eOn June 24, our video transcoding infrastructure reached maximum capacity. This created a backlog that delayed the publication of video podcast episodes for several hours. Creators reported that their episodes were not appearing on Spotify as expected. A queue built up and video podcast episodes that would normally be published within minutes were delayed for hours. Understandably, some creators re-uploaded episodes that had not appeared, which added further load. That is on us, not them: the system should have confirmed their upload was received and queued, and it did not.\u003c/p\u003e\n \u003cfigure class=\"figure-image\"\u003e\n \u003cimg \n src=\"//images.ctfassets.net/p762jor363g1/7MLYmEAuLL5pj8xOtqHT6I/bf82c56512ed295e1ddea673040e9f1a/BlogPostGraph.png\" \n style=\"width: 100%; max-width: 100%; height: auto;\"\n class=\"blog-image\"\n alt=\"BlogPostGraph\"\n /\u003e\n \n \u003c/figure\u003e\n \u003cp \u003eThe image above shows the build-up and subsequent emptying of the medium-priority (used for new episodes - shown in blue) and low-priority (used for updates to older episodes - shown in yellow) queues.\u003c/p\u003e\u003cp \u003eFour factors converged to create this situation:\u003c/p\u003e\u003cul\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003eOur transcoding infrastructure was running with insufficient headroom to handle large spikes of content delivery\u003c/b\u003e. During typical periods of content submission our transcoding systems were able to scale and support both low-priority transcoding as well as the high-priority publication of new content. However, there wasn’t sufficient headroom to scale and support large spikes caused by the bulk delivery of new content.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cul\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003eA scheduled batch processing job was running.\u003c/b\u003e On occasion, we need to re-process existing episodes to be compatible with changes or additions to Spotify’s playback systems. During the June 24 disruption, a routine batch job processing existing content was consuming additional capacity alongside the regular processing for new episodes. While this job appeared fine earlier in the day, it became problematic when combined with increased content submissions.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cul\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003eRecent improvements increased per-item processing cost.\u003c/b\u003e We had recently changed our video transcoding to deliver better quality at lower bitrates. That change increased the time and processing power each episode requires, and we did not fully account for that added demand in our capacity planning.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cul\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003eA software bug was underutilizing available compute resources.\u003c/b\u003e Following a recent infrastructure migration to more powerful hardware, a bug in our resource scheduling caused our systems to underuse available processing capacity, reducing throughput by about 10%.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cp \u003eWhen we identified the issue, we stopped the batch job, deployed a fix for the resource scheduling bug, and added additional processing capacity overnight. By the following morning, all backlogs had cleared and publishing was operating normally. We subsequently added further capacity to provide the headroom that had been missing.\u003c/p\u003e\u003ch2\u003eTimeline (UTC)\u003c/h2\u003e\u003cul\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003e13:30\u003c/b\u003e — Early alerts fire in our internal monitoring. Not immediately recognized as a broader capacity issue.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003e15:00\u003c/b\u003e — Video podcast delivery spike pushes transcoding close to maximum capacity.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003e16:35\u003c/b\u003e — Batch processing job stopped to free capacity.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003e17:31\u003c/b\u003e — First Creator report of an issue impacting podcast video publishing received.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003e17:34\u003c/b\u003e — Automated alerts confirm queue backlog exceeding thresholds. Incident response begins.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003e19:00\u003c/b\u003e — Creator reports escalated to incident team.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003e20:49\u003c/b\u003e — Software fix deployed to improve resource utilization.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003e00:14 (Jun 25)\u003c/b\u003e — Additional processing cluster brought online.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003e01:02\u003c/b\u003e — All queues cleared.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003e\u003cb\u003e07:30\u003c/b\u003e — Full confirmation: all publishing pipelines operating normally.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cp \u003eOne thing this timeline makes plain: roughly four hours passed between the first alerts and formal incident response. We want to do better. Engineers investigating the early alerts stopped the batch job at 16:35, but we did not recognize the full scope of the capacity problem until queues breached thresholds at 17:34. The monitoring improvements described below exist to close exactly that gap.\u003c/p\u003e\u003ch2\u003eWhere do we go from here?\u003c/h2\u003e\u003cp \u003eWe have already taken several steps to address this specific incident:\u003c/p\u003e\u003cul\u003e\u003cli\u003e\u003cp \u003eIncreased our transcoding capacity by approximately 67%, providing significantly more headroom for traffic spikes and batch operations.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003eFixed the resource scheduling bug that was leaving under-utilized compute capacity.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003eImproved our monitoring to alert earlier when capacity is approaching limits.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cp \u003eBeyond this specific incident, we are investing in broader improvements to the reliability of our podcast publishing pipeline. We\u0026#39;ve formed a dedicated cross-team effort focused on:\u003c/p\u003e\u003cul\u003e\u003cli\u003e\u003cp \u003eBuilding better capacity planning that accounts for not just steady-state traffic, but also burst capacity and incident recovery.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003eImproving prioritization across our publishing systems so that real-time content from creators is always processed ahead of background operations.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003eExtending rate limiting and backpressure mechanisms throughout the pipeline to handle unexpected load gracefully.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp \u003eDuring this incident, many creators learned something was wrong from their audiences before they heard anything from us. We are improving our processes and technical capabilities so creators get notified as soon as possible when things aren’t working. \u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cp \u003eWhen a creator hits publish, their audience is waiting, and hours matter. We fell short repeatedly this summer, and we know a report like this only counts if the next incident is handled better than the last. The work above is how we intend to earn that trust back.\u003c/p\u003e"])</script><script>self.__next_f.push([1,"5:[[\"$\",\"div\",null,{\"className\":\"single-post\",\"children\":[[\"$\",\"div\",null,{\"className\":\"container\",\"children\":[\"$\",\"h1\",null,{\"className\":\"single-post__title\",\"children\":\"Content Ingestion \u0026 Podcast Video Incident Report \"}]}],[\"$\",\"div\",null,{\"className\":\"container\",\"children\":[[\"$\",\"div\",null,{\"className\":\"single-post__content\",\"children\":[[\"$\",\"div\",null,{\"className\":\"single-post__featured-image\",\"children\":[\"$\",\"$L1c\",null,{\"src\":\"https://images.ctfassets.net/p762jor363g1/3AiqgaI6uFPcF6zc7lYX5t/bbed1b145db94423441163d9f149a02e/incident_report.png\",\"alt\":\"Feature Image\",\"width\":740,\"height\":404,\"priority\":true}]}],[\"$\",\"div\",null,{\"className\":\"html-content\",\"dangerouslySetInnerHTML\":{\"__html\":\"$1e\"}}],null,\"$L1f\"]}],\"$L20\"]}]]}],null]\n"])</script><script>self.__next_f.push([1,"21:I[58931,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\",\"/_next/static/chunks/0wni0lk_-haja.js\"],\"SocialShare\"]\n22:I[40367,[\"/_next/static/chunks/0a_f9~n73ra-c.js\",\"/_next/static/chunks/0d3shmwh5_nmn.js\",\"/_next/static/chunks/08c8uq~xsq31o.js\",\"/_next/static/chunks/0wni0lk_-haja.js\"],\"default\"]\n1f:[\"$\",\"$L21\",null,{\"heading\":\"SHARE THIS ARTICLE\",\"title\":\"Content Ingestion \u0026 Podcast Video Incident Report \"}]\n"])</script><script>self.__next_f.push([1,"20:[\"$\",\"$L22\",null,{\"blogPost\":{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"60dcNVzos5DWmTFQLUR5O2\",\"type\":\"Entry\",\"createdAt\":\"2026-07-20T16:24:48.925Z\",\"updatedAt\":\"2026-07-22T08:41:13.382Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":65,\"revision\":11,\"contentType\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"ContentType\",\"id\":\"blogPost\"}},\"locale\":\"en-US\"},\"fields\":{\"title\":\"Content Ingestion \u0026 Podcast Video Incident Report \",\"slug\":\"content-ingestion-and-podcast-video-incident-report\",\"content\":{\"nodeType\":\"document\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Over the past two months, podcast creators have experienced a series of reliability issues on Spotify. This report covers the most significant of these, the June 24 publishing delay, in full detail and describes the broader reliability program now underway across our publishing pipeline.\",\"marks\":[],\"data\":{}}]},{\"nodeType\":\"heading-2\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"What happened?\",\"marks\":[],\"data\":{}}]},{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"When a podcast creator publishes a new episode, the audio and video content goes through a series of processing steps before it becomes available to Spotify users. These steps include transcoding (converting media into the formats our apps need) and content analysis.\",\"marks\":[],\"data\":{}}]},{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"On June 24, our video transcoding infrastructure reached maximum capacity. This created a backlog that delayed the publication of video podcast episodes for several hours. Creators reported that their episodes were not appearing on Spotify as expected. A queue built up and video podcast episodes that would normally be published within minutes were delayed for hours. Understandably, some creators re-uploaded episodes that had not appeared, which added further load. That is on us, not them: the system should have confirmed their upload was received and queued, and it did not.\",\"marks\":[],\"data\":{}}]},{\"nodeType\":\"embedded-asset-block\",\"data\":{\"target\":{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"7MLYmEAuLL5pj8xOtqHT6I\",\"type\":\"Asset\",\"createdAt\":\"2026-07-20T17:19:36.399Z\",\"updatedAt\":\"2026-07-20T17:19:36.399Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":3,\"revision\":1,\"locale\":\"en-US\"},\"fields\":{\"title\":\"BlogPostGraph\",\"description\":\"\",\"file\":{\"url\":\"//images.ctfassets.net/p762jor363g1/7MLYmEAuLL5pj8xOtqHT6I/bf82c56512ed295e1ddea673040e9f1a/BlogPostGraph.png\",\"details\":{\"size\":322270,\"image\":{\"width\":2634,\"height\":1306}},\"fileName\":\"BlogPostGraph.png\",\"contentType\":\"image/png\"}}}},\"content\":[]},{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"The image above shows the build-up and subsequent emptying of the medium-priority (used for new episodes - shown in blue) and low-priority (used for updates to older episodes - shown in yellow) queues.\",\"marks\":[],\"data\":{}}]},{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Four factors converged to create this situation:\",\"marks\":[],\"data\":{}}]},{\"nodeType\":\"unordered-list\",\"data\":{},\"content\":[{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Our transcoding infrastructure was running with insufficient headroom to handle large spikes of content delivery\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\". During typical periods of content submission our transcoding systems were able to scale and support both low-priority transcoding as well as the high-priority publication of new content. However, there wasn’t sufficient headroom to scale and support large spikes caused by the bulk delivery of new content.\",\"marks\":[],\"data\":{}}]}]}]},{\"nodeType\":\"unordered-list\",\"data\":{},\"content\":[{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"A scheduled batch processing job was running.\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" On occasion, we need to re-process existing episodes to be compatible with changes or additions to Spotify’s playback systems. During the June 24 disruption, a routine batch job processing existing content was consuming additional capacity alongside the regular processing for new episodes. While this job appeared fine earlier in the day, it became problematic when combined with increased content submissions.\",\"marks\":[],\"data\":{}}]}]}]},{\"nodeType\":\"unordered-list\",\"data\":{},\"content\":[{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Recent improvements increased per-item processing cost.\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" We had recently changed our video transcoding to deliver better quality at lower bitrates. That change increased the time and processing power each episode requires, and we did not fully account for that added demand in our capacity planning.\",\"marks\":[],\"data\":{}}]}]}]},{\"nodeType\":\"unordered-list\",\"data\":{},\"content\":[{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"A software bug was underutilizing available compute resources.\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" Following a recent infrastructure migration to more powerful hardware, a bug in our resource scheduling caused our systems to underuse available processing capacity, reducing throughput by about 10%.\",\"marks\":[],\"data\":{}}]}]}]},{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"When we identified the issue, we stopped the batch job, deployed a fix for the resource scheduling bug, and added additional processing capacity overnight. By the following morning, all backlogs had cleared and publishing was operating normally. We subsequently added further capacity to provide the headroom that had been missing.\",\"marks\":[],\"data\":{}}]},{\"nodeType\":\"heading-2\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Timeline (UTC)\",\"marks\":[],\"data\":{}}]},{\"nodeType\":\"unordered-list\",\"data\":{},\"content\":[{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"13:30\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" — Early alerts fire in our internal monitoring. Not immediately recognized as a broader capacity issue.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"15:00\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" — Video podcast delivery spike pushes transcoding close to maximum capacity.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"16:35\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" — Batch processing job stopped to free capacity.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"17:31\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" — First Creator report of an issue impacting podcast video publishing received.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"17:34\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" — Automated alerts confirm queue backlog exceeding thresholds. Incident response begins.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"19:00\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" — Creator reports escalated to incident team.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"20:49\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" — Software fix deployed to improve resource utilization.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"00:14 (Jun 25)\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" — Additional processing cluster brought online.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"01:02\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" — All queues cleared.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"07:30\",\"marks\":[{\"type\":\"bold\"}],\"data\":{}},{\"nodeType\":\"text\",\"value\":\" — Full confirmation: all publishing pipelines operating normally.\",\"marks\":[],\"data\":{}}]}]}]},{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"One thing this timeline makes plain: roughly four hours passed between the first alerts and formal incident response. We want to do better. Engineers investigating the early alerts stopped the batch job at 16:35, but we did not recognize the full scope of the capacity problem until queues breached thresholds at 17:34. The monitoring improvements described below exist to close exactly that gap.\",\"marks\":[],\"data\":{}}]},{\"nodeType\":\"heading-2\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Where do we go from here?\",\"marks\":[],\"data\":{}}]},{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"We have already taken several steps to address this specific incident:\",\"marks\":[],\"data\":{}}]},{\"nodeType\":\"unordered-list\",\"data\":{},\"content\":[{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Increased our transcoding capacity by approximately 67%, providing significantly more headroom for traffic spikes and batch operations.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Fixed the resource scheduling bug that was leaving under-utilized compute capacity.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Improved our monitoring to alert earlier when capacity is approaching limits.\",\"marks\":[],\"data\":{}}]}]}]},{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Beyond this specific incident, we are investing in broader improvements to the reliability of our podcast publishing pipeline. We've formed a dedicated cross-team effort focused on:\",\"marks\":[],\"data\":{}}]},{\"nodeType\":\"unordered-list\",\"data\":{},\"content\":[{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Building better capacity planning that accounts for not just steady-state traffic, but also burst capacity and incident recovery.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Improving prioritization across our publishing systems so that real-time content from creators is always processed ahead of background operations.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"Extending rate limiting and backpressure mechanisms throughout the pipeline to handle unexpected load gracefully.\",\"marks\":[],\"data\":{}}]}]},{\"nodeType\":\"list-item\",\"data\":{},\"content\":[{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"During this incident, many creators learned something was wrong from their audiences before they heard anything from us. We are improving our processes and technical capabilities so creators get notified as soon as possible when things aren’t working. \",\"marks\":[],\"data\":{}}]}]}]},{\"nodeType\":\"paragraph\",\"data\":{},\"content\":[{\"nodeType\":\"text\",\"value\":\"When a creator hits publish, their audience is waiting, and hours matter. We fell short repeatedly this summer, and we know a report like this only counts if the next incident is handled better than the last. The work above is how we intend to earn that trust back.\",\"marks\":[],\"data\":{}}]}]},\"categories\":[{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"2OrtbFzOjVfpHo2KNrLqS9\",\"type\":\"Entry\",\"createdAt\":\"2026-07-20T16:25:39.884Z\",\"updatedAt\":\"2026-07-20T16:25:39.884Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":2,\"revision\":1,\"contentType\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"ContentType\",\"id\":\"blogCategory\"}},\"locale\":\"en-US\"},\"fields\":{\"name\":\"incident\",\"slug\":\"incident\",\"has_blogs\":false}}],\"featured_media\":{\"metadata\":{\"tags\":[],\"concepts\":[]},\"sys\":{\"space\":{\"sys\":{\"type\":\"Link\",\"linkType\":\"Space\",\"id\":\"p762jor363g1\"}},\"id\":\"3AiqgaI6uFPcF6zc7lYX5t\",\"type\":\"Asset\",\"createdAt\":\"2026-07-17T16:39:32.347Z\",\"updatedAt\":\"2026-07-17T16:39:32.347Z\",\"environment\":{\"sys\":{\"id\":\"spotify-engineering\",\"type\":\"Link\",\"linkType\":\"Environment\"}},\"publishedVersion\":3,\"revision\":1,\"locale\":\"en-US\"},\"fields\":{\"title\":\"incident report\",\"description\":\"\",\"file\":{\"url\":\"//images.ctfassets.net/p762jor363g1/3AiqgaI6uFPcF6zc7lYX5t/bbed1b145db94423441163d9f149a02e/incident_report.png\",\"details\":{\"size\":18420,\"image\":{\"width\":1200,\"height\":630}},\"fileName\":\"incident report.png\",\"contentType\":\"image/png\"}}},\"social_media\":\"$20:props:blogPost:fields:featured_media\",\"is_sticky\":true,\"is_old\":false,\"author\":\"Jim Whitehead (Director, Engineering), Ulrik Mikaelsson (Staff Engineer), John Lagomarsino (Product Manager II), and Saunak Jai Chakrabarti (Senior Director, Engineering)\"}}}]\n"])</script></body></html>