1347 lines
23 KiB
HTML
1347 lines
23 KiB
HTML
<!DOCTYPE html>
|
||
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en-us">
|
||
<head>
|
||
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>What is Backoff For? - Marc's Blog</title>
|
||
<meta name="author" content="Marc Brooker" />
|
||
|
||
<!-- Homepage CSS -->
|
||
<link rel="stylesheet" href="/blog/css/screen.css" type="text/css" media="screen, projection" />
|
||
<link rel="stylesheet" href="/blog/css/syntax.css" type="text/css" media="screen, projection" />
|
||
|
||
</head>
|
||
<body>
|
||
|
||
<div class="site">
|
||
<div class="title">
|
||
<h1><a href="/blog/">Marc's Blog</a></h1>
|
||
</div>
|
||
|
||
<div class="about">
|
||
<h1>About Me</h1>
|
||
My name is Marc Brooker. I like to build things that work, and do cool stuff. I like building big things. I also dabble in machining, welding, cooking, and skiing.<br/><br/>
|
||
|
||
I am an engineer at Amazon Web Services (AWS) in Seattle, where I work on agentic AI, especially safety and policy for agentic AI. Before that, I worked on EC2, EBS, databases, serverless, and serverless databases.<br/>
|
||
|
||
All opinions are my own.
|
||
<h1>Links</h1>
|
||
<a href="https://brooker.co.za/blog/publications.html">My Publications and Videos</a><br/>
|
||
<a rel="me" href="https://fediscience.org/@marcbrooker">@marcbrooker on Mastodon</a>
|
||
<a href="https://twitter.com/MarcJBrooker">@MarcJBrooker on Twitter</a>
|
||
|
||
<br/><br/><br/>
|
||
<a href="https://brooker.co.za/blog/2026/06/18/my-blog-and-ai.html">Is this blog written by AI?</a>
|
||
</div>
|
||
|
||
|
||
<div id="post">
|
||
<h1 id="what-is-backoff-for">What is Backoff For?</h1>
|
||
|
||
<p class="meta">Back off man, I'm a scientist.</p>
|
||
|
||
<p>Years ago I wrote a blog post about <a href="https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/">exponential backoff and jitter</a>, which has turned out to be enduringly popular. I like to believe that it’s influenced at least a couple of systems to add jitter, and become more stable. However, I do feel a little guilty about pushing the popularity of jitter without clearly explaining what backoff and jitter do, and do not do.</p>
|
||
|
||
<p>Here’s the pithy statement about backoff:</p>
|
||
|
||
<p><em>Backoff helps in the short term. It is only valuable in the long term if it reduces total work</em>.</p>
|
||
|
||
<p>Consider a system that suffers from <em>short spikes</em><sup><a href="#foot4">4</a></sup> of overload. That could be a flash sale, top-of-hour operational work, recovery after a brief network partition, etc. During the overload, some calls are failing, primarily due to the overload itself. Backing off in this case is extremely helpful: it spreads out the spike, and reduces the amount of chatter between clients and servers. Jitter is especially effective at helping broaden spikes. If you want to see this in action, look at the time series in <a href="https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/">the exponential backoff and jitter post</a>.</p>
|
||
|
||
<p>Now, consider a <em>long spike</em> of overload. There are two cases here.</p>
|
||
|
||
<p>One is where we have a large (maybe effectively unlimited) number of clients, and they’re independently sending work. For example, think about a website with a million customers, each visiting it about once a day. Each client backing off in this case <em>does not help</em>, because it does not reduce the work being sent to the system. Each client is still going to press F5 the same number of times, so delaying their presses doesn’t help.</p>
|
||
|
||
<p>The other is where we have a smaller number of clients, each sending a serial stream of requests. For example, think of a fleet of workers polling a queue. Each client backing off in this case <em>helps a lot</em> because they are serial. Backing off means they send less work, and less is asked of the service<sup><a href="#foot1">1</a></sup>.</p>
|
||
|
||
<p>That may sound like a subtle distinction, but the bottom line is this: does backoff actually reduce the work done? In the case of lots of clients, it doesn’t, because each new client entering the system doesn’t know other have backed off.</p>
|
||
|
||
<p><em>The only way to deal with long-term overload is to reduce load, deferring load does not work.</em></p>
|
||
|
||
<p>Now, on to retries. As I wrote about in <a href="https://brooker.co.za/blog/2022/02/28/retries.html">Fixing retries with token buckets and circuit breakers</a>, retries have an amplifying effect on the work a service is asked to do. During long-term overload, retries may increase the work to be done. The way to fix that is with a good retry policy, such as the token bucket based adaptive strategy described in the post.</p>
|
||
|
||
<p><em>Backoff is not a substitute for a good retry policy.</em></p>
|
||
|
||
<p>Backoff is not a good retry policy. Or, at least, is hard to use as one.</p>
|
||
|
||
<p>Backoff is only a good retry policy in systems with small numbers of sequential clients, where the introduced delay between retries delays <em>future first tries</em>. If this property is not true, and the next <em>first try</em> is going to come along at a time independent of retry backoff, then backing off retries <em>does nothing</em> to help long-term overload. It just defers work to a future time<sup><a href="#foot2">2</a></sup>.</p>
|
||
|
||
<p>That doesn’t mean that backing off between retries is a bad idea. It’s a good idea, but only helps for <em>short term</em> overload (spikes, etc). It does not reduce the total work in the system, or total amplification factor of retries.</p>
|
||
|
||
<p><em>A good approach to retries combines backoff, jitter, and a good retry policy.</em></p>
|
||
|
||
<p>These are complimentary mechanisms, and neither solves the whole problem.</p>
|
||
|
||
<p><strong>First tries and second tries</strong></p>
|
||
|
||
<p>The other way to think about this is to think about <em>first tries</em> and <em>second tries</em>. A first try is the first time a given client tries to do a piece of work. There are two ways systems can get overloaded: <em>too many first tries</em> and <em>too many second tries</em>.</p>
|
||
|
||
<p>If you have too many first tries, you need to have fewer. If you’ve got a bounded number of clients, getting each of them to back off is an effective strategy<sup><a href="#foot3">3</a></sup>. With a bounded number of clients, backoff is an effective way to do that. If you have an unbounded number of clients, backoff is not an effective way to do that. They only hear the bad news after their first try, so no amount of backoff will reduce their first try rate.</p>
|
||
|
||
<p>If you’ve got an OK number of first tries, but some error rate is driving up second try (retry traffic), then you need to reduce the number of second tries. Backoff is an effective way to reduce the number of second tries <em>now</em>, by <em>deferring them into the future</em>. If you think you’ll be able to handle them better in the future, that’s a win. But backoff is not an effective way to reduce the overall number of second tries <em>in total</em> for long-running overload. For that, you need something like the adaptive retry approach.</p>
|
||
|
||
<p>Unless, of course, your clients are relatively small in number, and their next <em>first try</em> is only going to get made after this round of <em>second tries</em> is done. Then backoff will reduce the overall rate of <em>second tries</em>.</p>
|
||
|
||
<p>This ‘number of clients’ thinking can be a bit confusing, because it’s not really about number of clients. Its about the number of parallel things doing work. Code that spawns a thread for each call, or dispatch each call to an event loop, can become an effectively unbounded number of things.</p>
|
||
|
||
<p><strong>Simulation Results</strong></p>
|
||
|
||
<p>We can validate these assertions by looking at some simulation results. First, let’s look at a simulation that compares four strategies: three retries with and without backoff, and adaptive with and without backoff, for a case with a very large number of clients. What we see in these results matches the assertions: in this case, which reflects a long-running overload with an unbounded number of clients, per-request retry backoff has nearly no effect on traffic amplification.</p>
|
||
|
||
<p>Again, the unbounded number of clients can mean lots of clients, or just clients that spawn threads or async work for requests. The important property is whether they wait on a request to be done (or fail) before they do the next one.</p>
|
||
|
||
<p><img src="https://mbrooker-blog-images.s3.amazonaws.com/backoff_sim_results.png" alt="" /></p>
|
||
|
||
<p>But the news is not all bad. As soon as we have a limited number of serial clients, we see that backoff is effective at avoiding amplification of retries, <em>and</em> at reducing the number of first tries. In this case, backoff is very effective at improving the behavior of the system. The results for adaptive retry are similar, and show that backoff is similarly useful.</p>
|
||
|
||
<p><img src="https://mbrooker-blog-images.s3.amazonaws.com/limited_client_backoff_results.png" alt="" /></p>
|
||
|
||
<p><strong>Footnotes</strong></p>
|
||
|
||
<ol>
|
||
<li><a name="foot1"></a> You may be wondering here about famous and super successful systems like TCP and Ethernet CSMA/CD which use backoff approaches like exponential backoff and AIMD very effectively. The same reasoning applies to them: their backoff strategies are only effective because the number of clients is relatively small, and slowing clients down reduces overall work in the system (helping it find a new dynamic set point).</li>
|
||
<li><a name="foot2"></a> It may even delay recovery after overload because those deferred backoffs are a kind of implicit queue of work that needs to be done before the system is fully recovered.</li>
|
||
<li><a name="foot3"></a> Again, this is what TCP does.</li>
|
||
<li><a name="foot4"></a> I keep saying <em>short</em> and <em>long</em> without a lot of details. Roughly, <em>short</em> means a time approximately around the time clients are willing to wait, and <em>long</em> is longer than that.</li>
|
||
</ol>
|
||
|
||
</div>
|
||
|
||
<div id="related">
|
||
« <a href="/blog">Back to the blog index</a><br>
|
||
<br>
|
||
<!-- Similar Posts -->
|
||
<h4>Similar Posts</h4>
|
||
<ul class="posts">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<li><span>21 Mar 2015</span> » <a href="/blog/2015/03/21/backoff.html">Jitter: Making Things Better With Randomness</a></li>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<li><span>28 Feb 2022</span> » <a href="/blog/2022/02/28/retries.html">Fixing retries with token buckets and circuit breakers</a></li>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<li><span>16 Feb 2022</span> » <a href="/blog/2022/02/16/circuit-breakers.html">Will circuit breakers solve my problems?</a></li>
|
||
|
||
|
||
|
||
|
||
|
||
</ul>
|
||
|
||
<!-- Dissimilar Posts -->
|
||
|
||
<h4>Something Completely Different</h4>
|
||
<ul class="posts">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<li><span>11 Aug 2021</span> » <a href="/blog/2021/08/11/arecibo.html">My Proposal for Arecibo: Drones</a></li>
|
||
|
||
|
||
|
||
|
||
</ul>
|
||
|
||
</div>
|
||
|
||
|
||
<div class="footer">
|
||
<div class="contact">
|
||
<p>
|
||
Marc Brooker<br />
|
||
The opinions on this site are my own. They do not necessarily represent those of my employer.<br />
|
||
marcbrooker@gmail.com
|
||
</p>
|
||
<p>
|
||
<a href="https://brooker.co.za/blog/rss.xml"><img src="/blog/images/feed-icon-14x14.png" /> RSS</a>
|
||
<a href="https://brooker.co.za/blog/atom.xml"><img src="/blog/images/feed-icon-14x14.png" /> Atom</a>
|
||
</p>
|
||
</div>
|
||
<div class="license">
|
||
<!-- <a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a><br /> -->
|
||
This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
</body>
|
||
</html>
|