Files
nexus/sreweekly/articles/78/02-what-can-developers-learn-from-being-on-call-julia-evans.html
2026-09-12 17:23:01 +08:00

370 lines
21 KiB
HTML

<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta charset="utf-8">
<title>What can developers learn from being on call?</title>
<meta name="author" content="Julia Evans">
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="description" content="What can developers learn from being on call?">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:title" content='What can developers learn from being on call?'>
<meta property="og:type" content="website" />
<meta property="og:url" content="https://jvns.ca/blog/2017/06/18/operate-your-software/" />
<meta property="og:site_name" content="Julia Evans" />
<link rel="canonical" href="https://jvns.ca/blog/2017/06/18/operate-your-software/">
<link href="/favicon.ico" rel="icon">
<link href="/stylesheets/screen.css" rel="preload" type="text/css" as="style">
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
<link href="/stylesheets/print.css" media="print" rel="stylesheet" type="text/css">
<link href="/atom.xml" rel="alternate" title="Julia Evans" type="application/atom+xml">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css" integrity="sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0" crossorigin="anonymous">
<script defer data-domain="jvns.ca" src="https://plausible.io/js/script.js"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js" integrity="sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4" crossorigin="anonymous"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
<script defer type="text/javascript">
window.heap=window.heap||[],heap.load=function(e,t){window.heap.appid=e,window.heap.config=t=t||{};var r=document.createElement("script");r.type="text/javascript",r.async=!0,r.src="https://cdn.heapanalytics.com/js/heap-"+e+".js";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(r,a);for(var n=function(e){return function(){heap.push([e].concat(Array.prototype.slice.call(arguments,0)))}},p=["addEventProperties","addUserProperties","clearEventProperties","identify","resetIdentity","removeEventProperty","setEventProperties","track","unsetEventProperty"],o=0;o<p.length;o++)heap[p[o]]=n(p[o])};
heap.load("2242143965");
</script>
</head>
<body>
<div id="skiptocontent">
<a href="#main">Skip to main content</a>
</div>
<div id="wrap">
<header role="banner">
<hgroup>
<h1><a href="/">Julia Evans</a></h1>
</hgroup>
<ul class="header-links">
<li><a href="/about">About</a></li>
<li><a href="/talks">Talks</a></li>
<li><a href="/projects/">Projects</a></li>
<li><a rel="me" href="https://social.jvns.ca/@b0rk">Mastodon</a></li>
<li><a href="https://bsky.app/profile/b0rk.jvns.ca">Bluesky</a></li>
<li><a href="https://github.com/jvns">Github</a></li>
</ul>
</header>
<nav role="navigation" class="header-nav"><ul class="main-navigation">
<li><a href="/categories/favorite/">Favorites</a></li>
<li><a href="/til/">TIL</a></li>
<li><a href="https://wizardzines.com">Zines</a></li>
<li class="subscription" data-subscription="rss"><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
</ul>
</nav>
<div id="main">
<div id="content">
<div>
<article class="hentry" role="article">
<header>
<h1 class="entry-title">What can developers learn from being on call?</h1>
<div class="post-tags">
</div>
<p class="meta sans">
<time class="date" datetime="2017-06-18T00:31:45" pubdate data-updated="true">
June 18, 2017
</time>
</p>
</header>
<main>
<p>We often talk about being on call as being a bad thing. For example, the night
before I wrote this my phone woke me up in the middle of the night because
something went wrong on a computer. That&rsquo;s no fun! I was grumpy.</p>
<p>In this post, though, we&rsquo;re going to talk about what you can learn from being
on call and how it can make you a better software engineer!. And to learn from
being on call you don&rsquo;t necessarily need to get woken up in the middle of the
night. By &ldquo;being on call&rdquo;, here, I mean &ldquo;being responsible for your code when
it breaks&rdquo;. It could mean waking up to issues that happened overnight and
needing to fix them during your workday!</p>
<p>Everything in here is synthesized from an amazing Twitter thread by Charity Majors where she asked &ldquo;How has being on call made you a better engineer?&rdquo;: <a href="https://twitter.com/mipsytipsy/status/847508734188191745">https://twitter.com/mipsytipsy/status/847508734188191745</a></p>
<h3 id="learn-what-kinds-of-production-problems-are-common-and-uncommon" class="post-heading">
<a href="#learn-what-kinds-of-production-problems-are-common-and-uncommon">
Learn what kinds of production problems are common and uncommon
</a>
</h3>
<p>When you&rsquo;re designing a system, you need to design for its error cases! When I
was just starting out as an engineer, I found coming up with error cases really
hard. ANYTHING could go wrong! But it&rsquo;s important to have a better model of
system failure than &ldquo;anything could go wrong, protect against everything!&rdquo;
because often you have to prioritize where to spend your time, and you should
spend your time worrying about edge cases that are actually likely to happen.</p>
<p>Being on call can teach you very fast what kinds of edge cases your system runs
into frequently!</p>
<p>For example, after seeing some software fail, I know that DNS queries can fail.
It&rsquo;s useful to have error handling for DNS queries (and network requests in
general), even if you think the servers you&rsquo;re talking to are mostly reliable!</p>
<p>I also know that in principle RAM can be faulty (when you set a value in memory, it can get set to something else!) but it&rsquo;s not something that&rsquo;s ever happened to me in practice (yet!) so I worry about it less. (this might be because servers use ECC memory?) This post <a href="https://googleprojectzero.blogspot.ca/2015/03/exploiting-dram-rowhammer-bug-to-gain.html">Exploiting the DRAM rowhammer bug to gain kernel privileges</a> is a good example about how you can use RAM being faulty to make an exploit.</p>
<h3 id="learn-to-build-in-monitoring-and-diagnostics-early" class="post-heading">
<a href="#learn-to-build-in-monitoring-and-diagnostics-early">
Learn to build in monitoring and diagnostics early
</a>
</h3>
<p>There&rsquo;s nothing quite like a system breaking, being in charge of fixing it, and
having no way of seeing what&rsquo;s wrong to convince you of the value of building
monitoring and logging into your application.</p>
<p>Being on call will teach you quickly what <em>kinds</em> of diagnostics you need to
debug your application. If you get paged because your application is taking an
abnormally long time to make database queries, you can start monitoring how
long your database queries take! Then next time it&rsquo;ll be much easier for the
person on call to see if that&rsquo;s the problem.</p>
<p>The great thing about this is that these lessons last even beyond your current
on-call rotations &ndash; you can notice &ldquo;hey, every time I write a program I end up
logging how long its database queries take, I&rsquo;ll just put that in at the
beginning this time!&rdquo;</p>
<h3 id="understand-the-parts-of-the-system-that-aren-t-yours" class="post-heading">
<a href="#understand-the-parts-of-the-system-that-aren-t-yours">
Understand the parts of the system that aren&rsquo;t yours
</a>
</h3>
<p>It&rsquo;s easy to think of the parts of the system you don&rsquo;t own as a black box. &ldquo;I
just make database queries and they work, it&rsquo;s fine, the database team is in
charge of the database&rdquo;.</p>
<p>But it&rsquo;s actually incredibly useful to have a basic understanding of the
limitations of the systems you work with! If you&rsquo;re working on the backend for
a web application, you want to know how many queries it&rsquo;s okay to make to your
database, approximately how much network bandwidth you have to work with, how
much it&rsquo;s okay to write to disk, and more.</p>
<p>If you get paged because your application is making too many database queries,
this is an awesome opportunity to learn more about the limitations of the
database you use! And then (can you see a pattern here?) the next time you work
on something that makes a lot of database queries, you can check up front to
make sure that it&rsquo;s okay.</p>
<h3 id="gain-confidence-in-your-judgement" class="post-heading">
<a href="#gain-confidence-in-your-judgement">
Gain confidence in your judgement
</a>
</h3>
<p>A couple great quotes from this thread:</p>
<blockquote>
<p>It helped me gain confidence in my own judgment. You have to make big calls, take scary actions, live through terrible decisions.</p>
</blockquote>
<blockquote>
<p>I stop second guessing myself. If I&rsquo;m getting paged, shits down and broken hard - no time to second guess yourself.</p>
</blockquote>
<h3 id="learn-what-needs-urgent-attention" class="post-heading">
<a href="#learn-what-needs-urgent-attention">
Learn what needs urgent attention
</a>
</h3>
<p>Some problems need to be fixed RIGHT NOW, and other problems&hellip; really don&rsquo;t.
It used to be really mysterious to me how some engineers could just tell you
&ldquo;yeah, that&rsquo;s not a big deal&rdquo; and be.. right about it?</p>
<p>This intuition is really important to build (otherwise you&rsquo;ll panic every time
there&rsquo;s an error and you&rsquo;ll never get anything done!). When you&rsquo;re on call for
a system, you see the urgent problems when they happen and you understand what
causes them. So you slowly gain intuition for &ldquo;oh, okay, when X happens it
often causes a serious issue, but when Y happens it&rsquo;s not a big deal&rdquo;.</p>
<p>This also lets you prevent upcoming problems proactively &ndash; if you see
something worrisome happening, you can fix it before anyone on your team has to
be woken up in the middle of the night.</p>
<h3 id="learn-to-design-reliable-systems" class="post-heading">
<a href="#learn-to-design-reliable-systems">
Learn to design reliable systems
</a>
</h3>
<p>There&rsquo;s been a common thread through all of this. A huge part of our jobs as
software engineers is to design systems that continues working for your
customers even when things don&rsquo;t happen quite as your expected. A great way to
learn how to design for failure is to be on call for your software.</p>
<p>Kamal pointed out to me that it&rsquo;s easy to have a system where the code is fine
(not too many bugs, etc), but because of some fundamental design choice it
doesn&rsquo;t run well in production. For example, you could design a system which
needs to make many database queries every time a user makes a request. So
having a good understanding of the production implications of different design
choices will help you design better systems!</p>
<h3 id="learn-how-to-make-minimum-effective-change" class="post-heading">
<a href="#learn-how-to-make-minimum-effective-change">
Learn how to make minimum effective change
</a>
</h3>
<p>When there&rsquo;s an
<a href="https://increment.com/on-call/when-the-pager-goes-off/">incident</a>, you want to
stabilize the system before fixing the root cause (ok, this server is on FIRE,
can we just divert traffic away from it before figuring out why?)! This is a
useful skill when you&rsquo;re being paged, but also when you have a system that
needs help but don&rsquo;t necessarily have the time/resources to completely fix it
right now.</p>
<h3 id="learn-about-distributed-systems-consistency-race-conditions" class="post-heading">
<a href="#learn-about-distributed-systems-consistency-race-conditions">
Learn about distributed systems &amp; consistency &amp; race conditions
</a>
</h3>
<blockquote>
<p>Being on call has taught me about race conditions</p>
</blockquote>
<p>Recently I got an alert that a job I&rsquo;d written was failing. I looked at it for
a while, and then I realized &ldquo;oh, this is happening because S3 list operations
are eventually consistent&rdquo; &ndash; my code was listing a prefix in S3, and the
result it was getting wasn&rsquo;t up to date. (and &ldquo;eventually consistent&rdquo; here
really means &ldquo;eventually&rdquo; &ndash; apparently sometimes you&rsquo;ll add / delete an object
from an S3 bucket and it won&rsquo;t show up in list operations for minutes)</p>
<p>This is how S3 is <em>supposed</em> to work, but I hadn&rsquo;t really thought about that
when I wrote the code. Arguably I should have read the docs more carefully,
but seeing issues like this in practice helps me understand what &ldquo;eventually
consistent&rdquo; systems look like when they fail and remember to write my code with
that in mind next time.</p>
<h3 id="other-quotes-i-liked" class="post-heading">
<a href="#other-quotes-i-liked">
Other quotes I liked
</a>
</h3>
<blockquote>
<p>I&rsquo;ve had teams that took on-call very seriously: each issue that paged
us was reviewed in a weekly meeting, and tasks were assigned to solve</p>
</blockquote>
<blockquote>
<p>The lesson for me is that processes are important, and working towards
continuous improvement is worth it.</p>
</blockquote>
<p>and</p>
<blockquote>
<p>Being on call means I can&rsquo;t pick and choose favorite/comfortable subjects
avoiding hard/unhappy ones. I&rsquo;m forced to stretch and learn.</p>
</blockquote>
<p>and</p>
<blockquote>
<p>Being able to put aside one&rsquo;s pride and say &ldquo;I need help with this
even though I&rsquo;m waking someone up to help me.&rdquo;</p>
</blockquote>
<p>and</p>
<blockquote>
<p>It made me much better at figuring out how to break up a complex
failure condition into smaller pieces that are easier to debug&hellip;</p>
</blockquote>
<h3 id="being-responsible-for-my-programs-operations-makes-me-a-better-developer" class="post-heading">
<a href="#being-responsible-for-my-programs-operations-makes-me-a-better-developer">
Being responsible for my programs&rsquo; operations makes me a better developer
</a>
</h3>
<p>I&rsquo;ve never really worked in a world where I wrote software and threw it over
the wall to be operated by another team. But I do feel like writing software
and then seeing how it fails in practice has been a good experience! I feel
like it&rsquo;s a great privilege to be able to write software and see how it
holds up in practice over the course of months/years.</p>
<p>That said &ndash; I&rsquo;ve never been on a particularly arduous on-call rotation
personally, the most I&rsquo;ve probably ever been paged is like.. 2-3 times per
week, once every 4 weeks. But I feel like I learned a lot from that still!</p>
<p>I&rsquo;ve probably left out many important things here but I wrote this 2 months ago
and so it&rsquo;s already being published far later than my usual &ldquo;write this and
publish it within 4 hours&rdquo;.</p>
</main>
<footer>
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; display: inline;}
#mc_embed_signup {
display: inline;
}
#mc_embed_signup input.button {
background: #ff5e00;
display: inline;
color: white;
padding: 6px 12px;
}
</style>
<div class="sharing">
<style>
.form-inline {
display:flex; flex-flow: row wrap; justify-content: center;
}
.form-inline input, .form-inline span {
padding: 10px;
}
.form-inline input {
display:inline;
max-width:30%;
margin: 0 10px 0 0;
background-color: #fff;
border: 1px solid #ddd;
border-radius: 5px;
padding: 10px;
}
button {
background-color: #f50;
box-shadow: none;
border: 0;
border-radius: 5px;
color: white;
padding: 5px 10px;
}
@media (max-width: 800px) {
.form-inline input {
margin: 10px 0;
max-width:100% !important;
}
.form-inline {
flex-direction: column;
align-items: stretch;
}
}
</style>
<div align="center">
<form class="form-inline" action="https://app.convertkit.com/forms/1052396/subscriptions" method="post" data-uid="8884355abb" data-format="inline" data-version="5">
<span> Want a weekly digest of this blog?</span>
<input name="email_address" type="text" placeholder="Email address" />
<button type="submit" data-element="submit">Subscribe</button>
</form>
</div>
</div>
<p class="meta">
<a class="basic-alignment left" href="https://jvns.ca/blog/2017/06/17/allison-parrish/" title="Previous Post: Awesome NLP tutorials by Allison Parrish">Awesome NLP tutorials by Allison Parrish</a>
<a class="basic-alignment right" href="https://jvns.ca/blog/2017/06/26/vue-js-fun/" title="Next Post: a tiny whack-a-mole game">a tiny whack-a-mole game</a>
</p>
</footer>
</article>
</div>
</div>
</div>
<nav role="navigation" class="footer-nav"> <a href="/">Archives</a>
</nav>
<footer role="contentinfo"><span class="credit">&copy; Julia Evans. </span>
<span>If you like this, you may like <a href="https://web.archive.org/web/20181228051203/http://www.uliaea.ca/">Ulia Ea</a> or, more seriously, this list of <a href="https://jvns.ca/blogroll">blogs I love</a> or some <a href="https://jvns.ca/bookshelf">books I've read</a>. <br>
<p class="rc-scout__text"><i class="rc-scout__logo"></i>
You might also like the <a class="rc-scout__link" href="https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8">Recurse Center</a>, my very favorite programming community <a href="/categories/hackerschool/">(my posts about it)</a></p>
</span>
<style class="rc-scout__style" type="text/css">.rc-scout{display:block;padding:0;border:0;margin:0;}.rc-scout__text{display:block;padding:0;border:0;margin:0;height:100%;font-size:100%;}.rc-scout__logo{display:inline-block;padding:0;border:0;margin:0;width:0.85em;height:0.85em;background:no-repeat center url('data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2012%2015%22%3E%3Crect%20x%3D%220%22%20y%3D%220%22%20width%3D%2212%22%20height%3D%2210%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%221%22%20width%3D%2210%22%20height%3D%228%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%222%22%20width%3D%228%22%20height%3D%226%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%229%22%20width%3D%224%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%2211%22%20width%3D%2210%22%20height%3D%224%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%220%22%20y%3D%2212%22%20width%3D%2212%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%225%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%227%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%228%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%229%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3C%2Fsvg%3E');}.rc-scout__link:link,.rc-scout__link:visited{color:#61ae24;text-decoration:underline;}.rc-scout__link:hover,.rc-scout__link:active{color:#4e8b1d;}</style>
</footer>
<script type="text/rocketscript">
(function(){
var twitterWidgets = document.createElement('script');
twitterWidgets.type = 'text/javascript';
twitterWidgets.async = true;
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
})();
</script>
</div>
</body>
</html>