370 lines
21 KiB
HTML
370 lines
21 KiB
HTML
<!DOCTYPE html>
|
|
|
|
|
|
<html class="no-js" lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<title>What can developers learn from being on call?</title>
|
|
<meta name="author" content="Julia Evans">
|
|
<meta name="HandheldFriendly" content="True">
|
|
<meta name="MobileOptimized" content="320">
|
|
<meta name="description" content="What can developers learn from being on call?">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
|
|
<meta property="og:title" content='What can developers learn from being on call?'>
|
|
<meta property="og:type" content="website" />
|
|
<meta property="og:url" content="https://jvns.ca/blog/2017/06/18/operate-your-software/" />
|
|
<meta property="og:site_name" content="Julia Evans" />
|
|
|
|
<link rel="canonical" href="https://jvns.ca/blog/2017/06/18/operate-your-software/">
|
|
<link href="/favicon.ico" rel="icon">
|
|
|
|
<link href="/stylesheets/screen.css" rel="preload" type="text/css" as="style">
|
|
|
|
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
|
|
<link href="/stylesheets/print.css" media="print" rel="stylesheet" type="text/css">
|
|
|
|
|
|
<link href="/atom.xml" rel="alternate" title="Julia Evans" type="application/atom+xml">
|
|
|
|
|
|
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css" integrity="sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0" crossorigin="anonymous">
|
|
<script defer data-domain="jvns.ca" src="https://plausible.io/js/script.js"></script>
|
|
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js" integrity="sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4" crossorigin="anonymous"></script>
|
|
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
|
|
|
|
<script defer type="text/javascript">
|
|
window.heap=window.heap||[],heap.load=function(e,t){window.heap.appid=e,window.heap.config=t=t||{};var r=document.createElement("script");r.type="text/javascript",r.async=!0,r.src="https://cdn.heapanalytics.com/js/heap-"+e+".js";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(r,a);for(var n=function(e){return function(){heap.push([e].concat(Array.prototype.slice.call(arguments,0)))}},p=["addEventProperties","addUserProperties","clearEventProperties","identify","resetIdentity","removeEventProperty","setEventProperties","track","unsetEventProperty"],o=0;o<p.length;o++)heap[p[o]]=n(p[o])};
|
|
heap.load("2242143965");
|
|
</script>
|
|
</head>
|
|
<body>
|
|
<div id="skiptocontent">
|
|
<a href="#main">Skip to main content</a>
|
|
</div>
|
|
<div id="wrap">
|
|
<header role="banner">
|
|
<hgroup>
|
|
<h1><a href="/">Julia Evans</a></h1>
|
|
</hgroup>
|
|
<ul class="header-links">
|
|
<li><a href="/about">About</a></li>
|
|
<li><a href="/talks">Talks</a></li>
|
|
<li><a href="/projects/">Projects</a></li>
|
|
<li><a rel="me" href="https://social.jvns.ca/@b0rk">Mastodon</a></li>
|
|
<li><a href="https://bsky.app/profile/b0rk.jvns.ca">Bluesky</a></li>
|
|
<li><a href="https://github.com/jvns">Github</a></li>
|
|
</ul>
|
|
</header>
|
|
<nav role="navigation" class="header-nav"><ul class="main-navigation">
|
|
<li><a href="/categories/favorite/">Favorites</a></li>
|
|
<li><a href="/til/">TIL</a></li>
|
|
<li><a href="https://wizardzines.com">Zines</a></li>
|
|
<li class="subscription" data-subscription="rss"><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
|
|
</ul>
|
|
</nav>
|
|
<div id="main">
|
|
<div id="content">
|
|
|
|
|
|
<div>
|
|
<article class="hentry" role="article">
|
|
<header>
|
|
<h1 class="entry-title">What can developers learn from being on call?</h1>
|
|
|
|
<div class="post-tags">
|
|
|
|
</div>
|
|
<p class="meta sans">
|
|
<time class="date" datetime="2017-06-18T00:31:45" pubdate data-updated="true">
|
|
|
|
June 18, 2017
|
|
|
|
</time>
|
|
</p>
|
|
</header>
|
|
<main>
|
|
<p>We often talk about being on call as being a bad thing. For example, the night
|
|
before I wrote this my phone woke me up in the middle of the night because
|
|
something went wrong on a computer. That’s no fun! I was grumpy.</p>
|
|
<p>In this post, though, we’re going to talk about what you can learn from being
|
|
on call and how it can make you a better software engineer!. And to learn from
|
|
being on call you don’t necessarily need to get woken up in the middle of the
|
|
night. By “being on call”, here, I mean “being responsible for your code when
|
|
it breaks”. It could mean waking up to issues that happened overnight and
|
|
needing to fix them during your workday!</p>
|
|
<p>Everything in here is synthesized from an amazing Twitter thread by Charity Majors where she asked “How has being on call made you a better engineer?”: <a href="https://twitter.com/mipsytipsy/status/847508734188191745">https://twitter.com/mipsytipsy/status/847508734188191745</a></p>
|
|
<h3 id="learn-what-kinds-of-production-problems-are-common-and-uncommon" class="post-heading">
|
|
<a href="#learn-what-kinds-of-production-problems-are-common-and-uncommon">
|
|
Learn what kinds of production problems are common and uncommon
|
|
</a>
|
|
</h3>
|
|
<p>When you’re designing a system, you need to design for its error cases! When I
|
|
was just starting out as an engineer, I found coming up with error cases really
|
|
hard. ANYTHING could go wrong! But it’s important to have a better model of
|
|
system failure than “anything could go wrong, protect against everything!”
|
|
because often you have to prioritize where to spend your time, and you should
|
|
spend your time worrying about edge cases that are actually likely to happen.</p>
|
|
<p>Being on call can teach you very fast what kinds of edge cases your system runs
|
|
into frequently!</p>
|
|
<p>For example, after seeing some software fail, I know that DNS queries can fail.
|
|
It’s useful to have error handling for DNS queries (and network requests in
|
|
general), even if you think the servers you’re talking to are mostly reliable!</p>
|
|
<p>I also know that in principle RAM can be faulty (when you set a value in memory, it can get set to something else!) but it’s not something that’s ever happened to me in practice (yet!) so I worry about it less. (this might be because servers use ECC memory?) This post <a href="https://googleprojectzero.blogspot.ca/2015/03/exploiting-dram-rowhammer-bug-to-gain.html">Exploiting the DRAM rowhammer bug to gain kernel privileges</a> is a good example about how you can use RAM being faulty to make an exploit.</p>
|
|
<h3 id="learn-to-build-in-monitoring-and-diagnostics-early" class="post-heading">
|
|
<a href="#learn-to-build-in-monitoring-and-diagnostics-early">
|
|
Learn to build in monitoring and diagnostics early
|
|
</a>
|
|
</h3>
|
|
<p>There’s nothing quite like a system breaking, being in charge of fixing it, and
|
|
having no way of seeing what’s wrong to convince you of the value of building
|
|
monitoring and logging into your application.</p>
|
|
<p>Being on call will teach you quickly what <em>kinds</em> of diagnostics you need to
|
|
debug your application. If you get paged because your application is taking an
|
|
abnormally long time to make database queries, you can start monitoring how
|
|
long your database queries take! Then next time it’ll be much easier for the
|
|
person on call to see if that’s the problem.</p>
|
|
<p>The great thing about this is that these lessons last even beyond your current
|
|
on-call rotations – you can notice “hey, every time I write a program I end up
|
|
logging how long its database queries take, I’ll just put that in at the
|
|
beginning this time!”</p>
|
|
<h3 id="understand-the-parts-of-the-system-that-aren-t-yours" class="post-heading">
|
|
<a href="#understand-the-parts-of-the-system-that-aren-t-yours">
|
|
Understand the parts of the system that aren’t yours
|
|
</a>
|
|
</h3>
|
|
<p>It’s easy to think of the parts of the system you don’t own as a black box. “I
|
|
just make database queries and they work, it’s fine, the database team is in
|
|
charge of the database”.</p>
|
|
<p>But it’s actually incredibly useful to have a basic understanding of the
|
|
limitations of the systems you work with! If you’re working on the backend for
|
|
a web application, you want to know how many queries it’s okay to make to your
|
|
database, approximately how much network bandwidth you have to work with, how
|
|
much it’s okay to write to disk, and more.</p>
|
|
<p>If you get paged because your application is making too many database queries,
|
|
this is an awesome opportunity to learn more about the limitations of the
|
|
database you use! And then (can you see a pattern here?) the next time you work
|
|
on something that makes a lot of database queries, you can check up front to
|
|
make sure that it’s okay.</p>
|
|
<h3 id="gain-confidence-in-your-judgement" class="post-heading">
|
|
<a href="#gain-confidence-in-your-judgement">
|
|
Gain confidence in your judgement
|
|
</a>
|
|
</h3>
|
|
<p>A couple great quotes from this thread:</p>
|
|
<blockquote>
|
|
<p>It helped me gain confidence in my own judgment. You have to make big calls, take scary actions, live through terrible decisions.</p>
|
|
</blockquote>
|
|
<blockquote>
|
|
<p>I stop second guessing myself. If I’m getting paged, shits down and broken hard - no time to second guess yourself.</p>
|
|
</blockquote>
|
|
<h3 id="learn-what-needs-urgent-attention" class="post-heading">
|
|
<a href="#learn-what-needs-urgent-attention">
|
|
Learn what needs urgent attention
|
|
</a>
|
|
</h3>
|
|
<p>Some problems need to be fixed RIGHT NOW, and other problems… really don’t.
|
|
It used to be really mysterious to me how some engineers could just tell you
|
|
“yeah, that’s not a big deal” and be.. right about it?</p>
|
|
<p>This intuition is really important to build (otherwise you’ll panic every time
|
|
there’s an error and you’ll never get anything done!). When you’re on call for
|
|
a system, you see the urgent problems when they happen and you understand what
|
|
causes them. So you slowly gain intuition for “oh, okay, when X happens it
|
|
often causes a serious issue, but when Y happens it’s not a big deal”.</p>
|
|
<p>This also lets you prevent upcoming problems proactively – if you see
|
|
something worrisome happening, you can fix it before anyone on your team has to
|
|
be woken up in the middle of the night.</p>
|
|
<h3 id="learn-to-design-reliable-systems" class="post-heading">
|
|
<a href="#learn-to-design-reliable-systems">
|
|
Learn to design reliable systems
|
|
</a>
|
|
</h3>
|
|
<p>There’s been a common thread through all of this. A huge part of our jobs as
|
|
software engineers is to design systems that continues working for your
|
|
customers even when things don’t happen quite as your expected. A great way to
|
|
learn how to design for failure is to be on call for your software.</p>
|
|
<p>Kamal pointed out to me that it’s easy to have a system where the code is fine
|
|
(not too many bugs, etc), but because of some fundamental design choice it
|
|
doesn’t run well in production. For example, you could design a system which
|
|
needs to make many database queries every time a user makes a request. So
|
|
having a good understanding of the production implications of different design
|
|
choices will help you design better systems!</p>
|
|
<h3 id="learn-how-to-make-minimum-effective-change" class="post-heading">
|
|
<a href="#learn-how-to-make-minimum-effective-change">
|
|
Learn how to make minimum effective change
|
|
</a>
|
|
</h3>
|
|
<p>When there’s an
|
|
<a href="https://increment.com/on-call/when-the-pager-goes-off/">incident</a>, you want to
|
|
stabilize the system before fixing the root cause (ok, this server is on FIRE,
|
|
can we just divert traffic away from it before figuring out why?)! This is a
|
|
useful skill when you’re being paged, but also when you have a system that
|
|
needs help but don’t necessarily have the time/resources to completely fix it
|
|
right now.</p>
|
|
<h3 id="learn-about-distributed-systems-consistency-race-conditions" class="post-heading">
|
|
<a href="#learn-about-distributed-systems-consistency-race-conditions">
|
|
Learn about distributed systems & consistency & race conditions
|
|
</a>
|
|
</h3>
|
|
<blockquote>
|
|
<p>Being on call has taught me about race conditions</p>
|
|
</blockquote>
|
|
<p>Recently I got an alert that a job I’d written was failing. I looked at it for
|
|
a while, and then I realized “oh, this is happening because S3 list operations
|
|
are eventually consistent” – my code was listing a prefix in S3, and the
|
|
result it was getting wasn’t up to date. (and “eventually consistent” here
|
|
really means “eventually” – apparently sometimes you’ll add / delete an object
|
|
from an S3 bucket and it won’t show up in list operations for minutes)</p>
|
|
<p>This is how S3 is <em>supposed</em> to work, but I hadn’t really thought about that
|
|
when I wrote the code. Arguably I should have read the docs more carefully,
|
|
but seeing issues like this in practice helps me understand what “eventually
|
|
consistent” systems look like when they fail and remember to write my code with
|
|
that in mind next time.</p>
|
|
<h3 id="other-quotes-i-liked" class="post-heading">
|
|
<a href="#other-quotes-i-liked">
|
|
Other quotes I liked
|
|
</a>
|
|
</h3>
|
|
<blockquote>
|
|
<p>I’ve had teams that took on-call very seriously: each issue that paged
|
|
us was reviewed in a weekly meeting, and tasks were assigned to solve</p>
|
|
</blockquote>
|
|
<blockquote>
|
|
<p>The lesson for me is that processes are important, and working towards
|
|
continuous improvement is worth it.</p>
|
|
</blockquote>
|
|
<p>and</p>
|
|
<blockquote>
|
|
<p>Being on call means I can’t pick and choose favorite/comfortable subjects
|
|
avoiding hard/unhappy ones. I’m forced to stretch and learn.</p>
|
|
</blockquote>
|
|
<p>and</p>
|
|
<blockquote>
|
|
<p>Being able to put aside one’s pride and say “I need help with this
|
|
even though I’m waking someone up to help me.”</p>
|
|
</blockquote>
|
|
<p>and</p>
|
|
<blockquote>
|
|
<p>It made me much better at figuring out how to break up a complex
|
|
failure condition into smaller pieces that are easier to debug…</p>
|
|
</blockquote>
|
|
<h3 id="being-responsible-for-my-programs-operations-makes-me-a-better-developer" class="post-heading">
|
|
<a href="#being-responsible-for-my-programs-operations-makes-me-a-better-developer">
|
|
Being responsible for my programs’ operations makes me a better developer
|
|
</a>
|
|
</h3>
|
|
<p>I’ve never really worked in a world where I wrote software and threw it over
|
|
the wall to be operated by another team. But I do feel like writing software
|
|
and then seeing how it fails in practice has been a good experience! I feel
|
|
like it’s a great privilege to be able to write software and see how it
|
|
holds up in practice over the course of months/years.</p>
|
|
<p>That said – I’ve never been on a particularly arduous on-call rotation
|
|
personally, the most I’ve probably ever been paged is like.. 2-3 times per
|
|
week, once every 4 weeks. But I feel like I learned a lot from that still!</p>
|
|
<p>I’ve probably left out many important things here but I wrote this 2 months ago
|
|
and so it’s already being published far later than my usual “write this and
|
|
publish it within 4 hours”.</p>
|
|
|
|
</main>
|
|
|
|
<footer>
|
|
|
|
<style type="text/css">
|
|
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; display: inline;}
|
|
#mc_embed_signup {
|
|
display: inline;
|
|
}
|
|
#mc_embed_signup input.button {
|
|
background: #ff5e00;
|
|
display: inline;
|
|
color: white;
|
|
padding: 6px 12px;
|
|
}
|
|
</style>
|
|
<div class="sharing">
|
|
|
|
<style>
|
|
.form-inline {
|
|
display:flex; flex-flow: row wrap; justify-content: center;
|
|
}
|
|
.form-inline input, .form-inline span {
|
|
padding: 10px;
|
|
}
|
|
.form-inline input {
|
|
display:inline;
|
|
max-width:30%;
|
|
margin: 0 10px 0 0;
|
|
background-color: #fff;
|
|
border: 1px solid #ddd;
|
|
border-radius: 5px;
|
|
padding: 10px;
|
|
}
|
|
button {
|
|
background-color: #f50;
|
|
box-shadow: none;
|
|
border: 0;
|
|
border-radius: 5px;
|
|
color: white;
|
|
padding: 5px 10px;
|
|
}
|
|
@media (max-width: 800px) {
|
|
.form-inline input {
|
|
margin: 10px 0;
|
|
max-width:100% !important;
|
|
}
|
|
.form-inline {
|
|
flex-direction: column;
|
|
align-items: stretch;
|
|
}
|
|
}
|
|
</style>
|
|
|
|
<div align="center">
|
|
<form class="form-inline" action="https://app.convertkit.com/forms/1052396/subscriptions" method="post" data-uid="8884355abb" data-format="inline" data-version="5">
|
|
<span> Want a weekly digest of this blog?</span>
|
|
<input name="email_address" type="text" placeholder="Email address" />
|
|
<button type="submit" data-element="submit">Subscribe</button>
|
|
</form>
|
|
</div>
|
|
|
|
|
|
</div>
|
|
|
|
<p class="meta">
|
|
|
|
<a class="basic-alignment left" href="https://jvns.ca/blog/2017/06/17/allison-parrish/" title="Previous Post: Awesome NLP tutorials by Allison Parrish">Awesome NLP tutorials by Allison Parrish</a>
|
|
|
|
|
|
<a class="basic-alignment right" href="https://jvns.ca/blog/2017/06/26/vue-js-fun/" title="Next Post: a tiny whack-a-mole game">a tiny whack-a-mole game</a>
|
|
|
|
</p>
|
|
</footer>
|
|
|
|
</article>
|
|
</div>
|
|
|
|
</div>
|
|
</div>
|
|
<nav role="navigation" class="footer-nav"> <a href="/">Archives</a>
|
|
</nav>
|
|
<footer role="contentinfo"><span class="credit">© Julia Evans. </span>
|
|
<span>If you like this, you may like <a href="https://web.archive.org/web/20181228051203/http://www.uliaea.ca/">Ulia Ea</a> or, more seriously, this list of <a href="https://jvns.ca/blogroll">blogs I love</a> or some <a href="https://jvns.ca/bookshelf">books I've read</a>. <br>
|
|
<p class="rc-scout__text"><i class="rc-scout__logo"></i>
|
|
You might also like the <a class="rc-scout__link" href="https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8">Recurse Center</a>, my very favorite programming community <a href="/categories/hackerschool/">(my posts about it)</a></p>
|
|
</span>
|
|
<style class="rc-scout__style" type="text/css">.rc-scout{display:block;padding:0;border:0;margin:0;}.rc-scout__text{display:block;padding:0;border:0;margin:0;height:100%;font-size:100%;}.rc-scout__logo{display:inline-block;padding:0;border:0;margin:0;width:0.85em;height:0.85em;background:no-repeat center url('data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2012%2015%22%3E%3Crect%20x%3D%220%22%20y%3D%220%22%20width%3D%2212%22%20height%3D%2210%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%221%22%20width%3D%2210%22%20height%3D%228%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%222%22%20width%3D%228%22%20height%3D%226%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%229%22%20width%3D%224%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%2211%22%20width%3D%2210%22%20height%3D%224%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%220%22%20y%3D%2212%22%20width%3D%2212%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%225%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%227%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%228%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%229%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3C%2Fsvg%3E');}.rc-scout__link:link,.rc-scout__link:visited{color:#61ae24;text-decoration:underline;}.rc-scout__link:hover,.rc-scout__link:active{color:#4e8b1d;}</style>
|
|
</footer>
|
|
<script type="text/rocketscript">
|
|
(function(){
|
|
var twitterWidgets = document.createElement('script');
|
|
twitterWidgets.type = 'text/javascript';
|
|
twitterWidgets.async = true;
|
|
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
|
|
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
|
|
})();
|
|
</script>
|
|
</div>
|
|
</body>
|
|
</html>
|
|
|