262 lines
18 KiB
HTML
262 lines
18 KiB
HTML
<!DOCTYPE html>
|
|
|
|
|
|
<html class="no-js" lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<title>Operations for software developers for beginners</title>
|
|
<meta name="author" content="Julia Evans">
|
|
<meta name="HandheldFriendly" content="True">
|
|
<meta name="MobileOptimized" content="320">
|
|
<meta name="description" content="Operations for software developers for beginners">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
|
|
<meta property="og:title" content='Operations for software developers for beginners'>
|
|
<meta property="og:type" content="website" />
|
|
<meta property="og:url" content="https://jvns.ca/blog/2016/10/15/operations-for-software-developers-for-beginners/" />
|
|
<meta property="og:site_name" content="Julia Evans" />
|
|
|
|
<link rel="canonical" href="https://jvns.ca/blog/2016/10/15/operations-for-software-developers-for-beginners/">
|
|
<link href="/favicon.ico" rel="icon">
|
|
|
|
<link href="/stylesheets/screen.css" rel="preload" type="text/css" as="style">
|
|
|
|
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
|
|
<link href="/stylesheets/print.css" media="print" rel="stylesheet" type="text/css">
|
|
|
|
|
|
<link href="/atom.xml" rel="alternate" title="Julia Evans" type="application/atom+xml">
|
|
|
|
|
|
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css" integrity="sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0" crossorigin="anonymous">
|
|
<script defer data-domain="jvns.ca" src="https://plausible.io/js/script.js"></script>
|
|
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js" integrity="sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4" crossorigin="anonymous"></script>
|
|
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
|
|
|
|
<script defer type="text/javascript">
|
|
window.heap=window.heap||[],heap.load=function(e,t){window.heap.appid=e,window.heap.config=t=t||{};var r=document.createElement("script");r.type="text/javascript",r.async=!0,r.src="https://cdn.heapanalytics.com/js/heap-"+e+".js";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(r,a);for(var n=function(e){return function(){heap.push([e].concat(Array.prototype.slice.call(arguments,0)))}},p=["addEventProperties","addUserProperties","clearEventProperties","identify","resetIdentity","removeEventProperty","setEventProperties","track","unsetEventProperty"],o=0;o<p.length;o++)heap[p[o]]=n(p[o])};
|
|
heap.load("2242143965");
|
|
</script>
|
|
</head>
|
|
<body>
|
|
<div id="skiptocontent">
|
|
<a href="#main">Skip to main content</a>
|
|
</div>
|
|
<div id="wrap">
|
|
<header role="banner">
|
|
<hgroup>
|
|
<h1><a href="/">Julia Evans</a></h1>
|
|
</hgroup>
|
|
<ul class="header-links">
|
|
<li><a href="/about">About</a></li>
|
|
<li><a href="/talks">Talks</a></li>
|
|
<li><a href="/projects/">Projects</a></li>
|
|
<li><a rel="me" href="https://social.jvns.ca/@b0rk">Mastodon</a></li>
|
|
<li><a href="https://bsky.app/profile/b0rk.jvns.ca">Bluesky</a></li>
|
|
<li><a href="https://github.com/jvns">Github</a></li>
|
|
</ul>
|
|
</header>
|
|
<nav role="navigation" class="header-nav"><ul class="main-navigation">
|
|
<li><a href="/categories/favorite/">Favorites</a></li>
|
|
<li><a href="/til/">TIL</a></li>
|
|
<li><a href="https://wizardzines.com">Zines</a></li>
|
|
<li class="subscription" data-subscription="rss"><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
|
|
</ul>
|
|
</nav>
|
|
<div id="main">
|
|
<div id="content">
|
|
|
|
|
|
<div>
|
|
<article class="hentry" role="article">
|
|
<header>
|
|
<h1 class="entry-title">Operations for software developers for beginners</h1>
|
|
|
|
<div class="post-tags">
|
|
|
|
</div>
|
|
<p class="meta sans">
|
|
<time class="date" datetime="2016-10-15T11:02:57" pubdate data-updated="true">
|
|
|
|
October 15, 2016
|
|
|
|
</time>
|
|
</p>
|
|
</header>
|
|
<main>
|
|
<p>I work as a software developer. A few years ago I had no idea what “operations” was. I had
|
|
never met anybody whose job it was to operate software. What does that even mean? Now I
|
|
know a tiny bit more about it so I want to write down what I’ve figured out.</p>
|
|
<h3 id="operations-what-even-is-it" class="post-heading">
|
|
<a href="#operations-what-even-is-it">
|
|
operations: what even is it?
|
|
</a>
|
|
</h3>
|
|
<p>I made up these 3 stages of operating software. These are stages of understanding about operations I am going through.</p>
|
|
<p><strong>Stage 1: your software just works. It’s fine.</strong></p>
|
|
<p>You’re a software developer. You are running software on computers. When you write your software, it generally works okay – you write tests, you make sure it works on localhost, you push it to production, everything is fine. You’re a good programmer!</p>
|
|
<p>Sometimes you push code with bugs to production. Someone tells you about the bugs, you fix
|
|
them, it’s not a big deal.</p>
|
|
<p>I used to work on projects which hardly anyone used. It wasn’t a big deal if there was a
|
|
small bug! I had no idea what operations was and it didn’t matter too much.</p>
|
|
<p><strong>Stage 2: omg anything can break at any time this is impossible</strong></p>
|
|
<p>You’re running a site with a lot of traffic. One day, you decide to upgrade your database over the weekend. <a href="https://blog.clevertap.com/sleepless-nights-with-mongodb-wiredtiger-and-our-return-to-mmapv1/">You have a bad weekend</a>. Charity writes a blog post saying <a href="https://charity.wtf/2016/10/02/the-accidental-dba/">you should have spent more than 3 days on a database upgrade</a>.</p>
|
|
<p>I think if in my “what even is operations” state somebody had told me “julia!! your site needs to be up 99.95% of the time” I would have just have hid under the bed.</p>
|
|
<p>Like, how can you make sure your site is up 99.95% of the time? ANYTHING CAN HAPPEN. You could have spikes in traffic, or some random piece of critical software you use could just stop working one day, or your database could catch fire. And what if I need to upgrade my database? How do I even do that safely? HELP.</p>
|
|
<p>I definitely went from “operations is trivial, whatever, how hard can keeping a site up be?” to “OMG THIS IS IMPOSSIBLE HOW DOES ANYONE EVER DO THIS”.</p>
|
|
<p><strong>Stage 2.5: learn to be scared</strong></p>
|
|
<p>I think learning to be scared is a really important skill – you <strong>should</strong> be worried
|
|
about upgrading a database safely, or about upgrading the version of Ruby you’re using in
|
|
production. These are dangerous changes!</p>
|
|
<p>But you can’t just <a href="/blog/2014/12/21/fear-makes-you-a-worse-programmer/">stop at being scared</a> – you need to learn to have a healthy concern about complex parts of your system, and then learn how to take the appropriate precautionary steps and then confidently make the upgrade or deploy the big change of whatever the thing you are appropriately scared of is.</p>
|
|
<p>If you stop here then you just end up using a super-old Ruby version for 4 years
|
|
because you were too scared to upgrade it. That is no good either!</p>
|
|
<p><strong>Stage 3: keeping your site up is possible</strong></p>
|
|
<p>So, it turns out that there is a huge body of knowledge about keeping your site up!</p>
|
|
<p>There are people who, when you show them a large complicated software system
|
|
running on thousands or tens of thousands of computers, and tell them “hey, this needs to
|
|
be up 99.9% of the time”, they’re like “yep, that is a normal problem I have worked on! Here’s the first step we can take!”</p>
|
|
<p>These people sometimes have the job title “operations engineer” or “SRE” or “devops engineer” or “software engineer” or “system administrator”. Like all things, it’s a skillset that you can learn, not a magical innate quality.</p>
|
|
<p>Charity is one of these people! That blog post ("<a href="https://charity.wtf/2016/10/02/the-accidental-dba/">The Accidental
|
|
DBA</a>")) I linked to before has a bunch
|
|
of extremely practical advice about how to upgrade a database safely. If you’re running a
|
|
database and you’re scared – you’re right! But you can learn about how to upgrade it from
|
|
someone like Charity and then it will go a lot better.</p>
|
|
<h3 id="getting-started-with-operations" class="post-heading">
|
|
<a href="#getting-started-with-operations">
|
|
getting started with operations
|
|
</a>
|
|
</h3>
|
|
<p>So, we’ve convinced ourselves that operations is important.</p>
|
|
<p>Last year I was on a team that had some software. It mostly ran okay, but infrequently it
|
|
would stop working or get super slow. There were a bunch of different reasons it had
|
|
problems! And it wasn’t a disaster, but it also wasn’t as awesome as we wanted it to be.</p>
|
|
<p>For me this was a really cool way to get a little bit better at operations! I worked on
|
|
making the service faster and more reliable. And it worked! I made a couple of good
|
|
improvements, and I was happy.</p>
|
|
<p>Some stuff that helped:</p>
|
|
<ul>
|
|
<li>work on a dashboard for the service that clearly shows its current state (this is surprisingly hard!)</li>
|
|
<li>move some complicated code that did a lot of database operations into a separate webservice so we could easily time it out if something went wrong</li>
|
|
<li>do some profiling and remove some unnecessarily slow code</li>
|
|
</ul>
|
|
<p>The most cool part of this, though, is that a much more experienced SRE later came in to
|
|
work with the team on making the same service operate better, and I got to see what he did
|
|
and what his process for improving things looked like!</p>
|
|
<p>It’s really helped me to realize that you don’t turn into a Magical Operations Person overnight. Instead, I can take whatever I’m working on right now, and make small improvements to make it operate better! That makes me a better programmer.</p>
|
|
<h3 id="you-can-make-operations-part-of-your-job" class="post-heading">
|
|
<a href="#you-can-make-operations-part-of-your-job">
|
|
you can make operations part of your job
|
|
</a>
|
|
</h3>
|
|
<p>As an industry, we used to have “software development” teams who wrote code and threw it over the wall to “operations teams” who ran that code. I feel like we’ve collectively decided that we want a different model (“devops”) – that we should have teams who both write code and know how to operate it. And there are a lot of details of how that works exactly (do you have “SRE"s?)</p>
|
|
<p>But as an individual software engineer, what does that mean for you? I thiiink it means that you get to LEARN COOL STUFF. You can learn about how to deploy changes safely, and observe what your code is doing. And then when something has gone wrong in production, you’ll both understand what the code is doing (because you wrote it!!) and you’ll have the skills to figure it out and systematically prevent it in the future (because you are better at operations!).</p>
|
|
<p>I have a lot more to say about this (how I really love being a generalist, how doing some operations work has been an awesome way to improve my debugging skills and my ability to reason about complex systems and plan how to build complicated software). And I need to write in the future about super useful Ideas For Operating Software Safely I’ve learned about (like dark reads and circuit breakers). But I’m going to stop here for now. If you want more reading <a href="http://www.susanjfowler.com/blog/2016/10/13/the-ops-identity-crisis">The Ops Identity Crisis</a> is a good post about software developers doing operations, from the point of view of an ops person.</p>
|
|
<p>This is my favorite paragraph from Charity’s “<a href="https://charity.wtf/2016/05/31/wtf-is-operations-serverless/">WTF is operations?</a>” blog post (which you should just go read instead of reading me):</p>
|
|
<blockquote>
|
|
<p>The best software engineers I know are the ones who consistently value the impact and lifecycle of the code they ship, and value deployment and instrumentation and observability. In other words, <strong>they rock at ops stuff</strong>.</p>
|
|
</blockquote>
|
|
|
|
</main>
|
|
|
|
<footer>
|
|
|
|
<style type="text/css">
|
|
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; display: inline;}
|
|
#mc_embed_signup {
|
|
display: inline;
|
|
}
|
|
#mc_embed_signup input.button {
|
|
background: #ff5e00;
|
|
display: inline;
|
|
color: white;
|
|
padding: 6px 12px;
|
|
}
|
|
</style>
|
|
<div class="sharing">
|
|
|
|
<style>
|
|
.form-inline {
|
|
display:flex; flex-flow: row wrap; justify-content: center;
|
|
}
|
|
.form-inline input, .form-inline span {
|
|
padding: 10px;
|
|
}
|
|
.form-inline input {
|
|
display:inline;
|
|
max-width:30%;
|
|
margin: 0 10px 0 0;
|
|
background-color: #fff;
|
|
border: 1px solid #ddd;
|
|
border-radius: 5px;
|
|
padding: 10px;
|
|
}
|
|
button {
|
|
background-color: #f50;
|
|
box-shadow: none;
|
|
border: 0;
|
|
border-radius: 5px;
|
|
color: white;
|
|
padding: 5px 10px;
|
|
}
|
|
@media (max-width: 800px) {
|
|
.form-inline input {
|
|
margin: 10px 0;
|
|
max-width:100% !important;
|
|
}
|
|
.form-inline {
|
|
flex-direction: column;
|
|
align-items: stretch;
|
|
}
|
|
}
|
|
</style>
|
|
|
|
<div align="center">
|
|
<form class="form-inline" action="https://app.convertkit.com/forms/1052396/subscriptions" method="post" data-uid="8884355abb" data-format="inline" data-version="5">
|
|
<span> Want a weekly digest of this blog?</span>
|
|
<input name="email_address" type="text" placeholder="Email address" />
|
|
<button type="submit" data-element="submit">Subscribe</button>
|
|
</form>
|
|
</div>
|
|
|
|
|
|
</div>
|
|
|
|
<p class="meta">
|
|
|
|
<a class="basic-alignment left" href="https://jvns.ca/blog/2016/10/10/what-even-is-a-container/" title="Previous Post: What even is a container: namespaces and cgroups">What even is a container: namespaces and cgroups</a>
|
|
|
|
|
|
<a class="basic-alignment right" href="https://jvns.ca/blog/2016/10/16/whats-devops/" title="Next Post: What's devops?">What's devops?</a>
|
|
|
|
</p>
|
|
</footer>
|
|
|
|
</article>
|
|
</div>
|
|
|
|
</div>
|
|
</div>
|
|
<nav role="navigation" class="footer-nav"> <a href="/">Archives</a>
|
|
</nav>
|
|
<footer role="contentinfo"><span class="credit">© Julia Evans. </span>
|
|
<span>If you like this, you may like <a href="https://web.archive.org/web/20181228051203/http://www.uliaea.ca/">Ulia Ea</a> or, more seriously, this list of <a href="https://jvns.ca/blogroll">blogs I love</a> or some <a href="https://jvns.ca/bookshelf">books I've read</a>. <br>
|
|
<p class="rc-scout__text"><i class="rc-scout__logo"></i>
|
|
You might also like the <a class="rc-scout__link" href="https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8">Recurse Center</a>, my very favorite programming community <a href="/categories/hackerschool/">(my posts about it)</a></p>
|
|
</span>
|
|
<style class="rc-scout__style" type="text/css">.rc-scout{display:block;padding:0;border:0;margin:0;}.rc-scout__text{display:block;padding:0;border:0;margin:0;height:100%;font-size:100%;}.rc-scout__logo{display:inline-block;padding:0;border:0;margin:0;width:0.85em;height:0.85em;background:no-repeat center url('data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2012%2015%22%3E%3Crect%20x%3D%220%22%20y%3D%220%22%20width%3D%2212%22%20height%3D%2210%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%221%22%20width%3D%2210%22%20height%3D%228%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%222%22%20width%3D%228%22%20height%3D%226%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%229%22%20width%3D%224%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%2211%22%20width%3D%2210%22%20height%3D%224%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%220%22%20y%3D%2212%22%20width%3D%2212%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%225%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%227%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%228%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%229%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3C%2Fsvg%3E');}.rc-scout__link:link,.rc-scout__link:visited{color:#61ae24;text-decoration:underline;}.rc-scout__link:hover,.rc-scout__link:active{color:#4e8b1d;}</style>
|
|
</footer>
|
|
<script type="text/rocketscript">
|
|
(function(){
|
|
var twitterWidgets = document.createElement('script');
|
|
twitterWidgets.type = 'text/javascript';
|
|
twitterWidgets.async = true;
|
|
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
|
|
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
|
|
})();
|
|
</script>
|
|
</div>
|
|
</body>
|
|
</html>
|
|
|