330 lines
14 KiB
HTML
330 lines
14 KiB
HTML
<!DOCTYPE HTML>
|
||
|
||
<html lang="en-us">
|
||
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<title>97 things every SRE should know - Part 01 - UnixDaemon: In search of (a) life</title>
|
||
<meta name="author" content="map[name:Dean Wilson]">
|
||
|
||
|
||
|
||
<meta name="description" content="">
|
||
|
||
|
||
|
||
<meta name="HandheldFriendly" content="True">
|
||
<meta name="MobileOptimized" content="320">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||
|
||
<link href='/index.xml' rel="alternate" title="UnixDaemon: In search of (a) life" type="application/atom+xml">
|
||
|
||
<link rel="canonical" href="https://www.unixdaemon.net/sysadmin/97-things-every-sre-01/">
|
||
<link href="https://www.unixdaemon.net/favicon.png" rel="shortcut icon">
|
||
<link href="https://www.unixdaemon.net/css/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
|
||
<link href="https://www.unixdaemon.net/css/font-awesome.min.css" media="screen" rel="stylesheet" type="text/css">
|
||
|
||
<link href='https://fonts.googleapis.com/css?family=Nunito:400,300,700' rel='stylesheet' type='text/css'>
|
||
<meta name="google-site-verification" content="xyisveClGOK3aUjA5QIPkeGjVV_TfExAxXhIh5yt5nU" />
|
||
|
||
|
||
<script type="text/javascript">
|
||
var _gaq = _gaq || [];
|
||
_gaq.push(['_setAccount', 'UA-46204499-1']);
|
||
_gaq.push(['_trackPageview']);
|
||
|
||
(function() {
|
||
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
|
||
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
|
||
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
|
||
})();
|
||
</script>
|
||
|
||
|
||
</head>
|
||
|
||
|
||
<body>
|
||
<div class="container">
|
||
<div class="left-col">
|
||
<div class="intrude-less">
|
||
<header id="header" class="inner"><div class="profilepic">
|
||
|
||
|
||
|
||
</div>
|
||
|
||
<nav id="main-nav"><ul class="main">
|
||
<li><a href="https://www.unixdaemon.net/">Home</a></li>
|
||
<li><a href="https://www.unixdaemon.net/about/">About</a></li>
|
||
</ul>
|
||
</nav>
|
||
<nav id="sub-nav">
|
||
<div class="social">
|
||
|
||
|
||
|
||
|
||
<a class="twitter" href='https://twitter.com/unixdaemon' title="Twitter">Twitter</a>
|
||
|
||
|
||
<a class="github" href='https://github.com/deanwilson' title="GitHub">GitHub</a>
|
||
|
||
|
||
|
||
|
||
<a class="linkedin" href='https://www.linkedin.com/in/deanswilson' title="LinkedIn">LinkedIn</a>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<a class="rss" href='/index.xml' title="RSS">RSS</a>
|
||
|
||
</div>
|
||
<nav id="main-nav"><ul class="main">
|
||
<li><a href="https://www.unixdaemon.net/ansible/">/ansible/</a>
|
||
<a href="https://www.unixdaemon.net/books/">/books/</a></li>
|
||
<li><a href="https://www.unixdaemon.net/career/">/career/</a>
|
||
<a href="https://www.unixdaemon.net/cloud/">/cloud/</a></li>
|
||
<li><a href="https://www.unixdaemon.net/commandline/">/commandline/</a>
|
||
<a href="https://www.unixdaemon.net/events/">/events/</a></li>
|
||
<li><a href="https://www.unixdaemon.net/firefox/">/firefox/</a>
|
||
<a href="https://www.unixdaemon.net/geekstuff/">/geekstuff/</a></li>
|
||
<li><a href="https://www.unixdaemon.net/gui/">/gui/</a>
|
||
<a href="https://www.unixdaemon.net/linux/">/linux/</a></li>
|
||
<li><a href="https://www.unixdaemon.net/meta/">/meta/</a>
|
||
<a href="https://www.unixdaemon.net/misctech/">/misctech/</a></li>
|
||
<li><a href="https://www.unixdaemon.net/movies/">/movies/</a>
|
||
<a href="https://www.unixdaemon.net/network/">/network/</a></li>
|
||
<li><a href="https://www.unixdaemon.net/nottech/">/nottech/</a>
|
||
<a href="https://www.unixdaemon.net/online/">/online/</a></li>
|
||
<li><a href="https://www.unixdaemon.net/perl/">/perl/</a>
|
||
<a href="https://www.unixdaemon.net/programming/">/programming/</a></li>
|
||
<li><a href="https://www.unixdaemon.net/puppet/">/puppet/</a>
|
||
<a href="https://www.unixdaemon.net/security/">/security/</a></li>
|
||
<li><a href="https://www.unixdaemon.net/sites/">/sites/</a>
|
||
<a href="https://www.unixdaemon.net/sysadmin/">/sysadmin/</a></li>
|
||
<li><a href="https://www.unixdaemon.net/tools/">/tools/</a>
|
||
<a href="https://www.unixdaemon.net/unixdaemon/">/unixdaemon/</a></li>
|
||
</ul>
|
||
</nav>
|
||
</nav>
|
||
</header>
|
||
</div>
|
||
</div>
|
||
<div class="mid-col">
|
||
|
||
<div class="mid-col-container">
|
||
<div id="content" class="inner">
|
||
<div itemscope itemtype="http://schema.org/Blog">
|
||
<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
|
||
<h1 class="title" itemprop="name">97 things every SRE should know - Part 01</h1>
|
||
|
||
<div class="date"><time datetime='Mon, Jan 11, 2021' data-updated="true" itemprop="datePublished">Mon, Jan 11, 2021</time>
|
||
</div>
|
||
<div class="tags">
|
||
|
||
|
||
|
||
|
||
<a href="https://www.unixdaemon.net/categories/sysadmin"> sysadmin </a>
|
||
|
||
|
||
|
||
</div>
|
||
|
||
<div class="entry-content" itemprop="articleBody"><p>A few people I follow on twitter mentioned they’d contributed to<br>
|
||
<a href="https://www.oreilly.com/library/view/97-things-every/9781492081487/">97 Things Every SRE Should Know</a>.
|
||
It’s a book full of short, 1-3 page chapters, focused on topics dear to
|
||
an SREs heart. So i had no choice but to buy it. In an attempt to be
|
||
more deliberate with my reading and what I’ve retained from the book
|
||
I’ve decided to create some reading notes for future me. This post is
|
||
broken down into a section per chapter.</p>
|
||
<h3 id="chapter-42---why-i-hate-our-playbooks">Chapter 42 - Why I Hate Our Playbooks</h3>
|
||
<p>There are some great quotes:</p>
|
||
<ul>
|
||
<li>
|
||
<p><em>Any playbook that can describe the exact steps to resolve an exact
|
||
circumstance should be an automated script instead.</em></p>
|
||
</li>
|
||
<li>
|
||
<p><em>We escalate to humans for a complex response, not a fast response.</em></p>
|
||
</li>
|
||
</ul>
|
||
<p>It also provides some good meta guidance on playbooks:</p>
|
||
<p>Ideally, a playbook should only contain:</p>
|
||
<ul>
|
||
<li>Why do I care? Severity and qualification of the user-visible impact.</li>
|
||
<li>What can I look at? Consoles, logs, and inspection tools.</li>
|
||
<li>What can I do? Mitigation tooling.</li>
|
||
</ul>
|
||
<p>As well as an example playbook work flow:</p>
|
||
<pre><code>|-> 1 Identify issue
|
||
| 2 Debug
|
||
| 3 Add alerts
|
||
| 4 Write documentation
|
||
| |-> 5 Automate resolution
|
||
| |-< 6 Update documentation
|
||
|<- 7 Have a different problem
|
||
</code></pre><p>My own views on playbooks are that you get out what you put in, and they
|
||
are often a last minute band aid rather than a full part of the product.
|
||
They should have a life cycle. A time to be useful and a time to <del>die</del>
|
||
be automated away. The trick is in knowing what stage one’s at.</p>
|
||
<p>One way to discover this is by capturing usage and relevance
|
||
information. A simple thumbs up thumbs down on each page gets you
|
||
started but ideally you’d track a little more. Has the page ever been
|
||
read? If so how long ago? Was it actually used? or did it just get
|
||
opened and immediately closed? When was it last reviewed? A simple
|
||
“helped” / “didn’t help” checkbox and a comment box at the bottom of
|
||
page can help you get started.</p>
|
||
<p>It’s also important any feedback you capture is as frictionless as
|
||
possible. Don’t make people change tab to a doc review system for
|
||
example but embed it in the document itself. Anything that adds friction
|
||
will stop people responding. I also think it’s better to gather some
|
||
data than none so don’t present a 4 page 50 question survey.</p>
|
||
<p>I found this chapter to be a great example on how you can approach
|
||
playbooks in a more consistent way.</p>
|
||
<h3 id="chapter-12---the-importance-of-a-management-interface---salim-virji">Chapter 12 - The Importance of a Management Interface - Salim Virji</h3>
|
||
<p>This chapter provides a high level overview of why you need to keep the
|
||
serving of user traffic serving separate from the control and
|
||
administration traffic and data flows. In short, it’s so you can
|
||
actually affect the system when events like high client requests happen.
|
||
Otherwise, you sit in the same queue as everyone else, waiting for the
|
||
fix to be processed.</p>
|
||
<p>There are some great quotes:</p>
|
||
<ul>
|
||
<li><em>During an outage, you care more about being able to control the system
|
||
than about the system answering all user-facing requests.</em></li>
|
||
<li><em>To implement this separation for software that’s already built, such
|
||
as third-party applications, you may need to add a separate service
|
||
that, like a sidecar, attaches to the core software and, through a
|
||
common interface such as an HTTP server, provides an endpoint for the
|
||
administrative API.</em></li>
|
||
</ul>
|
||
<p>You often learn of the merits of this separation approach when your
|
||
system becomes popular, or as it’s known, “The worst possible moment.
|
||
It’s interesting to compare this isolation approach against systems like
|
||
RDBMs that reserve connections for root but still have the same
|
||
connection point.</p>
|
||
<p>If you want to know more about control planes
|
||
<a href="https://www.youtube.com/watch?v=O8xLxNje30M">AWS re:Invent 2018: Close Loops & Opening Minds: How to Take Control of Systems,
|
||
Big & Small ARC337</a>
|
||
by @colmmacc was a great watch.</p>
|
||
<h3 id="chapter-63---effecting-sre-cultural-changes-in-enterprises---vanessa-yiu">Chapter 63 - Effecting SRE Cultural Changes in Enterprises - Vanessa Yiu</h3>
|
||
<p>For a short piece pretty much every paragraph had something i wanted to
|
||
quote or highlight. If you’re introducing SRE to a company I suspect
|
||
this one is worth a quarterly re-read until the practices are
|
||
established.</p>
|
||
<p>It dives right in with great advice if you’re just beginning your
|
||
journey to introduce SRE:</p>
|
||
<p><em>“To avoid this, focus initially on the few most critical behaviours to
|
||
adapt. In other words, find the key blockers to successful
|
||
implementation of SRE at your workplace. If a shared responsibility
|
||
model does not exist between developers and SRE, for instance, then
|
||
perhaps start here, because that is foundational to getting SRE right."</em></p>
|
||
<p>I still struggle with this, it’s reassuring to know other people
|
||
consider it essential.</p>
|
||
<p>Some of the quotables that most hit home with me are:</p>
|
||
<ul>
|
||
<li><em>“Bringing on SRE means overcoming inertia and requiring a substantial
|
||
investment of time to educate as well as continuous reinforcement of
|
||
practices and behaviours."</em> As always it’s not just about the technology.</li>
|
||
<li><em>“It is important to identify where the gaps are and then build a clear
|
||
roadmap to lay the required foundations first."</em></li>
|
||
<li><em>“Providing transparency and identifying the correct incentives are
|
||
critical to the success of any large-scale change program. People need
|
||
to see and believe the value for changes to stick. Be thoughtful about
|
||
which results matter and which indicators reflect successful changes in
|
||
behaviour."</em></li>
|
||
</ul>
|
||
<p>I found myself nodding along to pretty much this entire chapter as I’ve
|
||
now lived through it twice at the same organisation and this is a great
|
||
summary of my experience too. Some of its points might seem clichéd but
|
||
there’s a reason they became cliches.</p>
|
||
<h3 id="79---why-training-matters-to-an-sre-practice-and-sre-matters-to-your-training-program---jennifer-petoff">79 - Why Training Matters to an SRE Practice and SRE Matters to Your Training Program - Jennifer Petoff</h3>
|
||
<p>It’d be criminal of me not to bring one of the acronyms used in this
|
||
chapter to the forefront of my notes - “So where should you begin? I
|
||
have one acronym for you: ASSBAT”</p>
|
||
<p>A great mnemonic for an important concept. <em>“ASSBAT stands for a student
|
||
should be able to. ASSBATs are learning objectives focused on behaviours
|
||
you want to drive and observe. Understand the $foo service is a bad
|
||
ASSBAT”</em> by focusing on more user and task specific chunks you can build
|
||
competency and confidence on the things you value and will actually be
|
||
useful to the engineers when they actually start with the system. While
|
||
having a deep understanding of your data flows is important knowing
|
||
which command runs the loadshedder and which dashboard to review is more
|
||
immediately useful as a canned response at 3am.</p>
|
||
<h3 id="32---bootstrapping-sre-in-enterprises---vanessa-yiu">32 - Bootstrapping SRE in Enterprises - Vanessa Yiu</h3>
|
||
<p>Enterprises are internally very different beasts to startups and how
|
||
you try to coerce them into changing needs a different approach.
|
||
This chapter helps frame the subject well and if I’d highlighted all
|
||
the sections I nodded along to the page would have been a
|
||
magnificent rainbow.</p>
|
||
<p>Some excellent quotables include:</p>
|
||
<ul>
|
||
<li><em>“Standalone systems are rare in enterprises - the service you operate
|
||
will probably depend on a range of other upstream and downstream
|
||
services, so your success is directly correlated with your ability to
|
||
navigate, influence, and deliver across the organisation."</em></li>
|
||
<li><em>“services with higher service level maturity likely have
|
||
well-established operational practices in place already but perhaps more
|
||
instability due to usage growth (i.e., capacity constraints) and
|
||
increasing complexity over time compared to newer services."</em></li>
|
||
<li><em>“Take the time to do research; interview product managers and on-call
|
||
engineers to understand common challenges and leverage data sets (e.g.,
|
||
problem management database, incident postmortems) where available to
|
||
confirm trends and remove recency bias."</em></li>
|
||
<li><em>“At the outset, focus on solving a few key issues where SRE can
|
||
demonstrate the most impact in the short term. These early successes
|
||
build trust with the stakeholders mentioned earlier, demonstrate how SRE
|
||
adds value to those unfamiliar with the discipline”</em></li>
|
||
</ul>
|
||
<p>Vanessa also states <em>“Check in regularly on progress, more frequently
|
||
(e.g., weekly) at the start of any engagement until things reach steady
|
||
state."</em> I’d like to add, if you’re being checked in on, assume good
|
||
intent and that people care how things are going rather than it being a
|
||
sign of distrust. While you may be responsible for the work getting done
|
||
the stakeholders accountable for it will need a little time to build
|
||
trust before they can increase the scope of autonomy so upfront
|
||
communication and building rapport and showing successful delivery pays
|
||
off over time for both sides.</p>
|
||
</div>
|
||
</article>
|
||
|
||
<div class="share">
|
||
<div class="addthis_toolbox addthis_default_style ">
|
||
|
||
|
||
|
||
|
||
</div>
|
||
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<footer id="footer" class="inner">Copyright © 2025
|
||
|
||
Dean Wilson
|
||
|
||
|
||
|
||
Design credit: <a href="http://shashankmehta.in/archive/2012/greyshade.html">Shashank Mehta</a>
|
||
</footer>
|
||
</div>
|
||
</div>
|
||
|
||
</body>
|
||
</html>
|