193 lines
15 KiB
HTML
193 lines
15 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
|
|
|
<HTML>
|
|
<HEAD>
|
|
<TITLE>(A few) Ops Lessons We All Learn The Hard Way</TITLE>
|
|
|
|
<meta property="og:url" content="https://www.netmeister.org/blog/ops-lessons.html">
|
|
<meta name="robots" content="noai,noimageai">
|
|
<meta property="og:title" content="(A few) Ops Lessons We All Learn The Hard Way">
|
|
<meta property="og:description" content="Ops is hard. What have learned so far?">
|
|
<meta property="og:image" content="https://www.netmeister.org/blog/images/tar-is-not-a-plaything.png">
|
|
|
|
<LINK rev="made" href="mailto:jschauma@netmeister.org">
|
|
<link rel="alternate" type="application/rss+xml" title="Signs of Triviality" href="https://www.netmeister.org/blog/rss.xml">
|
|
<link rel="stylesheet" type="text/css" href="blog.css">
|
|
</HEAD>
|
|
|
|
<BODY>
|
|
<H1>Signs of Triviality</H1>
|
|
<em>Opinions, mostly my own, on the importance of being and other things. </em>
|
|
<HR WIDTH="100%" SIZE=2 ALIGN="CENTER" NOSHADE>
|
|
<small>
|
|
[<a href="../index.html">homepage</a>] [<a href="index.html">index</a>] [<a href="mailto:jschauma@netmeister.org">jschauma@netmeister.org</a>] [<a href="https://mstdn.social/@jschauma">@jschauma</a>] [<a href="rss.xml">RSS</a>]
|
|
</small>
|
|
<input type="checkbox" id="theme-checker" hidden>
|
|
<div class="container">
|
|
<label class="switch" for="theme-checker" title="Dark/Light mode">
|
|
<span class="slider round"></span>
|
|
</label>
|
|
</div>
|
|
<HR WIDTH="100%" SIZE=2 ALIGN="CENTER" NOSHADE>
|
|
<h2>(A few) Ops Lessons We All Learn The Hard Way</h2>
|
|
<table border="0" width="75%">
|
|
<tr>
|
|
<td>
|
|
<p><small>January 24th, 2020</small></p>
|
|
|
|
<p> Nope, not another <a
|
|
href="cs-falsehoods.html">Falsehoods</a> post, but not
|
|
entirely unlike one. Only here we have a few lessons
|
|
in <a href="defining-operations.html">operations</a>
|
|
that we all (eventually) (have to) learn, often the
|
|
hard way. Why things are the way they are, or what
|
|
the lessons <em>mean</em> is left to the reader to interpret,
|
|
agree, or disagree with. It's more fun that way.
|
|
Enjoy! </p>
|
|
|
|
<hr width="75%">
|
|
|
|
<ol>
|
|
<li><a href="http://stevens.netmeister.org/615/tar.html"><img src="images/tar-is-not-a-plaything.png" alt="Bart Simpson writing 'Tar is not a play thing' on the school chalkboard." align="right" border="0" title="It really isn't."></a>Email is the worst monitoring and alerting mechanism except for all the others.</li>
|
|
<li>Absence of a signal is itself a signal.</li>
|
|
<li>The severity of an incident is measured by the number of rules broken in resolving it.</li>
|
|
<li>The mobile hotspot you're paying for so you can leave your house while you're oncall only works at home and in the office.</li>
|
|
<li>The only other person who knows how this works is also on vacation.</li>
|
|
<li>If a post-mortem follow-up task is not picked up within a week, it's unlikely to be completed at all.</li>
|
|
<li>That janky script you put together during the outage -- the one that uses <tt>expect(1)</tt> and '<tt>ssh -t -t</tt>' -- now is the foundation of the entire team's toolchest.</li>
|
|
<li>NTP being off may not be a root cause, but it sure didn't help.</li>
|
|
<li><a href="https://utc-or-gtfo.creator-spring.com/">UTC or GTFO</a>.</li>
|
|
<li><a name="certs"></a>Your infrastructure uses a lot more self-signed certificates than you think. A <em>lot</em> more. In places that make you weep.</li>
|
|
<li>Self-signed certificates beget long lived certs, which beget lack of certificate validity monitoring, which begets <tt>curl -k</tt>, which begets a lack of certificate deployment automation, which begets self-signed certificates.</li>
|
|
<li>For any N applications, at most N/2+1 use the same certificate bundle.</li>
|
|
<li>The system you're troubleshooting doesn't use the one the tool you're <a href="debugging-certificate-errors.html">troubleshooting</a> it with does.</li>
|
|
<li>An <a href="/twitter/1086305575443542017">API without a reference</a> implementation and command-line client is called a gray box.</li>
|
|
<li>Restricted shells are not as restricted as you think.</li>
|
|
<li>Very few operations are truly idempotent.</li>
|
|
<li>"Asserting state" beats "monitoring for compliance" any day.</li>
|
|
<li><a href="https://docs.microsoft.com/en-us/archive/blogs/larryosterman/one-in-a-million-is-next-tuesday">One in a Million is next Tuesday.</a></li>
|
|
<li>People give talks at conferences not to convince others that their work is awesome and totally worth the time and effort they put in, but themselves.</li>
|
|
<li><img src="images/there-is-no-cloud.png" alt="There is no cloud, it's just someone else's computer." align="right">It's ok to use shell for complex stuff; it often times is easier, faster, and still less of a mess than juggling libraries and dependencies.</li>
|
|
<li>There's nothing wrong with Perl.</li>
|
|
<li>Okay, we all at times keep adding <tt>$</tt>, <tt>{</tt>, <tt>}</tt>, and <tt>@</tt> in random places trying to make things work, but still.</li>
|
|
<li>Serverless isn't.</li>
|
|
<li><a href="https://en.wikipedia.org/wiki/Year_2038_problem">Y38K</a> is <a href="https://twitter.com/jxxf/status/1219009308438024200">already here</a>, it's just not evenly distributed.</li>
|
|
<li>If you determine "<a href="humanerrno.html">human error</a>" as the root cause, then you're doing it wrong.</li>
|
|
<li>Your network team has a way into the network that your security team doesn't know about.</li>
|
|
<li>And don't even as much as <em>mention</em> the serial console and IPMI networks, but boy are you glad you have 'em.</li>
|
|
<li>Blocking TCP port 53 traffic leads to very strange failures. Don't.</li>
|
|
<li>Somewhere in your infrastructure a service you didn't know uses DNS for endpoint discovery in a very surprising way.</li>
|
|
<li>Do. Not. Monkey. Around. With. <tt>/etc/hosts</tt>.</li>
|
|
<li>If you break it, you own it - for now; if you fix it, you own it - forever.</li>
|
|
<li>Turning it off and on again is actually quite a reasonable way to fix many things.</li>
|
|
<li>A <tt>README.md</tt> in git is no substitute for a manual page that's shipped with your tool.</li>
|
|
<li>A search for a document you know exists will only turn up links to documents referencing <em>but not actually linking to</em> the one you're looking for.</li>
|
|
<li>The document you're looking for was marked as obsolete and not migrated to the new content management solution.</li>
|
|
<li><img src="images/hot-water-pipes.jpg" alt="Connected hot water pipes." align="right" title="Illustration of ssh port forwarding.">Sure, your current content management system sucks, but it's still better than the one you're moving to.</li>
|
|
<li>Nobody knows how git works; everybody simply <tt>rm -fr && git checkout</tt>'s periodically.</li>
|
|
<li>There are very few network restrictions creative and determined use of <tt>ssh(1)</tt> port forwarding can't overcome.</li>
|
|
<li>This is both incredibly useful and concerning.</li>
|
|
<li>It is tempting to jump right into implementing a solution when the right thing may well be to not do the thing that requires the solution in the first place.</li>
|
|
<li>Turning things off permanently is <a href="/twitter/1074684924588974080">surprisingly difficult</a>.</li>
|
|
<li>"<em>Ancient</em>" is a very relative term when it comes to software and protocols.</li>
|
|
<li>"<em>Obsolete</em>" doesn't mean it's not in use and relied on.</li>
|
|
<li>The sets of systems online before and after a data center power outage only intersect. Some of the old systems coming online will immediately cause a different outage.</li>
|
|
<li>Some of your most critical services are kept alive by a handful of people whose job description does not mention those services at all.</li>
|
|
<!-- <li>The <a
|
|
href="/twitter/816409198116433920">evolution of configuration management</a>
|
|
<em>always</em> follows the 'ssh -> rsync -> parallel ssh -> switch to pull -> switch back to daemon pull -> deploy $product -> find edge-case not met -> goto 1' route. Multiple times.</li> -->
|
|
<li>After the initial "down for everybody or just me ermahgehrd Slack is down" drop, productivity increases linearly throughout the duration of the outage.</li>
|
|
<li>You're bound by the <a href="https://en.wikipedia.org/wiki/CAP_theorem">CAP theorem</a> much more often than you may think. <a href="https://en.wikipedia.org/wiki/Halting_problem">Halting Problem</a>'s a bitch, too.</li>
|
|
<li>Eventual consistency doesn't help when the system you're debugging hasn't converged yet.</li>
|
|
<li>The source you're looking at is not the code running in production.</li>
|
|
<li><tt>strace(1)</tt>/<tt>ktrace(1)</tt> doesn't lie.</li>
|
|
<li>Unless somebody's been playing <tt>LD_PRELOAD</tt> games.</li>
|
|
<li>Schrödinger's Backup -- "The condition of any backup is unknown until a restore is attempted." -- is overly optimistic.</li>
|
|
<li>There's <a href="https://xkcd.com/305/">an xkcd</a> for the precise situation you find yourself in. (There's also one for at least half of these.)</li>
|
|
<li>At some point in your career you will implement <a href="/twitter/643444266300239873">half of kerberos</a>. Poorly.</li>
|
|
<li>Any sufficiently successful product launch is indistinguishable from a DDoS; any sufficiently advanced user indistinguishable from an attacker.</li>
|
|
<li>Debugging any sufficiently complex open source product is indistinguishable from reverse engineering a black box.</li>
|
|
<li>"We've always done it this way." is not a good reason by itself, but there's bound to be one for why.</li>
|
|
<li>That reason may or may not be valid any longer, however.</li>
|
|
<li><img src="images/inferno.jpg" alt="Circles of Hell from Dante's Inferno" align="right" title="You see, it's a metaphor...">A junior engineer asking "why" and pointing out the docs don't reflect reality is at least as valuable as the senior engineer working blindly off tribal knowledge.</li>
|
|
<li>Your herculean efforts to upgrade the OS across your entire fleet completed just in time for the EOL announcement of the version you upgraded to.</li>
|
|
<li>This phenomenon was first described in Dante's <em>Inferno</em> as the Ninth Circle of Hell, Ring Four, aka <tt>RedHat Canto XXXIV</tt>.</li>
|
|
<li>Containers create at least as many problems as they solve.</li>
|
|
<li>The most ninja move the expert you hired for that third party black box product you rely on is to say "Let me ping the support team".</li>
|
|
<li>Somewhere, somebody ran into this <em>exact</em> problem, but they never bothered to post a solution.</li>
|
|
<li>That completely automated solution you set up requires at least three manual steps you didn't document.</li>
|
|
<li>CAPEX budget always increases, OPEX budget always decreases.</li>
|
|
<li>CAPEX costs can be reasonably estimated, OPEX costs can only be ballparked.</li>
|
|
<li>Doubling your time estimate in the hopes of beating expectations won't work because your manager takes your estimate, has a hardy laugh, and then resets it back to what they already promised upchain.</li>
|
|
<li>Your quarterly planning means bubkes when the next re-org rolls around.</li>
|
|
<li>Most of your actual work is not covered by your <a href="okr-distractions.html">OKRs</a>.</li>
|
|
<li>Recursively applying the <a href="https://en.wikipedia.org/wiki/Pareto_principle">Pareto Principle</a> is a surprisingly accurate way to gauge your low hanging fruit, determine your high impact objectives, and ballpark your required effort.</li>
|
|
<li>Although, to be honest, it <a href="/twitter/962448995690967041">only works in about 80% of cases</a>.</li>
|
|
<!-- <li>Say "OKR" one more time, I dare you, I double dare you motherfucker, say "OKR" one more goddamn time!</li> -->
|
|
<li>Management will always happily <a href="crazy-like-a-fox.html">spend $$$ on outside consultants</a> to tell them what you've been saying for years.</li>
|
|
<li>Management will much rather invest in inventing a new, square wheel than fixing an old round one.</li>
|
|
<li>In any organization practicing <a href="/twitter/1225455309919006720">continuous integration</a>, half of all commits are to fake out CI tests.</li>
|
|
<li>Good software development practices do not always translate well to ops and friends.</li>
|
|
<li>Mandatory <a href="/twitter/1019410471999467525">code reviews</a> do not automatically improve code quality nor reduce the frequency of incidents.</li>
|
|
<li>Every new paradigm tends to mostly add layers of abstractions; cutting through them and identifying what basic principles continue to apply is half the battle.</li>
|
|
<li><a href="https://en.wikipedia.org/wiki/Layer_8"><img align="right" src="images/osi-stack2.png" width="250" border="0" alt="OSI stack with layers 8 (Financial) and 9 (Political) added" title="Not to be confused with the 9 circles of hell..."></a>
|
|
Real change can only be implemented <a href="https://en.wikipedia.org/wiki/Layer_8">above layer 7</a>.</li>
|
|
<li>"Prod" is just another name for "staging".</li>
|
|
<li>Your source of truth lies.</li>
|
|
<li>Also: it's incomplete.</li>
|
|
<li><tt>pcap</tt> or it didn't happen.</li>
|
|
<li><tt>grep(1)</tt> > Splunk (there, I said it)</li>
|
|
<li>Multithreading is rarely worth the added complexity.</li>
|
|
<li>Parallelism is not Concurrency.</li>
|
|
<li>Simplicity is King.</li>
|
|
<li>Nobody knows what exactly it is you do.</li>
|
|
</ol>
|
|
|
|
<p><small>January 24th, 2020</small></p>
|
|
|
|
<p><small>Related:</small></p>
|
|
|
|
<ul>
|
|
<li><small><a href="https://ops-lessons.creator-spring.com/">This blog post as a Teespring store</a> (i.e., stickers'n'stuff)</small></li>
|
|
<li><small><a href="software-engineering-laws.html">10
|
|
Software Engineering Laws Everybody Loves To Ignore</a></small></li>
|
|
<li><small><a href="cs-falsehoods.html">Falsehoods CS Students (Still) Believe Upon Graduating</a></small></li>
|
|
<li><small><a href="semper-ubi-sub-ubi.html">Things They Don't Teach You In School</a></small></li>
|
|
<li><small><a href="defining-operations.html">Defining Operations</a></small></li>
|
|
<li><small><a href="humanerrno.html">Root Cause: Human Errno</a></small></li>
|
|
<li><small><a href="infosec-competencies.html">(Technical) Infosec Core Competencies</a></small></li>
|
|
<li><small>Discussions / Comments:</small>
|
|
<ul>
|
|
<li><small><a href="https://news.ycombinator.com/item?id=32550670">Discussion on Hacker News</a></small></li>
|
|
<li><small><a href="https://lobste.rs/s/bzwjia/few_ops_lessons_we_all_learn_hard_way">Discussion on Lobsters</a></small></li>
|
|
<li><small><a href="https://www.reddit.com/r/devops/comments/eu7wv2/a_few_ops_lessons_we_all_learn_the_hard/">Discussion on r/devops</a>; <a href="https://www.reddit.com/r/sysadmin/comments/evecz9/a_few_ops_lessons_we_all_learn_the_hard_way/">r/sysadmin</a>; <a href="https://www.reddit.com/r/SysAdminBlogs/comments/exqqye/a_few_ops_lessons_we_all_learn_the_hard_way/">r/SysAdminBlogs</a></small></li>
|
|
<li><small><a href="https://bbs.boingboing.net/t/the-ops-lessons-we-all-learn-the-hard-way/159939">Comments on Boing Boing</a></small></li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
<HR WIDTH="100%" SIZE=2 ALIGN="CENTER" NOSHADE>
|
|
<small>
|
|
← [<a href="stdarg.html"><tt>stdarg</tt> And The Case Of The Forgotten Registers</a>]
|
|
<div style="float: right;">[<a href="browser-startup.html">Browser Startup Comparison</a>] →</div>
|
|
</small>
|
|
<hr class="noshade" style="width:100%;">
|
|
<small>
|
|
[<a href="../index.html">homepage</a>]
|
|
[<a href="index.html">blog</a>]
|
|
[<a href="mailto:jschauma@netmeister.org">jschauma@netmeister.org</a>]
|
|
[<a href="https://mstdn.social/@jschauma/">@jschauma</a>]
|
|
[<a href="rss.xml">RSS</a>]
|
|
</small>
|
|
<div class="container">
|
|
<label class="switch" for="theme-checker" title="Dark/Light mode">
|
|
<span class="slider round"></span>
|
|
</label>
|
|
</div>
|
|
<HR WIDTH="100%" SIZE=2 ALIGN="CENTER" NOSHADE>
|
|
</body>
|
|
</html>
|