213 lines
7.6 KiB
HTML
213 lines
7.6 KiB
HTML
<!DOCTYPE HTML>
|
|
|
|
<html lang="en-us">
|
|
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<title>Scary sysadmin Halloween stories - UnixDaemon: In search of (a) life</title>
|
|
<meta name="author" content="map[name:Dean Wilson]">
|
|
|
|
|
|
|
|
<meta name="description" content="">
|
|
|
|
|
|
|
|
<meta name="HandheldFriendly" content="True">
|
|
<meta name="MobileOptimized" content="320">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
|
|
<link href='/index.xml' rel="alternate" title="UnixDaemon: In search of (a) life" type="application/atom+xml">
|
|
|
|
<link rel="canonical" href="https://www.unixdaemon.net/sysadmin/scary-sysadmin-halloween-stories/">
|
|
<link href="https://www.unixdaemon.net/favicon.png" rel="shortcut icon">
|
|
<link href="https://www.unixdaemon.net/css/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
|
|
<link href="https://www.unixdaemon.net/css/font-awesome.min.css" media="screen" rel="stylesheet" type="text/css">
|
|
|
|
<link href='https://fonts.googleapis.com/css?family=Nunito:400,300,700' rel='stylesheet' type='text/css'>
|
|
<meta name="google-site-verification" content="xyisveClGOK3aUjA5QIPkeGjVV_TfExAxXhIh5yt5nU" />
|
|
|
|
|
|
<script type="text/javascript">
|
|
var _gaq = _gaq || [];
|
|
_gaq.push(['_setAccount', 'UA-46204499-1']);
|
|
_gaq.push(['_trackPageview']);
|
|
|
|
(function() {
|
|
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
|
|
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
|
|
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
|
|
})();
|
|
</script>
|
|
|
|
|
|
</head>
|
|
|
|
|
|
<body>
|
|
<div class="container">
|
|
<div class="left-col">
|
|
<div class="intrude-less">
|
|
<header id="header" class="inner"><div class="profilepic">
|
|
|
|
|
|
|
|
</div>
|
|
|
|
<nav id="main-nav"><ul class="main">
|
|
<li><a href="https://www.unixdaemon.net/">Home</a></li>
|
|
<li><a href="https://www.unixdaemon.net/about/">About</a></li>
|
|
</ul>
|
|
</nav>
|
|
<nav id="sub-nav">
|
|
<div class="social">
|
|
|
|
|
|
|
|
|
|
<a class="twitter" href='https://twitter.com/unixdaemon' title="Twitter">Twitter</a>
|
|
|
|
|
|
<a class="github" href='https://github.com/deanwilson' title="GitHub">GitHub</a>
|
|
|
|
|
|
|
|
|
|
<a class="linkedin" href='https://www.linkedin.com/in/deanswilson' title="LinkedIn">LinkedIn</a>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<a class="rss" href='/index.xml' title="RSS">RSS</a>
|
|
|
|
</div>
|
|
<nav id="main-nav"><ul class="main">
|
|
<li><a href="https://www.unixdaemon.net/ansible/">/ansible/</a>
|
|
<a href="https://www.unixdaemon.net/books/">/books/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/career/">/career/</a>
|
|
<a href="https://www.unixdaemon.net/cloud/">/cloud/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/commandline/">/commandline/</a>
|
|
<a href="https://www.unixdaemon.net/events/">/events/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/firefox/">/firefox/</a>
|
|
<a href="https://www.unixdaemon.net/geekstuff/">/geekstuff/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/gui/">/gui/</a>
|
|
<a href="https://www.unixdaemon.net/linux/">/linux/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/meta/">/meta/</a>
|
|
<a href="https://www.unixdaemon.net/misctech/">/misctech/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/movies/">/movies/</a>
|
|
<a href="https://www.unixdaemon.net/network/">/network/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/nottech/">/nottech/</a>
|
|
<a href="https://www.unixdaemon.net/online/">/online/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/perl/">/perl/</a>
|
|
<a href="https://www.unixdaemon.net/programming/">/programming/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/puppet/">/puppet/</a>
|
|
<a href="https://www.unixdaemon.net/security/">/security/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/sites/">/sites/</a>
|
|
<a href="https://www.unixdaemon.net/sysadmin/">/sysadmin/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/tools/">/tools/</a>
|
|
<a href="https://www.unixdaemon.net/unixdaemon/">/unixdaemon/</a></li>
|
|
</ul>
|
|
</nav>
|
|
</nav>
|
|
</header>
|
|
</div>
|
|
</div>
|
|
<div class="mid-col">
|
|
|
|
<div class="mid-col-container">
|
|
<div id="content" class="inner">
|
|
<div itemscope itemtype="http://schema.org/Blog">
|
|
<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
|
|
<h1 class="title" itemprop="name">Scary sysadmin Halloween stories</h1>
|
|
|
|
<div class="date"><time datetime='Mon, Dec 7, 2020' data-updated="true" itemprop="datePublished">Mon, Dec 7, 2020</time>
|
|
</div>
|
|
<div class="tags">
|
|
|
|
|
|
|
|
|
|
<a href="https://www.unixdaemon.net/categories/sysadmin"> sysadmin </a>
|
|
|
|
|
|
|
|
</div>
|
|
|
|
<div class="entry-content" itemprop="articleBody"><p>Gremlin recently ran a small twitter hashtag challenge called
|
|
“#talesfromtheNOC” where people were invited to share their scary
|
|
sysadmin stories. Reading through of the entries I was reminded of one
|
|
slightly less than welcoming environment that led to a lot of learning,
|
|
frustration and trepidation. I’ve captured my posts here.</p>
|
|
<h2 id="opening-volley">Opening Volley</h2>
|
|
<p>I was hired as the new sysadmin at a financial services company and had
|
|
no hand over as my predecessor had apparently ‘left on short notice’. On
|
|
my first day the CTO and I took a trip to the colo to check the asset register
|
|
against the running machines.</p>
|
|
<p>After unlocking the back of the first rack I gave the door a little tug
|
|
and it didn’t budge. “It sticks sometimes. Give it a decent pull.” said
|
|
the CTO helpfully, so I put a <em>little</em> more force behind it and yanked
|
|
the door open. Luckily the company never skimped on cable tie quality so
|
|
I managed to pop out every individually tied to the rack door(!) power
|
|
cable in one swing. It took a moment for us to realise what had happened
|
|
as dozens of fans were suddenly silenced…</p>
|
|
<p>I later learned more about the meaning of ‘left on short notice’ and
|
|
found a number of other little presents over my first few months in
|
|
that role.</p>
|
|
<h2 id="the-second-incursion">The second incursion</h2>
|
|
<p>Same place as the previous story: I plugged the KVM into the management
|
|
machine and was greeted by a full X-windows session. Unusual but not the
|
|
first time I’d seen it. I logged in, decided to see if the previous
|
|
admin still had access and ran <code>visudo</code>. The process seemed to hang and
|
|
I paused for a few moments to consider what was going on.</p>
|
|
<p>Shuddering slowly to life on a heavily underspecced machine was a full
|
|
install and process of Eclipse with <code>/etc/sudoers</code> open. <code>$EDITOR</code> was set to
|
|
<code>eclipse</code>. What kind of person does that? A
|
|
<a href="https://www.youtube.com/watch?v=o7X3LYGV_N8&t=40">MONSTER!</a></p>
|
|
<h2 id="saboteur-strikes">Saboteur Strikes</h2>
|
|
<p>To finish the ‘trilogy of twattery’ the same person used to leave a
|
|
sendmail m4 file on the machine that DID NOT generate the running <code>.cf</code>
|
|
file. So if you didn’t move it out before the machines rebooted the
|
|
daemon regenerated its config and you came up with broken mail servers.</p>
|
|
<p>That was made slightly more annoying coupled with the booby trapped
|
|
power cabling previously mentioned.</p>
|
|
<p>It was a very long first day.</p>
|
|
</div>
|
|
</article>
|
|
|
|
<div class="share">
|
|
<div class="addthis_toolbox addthis_default_style ">
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<footer id="footer" class="inner">Copyright © 2025
|
|
|
|
Dean Wilson
|
|
|
|
|
|
|
|
Design credit: <a href="http://shashankmehta.in/archive/2012/greyshade.html">Shashank Mehta</a>
|
|
</footer>
|
|
</div>
|
|
</div>
|
|
|
|
</body>
|
|
</html>
|