244 lines
9.7 KiB
HTML
244 lines
9.7 KiB
HTML
<!DOCTYPE HTML>
|
|
|
|
<html lang="en-us">
|
|
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<title>Low hanging BCP and DR scenarios - UnixDaemon: In search of (a) life</title>
|
|
<meta name="author" content="map[name:Dean Wilson]">
|
|
|
|
|
|
|
|
<meta name="description" content="">
|
|
|
|
|
|
|
|
<meta name="HandheldFriendly" content="True">
|
|
<meta name="MobileOptimized" content="320">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
|
|
<link href='/index.xml' rel="alternate" title="UnixDaemon: In search of (a) life" type="application/atom+xml">
|
|
|
|
<link rel="canonical" href="https://www.unixdaemon.net/sysadmin/low-hanging-bcp-and-dr-scenarios/">
|
|
<link href="https://www.unixdaemon.net/favicon.png" rel="shortcut icon">
|
|
<link href="https://www.unixdaemon.net/css/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
|
|
<link href="https://www.unixdaemon.net/css/font-awesome.min.css" media="screen" rel="stylesheet" type="text/css">
|
|
|
|
<link href='https://fonts.googleapis.com/css?family=Nunito:400,300,700' rel='stylesheet' type='text/css'>
|
|
<meta name="google-site-verification" content="xyisveClGOK3aUjA5QIPkeGjVV_TfExAxXhIh5yt5nU" />
|
|
|
|
|
|
<script type="text/javascript">
|
|
var _gaq = _gaq || [];
|
|
_gaq.push(['_setAccount', 'UA-46204499-1']);
|
|
_gaq.push(['_trackPageview']);
|
|
|
|
(function() {
|
|
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
|
|
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
|
|
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
|
|
})();
|
|
</script>
|
|
|
|
|
|
</head>
|
|
|
|
|
|
<body>
|
|
<div class="container">
|
|
<div class="left-col">
|
|
<div class="intrude-less">
|
|
<header id="header" class="inner"><div class="profilepic">
|
|
|
|
|
|
|
|
</div>
|
|
|
|
<nav id="main-nav"><ul class="main">
|
|
<li><a href="https://www.unixdaemon.net/">Home</a></li>
|
|
<li><a href="https://www.unixdaemon.net/about/">About</a></li>
|
|
</ul>
|
|
</nav>
|
|
<nav id="sub-nav">
|
|
<div class="social">
|
|
|
|
|
|
|
|
|
|
<a class="twitter" href='https://twitter.com/unixdaemon' title="Twitter">Twitter</a>
|
|
|
|
|
|
<a class="github" href='https://github.com/deanwilson' title="GitHub">GitHub</a>
|
|
|
|
|
|
|
|
|
|
<a class="linkedin" href='https://www.linkedin.com/in/deanswilson' title="LinkedIn">LinkedIn</a>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<a class="rss" href='/index.xml' title="RSS">RSS</a>
|
|
|
|
</div>
|
|
<nav id="main-nav"><ul class="main">
|
|
<li><a href="https://www.unixdaemon.net/ansible/">/ansible/</a>
|
|
<a href="https://www.unixdaemon.net/books/">/books/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/career/">/career/</a>
|
|
<a href="https://www.unixdaemon.net/cloud/">/cloud/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/commandline/">/commandline/</a>
|
|
<a href="https://www.unixdaemon.net/events/">/events/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/firefox/">/firefox/</a>
|
|
<a href="https://www.unixdaemon.net/geekstuff/">/geekstuff/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/gui/">/gui/</a>
|
|
<a href="https://www.unixdaemon.net/linux/">/linux/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/meta/">/meta/</a>
|
|
<a href="https://www.unixdaemon.net/misctech/">/misctech/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/movies/">/movies/</a>
|
|
<a href="https://www.unixdaemon.net/network/">/network/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/nottech/">/nottech/</a>
|
|
<a href="https://www.unixdaemon.net/online/">/online/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/perl/">/perl/</a>
|
|
<a href="https://www.unixdaemon.net/programming/">/programming/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/puppet/">/puppet/</a>
|
|
<a href="https://www.unixdaemon.net/security/">/security/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/sites/">/sites/</a>
|
|
<a href="https://www.unixdaemon.net/sysadmin/">/sysadmin/</a></li>
|
|
<li><a href="https://www.unixdaemon.net/tools/">/tools/</a>
|
|
<a href="https://www.unixdaemon.net/unixdaemon/">/unixdaemon/</a></li>
|
|
</ul>
|
|
</nav>
|
|
</nav>
|
|
</header>
|
|
</div>
|
|
</div>
|
|
<div class="mid-col">
|
|
|
|
<div class="mid-col-container">
|
|
<div id="content" class="inner">
|
|
<div itemscope itemtype="http://schema.org/Blog">
|
|
<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
|
|
<h1 class="title" itemprop="name">Low hanging BCP and DR scenarios</h1>
|
|
|
|
<div class="date"><time datetime='Fri, Feb 7, 2020' data-updated="true" itemprop="datePublished">Fri, Feb 7, 2020</time>
|
|
</div>
|
|
<div class="tags">
|
|
|
|
|
|
|
|
|
|
<a href="https://www.unixdaemon.net/categories/sysadmin"> sysadmin </a>
|
|
|
|
|
|
|
|
</div>
|
|
|
|
<div class="entry-content" itemprop="articleBody"><p>Just before Christmas I had to do some work on new business continuity
|
|
plans (BCP) and disaster recovery (DR) documents. To help warm up and
|
|
get myself in the right frame of mind I posted a few easy opening
|
|
scenarios to Twitter for comment and I’ve decided to collect them back
|
|
up and post here, in my external memory, for posterity.</p>
|
|
<p>Each of these ideas should be considered the most generic and low
|
|
hanging fruit of your plans. They will not be enough on their own but
|
|
if you don’t have something covering each of them you may well have
|
|
some gaps.</p>
|
|
<h3 id="unavailable-offices">Unavailable offices</h3>
|
|
<p>What would happen if one of your office buildings became unavailable for
|
|
a few days to a couple of weeks? Do you have a
|
|
<a href="https://en.wiktionary.org/wiki/call_tree">call tree</a> in place to ensure
|
|
everyone knows not to make the journey in and why? Do you have anything
|
|
that’s only available in that location? Visual displays you can’t see the
|
|
output of via any comparable route? A safe with AWS root credentials or
|
|
physical security tokens? The easiest remedy is to be remote friendly
|
|
and practice the occasional enforced remote working week and game day.</p>
|
|
<h3 id="unavailable-third-party">Unavailable third party</h3>
|
|
<p>What would happen if your main hosting provider became unavailable to
|
|
you? There are a couple of ways to look at this one, someone compromises
|
|
your accounts, deletes everything and closes them (and makes S3 buckets
|
|
using the same names as your deleted ones, the scoundrels!). A new law
|
|
comes in preventing you from storing your data in their main
|
|
jurisdiction or your finance team makes a mistake and your systems are
|
|
shutdown due to lack of payment. You should look at this in a different
|
|
way to how you’d consider resilience in the cloud, being multi-region
|
|
doesn’t help if Google shuts you down for lack of contract agreement or
|
|
late payment.</p>
|
|
<p>On a related but smaller scale, pick any of your more focused vendors
|
|
and run the same thought experiment. What would happen if you lost
|
|
access to DataDog or PagerDuty for a week? What about Dockerhub? We’ll
|
|
ignore the loss of Slack as a ‘happy accident’. Maybe one of your
|
|
employees finally snaps from one too many <code>if err != nil {</code> and after
|
|
going on an emoji fuelled rampage gets your organisation banned from
|
|
GitHub. You might have the code locally but which processes have to
|
|
change? What things need to be migrated immediately and which can be
|
|
left in a degraded state for a few days? Collecting information on these
|
|
aspects can help inform the rebuild order and priorities if you do
|
|
suffer from a BCP level issue.</p>
|
|
<h3 id="people-its-always-the-people">People. It’s always the people.</h3>
|
|
<p>How many people, in which roles, can you afford to not have available?
|
|
From the risk of an employee leaving lunch laying out an entire team
|
|
with food poisoning (which I’ve had happen) to the positively spun
|
|
scenario of the office lottery syndicate winning and all leaving without
|
|
handing back their yubikeys. Do you have any roles with special sign off
|
|
powers for compliance or regulatory reasons? What is your response?
|
|
There’s nothing wrong with the answer being ‘We can’t afford to have the
|
|
extra headcount so we’d enter “maintenance mode” until they recover’ but
|
|
it’s always nicer to think about these things when you’re not in the
|
|
middle of one of them.</p>
|
|
<p>How about crossing the streams? Both people that work at the small
|
|
vendor you use to run your CI/CD lava lamps decide they’d rather do
|
|
something non-technical and walk off into the sunset. What’s your
|
|
response? Questions like these are useful things to include as part of
|
|
your procurement process so you always (assuming you do the occasionally
|
|
review) have a deliberate fallback plan.</p>
|
|
<p>One of my favourite comments on the original twitter thread, and related to people, was from
|
|
<a href="https://twitter.com/tgulfie/status/1206660189476147201">tgulfie on Decimation</a>.
|
|
“Get a 10 sided die, when people arrive on Monday they roll it. When
|
|
someone rolls a 10, they get an orange arm band. Anyone with an orange
|
|
arm band is unavailable for any business operations, they can however
|
|
watch and learn what happens when they are not there.” This could be
|
|
that mythical “I’ll go back and write documentation for $foo” time we’ve
|
|
always dreamed of.</p>
|
|
<p>None of these examples are really industry or company specific and they
|
|
don’t take local context into consideration. Hopefully they widely
|
|
applicable enough to help kick start your own thinking and lead to more
|
|
relevant and important questions for your own platform or company.</p>
|
|
</div>
|
|
</article>
|
|
|
|
<div class="share">
|
|
<div class="addthis_toolbox addthis_default_style ">
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<footer id="footer" class="inner">Copyright © 2025
|
|
|
|
Dean Wilson
|
|
|
|
|
|
|
|
Design credit: <a href="http://shashankmehta.in/archive/2012/greyshade.html">Shashank Mehta</a>
|
|
</footer>
|
|
</div>
|
|
</div>
|
|
|
|
</body>
|
|
</html>
|