218 lines
14 KiB
HTML
218 lines
14 KiB
HTML
<!doctype html>
|
|
<html lang="en-US">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<title>Detecting who used the EC2 metadata server with BCC - fREW Schmidt's Foolish Manifesto</title>
|
|
<meta property="og:title" content="Detecting who used the EC2 metadata server with BCC" />
|
|
<meta name="twitter:title" content="Detecting who used the EC2 metadata server with BCC" />
|
|
<meta name="description" content="Recently at work we had a minor incident involving exhaustion of the EC2 metadata server on some of our hosts. I was able to get enough detail to delegate the rest to a team to fix the issue. ">
|
|
<meta property="og:description" content="Recently at work we had a minor incident involving exhaustion of the EC2 metadata server on some of our hosts. I was able to get enough detail to delegate the rest to a team to fix the issue. ">
|
|
<meta name="twitter:description" content="Recently at work we had a minor incident involving exhaustion of the EC2 metadata server on some of our hosts. I was able to get enough detail to delegate the rest to a team to fix the issue. ">
|
|
<meta name="author" content="Arthur Axel fREW Schmidt"/>
|
|
<meta name="twitter:card" content="summary" />
|
|
<meta name="twitter:site" content="@frioux" />
|
|
<meta name="twitter:creator" content="@frioux" />
|
|
<meta property="og:url" content="https://blog.afoolishmanifesto.com/posts/detecting-who-used-ec2-metadata-server-bcc/" />
|
|
<meta property="og:type" content="website" />
|
|
<meta property="og:site_name" content="fREW Schmidt's Foolish Manifesto" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<link href="/static/css/bootstrap-replacement.css" rel="stylesheet"/>
|
|
<link href="/static/css/styles.css" rel="stylesheet"/>
|
|
<link href="/static/img/fav.png" rel='icon' type='image/x-icon'/>
|
|
|
|
</head>
|
|
<body>
|
|
|
|
<nav class="navbar navbar-inverse navbar-fixed-top" role="navigation">
|
|
<input type="checkbox" id="nav-toggle" class="nav-toggle-checkbox" aria-label="Toggle navigation">
|
|
<div class="navbar-header">
|
|
<label for="nav-toggle" class="navbar-toggle">
|
|
<span class="sr-only">Toggle navigation</span>
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
</label>
|
|
<a class="navbar-brand" href="/">fREW Schmidt's Foolish Manifesto</a>
|
|
</div>
|
|
|
|
<div class="navbar-collapse">
|
|
<ul class="nav navbar-nav navbar-right">
|
|
<li><a class="sigil" href="https://github.com/frioux"><img alt="github profile" width=23 height=23 src="/static/img/GitHub-Mark-Light-120px-plus.png" \></a></li>
|
|
<li><a class="sigil" href="https://twitter.com/frioux"><img alt="twitter profile" width=23 height=23 src="/static/img/Twitter social icons - circle - white.svg" \></a></li>
|
|
</ul>
|
|
</div>
|
|
</nav>
|
|
|
|
|
|
<div class="container" id="main">
|
|
|
|
<h1>Detecting who used the EC2 metadata server with BCC</h1><p>Recently at work we had a minor incident involving exhaustion of the EC2
|
|
metadata server on some of our hosts. I was able to get enough detail to
|
|
delegate the rest to a team to fix the issue.</p>
|
|
|
|
<p></p>
|
|
|
|
<p>AWS EC2 has this thing called the metadata server. As far as the user can tell
|
|
it runs inside the hypervisor and is exposed directly to your host via an http
|
|
server at 169.254.169.254. You can use <code>curl(1)</code> to get basic info about your
|
|
server, like what instance type it is, for example.</p>
|
|
|
|
<p>The metadata server <em>also</em> hosts per instance authentication data. Your hosts
|
|
reach out to the metadata server, get some auth material, and use that auth
|
|
material for each request that interacts with AWS.</p>
|
|
|
|
<p>This means that if you somehow <em>exhaust</em> the metadata server processes will not
|
|
be able to authenticate with AWS. I haven’t dove in deeply to understand if
|
|
it’s a rate limit or a concurrency limit, but I can say that in any case we ran
|
|
into it.</p>
|
|
|
|
<h2 id="detecting-the-bad-actor"><a href="#detecting-the-bad-actor" class="hanchor" ariaLabel="Anchor"> 🔗 </a> Detecting the Bad Actor</h2>
|
|
|
|
<p>Initially I was going to use <code>tcpdump(1)</code> to figure out what was happening, but
|
|
as far as I know it does not expose process ids, and even if it did I suspect
|
|
you’d have to do a dump per process.</p>
|
|
|
|
<p>My go to for “stuff lower level than <code>strace(1)</code> is
|
|
<a href="https://github.com/iovisor/bcc">BCC</a>. BCC is a Linux-ish DTrace; and I say
|
|
that in every sense. Just like Zones are a single, standalone thing in Solaris,
|
|
and containers are a combination of two or more complex Linux features, BCC
|
|
takes advantage of two or more compilers, kprobes, uprobes, and surely more. I
|
|
don’t know everything there is to know about BCC, but generally speaking I don’t
|
|
have to because there is a nice suite of tools to give you what you want.</p>
|
|
|
|
<p>I loaded up the <a href="https://github.com/iovisor/bcc/tree/master/tools">tool
|
|
listing</a>, searched for <code>tcp</code>,
|
|
and the second tool is <code>tcpconnect</code>. Here’s a basic example:</p>
|
|
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ sudo /usr/share/bcc/tools/tcpconnect
|
|
PID COMM IP SADDR DADDR DPORT
|
|
<span style="color:#ae81ff">15100</span> curl <span style="color:#ae81ff">4</span> <span style="color:#ae81ff">10</span>.1.18.45 <span style="color:#ae81ff">192</span>.30.255.112 <span style="color:#ae81ff">80</span>
|
|
<span style="color:#ae81ff">15110</span> curl <span style="color:#ae81ff">4</span> <span style="color:#ae81ff">10</span>.1.18.45 <span style="color:#ae81ff">216</span>.58.192.14 <span style="color:#ae81ff">80</span> </code></pre></div>
|
|
<p>In the actual incident though I only wanted <code>169.254.169.254</code>, so I changed my
|
|
command to:</p>
|
|
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ sudo /usr/share/bcc/tools/tcpconnect | grep -F <span style="color:#ae81ff">169</span>.254.169.254</code></pre></div>
|
|
<p>I stopped getting any output at all, but from experience I know that’s because
|
|
that <code>tcpconnect</code> is now buffering. <a href="https://blog.plover.com/Unix/stdio-buffering.html">Dominus recently had a blog post that
|
|
discusses this</a>, including
|
|
solutions, so I tweaked the command to be:</p>
|
|
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ sudo stdbuf -oL /usr/share/bcc/tools/tcpconnect |
|
|
stdbuf -oL grep -F <span style="color:#ae81ff">169</span>.254.169.254</code></pre></div>
|
|
<p>I should figure out if I can just do something like <code>exec stdbuf -oL $SHELL</code>,
|
|
but anyway the above works. So now the output will be something like this,
|
|
printed as the connections are made:</p>
|
|
|
|
<pre><code>15100 curl 4 10.1.18.45 169.254.169.254 80
|
|
15110 curl 4 10.1.18.45 169.254.169.254 80
|
|
</code></pre>
|
|
|
|
<h2 id="getting-more-detail"><a href="#getting-more-detail" class="hanchor" ariaLabel="Anchor"> 🔗 </a> Getting More Detail</h2>
|
|
|
|
<p>This is great, but our processes set their name and the COMM field above
|
|
truncates it. Side note: if you are running a fork based service, <em>set your
|
|
process name to something relevant</em>. It’s really useful and basically free.</p>
|
|
|
|
<p>In Perl you can set it by simply doing <code>$0 = "..."</code>.</p>
|
|
|
|
<p>My next step was to add a dash of Perl to grab the full process name. Here’s
|
|
what I ended up with:</p>
|
|
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ sudo stdbuf -oL /usr/share/bcc/tools/tcpconnect |
|
|
stdbuf -oL grep -F <span style="color:#ae81ff">169</span>.254.169.254 |
|
|
stdbuf -oL perl -pae<span style="color:#e6db74">'$F[1] = `cat /proc/$F[0]/cmdline`;
|
|
</span><span style="color:#e6db74"> $_ = join("\t", scalar(localtime), @F) . "\n"'</span></code></pre></div>
|
|
<p>Perl’s <code>-a</code> flag makes it act a bit like <code>awk(1)</code>, in that it tokenizes input on
|
|
whitespace and populates <code>@F</code> with your data. So <code>$F[0]</code> is the pid, <code>$F[1]</code>
|
|
becomes the untruncated name. I also added the timestamp. Here’s a (sanitized) example of the output:</p>
|
|
|
|
<pre><code>Wed Jun 20 14:21:33 2018 perform-queued-tasks-manager send-welcome-email 4 10.0.1.186 169.254.169.254 80
|
|
Wed Jun 20 14:21:33 2018 perform-queued-tasks-manager send-welcome-email 4 10.0.1.186 169.254.169.254 80
|
|
Wed Jun 20 14:21:33 2018 perform-queued-tasks-manager send-welcome-email 4 10.0.1.186 169.254.169.254 80
|
|
Wed Jun 20 14:21:33 2018 perform-queued-tasks-manager send-welcome-email 4 10.0.1.186 169.254.169.254 80
|
|
Wed Jun 20 14:21:33 2018 perform-queued-tasks-manager send-welcome-email 4 10.0.1.186 169.254.169.254 80
|
|
</code></pre>
|
|
|
|
<hr />
|
|
|
|
<p>I’ve wanted to use low level Linux instrumentation in anger for years, and the
|
|
fact that I did without thinking much about it is delightful. Thankfully I
|
|
don’t need this kind of information very often, but having it available is
|
|
great.</p>
|
|
|
|
<hr />
|
|
|
|
<p>(The following includes affiliate links.)</p>
|
|
|
|
<p>I don’t think there is a book about BCC (yet.) I think the closest thing would
|
|
be Brendan Gregg’s
|
|
<a target="_blank" href="https://www.amazon.com/gp/product/0133390098/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0133390098&linkCode=as2&tag=afoolishmanif-20&linkId=20dafcbf13582f9fe5049d9fde39dd79">Systems Performance</a><img src="//ir-na.amazon-adsystem.com/e/ir?t=afoolishmanif-20&l=am2&o=1&a=0133390098" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" />.
|
|
It’s got a ton of detail and a good helping of methodology that will help with
|
|
the kind of stuff that one tends to use BCC for.</p>
|
|
|
|
<p>BCC is very much implemented atop Linux, so it is worth knowing Linux and Unix
|
|
if you ever need to do something more advanced than use an out-of-the-box tool.
|
|
I suggest reading
|
|
<a target="_blank" href="https://www.amazon.com/gp/product/1593272200/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1593272200&linkCode=as2&tag=afoolishmanif-20&linkId=afca82c8c1ccaa7f97bd25b0c8e6a062">The Linux Programming Interface</a><img src="//ir-na.amazon-adsystem.com/e/ir?t=afoolishmanif-20&l=am2&o=1&a=1593272200" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" />
|
|
for that kind of information.</p>Posted Thu, Jun 21, 2018<br />
|
|
|
|
|
|
<hr>
|
|
|
|
<p>If you're interested in being notified when new posts are published,
|
|
<a href="/cdn-cgi/l/email-protection#06646a696146676069696a6f756e6b67686f60637572692865696b397573646c6365723b5573647565746f6463206469627f3b506f672334364263726365726f6861233436716e6923343673756362233436726e632334364345342334366b63726762677267233436756374706374233436716f726e233436444545">you can subscribe here</a>; you'll get an email once a
|
|
week at the most.</p>
|
|
|
|
<div id="disqus_thread"></div>
|
|
<div id="disqus_loader" style="text-align: center">
|
|
<button style="width:100%" onclick="if (!window.__cfRLUnblockHandlers) return false; load_disqus()" data-cf-modified-903a19e17f535a0029e90998-="">Load Comments</button>
|
|
<script data-cfasync="false" src="/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js"></script><script type="903a19e17f535a0029e90998-text/javascript">
|
|
var disqus_loaded = false;
|
|
function load_disqus() {
|
|
disqus_loaded = true;
|
|
|
|
var disqus_shortname = 'afoolishmanifesto';
|
|
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
|
|
dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
|
|
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
|
|
var ldr = document.getElementById('disqus_loader');
|
|
ldr.parentNode.removeChild(ldr);
|
|
}
|
|
|
|
if (window.location.hash.match(/(?:disqus_thread|comment-\d+)/)) {
|
|
load_disqus();
|
|
}
|
|
|
|
window.onscroll = function(e) {
|
|
if ((window.innerHeight + window.scrollY) >= document.body.offsetHeight) {
|
|
|
|
if (disqus_loaded==false){ load_disqus() };
|
|
}
|
|
};
|
|
</script>
|
|
</div>
|
|
|
|
</div>
|
|
|
|
<div class="container">
|
|
<hr>
|
|
<footer id="footer">
|
|
<p class="pull-right"><a href="#top">Back to top</a></p>
|
|
<ul id="tags">
|
|
|
|
<li><a href="/tags/iam">iam</a> </li>
|
|
|
|
<li><a href="/tags/perl">perl</a> </li>
|
|
|
|
<li><a href="/tags/bcc">bcc</a> </li>
|
|
|
|
<li><a href="/tags/instrumentation">instrumentation</a> </li>
|
|
|
|
<li><a href="/tags/linux">linux</a> </li>
|
|
|
|
<li><a href="/tags">all tags</a></li>
|
|
</ul>
|
|
</footer>
|
|
</div>
|
|
<script src="/cdn-cgi/scripts/7d0fa10a/cloudflare-static/rocket-loader.min.js" data-cf-settings="903a19e17f535a0029e90998-|49" defer></script><script type="module" src="https://static.cloudflareinsights.com/beacon.min.js/v31edd6df95cf4e85bb4c19e7a9bdbcba1788362987495" integrity="sha512-iIg7k2xntmwu6/uSb5tpc/hySgZc4eoL31yB29W6tJFo2akwjPWcEqnCEdJvGexCL0KEQwVYv5BlowfhVz26hg==" data-cf-beacon='{"version":"2024.11.0","token":"8e46d7ced7ef4a8db880960c160d77d3","r":1,"spa":2}' crossorigin="anonymous"></script>
|
|
</body>
|
|
|
|
</html>
|
|
|