Files
nexus/sreweekly/articles/305/07-some-ways-dns-can-break.html
2026-09-12 17:23:01 +08:00

421 lines
24 KiB
HTML

<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta charset="utf-8">
<title>Some ways DNS can break</title>
<meta name="author" content="Julia Evans">
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="description" content="Some ways DNS can break">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:title" content='Some ways DNS can break'>
<meta property="og:type" content="website" />
<meta property="og:url" content="https://jvns.ca/blog/2022/01/15/some-ways-dns-can-break/" />
<meta property="og:site_name" content="Julia Evans" />
<link rel="canonical" href="https://jvns.ca/blog/2022/01/15/some-ways-dns-can-break/">
<link href="/favicon.ico" rel="icon">
<link href="/stylesheets/screen.css" rel="preload" type="text/css" as="style">
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
<link href="/stylesheets/print.css" media="print" rel="stylesheet" type="text/css">
<link href="/atom.xml" rel="alternate" title="Julia Evans" type="application/atom+xml">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css" integrity="sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0" crossorigin="anonymous">
<script defer data-domain="jvns.ca" src="https://plausible.io/js/script.js"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js" integrity="sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4" crossorigin="anonymous"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
<script defer type="text/javascript">
window.heap=window.heap||[],heap.load=function(e,t){window.heap.appid=e,window.heap.config=t=t||{};var r=document.createElement("script");r.type="text/javascript",r.async=!0,r.src="https://cdn.heapanalytics.com/js/heap-"+e+".js";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(r,a);for(var n=function(e){return function(){heap.push([e].concat(Array.prototype.slice.call(arguments,0)))}},p=["addEventProperties","addUserProperties","clearEventProperties","identify","resetIdentity","removeEventProperty","setEventProperties","track","unsetEventProperty"],o=0;o<p.length;o++)heap[p[o]]=n(p[o])};
heap.load("2242143965");
</script>
</head>
<body>
<div id="skiptocontent">
<a href="#main">Skip to main content</a>
</div>
<div id="wrap">
<header role="banner">
<hgroup>
<h1><a href="/">Julia Evans</a></h1>
</hgroup>
<ul class="header-links">
<li><a href="/about">About</a></li>
<li><a href="/talks">Talks</a></li>
<li><a href="/projects/">Projects</a></li>
<li><a rel="me" href="https://social.jvns.ca/@b0rk">Mastodon</a></li>
<li><a href="https://bsky.app/profile/b0rk.jvns.ca">Bluesky</a></li>
<li><a href="https://github.com/jvns">Github</a></li>
</ul>
</header>
<nav role="navigation" class="header-nav"><ul class="main-navigation">
<li><a href="/categories/favorite/">Favorites</a></li>
<li><a href="/til/">TIL</a></li>
<li><a href="https://wizardzines.com">Zines</a></li>
<li class="subscription" data-subscription="rss"><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
</ul>
</nav>
<div id="main">
<div id="content">
<div>
<article class="hentry" role="article">
<header>
<h1 class="entry-title">Some ways DNS can break</h1>
<div class="post-tags">
•
<a class="post-tag" href="/categories/dns">dns</a> •
</div>
<p class="meta sans">
<time class="date" datetime="2022-01-15T12:25:37" pubdate data-updated="true">
January 15, 2022
</time>
</p>
</header>
<main>
<p>When I first learned about it, DNS didn&rsquo;t seem like it should be THAT
complicated. Like, there are DNS records, they&rsquo;re stored on a server, what&rsquo;s
the big deal?</p>
<p>But with DNS, reading about how it works in a textbook doesn&rsquo;t prepare you for
the sheer volume of different ways DNS can break your system in practice. It&rsquo;s
not just caching problems!</p>
<p>So I <a href="https://twitter.com/b0rk/status/1481265429897261058">asked people on Twitter</a> for
example of DNS problems they&rsquo;ve run into, especially DNS problems that <strong>didn&rsquo;t
initially appear to be DNS problems</strong>. (the popular &ldquo;it&rsquo;s always DNS&rdquo; meme)</p>
<p>I&rsquo;m not going to discuss how to solve or avoid any of these problems in this
post, but I&rsquo;ve linked to webpages discussing the problem where I could find
them.</p>
<h3 id="problem-slow-network-requests" class="post-heading">
<a href="#problem-slow-network-requests">
problem: slow network requests
</a>
</h3>
<p>Your network requests are a little bit slower than expected, and it&rsquo;s actually
because your DNS resolver is slow for some reason. This might be because the
resolver is under a lot of load, or it has a memory leak, or something else.</p>
<p>I&rsquo;ve run into this before with my router&rsquo;s DNS forwarder &ndash; all of my DNS
requests were slow, and I restarted my router and that fixed the problem.</p>
<h3 id="problem-dns-timeouts" class="post-heading">
<a href="#problem-dns-timeouts">
problem: DNS timeouts
</a>
</h3>
<p>A couple of people mentioned network requests that were taking 2+ seconds or 30
seconds because of DNS queries that were timing out. This is sort of the same
as &ldquo;slow requests&rdquo;, but it&rsquo;s worse because queries can take several seconds to
time out.</p>
<p>Sophie Haskins has a great blog post <a href="https://blog.sophaskins.net/blog/misadventures-with-kube-dns/">Misadventures with Kube DNS</a> about DNS
timeouts with Kubernetes.</p>
<h3 id="problem-ndots" class="post-heading">
<a href="#problem-ndots">
problem: ndots
</a>
</h3>
<p>A few people mentioned a specific issue where Kubernetes sets <code>ndots:5</code> in its <code>/etc/resolv.conf</code></p>
<p>Here&rsquo;s an example /etc/resolv.conf from <a href="https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html">Kubernetes pods /etc/resolv.conf ndots:5 option and why it may negatively affect your application performances</a>.</p>
<pre><code>nameserver 100.64.0.10
search namespace.svc.cluster.local svc.cluster.local cluster.local eu-west-1.compute.internal
options ndots:5
</code></pre>
<p>My understanding is that if this is your <code>/etc/resolv.conf</code> and you look up
<code>google.com</code>, your application will call the C <code>getaddrinfo</code> function, and
<code>getaddrinfo</code> will:</p>
<ol>
<li>look up <code>google.com.namespace.svc.cluster.local.</code></li>
<li>look up <code>google.com.svc.cluster.local.</code></li>
<li>look up <code>google.com.cluster.local.</code></li>
<li>look up <code>google.com.eu-west-1.compute.internal.</code></li>
<li>look up <code>google.com.</code></li>
</ol>
<p>Basically it checks if <code>google.com</code> is actually a subdomain of everything on the <code>search</code> line.</p>
<p>So every time you make a DNS query, you need to wait for 4 DNS queries to fail
before you can get to the actual real DNS query that succeeds.</p>
<h3 id="problem-it-s-hard-to-tell-what-dns-resolver-s-your-system-is-using" class="post-heading">
<a href="#problem-it-s-hard-to-tell-what-dns-resolver-s-your-system-is-using">
problem: it&rsquo;s hard to tell what DNS resolver(s) your system is using
</a>
</h3>
<p>This isn&rsquo;t a bug by itself, but when you run into a problem with DNS, often
it&rsquo;s related in some way to your DNS resolver. I don&rsquo;t know of any foolproof
way to tell what DNS resolver is being used.</p>
<p>A few things I know:</p>
<ul>
<li>on Linux, I think that most things use /etc/resolv.conf to choose a DNS
resolver. There are definitely exceptions though, for example your browser
might ignore /etc/resolv.conf and use a different DNS-over-HTTPS service
instead.</li>
<li>if you&rsquo;re using UDP DNS, you can use <code>sudo tcpdump port 53</code> to see where DNS
requests are being sent. This doesn&rsquo;t work if you&rsquo;re using DNS over HTTPS or
DNS over TLS though.</li>
</ul>
<p>I also vaguely remember it being even more confusing on MacOS than on Linux,
though I don&rsquo;t know why.</p>
<h3 id="problem-dns-servers-that-return-nxdomain-instead-of-noerror" class="post-heading">
<a href="#problem-dns-servers-that-return-nxdomain-instead-of-noerror">
problem: DNS servers that return NXDOMAIN instead of NOERROR
</a>
</h3>
<p>Here&rsquo;s a problem that I ran into once, where nginx couldn&rsquo;t resolve a domain.</p>
<ul>
<li>I set up nginx to use a specific DNS server to resolve DNS queries</li>
<li>when visiting the domain, nginx made 2 queries, one for an <code>A</code> record, and one for an <code>AAAA</code> record</li>
<li>the DNS server returned a <code>NXDOMAIN</code> reply for the <code>A</code> query</li>
<li>nginx decided &ldquo;ok, that domain doesn&rsquo;t exist&rdquo;, and gave up</li>
<li>the DNS server returned a successful reply for the <code>AAAA</code> query</li>
<li>nginx ignored the <code>AAAA</code> record because it had already given up</li>
</ul>
<p>The problem was that the DNS server should have returned <code>NOERROR</code> &ndash; that
domain <em>did</em> exist, it was just that there weren&rsquo;t any <code>A</code> records for it. I
reported the bug, they fixed it, and that fixed the problem.</p>
<p>I&rsquo;ve implemented this bug myself too, so I understand why it happens &ndash; it&rsquo;s
easy to think &ldquo;there aren&rsquo;t any records for this query, I should return an
<code>NXDOMAIN</code> error&rdquo;.</p>
<h3 id="problem-negative-dns-caching" class="post-heading">
<a href="#problem-negative-dns-caching">
problem: negative DNS caching
</a>
</h3>
<p>If you visit a domain before creating a DNS record for it, the <strong>absence</strong> of
the record will be cached. This is very surprising the first time your run into
it &ndash; I only learned about this last year!</p>
<p>The TTL for cache entry is the TTL of the domain&rsquo;s SOA record &ndash; for example
for <code>jvns.ca</code>, it&rsquo;s an hour.</p>
<h3 id="problem-nginx-caching-dns-records-forever" class="post-heading">
<a href="#problem-nginx-caching-dns-records-forever">
problem: nginx caching DNS records forever
</a>
</h3>
<p>If you put this in your nginx config:</p>
<pre><code>location / {
proxy_pass https://some.domain.com;
}
</code></pre>
<p>then nginx will resolve <code>some.domain.com</code> once on startup and never again. This
is especially dangerous if the IP address for <code>some.domain.com</code> changes
infrequently, because it might keep happily working for months and then
suddenly break at 2am one day.</p>
<p>There are pretty well-known ways to fix this and this post isn&rsquo;t about nginx so
I won&rsquo;t get into it, but it&rsquo;s surprising the first time you run into it.</p>
<p>Here&rsquo;s a <a href="https://medium.com/driven-by-code/dynamic-dns-resolution-in-nginx-22133c22e3ab">blog post</a> with a story of how this happened to someone with an AWS load balancer.</p>
<h3 id="problem-java-caching-dns-records-forever" class="post-heading">
<a href="#problem-java-caching-dns-records-forever">
problem: Java caching DNS records forever
</a>
</h3>
<p>Same thing, but for Java: <a href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html">Apparently</a>
depending on how you configure Java, &ldquo;the JVM default TTL [might be] set so
that it will never refresh DNS entries until the JVM is restarted.&rdquo;</p>
<p>I haven&rsquo;t run into this myself but I asked a friend about it who writes more
Java than me and they told me that it&rsquo;s happened to them.</p>
<p>Of course, literally any software could have this problem of caching DNS
records forever, but the main cases I&rsquo;ve heard of in practice are nginx and
Java.</p>
<h3 id="problem-that-entry-in-etc-hosts-you-forgot-about" class="post-heading">
<a href="#problem-that-entry-in-etc-hosts-you-forgot-about">
problem: that entry in /etc/hosts you forgot about
</a>
</h3>
<p>Another variant on caching issues: entries in <code>/etc/hosts</code> that override your
usual DNS settings!</p>
<p>This is extra confusing because <code>dig</code> ignores <code>/etc/hosts</code>, so everything SEEMS
like it should be fine (&quot;<code>dig whatever.com</code> is working!&quot;).</p>
<h3 id="problem-your-email-isn-t-being-sent-is-going-to-spam" class="post-heading">
<a href="#problem-your-email-isn-t-being-sent-is-going-to-spam">
problem: your email isn&rsquo;t being sent / is going to spam
</a>
</h3>
<p>The way email is sent and validated is through DNS (MX records, SPF records,
DKIM records), so a lot of email problems are DNS problems.</p>
<h3 id="problem-internationalized-domain-names-don-t-work" class="post-heading">
<a href="#problem-internationalized-domain-names-don-t-work">
problem: internationalized domain names don&rsquo;t work
</a>
</h3>
<p>You can register domain names with non-ASCII characters or emoji like <a href="https://%F0%9F%92%A9.la/">https://💩.la</a>.</p>
<p>The way this works with DNS is that <code>💩.la</code> gets translated into <code>xn--ls8h.la</code> with an encoding called &ldquo;punycode&rdquo;.</p>
<p>But even though there&rsquo;s a clear standard for how they should work with DNS, a lot of software doesn&rsquo;t handle internationalized domain names well!
There&rsquo;s a fun story about this in Julian Squires&rsquo; great talk <a href="https://www.youtube.com/watch?v=UE-fJjMasec">The emoji that Killed Chrome!!</a>.</p>
<h3 id="problem-tcp-dns-is-blocked-by-a-firewall" class="post-heading">
<a href="#problem-tcp-dns-is-blocked-by-a-firewall">
problem: TCP DNS is blocked by a firewall
</a>
</h3>
<p>A couple of people mentioned that some firewalls allow UDP port 53 but not TCP
port 53. But large DNS queries need to use TCP port 53, so this can cause weird
intermittent problems that are hard to debug.</p>
<h3 id="problem-musl-doesn-t-support-tcp-dns" class="post-heading">
<a href="#problem-musl-doesn-t-support-tcp-dns">
problem: musl doesn&rsquo;t support TCP DNS
</a>
</h3>
<p>A lot of applications use libc&rsquo;s <code>getaddrinfo</code> to make DNS queries. musl is an
alternative to <code>glibc</code> that&rsquo;s used in Alpine Docker container which doesn&rsquo;t
support TCP DNS. This can cause problems if you make DNS queries where the
response would be too big to fit inside a regular DNS UDP packet (512 bytes).</p>
<p>I&rsquo;m still a bit fuzzy on this so I might have it wrong, but my understanding of how this can break is:</p>
<ol>
<li>musl&rsquo;s getaddrinfo makes a DNS query</li>
<li>the DNS server notices that the response is too big to fit in a single DNS response packet</li>
<li>the DNS server returns an <strong>empty</strong> truncated response, expecting that the client will retry by making a TCP DNS query</li>
<li><code>musl</code> does not support TCP so it does not retry</li>
</ol>
<p>A blog post about this: <a href="https://christoph.luppri.ch/fixing-dns-resolution-for-ruby-on-alpine-linux">DNS resolution issue in Alpine Linux</a></p>
<h3 id="problem-round-robin-dns-doesn-t-work-with-getaddrinfo" class="post-heading">
<a href="#problem-round-robin-dns-doesn-t-work-with-getaddrinfo">
problem: round robin DNS doesn&rsquo;t work with <code>getaddrinfo</code>
</a>
</h3>
<p>One way you could approach load balancing is to use &ldquo;round robin DNS&rdquo;. The idea
is that every time you make a DNS query, you get a different IP address.
Apparently this works if you use <code>gethostbyname</code> to make DNS queries, but it
does not work if you use <code>getaddrinfo</code> because <code>getaddrinfo</code> sorts the IP
responses it receives.</p>
<p>So you could run into an upsetting problem if you switch from <code>gethostbyname</code> to <code>getaddrinfo</code> behind the scenes without realising that this will break your DNS load balancing.</p>
<p>This is especially insidious because you might not realize that you&rsquo;re
switching to <code>gethostbyname</code> to <code>getaddrinfo</code> at all &ndash; if you&rsquo;re not writing a
C program, those functions calls are hidden inside some library. So it could be
part of a seemingly innocuous upgrade.</p>
<p>Here are a couple of pages discussing this:</p>
<ul>
<li><a href="https://groups.google.com/g/consul-tool/c/AGgPjrrkw3g">getaddrinfo breaks round robin DNS</a></li>
<li><a href="https://daniel.haxx.se/blog/2012/01/03/getaddrinfo-with-round-robin-dns-and-happy-eyeballs/">getaddrinfo with round robin DNS and happy eyeballs</a></li>
</ul>
<h3 id="problem-a-race-condition-when-starting-a-service" class="post-heading">
<a href="#problem-a-race-condition-when-starting-a-service">
problem: a race condition when starting a service
</a>
</h3>
<p>A problem someone <a href="https://mobile.twitter.com/omatskiv/status/1481305175440646148">mentioned</a>
with Kubernetes DNS: they had 2 containers which started simultaneously and
immediately tried to resolve each other. But the DNS lookup failed because the
Kubernetes DNS change hadn&rsquo;t happened yet, and then the failure was cached so
it kept failing.</p>
<h3 id="that-s-all" class="post-heading">
<a href="#that-s-all">
that&rsquo;s all!
</a>
</h3>
<p>I&rsquo;ve definitely missed some important DNS problems here, so I&rsquo;d love to hear
what I&rsquo;ve missed. I&rsquo;d also love links to blog posts that write up examples of
these problems &ndash; I think it&rsquo;s really useful to see how the problem
specifically manifests in practice and how people debugged it.</p>
</main>
<footer>
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; display: inline;}
#mc_embed_signup {
display: inline;
}
#mc_embed_signup input.button {
background: #ff5e00;
display: inline;
color: white;
padding: 6px 12px;
}
</style>
<div class="sharing">
<style>
.form-inline {
display:flex; flex-flow: row wrap; justify-content: center;
}
.form-inline input, .form-inline span {
padding: 10px;
}
.form-inline input {
display:inline;
max-width:30%;
margin: 0 10px 0 0;
background-color: #fff;
border: 1px solid #ddd;
border-radius: 5px;
padding: 10px;
}
button {
background-color: #f50;
box-shadow: none;
border: 0;
border-radius: 5px;
color: white;
padding: 5px 10px;
}
@media (max-width: 800px) {
.form-inline input {
margin: 10px 0;
max-width:100% !important;
}
.form-inline {
flex-direction: column;
align-items: stretch;
}
}
</style>
<div align="center">
<form class="form-inline" action="https://app.convertkit.com/forms/1052396/subscriptions" method="post" data-uid="8884355abb" data-format="inline" data-version="5">
<span> Want a weekly digest of this blog?</span>
<input name="email_address" type="text" placeholder="Email address" />
<button type="submit" data-element="submit">Subscribe</button>
</form>
</div>
</div>
<p class="meta">
<a class="basic-alignment left" href="https://jvns.ca/blog/2022/01/11/how-to-find-a-domain-s-authoritative-nameserver/" title="Previous Post: How to find a domain&#39;s authoritative nameservers">How to find a domain&#39;s authoritative nameservers</a>
<a class="basic-alignment right" href="https://jvns.ca/blog/2022/01/24/hosting-my-static-sites-with-nginx/" title="Next Post: Hosting my static sites with nginx">Hosting my static sites with nginx</a>
</p>
</footer>
</article>
</div>
</div>
</div>
<nav role="navigation" class="footer-nav"> <a href="/">Archives</a>
</nav>
<footer role="contentinfo"><span class="credit">&copy; Julia Evans. </span>
<span>If you like this, you may like <a href="https://web.archive.org/web/20181228051203/http://www.uliaea.ca/">Ulia Ea</a> or, more seriously, this list of <a href="https://jvns.ca/blogroll">blogs I love</a> or some <a href="https://jvns.ca/bookshelf">books I've read</a>. <br>
<p class="rc-scout__text"><i class="rc-scout__logo"></i>
You might also like the <a class="rc-scout__link" href="https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8">Recurse Center</a>, my very favorite programming community <a href="/categories/hackerschool/">(my posts about it)</a></p>
</span>
<style class="rc-scout__style" type="text/css">.rc-scout{display:block;padding:0;border:0;margin:0;}.rc-scout__text{display:block;padding:0;border:0;margin:0;height:100%;font-size:100%;}.rc-scout__logo{display:inline-block;padding:0;border:0;margin:0;width:0.85em;height:0.85em;background:no-repeat center url('data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2012%2015%22%3E%3Crect%20x%3D%220%22%20y%3D%220%22%20width%3D%2212%22%20height%3D%2210%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%221%22%20width%3D%2210%22%20height%3D%228%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%222%22%20width%3D%228%22%20height%3D%226%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%229%22%20width%3D%224%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%2211%22%20width%3D%2210%22%20height%3D%224%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%220%22%20y%3D%2212%22%20width%3D%2212%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%225%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%227%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%228%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%229%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3C%2Fsvg%3E');}.rc-scout__link:link,.rc-scout__link:visited{color:#61ae24;text-decoration:underline;}.rc-scout__link:hover,.rc-scout__link:active{color:#4e8b1d;}</style>
</footer>
<script type="text/rocketscript">
(function(){
var twitterWidgets = document.createElement('script');
twitterWidgets.type = 'text/javascript';
twitterWidgets.async = true;
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
})();
</script>
</div>
</body>
</html>