421 lines
24 KiB
HTML
421 lines
24 KiB
HTML
<!DOCTYPE html>
|
|
|
|
|
|
<html class="no-js" lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<title>Some ways DNS can break</title>
|
|
<meta name="author" content="Julia Evans">
|
|
<meta name="HandheldFriendly" content="True">
|
|
<meta name="MobileOptimized" content="320">
|
|
<meta name="description" content="Some ways DNS can break">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
|
|
<meta property="og:title" content='Some ways DNS can break'>
|
|
<meta property="og:type" content="website" />
|
|
<meta property="og:url" content="https://jvns.ca/blog/2022/01/15/some-ways-dns-can-break/" />
|
|
<meta property="og:site_name" content="Julia Evans" />
|
|
|
|
<link rel="canonical" href="https://jvns.ca/blog/2022/01/15/some-ways-dns-can-break/">
|
|
<link href="/favicon.ico" rel="icon">
|
|
|
|
<link href="/stylesheets/screen.css" rel="preload" type="text/css" as="style">
|
|
|
|
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
|
|
<link href="/stylesheets/print.css" media="print" rel="stylesheet" type="text/css">
|
|
|
|
|
|
<link href="/atom.xml" rel="alternate" title="Julia Evans" type="application/atom+xml">
|
|
|
|
|
|
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css" integrity="sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0" crossorigin="anonymous">
|
|
<script defer data-domain="jvns.ca" src="https://plausible.io/js/script.js"></script>
|
|
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js" integrity="sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4" crossorigin="anonymous"></script>
|
|
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
|
|
|
|
<script defer type="text/javascript">
|
|
window.heap=window.heap||[],heap.load=function(e,t){window.heap.appid=e,window.heap.config=t=t||{};var r=document.createElement("script");r.type="text/javascript",r.async=!0,r.src="https://cdn.heapanalytics.com/js/heap-"+e+".js";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(r,a);for(var n=function(e){return function(){heap.push([e].concat(Array.prototype.slice.call(arguments,0)))}},p=["addEventProperties","addUserProperties","clearEventProperties","identify","resetIdentity","removeEventProperty","setEventProperties","track","unsetEventProperty"],o=0;o<p.length;o++)heap[p[o]]=n(p[o])};
|
|
heap.load("2242143965");
|
|
</script>
|
|
</head>
|
|
<body>
|
|
<div id="skiptocontent">
|
|
<a href="#main">Skip to main content</a>
|
|
</div>
|
|
<div id="wrap">
|
|
<header role="banner">
|
|
<hgroup>
|
|
<h1><a href="/">Julia Evans</a></h1>
|
|
</hgroup>
|
|
<ul class="header-links">
|
|
<li><a href="/about">About</a></li>
|
|
<li><a href="/talks">Talks</a></li>
|
|
<li><a href="/projects/">Projects</a></li>
|
|
<li><a rel="me" href="https://social.jvns.ca/@b0rk">Mastodon</a></li>
|
|
<li><a href="https://bsky.app/profile/b0rk.jvns.ca">Bluesky</a></li>
|
|
<li><a href="https://github.com/jvns">Github</a></li>
|
|
</ul>
|
|
</header>
|
|
<nav role="navigation" class="header-nav"><ul class="main-navigation">
|
|
<li><a href="/categories/favorite/">Favorites</a></li>
|
|
<li><a href="/til/">TIL</a></li>
|
|
<li><a href="https://wizardzines.com">Zines</a></li>
|
|
<li class="subscription" data-subscription="rss"><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
|
|
</ul>
|
|
</nav>
|
|
<div id="main">
|
|
<div id="content">
|
|
|
|
|
|
<div>
|
|
<article class="hentry" role="article">
|
|
<header>
|
|
<h1 class="entry-title">Some ways DNS can break</h1>
|
|
|
|
<div class="post-tags">
|
|
|
|
•
|
|
|
|
<a class="post-tag" href="/categories/dns">dns</a> •
|
|
|
|
|
|
</div>
|
|
<p class="meta sans">
|
|
<time class="date" datetime="2022-01-15T12:25:37" pubdate data-updated="true">
|
|
|
|
January 15, 2022
|
|
|
|
</time>
|
|
</p>
|
|
</header>
|
|
<main>
|
|
<p>When I first learned about it, DNS didn’t seem like it should be THAT
|
|
complicated. Like, there are DNS records, they’re stored on a server, what’s
|
|
the big deal?</p>
|
|
<p>But with DNS, reading about how it works in a textbook doesn’t prepare you for
|
|
the sheer volume of different ways DNS can break your system in practice. It’s
|
|
not just caching problems!</p>
|
|
<p>So I <a href="https://twitter.com/b0rk/status/1481265429897261058">asked people on Twitter</a> for
|
|
example of DNS problems they’ve run into, especially DNS problems that <strong>didn’t
|
|
initially appear to be DNS problems</strong>. (the popular “it’s always DNS” meme)</p>
|
|
<p>I’m not going to discuss how to solve or avoid any of these problems in this
|
|
post, but I’ve linked to webpages discussing the problem where I could find
|
|
them.</p>
|
|
<h3 id="problem-slow-network-requests" class="post-heading">
|
|
<a href="#problem-slow-network-requests">
|
|
problem: slow network requests
|
|
</a>
|
|
</h3>
|
|
<p>Your network requests are a little bit slower than expected, and it’s actually
|
|
because your DNS resolver is slow for some reason. This might be because the
|
|
resolver is under a lot of load, or it has a memory leak, or something else.</p>
|
|
<p>I’ve run into this before with my router’s DNS forwarder – all of my DNS
|
|
requests were slow, and I restarted my router and that fixed the problem.</p>
|
|
<h3 id="problem-dns-timeouts" class="post-heading">
|
|
<a href="#problem-dns-timeouts">
|
|
problem: DNS timeouts
|
|
</a>
|
|
</h3>
|
|
<p>A couple of people mentioned network requests that were taking 2+ seconds or 30
|
|
seconds because of DNS queries that were timing out. This is sort of the same
|
|
as “slow requests”, but it’s worse because queries can take several seconds to
|
|
time out.</p>
|
|
<p>Sophie Haskins has a great blog post <a href="https://blog.sophaskins.net/blog/misadventures-with-kube-dns/">Misadventures with Kube DNS</a> about DNS
|
|
timeouts with Kubernetes.</p>
|
|
<h3 id="problem-ndots" class="post-heading">
|
|
<a href="#problem-ndots">
|
|
problem: ndots
|
|
</a>
|
|
</h3>
|
|
<p>A few people mentioned a specific issue where Kubernetes sets <code>ndots:5</code> in its <code>/etc/resolv.conf</code></p>
|
|
<p>Here’s an example /etc/resolv.conf from <a href="https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html">Kubernetes pods /etc/resolv.conf ndots:5 option and why it may negatively affect your application performances</a>.</p>
|
|
<pre><code>nameserver 100.64.0.10
|
|
search namespace.svc.cluster.local svc.cluster.local cluster.local eu-west-1.compute.internal
|
|
options ndots:5
|
|
</code></pre>
|
|
<p>My understanding is that if this is your <code>/etc/resolv.conf</code> and you look up
|
|
<code>google.com</code>, your application will call the C <code>getaddrinfo</code> function, and
|
|
<code>getaddrinfo</code> will:</p>
|
|
<ol>
|
|
<li>look up <code>google.com.namespace.svc.cluster.local.</code></li>
|
|
<li>look up <code>google.com.svc.cluster.local.</code></li>
|
|
<li>look up <code>google.com.cluster.local.</code></li>
|
|
<li>look up <code>google.com.eu-west-1.compute.internal.</code></li>
|
|
<li>look up <code>google.com.</code></li>
|
|
</ol>
|
|
<p>Basically it checks if <code>google.com</code> is actually a subdomain of everything on the <code>search</code> line.</p>
|
|
<p>So every time you make a DNS query, you need to wait for 4 DNS queries to fail
|
|
before you can get to the actual real DNS query that succeeds.</p>
|
|
<h3 id="problem-it-s-hard-to-tell-what-dns-resolver-s-your-system-is-using" class="post-heading">
|
|
<a href="#problem-it-s-hard-to-tell-what-dns-resolver-s-your-system-is-using">
|
|
problem: it’s hard to tell what DNS resolver(s) your system is using
|
|
</a>
|
|
</h3>
|
|
<p>This isn’t a bug by itself, but when you run into a problem with DNS, often
|
|
it’s related in some way to your DNS resolver. I don’t know of any foolproof
|
|
way to tell what DNS resolver is being used.</p>
|
|
<p>A few things I know:</p>
|
|
<ul>
|
|
<li>on Linux, I think that most things use /etc/resolv.conf to choose a DNS
|
|
resolver. There are definitely exceptions though, for example your browser
|
|
might ignore /etc/resolv.conf and use a different DNS-over-HTTPS service
|
|
instead.</li>
|
|
<li>if you’re using UDP DNS, you can use <code>sudo tcpdump port 53</code> to see where DNS
|
|
requests are being sent. This doesn’t work if you’re using DNS over HTTPS or
|
|
DNS over TLS though.</li>
|
|
</ul>
|
|
<p>I also vaguely remember it being even more confusing on MacOS than on Linux,
|
|
though I don’t know why.</p>
|
|
<h3 id="problem-dns-servers-that-return-nxdomain-instead-of-noerror" class="post-heading">
|
|
<a href="#problem-dns-servers-that-return-nxdomain-instead-of-noerror">
|
|
problem: DNS servers that return NXDOMAIN instead of NOERROR
|
|
</a>
|
|
</h3>
|
|
<p>Here’s a problem that I ran into once, where nginx couldn’t resolve a domain.</p>
|
|
<ul>
|
|
<li>I set up nginx to use a specific DNS server to resolve DNS queries</li>
|
|
<li>when visiting the domain, nginx made 2 queries, one for an <code>A</code> record, and one for an <code>AAAA</code> record</li>
|
|
<li>the DNS server returned a <code>NXDOMAIN</code> reply for the <code>A</code> query</li>
|
|
<li>nginx decided “ok, that domain doesn’t exist”, and gave up</li>
|
|
<li>the DNS server returned a successful reply for the <code>AAAA</code> query</li>
|
|
<li>nginx ignored the <code>AAAA</code> record because it had already given up</li>
|
|
</ul>
|
|
<p>The problem was that the DNS server should have returned <code>NOERROR</code> – that
|
|
domain <em>did</em> exist, it was just that there weren’t any <code>A</code> records for it. I
|
|
reported the bug, they fixed it, and that fixed the problem.</p>
|
|
<p>I’ve implemented this bug myself too, so I understand why it happens – it’s
|
|
easy to think “there aren’t any records for this query, I should return an
|
|
<code>NXDOMAIN</code> error”.</p>
|
|
<h3 id="problem-negative-dns-caching" class="post-heading">
|
|
<a href="#problem-negative-dns-caching">
|
|
problem: negative DNS caching
|
|
</a>
|
|
</h3>
|
|
<p>If you visit a domain before creating a DNS record for it, the <strong>absence</strong> of
|
|
the record will be cached. This is very surprising the first time your run into
|
|
it – I only learned about this last year!</p>
|
|
<p>The TTL for cache entry is the TTL of the domain’s SOA record – for example
|
|
for <code>jvns.ca</code>, it’s an hour.</p>
|
|
<h3 id="problem-nginx-caching-dns-records-forever" class="post-heading">
|
|
<a href="#problem-nginx-caching-dns-records-forever">
|
|
problem: nginx caching DNS records forever
|
|
</a>
|
|
</h3>
|
|
<p>If you put this in your nginx config:</p>
|
|
<pre><code>location / {
|
|
proxy_pass https://some.domain.com;
|
|
}
|
|
</code></pre>
|
|
<p>then nginx will resolve <code>some.domain.com</code> once on startup and never again. This
|
|
is especially dangerous if the IP address for <code>some.domain.com</code> changes
|
|
infrequently, because it might keep happily working for months and then
|
|
suddenly break at 2am one day.</p>
|
|
<p>There are pretty well-known ways to fix this and this post isn’t about nginx so
|
|
I won’t get into it, but it’s surprising the first time you run into it.</p>
|
|
<p>Here’s a <a href="https://medium.com/driven-by-code/dynamic-dns-resolution-in-nginx-22133c22e3ab">blog post</a> with a story of how this happened to someone with an AWS load balancer.</p>
|
|
<h3 id="problem-java-caching-dns-records-forever" class="post-heading">
|
|
<a href="#problem-java-caching-dns-records-forever">
|
|
problem: Java caching DNS records forever
|
|
</a>
|
|
</h3>
|
|
<p>Same thing, but for Java: <a href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html">Apparently</a>
|
|
depending on how you configure Java, “the JVM default TTL [might be] set so
|
|
that it will never refresh DNS entries until the JVM is restarted.”</p>
|
|
<p>I haven’t run into this myself but I asked a friend about it who writes more
|
|
Java than me and they told me that it’s happened to them.</p>
|
|
<p>Of course, literally any software could have this problem of caching DNS
|
|
records forever, but the main cases I’ve heard of in practice are nginx and
|
|
Java.</p>
|
|
<h3 id="problem-that-entry-in-etc-hosts-you-forgot-about" class="post-heading">
|
|
<a href="#problem-that-entry-in-etc-hosts-you-forgot-about">
|
|
problem: that entry in /etc/hosts you forgot about
|
|
</a>
|
|
</h3>
|
|
<p>Another variant on caching issues: entries in <code>/etc/hosts</code> that override your
|
|
usual DNS settings!</p>
|
|
<p>This is extra confusing because <code>dig</code> ignores <code>/etc/hosts</code>, so everything SEEMS
|
|
like it should be fine ("<code>dig whatever.com</code> is working!").</p>
|
|
<h3 id="problem-your-email-isn-t-being-sent-is-going-to-spam" class="post-heading">
|
|
<a href="#problem-your-email-isn-t-being-sent-is-going-to-spam">
|
|
problem: your email isn’t being sent / is going to spam
|
|
</a>
|
|
</h3>
|
|
<p>The way email is sent and validated is through DNS (MX records, SPF records,
|
|
DKIM records), so a lot of email problems are DNS problems.</p>
|
|
<h3 id="problem-internationalized-domain-names-don-t-work" class="post-heading">
|
|
<a href="#problem-internationalized-domain-names-don-t-work">
|
|
problem: internationalized domain names don’t work
|
|
</a>
|
|
</h3>
|
|
<p>You can register domain names with non-ASCII characters or emoji like <a href="https://%F0%9F%92%A9.la/">https://💩.la</a>.</p>
|
|
<p>The way this works with DNS is that <code>💩.la</code> gets translated into <code>xn--ls8h.la</code> with an encoding called “punycode”.</p>
|
|
<p>But even though there’s a clear standard for how they should work with DNS, a lot of software doesn’t handle internationalized domain names well!
|
|
There’s a fun story about this in Julian Squires’ great talk <a href="https://www.youtube.com/watch?v=UE-fJjMasec">The emoji that Killed Chrome!!</a>.</p>
|
|
<h3 id="problem-tcp-dns-is-blocked-by-a-firewall" class="post-heading">
|
|
<a href="#problem-tcp-dns-is-blocked-by-a-firewall">
|
|
problem: TCP DNS is blocked by a firewall
|
|
</a>
|
|
</h3>
|
|
<p>A couple of people mentioned that some firewalls allow UDP port 53 but not TCP
|
|
port 53. But large DNS queries need to use TCP port 53, so this can cause weird
|
|
intermittent problems that are hard to debug.</p>
|
|
<h3 id="problem-musl-doesn-t-support-tcp-dns" class="post-heading">
|
|
<a href="#problem-musl-doesn-t-support-tcp-dns">
|
|
problem: musl doesn’t support TCP DNS
|
|
</a>
|
|
</h3>
|
|
<p>A lot of applications use libc’s <code>getaddrinfo</code> to make DNS queries. musl is an
|
|
alternative to <code>glibc</code> that’s used in Alpine Docker container which doesn’t
|
|
support TCP DNS. This can cause problems if you make DNS queries where the
|
|
response would be too big to fit inside a regular DNS UDP packet (512 bytes).</p>
|
|
<p>I’m still a bit fuzzy on this so I might have it wrong, but my understanding of how this can break is:</p>
|
|
<ol>
|
|
<li>musl’s getaddrinfo makes a DNS query</li>
|
|
<li>the DNS server notices that the response is too big to fit in a single DNS response packet</li>
|
|
<li>the DNS server returns an <strong>empty</strong> truncated response, expecting that the client will retry by making a TCP DNS query</li>
|
|
<li><code>musl</code> does not support TCP so it does not retry</li>
|
|
</ol>
|
|
<p>A blog post about this: <a href="https://christoph.luppri.ch/fixing-dns-resolution-for-ruby-on-alpine-linux">DNS resolution issue in Alpine Linux</a></p>
|
|
<h3 id="problem-round-robin-dns-doesn-t-work-with-getaddrinfo" class="post-heading">
|
|
<a href="#problem-round-robin-dns-doesn-t-work-with-getaddrinfo">
|
|
problem: round robin DNS doesn’t work with <code>getaddrinfo</code>
|
|
</a>
|
|
</h3>
|
|
<p>One way you could approach load balancing is to use “round robin DNS”. The idea
|
|
is that every time you make a DNS query, you get a different IP address.
|
|
Apparently this works if you use <code>gethostbyname</code> to make DNS queries, but it
|
|
does not work if you use <code>getaddrinfo</code> because <code>getaddrinfo</code> sorts the IP
|
|
responses it receives.</p>
|
|
<p>So you could run into an upsetting problem if you switch from <code>gethostbyname</code> to <code>getaddrinfo</code> behind the scenes without realising that this will break your DNS load balancing.</p>
|
|
<p>This is especially insidious because you might not realize that you’re
|
|
switching to <code>gethostbyname</code> to <code>getaddrinfo</code> at all – if you’re not writing a
|
|
C program, those functions calls are hidden inside some library. So it could be
|
|
part of a seemingly innocuous upgrade.</p>
|
|
<p>Here are a couple of pages discussing this:</p>
|
|
<ul>
|
|
<li><a href="https://groups.google.com/g/consul-tool/c/AGgPjrrkw3g">getaddrinfo breaks round robin DNS</a></li>
|
|
<li><a href="https://daniel.haxx.se/blog/2012/01/03/getaddrinfo-with-round-robin-dns-and-happy-eyeballs/">getaddrinfo with round robin DNS and happy eyeballs</a></li>
|
|
</ul>
|
|
<h3 id="problem-a-race-condition-when-starting-a-service" class="post-heading">
|
|
<a href="#problem-a-race-condition-when-starting-a-service">
|
|
problem: a race condition when starting a service
|
|
</a>
|
|
</h3>
|
|
<p>A problem someone <a href="https://mobile.twitter.com/omatskiv/status/1481305175440646148">mentioned</a>
|
|
with Kubernetes DNS: they had 2 containers which started simultaneously and
|
|
immediately tried to resolve each other. But the DNS lookup failed because the
|
|
Kubernetes DNS change hadn’t happened yet, and then the failure was cached so
|
|
it kept failing.</p>
|
|
<h3 id="that-s-all" class="post-heading">
|
|
<a href="#that-s-all">
|
|
that’s all!
|
|
</a>
|
|
</h3>
|
|
<p>I’ve definitely missed some important DNS problems here, so I’d love to hear
|
|
what I’ve missed. I’d also love links to blog posts that write up examples of
|
|
these problems – I think it’s really useful to see how the problem
|
|
specifically manifests in practice and how people debugged it.</p>
|
|
|
|
</main>
|
|
|
|
<footer>
|
|
|
|
<style type="text/css">
|
|
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; display: inline;}
|
|
#mc_embed_signup {
|
|
display: inline;
|
|
}
|
|
#mc_embed_signup input.button {
|
|
background: #ff5e00;
|
|
display: inline;
|
|
color: white;
|
|
padding: 6px 12px;
|
|
}
|
|
</style>
|
|
<div class="sharing">
|
|
|
|
<style>
|
|
.form-inline {
|
|
display:flex; flex-flow: row wrap; justify-content: center;
|
|
}
|
|
.form-inline input, .form-inline span {
|
|
padding: 10px;
|
|
}
|
|
.form-inline input {
|
|
display:inline;
|
|
max-width:30%;
|
|
margin: 0 10px 0 0;
|
|
background-color: #fff;
|
|
border: 1px solid #ddd;
|
|
border-radius: 5px;
|
|
padding: 10px;
|
|
}
|
|
button {
|
|
background-color: #f50;
|
|
box-shadow: none;
|
|
border: 0;
|
|
border-radius: 5px;
|
|
color: white;
|
|
padding: 5px 10px;
|
|
}
|
|
@media (max-width: 800px) {
|
|
.form-inline input {
|
|
margin: 10px 0;
|
|
max-width:100% !important;
|
|
}
|
|
.form-inline {
|
|
flex-direction: column;
|
|
align-items: stretch;
|
|
}
|
|
}
|
|
</style>
|
|
|
|
<div align="center">
|
|
<form class="form-inline" action="https://app.convertkit.com/forms/1052396/subscriptions" method="post" data-uid="8884355abb" data-format="inline" data-version="5">
|
|
<span> Want a weekly digest of this blog?</span>
|
|
<input name="email_address" type="text" placeholder="Email address" />
|
|
<button type="submit" data-element="submit">Subscribe</button>
|
|
</form>
|
|
</div>
|
|
|
|
|
|
</div>
|
|
|
|
<p class="meta">
|
|
|
|
<a class="basic-alignment left" href="https://jvns.ca/blog/2022/01/11/how-to-find-a-domain-s-authoritative-nameserver/" title="Previous Post: How to find a domain's authoritative nameservers">How to find a domain's authoritative nameservers</a>
|
|
|
|
|
|
<a class="basic-alignment right" href="https://jvns.ca/blog/2022/01/24/hosting-my-static-sites-with-nginx/" title="Next Post: Hosting my static sites with nginx">Hosting my static sites with nginx</a>
|
|
|
|
</p>
|
|
</footer>
|
|
|
|
</article>
|
|
</div>
|
|
|
|
</div>
|
|
</div>
|
|
<nav role="navigation" class="footer-nav"> <a href="/">Archives</a>
|
|
</nav>
|
|
<footer role="contentinfo"><span class="credit">© Julia Evans. </span>
|
|
<span>If you like this, you may like <a href="https://web.archive.org/web/20181228051203/http://www.uliaea.ca/">Ulia Ea</a> or, more seriously, this list of <a href="https://jvns.ca/blogroll">blogs I love</a> or some <a href="https://jvns.ca/bookshelf">books I've read</a>. <br>
|
|
<p class="rc-scout__text"><i class="rc-scout__logo"></i>
|
|
You might also like the <a class="rc-scout__link" href="https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8">Recurse Center</a>, my very favorite programming community <a href="/categories/hackerschool/">(my posts about it)</a></p>
|
|
</span>
|
|
<style class="rc-scout__style" type="text/css">.rc-scout{display:block;padding:0;border:0;margin:0;}.rc-scout__text{display:block;padding:0;border:0;margin:0;height:100%;font-size:100%;}.rc-scout__logo{display:inline-block;padding:0;border:0;margin:0;width:0.85em;height:0.85em;background:no-repeat center url('data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2012%2015%22%3E%3Crect%20x%3D%220%22%20y%3D%220%22%20width%3D%2212%22%20height%3D%2210%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%221%22%20width%3D%2210%22%20height%3D%228%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%222%22%20width%3D%228%22%20height%3D%226%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%229%22%20width%3D%224%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%2211%22%20width%3D%2210%22%20height%3D%224%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%220%22%20y%3D%2212%22%20width%3D%2212%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%225%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%227%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%228%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%229%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3C%2Fsvg%3E');}.rc-scout__link:link,.rc-scout__link:visited{color:#61ae24;text-decoration:underline;}.rc-scout__link:hover,.rc-scout__link:active{color:#4e8b1d;}</style>
|
|
</footer>
|
|
<script type="text/rocketscript">
|
|
(function(){
|
|
var twitterWidgets = document.createElement('script');
|
|
twitterWidgets.type = 'text/javascript';
|
|
twitterWidgets.async = true;
|
|
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
|
|
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
|
|
})();
|
|
</script>
|
|
</div>
|
|
</body>
|
|
</html>
|
|
|