353 lines
22 KiB
HTML
353 lines
22 KiB
HTML
<!DOCTYPE html>
|
||
|
||
|
||
<html class="no-js" lang="en">
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<title>Reasons Kubernetes is cool</title>
|
||
<meta name="author" content="Julia Evans">
|
||
<meta name="HandheldFriendly" content="True">
|
||
<meta name="MobileOptimized" content="320">
|
||
<meta name="description" content="Reasons Kubernetes is cool">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||
|
||
<meta property="og:title" content='Reasons Kubernetes is cool'>
|
||
<meta property="og:type" content="website" />
|
||
<meta property="og:url" content="https://jvns.ca/blog/2017/10/05/reasons-kubernetes-is-cool/" />
|
||
<meta property="og:site_name" content="Julia Evans" />
|
||
|
||
<link rel="canonical" href="https://jvns.ca/blog/2017/10/05/reasons-kubernetes-is-cool/">
|
||
<link href="/favicon.ico" rel="icon">
|
||
|
||
<link href="/stylesheets/screen.css" rel="preload" type="text/css" as="style">
|
||
|
||
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
|
||
<link href="/stylesheets/print.css" media="print" rel="stylesheet" type="text/css">
|
||
|
||
|
||
<link href="/atom.xml" rel="alternate" title="Julia Evans" type="application/atom+xml">
|
||
|
||
|
||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css" integrity="sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0" crossorigin="anonymous">
|
||
<script defer data-domain="jvns.ca" src="https://plausible.io/js/script.js"></script>
|
||
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js" integrity="sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4" crossorigin="anonymous"></script>
|
||
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
|
||
|
||
<script defer type="text/javascript">
|
||
window.heap=window.heap||[],heap.load=function(e,t){window.heap.appid=e,window.heap.config=t=t||{};var r=document.createElement("script");r.type="text/javascript",r.async=!0,r.src="https://cdn.heapanalytics.com/js/heap-"+e+".js";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(r,a);for(var n=function(e){return function(){heap.push([e].concat(Array.prototype.slice.call(arguments,0)))}},p=["addEventProperties","addUserProperties","clearEventProperties","identify","resetIdentity","removeEventProperty","setEventProperties","track","unsetEventProperty"],o=0;o<p.length;o++)heap[p[o]]=n(p[o])};
|
||
heap.load("2242143965");
|
||
</script>
|
||
</head>
|
||
<body>
|
||
<div id="skiptocontent">
|
||
<a href="#main">Skip to main content</a>
|
||
</div>
|
||
<div id="wrap">
|
||
<header role="banner">
|
||
<hgroup>
|
||
<h1><a href="/">Julia Evans</a></h1>
|
||
</hgroup>
|
||
<ul class="header-links">
|
||
<li><a href="/about">About</a></li>
|
||
<li><a href="/talks">Talks</a></li>
|
||
<li><a href="/projects/">Projects</a></li>
|
||
<li><a rel="me" href="https://social.jvns.ca/@b0rk">Mastodon</a></li>
|
||
<li><a href="https://bsky.app/profile/b0rk.jvns.ca">Bluesky</a></li>
|
||
<li><a href="https://github.com/jvns">Github</a></li>
|
||
</ul>
|
||
</header>
|
||
<nav role="navigation" class="header-nav"><ul class="main-navigation">
|
||
<li><a href="/categories/favorite/">Favorites</a></li>
|
||
<li><a href="/til/">TIL</a></li>
|
||
<li><a href="https://wizardzines.com">Zines</a></li>
|
||
<li class="subscription" data-subscription="rss"><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
|
||
</ul>
|
||
</nav>
|
||
<div id="main">
|
||
<div id="content">
|
||
|
||
|
||
<div>
|
||
<article class="hentry" role="article">
|
||
<header>
|
||
<h1 class="entry-title">Reasons Kubernetes is cool</h1>
|
||
|
||
<div class="post-tags">
|
||
|
||
•
|
||
|
||
<a class="post-tag" href="/categories/kubernetes">kubernetes</a> •
|
||
|
||
|
||
</div>
|
||
<p class="meta sans">
|
||
<time class="date" datetime="2017-10-05T21:23:46" pubdate data-updated="true">
|
||
|
||
October 5, 2017
|
||
|
||
</time>
|
||
</p>
|
||
</header>
|
||
<main>
|
||
<p>When I first learned about Kubernetes (a year and a half ago?) I really didn’t understand why I
|
||
should care about it.</p>
|
||
<p>I’ve been working full time with Kubernetes for 3 months or so and now have some thoughts about why
|
||
I think it’s useful. (I’m still very far from being a Kubernetes expert!) Hopefully this will help a
|
||
little in your journey to understand what even is going on with Kubernetes!</p>
|
||
<p>I will try to explain some reason I think Kubenetes is interesting without using the words “cloud native”,
|
||
“orchestration”, “container”, or any Kubernetes-specific terminology :). I’m going to explain this
|
||
mostly from the perspective of a kubernetes operator / infrastructure engineer, since my job right
|
||
now is to set up Kubernetes and make it work well.</p>
|
||
<p>I’m not going to try to address the question of “should you use kubernetes for your production
|
||
systems?” at all, that is a very complicated question. (not least because “in production” has
|
||
totally different requirements depending on what you’re doing)</p>
|
||
<h3 id="kubernetes-lets-you-run-code-in-production-without-setting-up-new-servers" class="post-heading">
|
||
<a href="#kubernetes-lets-you-run-code-in-production-without-setting-up-new-servers">
|
||
Kubernetes lets you run code in production without setting up new servers
|
||
</a>
|
||
</h3>
|
||
<p>The first pitch I got for Kubernetes was the following conversation with my partner Kamal:</p>
|
||
<p>Here’s an approximate transcript:</p>
|
||
<ul>
|
||
<li>Kamal: With Kubernetes you can set up a new service with a single command</li>
|
||
<li>Julia: I don’t understand how that’s possible.</li>
|
||
<li>Kamal: Like, you just write 1 configuration file, apply it, and then you have a HTTP service running in production</li>
|
||
<li>Julia: But today I need to create new AWS instances, write a puppet manifest, set up service discovery, configure my load balancers, configure our deployment software, and make sure DNS is working, it takes at least 4 hours if nothing goes wrong.</li>
|
||
<li>Kamal: Yeah. With Kubernetes you don’t have to do any of that, you can set up a new HTTP service in 5 minutes and it’ll just automatically run. As long as you have spare capacity in your cluster it just works!</li>
|
||
<li>Julia: There must be a trap</li>
|
||
</ul>
|
||
<p>There kind of is a trap, setting up a production Kubernetes cluster is (in my experience) is
|
||
definitely not easy. (see <a href="https://github.com/kelseyhightower/kubernetes-the-hard-way">Kubernetes The Hard Way</a> for what’s involved to get
|
||
started). But we’re not going to go into that right now!</p>
|
||
<p>So the first cool thing about Kubernetes is that it has the potential to make life way easier for
|
||
developers who want to deploy new software into production. That’s cool, and it’s actually true,
|
||
once you have a working Kubernetes cluster you really can set up a production HTTP service (“run 5
|
||
of this application, set up a load balancer, give it this DNS name, done”) with just one
|
||
configuration file. It’s really fun to see.</p>
|
||
<h3 id="kubernetes-gives-you-easy-visibility-control-of-what-code-you-have-running-in-production" class="post-heading">
|
||
<a href="#kubernetes-gives-you-easy-visibility-control-of-what-code-you-have-running-in-production">
|
||
Kubernetes gives you easy visibility & control of what code you have running in production
|
||
</a>
|
||
</h3>
|
||
<p>IMO you can’t understand Kubernetes without understanding etcd. So let’s talk about etcd!</p>
|
||
<p>Imagine that I asked you today “hey, tell me every application you have running in production, what
|
||
host it’s running on, whether it’s healthy or not, and whether or not it has a DNS name attached to
|
||
it”. I don’t know about you but I would need to go look in a bunch of different places to answer
|
||
this question and it would take me quite a while to figure out. I definitely can’t query just one
|
||
API.</p>
|
||
<p>In Kubernetes, all the state in your cluster – applications running (“pods”), nodes, DNS names,
|
||
cron jobs, and more – is stored in a single database (etcd). Every Kubernetes component is
|
||
stateless, and basically works by</p>
|
||
<ul>
|
||
<li>Reading state from etcd (eg “the list of pods assigned to node 1”)</li>
|
||
<li>Making changes (eg “actually start running pod A on node 1”)</li>
|
||
<li>Updating the state in etcd (eg “set the state of pod A to ‘running’”)</li>
|
||
</ul>
|
||
<p>This means that if you want to answer a question like “hey, how many nginx pods do I have running
|
||
right now in that availabliity zone?” you can answer it by querying a single unified API (the
|
||
Kubernetes API!). And you have exactly the same access to that API that every other Kubernetes
|
||
component does.</p>
|
||
<p>This also means that you have easy control of everything running in Kubernetes. If you want to, say,</p>
|
||
<ul>
|
||
<li>Implement a complicated custom rollout strategy for deployments (deploy 1 thing, wait 2 minutes, deploy 5 more, wait 3.7 minutes, etc)</li>
|
||
<li>Automatically <a href="https://github.com/kamalmarhubi/kubereview">start a new webserver</a> every time a branch is pushed to github</li>
|
||
<li>Monitor all your running applications to make sure all of them have a reasonable cgroups memory limit</li>
|
||
</ul>
|
||
<p>all you need to do is to write a program that talks to the Kubernetes API. (a “controller”)</p>
|
||
<p>Another very exciting thing about the Kubernetes API is that you’re not limited to just
|
||
functionality that Kubernetes provides! If you decide that you have your own opinions about how your
|
||
software should be deployed / created / monitored, then you can write code that uses the Kubernetes
|
||
API to do it! It lets you do everything you need.</p>
|
||
<h3 id="if-every-kubernetes-component-dies-your-code-will-still-keep-running" class="post-heading">
|
||
<a href="#if-every-kubernetes-component-dies-your-code-will-still-keep-running">
|
||
If every Kubernetes component dies, your code will still keep running
|
||
</a>
|
||
</h3>
|
||
<p>One thing I was originally promised (by various blog posts :)) about Kubernetes was “hey, if the
|
||
Kubernetes apiserver and everything else dies, it’s ok, your code will just keep running”. I thought
|
||
this sounded cool in theory but I wasn’t sure if it was actually true.</p>
|
||
<p>So far it seems to be actually true!</p>
|
||
<p>I’ve been through some etcd outages now, and what happens is</p>
|
||
<ol>
|
||
<li>All the code that was running keeps running</li>
|
||
<li>Nothing <em>new</em> happens (you can’t deploy new code or make changes, cron jobs will stop working)</li>
|
||
<li>When everything comes back, the cluster will catch up on whatever it missed</li>
|
||
</ol>
|
||
<p>This does mean that if etcd goes down and one of your applications crashes or something, it can’t come back up until etcd returns.</p>
|
||
<h3 id="kubernetes-design-is-pretty-resilient-to-bugs" class="post-heading">
|
||
<a href="#kubernetes-design-is-pretty-resilient-to-bugs">
|
||
Kubernetes’ design is pretty resilient to bugs
|
||
</a>
|
||
</h3>
|
||
<p>Like any piece of software, Kubernetes has bugs. For example right now in our cluster the controller manager has a memory leak, and the scheduler crashes pretty regularly. Bugs obviously aren’t good but so far I’ve found that Kubernetes’ design helps mitigate a lot of the bugs in its core components really well.</p>
|
||
<p>If you restart any component, what happens is:</p>
|
||
<ul>
|
||
<li>It reads all its relevant state from etcd</li>
|
||
<li>It starts doing the necessary things it’s supposed to be doing based on that state (scheduling pods, garbage collecting completed pods, scheduling cronjobs, deploying daemonsets, whatever)</li>
|
||
</ul>
|
||
<p>Because all the components don’t keep any state in memory, you can just restart them at any time and that can help mitigate a variety of bugs.</p>
|
||
<p>For example! Let’s say you have a memory leak in your controller manager. Because the controller
|
||
manager is stateless, you can just periodically restart it every hour or something and feel
|
||
confident that you won’t cause any consistency issues. Or we ran into a bug in the scheduler where
|
||
it would sometimes just forget about pods and never schedule them. You can sort of mitigate this
|
||
just by restarting the scheduler every 10 minutes. (we didn’t do that, we fixed the bug instead, but
|
||
you <em>could</em> :) )</p>
|
||
<p>So I feel like I can trust Kubernetes’ design to help make sure the state in the cluster is
|
||
consistent even when there are bugs in its core components. And in general I think the software is
|
||
generally improving over time. The only stateful thing you have to operate is etcd</p>
|
||
<p>Not to harp on this “state” thing too much but – I think it’s cool that in Kubernetes the only
|
||
thing you have to come up with backup/restore plans for is etcd (unless you use persistent volumes
|
||
for your pods). I think it makes kubernetes operations a lot easier to think about.</p>
|
||
<h3 id="implementing-new-distributed-systems-on-top-of-kubernetes-is-relatively-easy" class="post-heading">
|
||
<a href="#implementing-new-distributed-systems-on-top-of-kubernetes-is-relatively-easy">
|
||
Implementing new distributed systems on top of Kubernetes is relatively easy
|
||
</a>
|
||
</h3>
|
||
<p>Suppose you want to implement a distributed cron job scheduling system! Doing that from scratch is a
|
||
ton of work. But implementing a distributed cron job scheduling system inside Kubernetes is much
|
||
easier! (still not trivial, it’s still a distributed system)</p>
|
||
<p>The first time I read the code for the Kubernetes cronjob controller I was really delighted by how
|
||
simple it was. Here, go read it! The main logic is like 400 lines of Go. Go ahead, read it! =>
|
||
<a href="https://github.com/kubernetes/kubernetes/blob/e4551d50e57c089aab6f67333412d3ca64bc09ae/pkg/controller/cronjob/cronjob_controller.go">cronjob_controller.go</a> <=</p>
|
||
<p>Basically what the cronjob controller does is:</p>
|
||
<ul>
|
||
<li>Every 10 seconds:
|
||
<ul>
|
||
<li>Lists all the cronjobs that exist</li>
|
||
<li>Checks if any of them need to run right now</li>
|
||
<li>If so, creates a new Job object to be scheduled & actually run by other Kubernetes controllers</li>
|
||
<li>Clean up finished jobs</li>
|
||
<li>Repeat</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<p>The Kubernetes model is pretty constrained (it has this pattern of resources are defined in etcd,
|
||
controllers read those resources and update etcd), and I think having this relatively
|
||
opinionated/constrained model makes it easier to develop your own distributed systems inside the
|
||
Kubernetes framework.</p>
|
||
<p>Kamal introduced me to this idea of “Kubernetes is a good platform for writing your own distributed systems” instead of just “Kubernetes is a distributed system you can use” and I think it’s really interesting. He has a prototype of a <a href="https://github.com/kamalmarhubi/kubereview">system to run an HTTP service for every branch you push to github</a>. It took him a weekend and is like 800 lines of Go, which I thought was impressive!</p>
|
||
<h3 id="kubernetes-lets-you-do-some-amazing-things-but-isn-t-easy" class="post-heading">
|
||
<a href="#kubernetes-lets-you-do-some-amazing-things-but-isn-t-easy">
|
||
Kubernetes lets you do some amazing things (but isn’t easy)
|
||
</a>
|
||
</h3>
|
||
<p>I started out by saying “kubernetes lets you do these magical things, you can just spin up so much
|
||
infrastructure with a single configuration file, it’s amazing”. And that’s true!</p>
|
||
<p>What I mean by “Kubernetes isn’t easy” is that Kubernetes has a lot of moving parts learning how to
|
||
successfully operate a highly available Kubernetes cluster is a lot of work. Like I find that with a
|
||
lot of the abstractions it gives me, I need to understand what is underneath those abstractions in
|
||
order to debug issues and configure things properly. I love learning new things so this doesn’t make
|
||
me angry or anything, I just think it’s important to know :)</p>
|
||
<p>One specific example of “I can’t just rely on the abstractions” that I’ve struggled with is that I
|
||
needed to learn a LOT <a href="https://jvns.ca/blog/2016/12/22/container-networking/">about how networking works on Linux</a> to feel confident with setting up
|
||
Kubernetes networking, way more than I’d ever had to learn about networking before. This was very
|
||
fun but pretty time consuming. I might write more about what is hard/interesting about setting up Kubernetes networking at some point.</p>
|
||
<p>Or I wrote a <a href="https://jvns.ca/blog/2017/08/05/how-kubernetes-certificates-work/">2000 word blog post</a>
|
||
about everything I had to learn about Kubernetes’ different options for certificate authorities to
|
||
be able to set up my Kubernetes CAs successfully.</p>
|
||
<p>I think some of these managed Kubernetes systems like GKE (google’s kubernetes product) may be
|
||
simpler since they make a lot of decisions for you but I haven’t tried any of them.</p>
|
||
|
||
</main>
|
||
|
||
<footer>
|
||
|
||
<style type="text/css">
|
||
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; display: inline;}
|
||
#mc_embed_signup {
|
||
display: inline;
|
||
}
|
||
#mc_embed_signup input.button {
|
||
background: #ff5e00;
|
||
display: inline;
|
||
color: white;
|
||
padding: 6px 12px;
|
||
}
|
||
</style>
|
||
<div class="sharing">
|
||
|
||
<style>
|
||
.form-inline {
|
||
display:flex; flex-flow: row wrap; justify-content: center;
|
||
}
|
||
.form-inline input, .form-inline span {
|
||
padding: 10px;
|
||
}
|
||
.form-inline input {
|
||
display:inline;
|
||
max-width:30%;
|
||
margin: 0 10px 0 0;
|
||
background-color: #fff;
|
||
border: 1px solid #ddd;
|
||
border-radius: 5px;
|
||
padding: 10px;
|
||
}
|
||
button {
|
||
background-color: #f50;
|
||
box-shadow: none;
|
||
border: 0;
|
||
border-radius: 5px;
|
||
color: white;
|
||
padding: 5px 10px;
|
||
}
|
||
@media (max-width: 800px) {
|
||
.form-inline input {
|
||
margin: 10px 0;
|
||
max-width:100% !important;
|
||
}
|
||
.form-inline {
|
||
flex-direction: column;
|
||
align-items: stretch;
|
||
}
|
||
}
|
||
</style>
|
||
|
||
<div align="center">
|
||
<form class="form-inline" action="https://app.convertkit.com/forms/1052396/subscriptions" method="post" data-uid="8884355abb" data-format="inline" data-version="5">
|
||
<span> Want a weekly digest of this blog?</span>
|
||
<input name="email_address" type="text" placeholder="Email address" />
|
||
<button type="submit" data-element="submit">Subscribe</button>
|
||
</form>
|
||
</div>
|
||
|
||
|
||
</div>
|
||
|
||
<p class="meta">
|
||
|
||
<a class="basic-alignment left" href="https://jvns.ca/teach-tech-with-cartoons/" title="Previous Post: How to teach technical concepts with cartoons">How to teach technical concepts with cartoons</a>
|
||
|
||
|
||
<a class="basic-alignment right" href="https://jvns.ca/blog/2017/10/10/operating-a-kubernetes-network/" title="Next Post: Operating a Kubernetes network">Operating a Kubernetes network</a>
|
||
|
||
</p>
|
||
</footer>
|
||
|
||
</article>
|
||
</div>
|
||
|
||
</div>
|
||
</div>
|
||
<nav role="navigation" class="footer-nav"> <a href="/">Archives</a>
|
||
</nav>
|
||
<footer role="contentinfo"><span class="credit">© Julia Evans. </span>
|
||
<span>If you like this, you may like <a href="https://web.archive.org/web/20181228051203/http://www.uliaea.ca/">Ulia Ea</a> or, more seriously, this list of <a href="https://jvns.ca/blogroll">blogs I love</a> or some <a href="https://jvns.ca/bookshelf">books I've read</a>. <br>
|
||
<p class="rc-scout__text"><i class="rc-scout__logo"></i>
|
||
You might also like the <a class="rc-scout__link" href="https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8">Recurse Center</a>, my very favorite programming community <a href="/categories/hackerschool/">(my posts about it)</a></p>
|
||
</span>
|
||
<style class="rc-scout__style" type="text/css">.rc-scout{display:block;padding:0;border:0;margin:0;}.rc-scout__text{display:block;padding:0;border:0;margin:0;height:100%;font-size:100%;}.rc-scout__logo{display:inline-block;padding:0;border:0;margin:0;width:0.85em;height:0.85em;background:no-repeat center url('data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2012%2015%22%3E%3Crect%20x%3D%220%22%20y%3D%220%22%20width%3D%2212%22%20height%3D%2210%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%221%22%20width%3D%2210%22%20height%3D%228%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%222%22%20width%3D%228%22%20height%3D%226%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%229%22%20width%3D%224%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%2211%22%20width%3D%2210%22%20height%3D%224%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%220%22%20y%3D%2212%22%20width%3D%2212%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%225%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%227%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%228%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%229%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3C%2Fsvg%3E');}.rc-scout__link:link,.rc-scout__link:visited{color:#61ae24;text-decoration:underline;}.rc-scout__link:hover,.rc-scout__link:active{color:#4e8b1d;}</style>
|
||
</footer>
|
||
<script type="text/rocketscript">
|
||
(function(){
|
||
var twitterWidgets = document.createElement('script');
|
||
twitterWidgets.type = 'text/javascript';
|
||
twitterWidgets.async = true;
|
||
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
|
||
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
|
||
})();
|
||
</script>
|
||
</div>
|
||
</body>
|
||
</html>
|
||
|