Files
nexus/sreweekly/articles/93/01-reasons-kubernetes-is-cool.html
2026-09-12 17:23:01 +08:00

353 lines
22 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta charset="utf-8">
<title>Reasons Kubernetes is cool</title>
<meta name="author" content="Julia Evans">
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="description" content="Reasons Kubernetes is cool">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:title" content='Reasons Kubernetes is cool'>
<meta property="og:type" content="website" />
<meta property="og:url" content="https://jvns.ca/blog/2017/10/05/reasons-kubernetes-is-cool/" />
<meta property="og:site_name" content="Julia Evans" />
<link rel="canonical" href="https://jvns.ca/blog/2017/10/05/reasons-kubernetes-is-cool/">
<link href="/favicon.ico" rel="icon">
<link href="/stylesheets/screen.css" rel="preload" type="text/css" as="style">
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
<link href="/stylesheets/print.css" media="print" rel="stylesheet" type="text/css">
<link href="/atom.xml" rel="alternate" title="Julia Evans" type="application/atom+xml">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css" integrity="sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0" crossorigin="anonymous">
<script defer data-domain="jvns.ca" src="https://plausible.io/js/script.js"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js" integrity="sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4" crossorigin="anonymous"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
<script defer type="text/javascript">
window.heap=window.heap||[],heap.load=function(e,t){window.heap.appid=e,window.heap.config=t=t||{};var r=document.createElement("script");r.type="text/javascript",r.async=!0,r.src="https://cdn.heapanalytics.com/js/heap-"+e+".js";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(r,a);for(var n=function(e){return function(){heap.push([e].concat(Array.prototype.slice.call(arguments,0)))}},p=["addEventProperties","addUserProperties","clearEventProperties","identify","resetIdentity","removeEventProperty","setEventProperties","track","unsetEventProperty"],o=0;o<p.length;o++)heap[p[o]]=n(p[o])};
heap.load("2242143965");
</script>
</head>
<body>
<div id="skiptocontent">
<a href="#main">Skip to main content</a>
</div>
<div id="wrap">
<header role="banner">
<hgroup>
<h1><a href="/">Julia Evans</a></h1>
</hgroup>
<ul class="header-links">
<li><a href="/about">About</a></li>
<li><a href="/talks">Talks</a></li>
<li><a href="/projects/">Projects</a></li>
<li><a rel="me" href="https://social.jvns.ca/@b0rk">Mastodon</a></li>
<li><a href="https://bsky.app/profile/b0rk.jvns.ca">Bluesky</a></li>
<li><a href="https://github.com/jvns">Github</a></li>
</ul>
</header>
<nav role="navigation" class="header-nav"><ul class="main-navigation">
<li><a href="/categories/favorite/">Favorites</a></li>
<li><a href="/til/">TIL</a></li>
<li><a href="https://wizardzines.com">Zines</a></li>
<li class="subscription" data-subscription="rss"><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
</ul>
</nav>
<div id="main">
<div id="content">
<div>
<article class="hentry" role="article">
<header>
<h1 class="entry-title">Reasons Kubernetes is cool</h1>
<div class="post-tags">
•
<a class="post-tag" href="/categories/kubernetes">kubernetes</a> •
</div>
<p class="meta sans">
<time class="date" datetime="2017-10-05T21:23:46" pubdate data-updated="true">
October 5, 2017
</time>
</p>
</header>
<main>
<p>When I first learned about Kubernetes (a year and a half ago?) I really didn&rsquo;t understand why I
should care about it.</p>
<p>I&rsquo;ve been working full time with Kubernetes for 3 months or so and now have some thoughts about why
I think it&rsquo;s useful. (I&rsquo;m still very far from being a Kubernetes expert!) Hopefully this will help a
little in your journey to understand what even is going on with Kubernetes!</p>
<p>I will try to explain some reason I think Kubenetes is interesting without using the words &ldquo;cloud native&rdquo;,
&ldquo;orchestration&rdquo;, &ldquo;container&rdquo;, or any Kubernetes-specific terminology :). I&rsquo;m going to explain this
mostly from the perspective of a kubernetes operator / infrastructure engineer, since my job right
now is to set up Kubernetes and make it work well.</p>
<p>I&rsquo;m not going to try to address the question of &ldquo;should you use kubernetes for your production
systems?&rdquo; at all, that is a very complicated question. (not least because &ldquo;in production&rdquo; has
totally different requirements depending on what you&rsquo;re doing)</p>
<h3 id="kubernetes-lets-you-run-code-in-production-without-setting-up-new-servers" class="post-heading">
<a href="#kubernetes-lets-you-run-code-in-production-without-setting-up-new-servers">
Kubernetes lets you run code in production without setting up new servers
</a>
</h3>
<p>The first pitch I got for Kubernetes was the following conversation with my partner Kamal:</p>
<p>Here&rsquo;s an approximate transcript:</p>
<ul>
<li>Kamal: With Kubernetes you can set up a new service with a single command</li>
<li>Julia: I don&rsquo;t understand how that&rsquo;s possible.</li>
<li>Kamal: Like, you just write 1 configuration file, apply it, and then you have a HTTP service running in production</li>
<li>Julia: But today I need to create new AWS instances, write a puppet manifest, set up service discovery, configure my load balancers, configure our deployment software, and make sure DNS is working, it takes at least 4 hours if nothing goes wrong.</li>
<li>Kamal: Yeah. With Kubernetes you don&rsquo;t have to do any of that, you can set up a new HTTP service in 5 minutes and it&rsquo;ll just automatically run. As long as you have spare capacity in your cluster it just works!</li>
<li>Julia: There must be a trap</li>
</ul>
<p>There kind of is a trap, setting up a production Kubernetes cluster is (in my experience) is
definitely not easy. (see <a href="https://github.com/kelseyhightower/kubernetes-the-hard-way">Kubernetes The Hard Way</a> for what&rsquo;s involved to get
started). But we&rsquo;re not going to go into that right now!</p>
<p>So the first cool thing about Kubernetes is that it has the potential to make life way easier for
developers who want to deploy new software into production. That&rsquo;s cool, and it&rsquo;s actually true,
once you have a working Kubernetes cluster you really can set up a production HTTP service (&ldquo;run 5
of this application, set up a load balancer, give it this DNS name, done&rdquo;) with just one
configuration file. It&rsquo;s really fun to see.</p>
<h3 id="kubernetes-gives-you-easy-visibility-control-of-what-code-you-have-running-in-production" class="post-heading">
<a href="#kubernetes-gives-you-easy-visibility-control-of-what-code-you-have-running-in-production">
Kubernetes gives you easy visibility &amp; control of what code you have running in production
</a>
</h3>
<p>IMO you can&rsquo;t understand Kubernetes without understanding etcd. So let&rsquo;s talk about etcd!</p>
<p>Imagine that I asked you today &ldquo;hey, tell me every application you have running in production, what
host it&rsquo;s running on, whether it&rsquo;s healthy or not, and whether or not it has a DNS name attached to
it&rdquo;. I don&rsquo;t know about you but I would need to go look in a bunch of different places to answer
this question and it would take me quite a while to figure out. I definitely can&rsquo;t query just one
API.</p>
<p>In Kubernetes, all the state in your cluster &ndash; applications running (&ldquo;pods&rdquo;), nodes, DNS names,
cron jobs, and more &ndash; is stored in a single database (etcd). Every Kubernetes component is
stateless, and basically works by</p>
<ul>
<li>Reading state from etcd (eg &ldquo;the list of pods assigned to node 1&rdquo;)</li>
<li>Making changes (eg &ldquo;actually start running pod A on node 1&rdquo;)</li>
<li>Updating the state in etcd (eg &ldquo;set the state of pod A to ‘running&rsquo;&rdquo;)</li>
</ul>
<p>This means that if you want to answer a question like &ldquo;hey, how many nginx pods do I have running
right now in that availabliity zone?&rdquo; you can answer it by querying a single unified API (the
Kubernetes API!). And you have exactly the same access to that API that every other Kubernetes
component does.</p>
<p>This also means that you have easy control of everything running in Kubernetes. If you want to, say,</p>
<ul>
<li>Implement a complicated custom rollout strategy for deployments (deploy 1 thing, wait 2 minutes, deploy 5 more, wait 3.7 minutes, etc)</li>
<li>Automatically <a href="https://github.com/kamalmarhubi/kubereview">start a new webserver</a> every time a branch is pushed to github</li>
<li>Monitor all your running applications to make sure all of them have a reasonable cgroups memory limit</li>
</ul>
<p>all you need to do is to write a program that talks to the Kubernetes API. (a &ldquo;controller&rdquo;)</p>
<p>Another very exciting thing about the Kubernetes API is that you&rsquo;re not limited to just
functionality that Kubernetes provides! If you decide that you have your own opinions about how your
software should be deployed / created / monitored, then you can write code that uses the Kubernetes
API to do it! It lets you do everything you need.</p>
<h3 id="if-every-kubernetes-component-dies-your-code-will-still-keep-running" class="post-heading">
<a href="#if-every-kubernetes-component-dies-your-code-will-still-keep-running">
If every Kubernetes component dies, your code will still keep running
</a>
</h3>
<p>One thing I was originally promised (by various blog posts :)) about Kubernetes was &ldquo;hey, if the
Kubernetes apiserver and everything else dies, it&rsquo;s ok, your code will just keep running&rdquo;. I thought
this sounded cool in theory but I wasn&rsquo;t sure if it was actually true.</p>
<p>So far it seems to be actually true!</p>
<p>I&rsquo;ve been through some etcd outages now, and what happens is</p>
<ol>
<li>All the code that was running keeps running</li>
<li>Nothing <em>new</em> happens (you can&rsquo;t deploy new code or make changes, cron jobs will stop working)</li>
<li>When everything comes back, the cluster will catch up on whatever it missed</li>
</ol>
<p>This does mean that if etcd goes down and one of your applications crashes or something, it can&rsquo;t come back up until etcd returns.</p>
<h3 id="kubernetes-design-is-pretty-resilient-to-bugs" class="post-heading">
<a href="#kubernetes-design-is-pretty-resilient-to-bugs">
Kubernetes&rsquo; design is pretty resilient to bugs
</a>
</h3>
<p>Like any piece of software, Kubernetes has bugs. For example right now in our cluster the controller manager has a memory leak, and the scheduler crashes pretty regularly. Bugs obviously aren&rsquo;t good but so far I&rsquo;ve found that Kubernetes&rsquo; design helps mitigate a lot of the bugs in its core components really well.</p>
<p>If you restart any component, what happens is:</p>
<ul>
<li>It reads all its relevant state from etcd</li>
<li>It starts doing the necessary things it&rsquo;s supposed to be doing based on that state (scheduling pods, garbage collecting completed pods, scheduling cronjobs, deploying daemonsets, whatever)</li>
</ul>
<p>Because all the components don&rsquo;t keep any state in memory, you can just restart them at any time and that can help mitigate a variety of bugs.</p>
<p>For example! Let&rsquo;s say you have a memory leak in your controller manager. Because the controller
manager is stateless, you can just periodically restart it every hour or something and feel
confident that you won&rsquo;t cause any consistency issues. Or we ran into a bug in the scheduler where
it would sometimes just forget about pods and never schedule them. You can sort of mitigate this
just by restarting the scheduler every 10 minutes. (we didn&rsquo;t do that, we fixed the bug instead, but
you <em>could</em> :) )</p>
<p>So I feel like I can trust Kubernetes&rsquo; design to help make sure the state in the cluster is
consistent even when there are bugs in its core components. And in general I think the software is
generally improving over time. The only stateful thing you have to operate is etcd</p>
<p>Not to harp on this &ldquo;state&rdquo; thing too much but &ndash; I think it&rsquo;s cool that in Kubernetes the only
thing you have to come up with backup/restore plans for is etcd (unless you use persistent volumes
for your pods). I think it makes kubernetes operations a lot easier to think about.</p>
<h3 id="implementing-new-distributed-systems-on-top-of-kubernetes-is-relatively-easy" class="post-heading">
<a href="#implementing-new-distributed-systems-on-top-of-kubernetes-is-relatively-easy">
Implementing new distributed systems on top of Kubernetes is relatively easy
</a>
</h3>
<p>Suppose you want to implement a distributed cron job scheduling system! Doing that from scratch is a
ton of work. But implementing a distributed cron job scheduling system inside Kubernetes is much
easier! (still not trivial, it&rsquo;s still a distributed system)</p>
<p>The first time I read the code for the Kubernetes cronjob controller I was really delighted by how
simple it was. Here, go read it! The main logic is like 400 lines of Go. Go ahead, read it! =&gt;
<a href="https://github.com/kubernetes/kubernetes/blob/e4551d50e57c089aab6f67333412d3ca64bc09ae/pkg/controller/cronjob/cronjob_controller.go">cronjob_controller.go</a> &lt;=</p>
<p>Basically what the cronjob controller does is:</p>
<ul>
<li>Every 10 seconds:
<ul>
<li>Lists all the cronjobs that exist</li>
<li>Checks if any of them need to run right now</li>
<li>If so, creates a new Job object to be scheduled &amp; actually run by other Kubernetes controllers</li>
<li>Clean up finished jobs</li>
<li>Repeat</li>
</ul>
</li>
</ul>
<p>The Kubernetes model is pretty constrained (it has this pattern of resources are defined in etcd,
controllers read those resources and update etcd), and I think having this relatively
opinionated/constrained model makes it easier to develop your own distributed systems inside the
Kubernetes framework.</p>
<p>Kamal introduced me to this idea of &ldquo;Kubernetes is a good platform for writing your own distributed systems&rdquo; instead of just &ldquo;Kubernetes is a distributed system you can use&rdquo; and I think it&rsquo;s really interesting. He has a prototype of a <a href="https://github.com/kamalmarhubi/kubereview">system to run an HTTP service for every branch you push to github</a>. It took him a weekend and is like 800 lines of Go, which I thought was impressive!</p>
<h3 id="kubernetes-lets-you-do-some-amazing-things-but-isn-t-easy" class="post-heading">
<a href="#kubernetes-lets-you-do-some-amazing-things-but-isn-t-easy">
Kubernetes lets you do some amazing things (but isn&rsquo;t easy)
</a>
</h3>
<p>I started out by saying &ldquo;kubernetes lets you do these magical things, you can just spin up so much
infrastructure with a single configuration file, it&rsquo;s amazing&rdquo;. And that&rsquo;s true!</p>
<p>What I mean by &ldquo;Kubernetes isn&rsquo;t easy&rdquo; is that Kubernetes has a lot of moving parts learning how to
successfully operate a highly available Kubernetes cluster is a lot of work. Like I find that with a
lot of the abstractions it gives me, I need to understand what is underneath those abstractions in
order to debug issues and configure things properly. I love learning new things so this doesn&rsquo;t make
me angry or anything, I just think it&rsquo;s important to know :)</p>
<p>One specific example of &ldquo;I can&rsquo;t just rely on the abstractions&rdquo; that I&rsquo;ve struggled with is that I
needed to learn a LOT <a href="https://jvns.ca/blog/2016/12/22/container-networking/">about how networking works on Linux</a> to feel confident with setting up
Kubernetes networking, way more than I&rsquo;d ever had to learn about networking before. This was very
fun but pretty time consuming. I might write more about what is hard/interesting about setting up Kubernetes networking at some point.</p>
<p>Or I wrote a <a href="https://jvns.ca/blog/2017/08/05/how-kubernetes-certificates-work/">2000 word blog post</a>
about everything I had to learn about Kubernetes&rsquo; different options for certificate authorities to
be able to set up my Kubernetes CAs successfully.</p>
<p>I think some of these managed Kubernetes systems like GKE (google&rsquo;s kubernetes product) may be
simpler since they make a lot of decisions for you but I haven&rsquo;t tried any of them.</p>
</main>
<footer>
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; display: inline;}
#mc_embed_signup {
display: inline;
}
#mc_embed_signup input.button {
background: #ff5e00;
display: inline;
color: white;
padding: 6px 12px;
}
</style>
<div class="sharing">
<style>
.form-inline {
display:flex; flex-flow: row wrap; justify-content: center;
}
.form-inline input, .form-inline span {
padding: 10px;
}
.form-inline input {
display:inline;
max-width:30%;
margin: 0 10px 0 0;
background-color: #fff;
border: 1px solid #ddd;
border-radius: 5px;
padding: 10px;
}
button {
background-color: #f50;
box-shadow: none;
border: 0;
border-radius: 5px;
color: white;
padding: 5px 10px;
}
@media (max-width: 800px) {
.form-inline input {
margin: 10px 0;
max-width:100% !important;
}
.form-inline {
flex-direction: column;
align-items: stretch;
}
}
</style>
<div align="center">
<form class="form-inline" action="https://app.convertkit.com/forms/1052396/subscriptions" method="post" data-uid="8884355abb" data-format="inline" data-version="5">
<span> Want a weekly digest of this blog?</span>
<input name="email_address" type="text" placeholder="Email address" />
<button type="submit" data-element="submit">Subscribe</button>
</form>
</div>
</div>
<p class="meta">
<a class="basic-alignment left" href="https://jvns.ca/teach-tech-with-cartoons/" title="Previous Post: How to teach technical concepts with cartoons">How to teach technical concepts with cartoons</a>
<a class="basic-alignment right" href="https://jvns.ca/blog/2017/10/10/operating-a-kubernetes-network/" title="Next Post: Operating a Kubernetes network">Operating a Kubernetes network</a>
</p>
</footer>
</article>
</div>
</div>
</div>
<nav role="navigation" class="footer-nav"> <a href="/">Archives</a>
</nav>
<footer role="contentinfo"><span class="credit">&copy; Julia Evans. </span>
<span>If you like this, you may like <a href="https://web.archive.org/web/20181228051203/http://www.uliaea.ca/">Ulia Ea</a> or, more seriously, this list of <a href="https://jvns.ca/blogroll">blogs I love</a> or some <a href="https://jvns.ca/bookshelf">books I've read</a>. <br>
<p class="rc-scout__text"><i class="rc-scout__logo"></i>
You might also like the <a class="rc-scout__link" href="https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8">Recurse Center</a>, my very favorite programming community <a href="/categories/hackerschool/">(my posts about it)</a></p>
</span>
<style class="rc-scout__style" type="text/css">.rc-scout{display:block;padding:0;border:0;margin:0;}.rc-scout__text{display:block;padding:0;border:0;margin:0;height:100%;font-size:100%;}.rc-scout__logo{display:inline-block;padding:0;border:0;margin:0;width:0.85em;height:0.85em;background:no-repeat center url('data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2012%2015%22%3E%3Crect%20x%3D%220%22%20y%3D%220%22%20width%3D%2212%22%20height%3D%2210%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%221%22%20width%3D%2210%22%20height%3D%228%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%222%22%20width%3D%228%22%20height%3D%226%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%229%22%20width%3D%224%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%2211%22%20width%3D%2210%22%20height%3D%224%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%220%22%20y%3D%2212%22%20width%3D%2212%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%225%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%227%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%228%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%229%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3C%2Fsvg%3E');}.rc-scout__link:link,.rc-scout__link:visited{color:#61ae24;text-decoration:underline;}.rc-scout__link:hover,.rc-scout__link:active{color:#4e8b1d;}</style>
</footer>
<script type="text/rocketscript">
(function(){
var twitterWidgets = document.createElement('script');
twitterWidgets.type = 'text/javascript';
twitterWidgets.async = true;
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
})();
</script>
</div>
</body>
</html>