281 lines
15 KiB
HTML
281 lines
15 KiB
HTML
<!DOCTYPE html>
|
|
|
|
|
|
<html class="no-js" lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<title>Swapping, memory limits, and cgroups</title>
|
|
<meta name="author" content="Julia Evans">
|
|
<meta name="HandheldFriendly" content="True">
|
|
<meta name="MobileOptimized" content="320">
|
|
<meta name="description" content="Swapping, memory limits, and cgroups">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
|
|
<meta property="og:title" content='Swapping, memory limits, and cgroups'>
|
|
<meta property="og:type" content="website" />
|
|
<meta property="og:url" content="https://jvns.ca/blog/2017/02/17/mystery-swap/" />
|
|
<meta property="og:site_name" content="Julia Evans" />
|
|
|
|
<link rel="canonical" href="https://jvns.ca/blog/2017/02/17/mystery-swap/">
|
|
<link href="/favicon.ico" rel="icon">
|
|
|
|
<link href="/stylesheets/screen.css" rel="preload" type="text/css" as="style">
|
|
|
|
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
|
|
<link href="/stylesheets/print.css" media="print" rel="stylesheet" type="text/css">
|
|
|
|
|
|
<link href="/atom.xml" rel="alternate" title="Julia Evans" type="application/atom+xml">
|
|
|
|
|
|
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css" integrity="sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0" crossorigin="anonymous">
|
|
<script defer data-domain="jvns.ca" src="https://plausible.io/js/script.js"></script>
|
|
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js" integrity="sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4" crossorigin="anonymous"></script>
|
|
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
|
|
|
|
<script defer type="text/javascript">
|
|
window.heap=window.heap||[],heap.load=function(e,t){window.heap.appid=e,window.heap.config=t=t||{};var r=document.createElement("script");r.type="text/javascript",r.async=!0,r.src="https://cdn.heapanalytics.com/js/heap-"+e+".js";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(r,a);for(var n=function(e){return function(){heap.push([e].concat(Array.prototype.slice.call(arguments,0)))}},p=["addEventProperties","addUserProperties","clearEventProperties","identify","resetIdentity","removeEventProperty","setEventProperties","track","unsetEventProperty"],o=0;o<p.length;o++)heap[p[o]]=n(p[o])};
|
|
heap.load("2242143965");
|
|
</script>
|
|
</head>
|
|
<body>
|
|
<div id="skiptocontent">
|
|
<a href="#main">Skip to main content</a>
|
|
</div>
|
|
<div id="wrap">
|
|
<header role="banner">
|
|
<hgroup>
|
|
<h1><a href="/">Julia Evans</a></h1>
|
|
</hgroup>
|
|
<ul class="header-links">
|
|
<li><a href="/about">About</a></li>
|
|
<li><a href="/talks">Talks</a></li>
|
|
<li><a href="/projects/">Projects</a></li>
|
|
<li><a rel="me" href="https://social.jvns.ca/@b0rk">Mastodon</a></li>
|
|
<li><a href="https://bsky.app/profile/b0rk.jvns.ca">Bluesky</a></li>
|
|
<li><a href="https://github.com/jvns">Github</a></li>
|
|
</ul>
|
|
</header>
|
|
<nav role="navigation" class="header-nav"><ul class="main-navigation">
|
|
<li><a href="/categories/favorite/">Favorites</a></li>
|
|
<li><a href="/til/">TIL</a></li>
|
|
<li><a href="https://wizardzines.com">Zines</a></li>
|
|
<li class="subscription" data-subscription="rss"><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
|
|
</ul>
|
|
</nav>
|
|
<div id="main">
|
|
<div id="content">
|
|
|
|
|
|
<div>
|
|
<article class="hentry" role="article">
|
|
<header>
|
|
<h1 class="entry-title">Swapping, memory limits, and cgroups</h1>
|
|
|
|
<div class="post-tags">
|
|
|
|
</div>
|
|
<p class="meta sans">
|
|
<time class="date" datetime="2017-02-17T19:47:27" pubdate data-updated="true">
|
|
|
|
February 17, 2017
|
|
|
|
</time>
|
|
</p>
|
|
</header>
|
|
<main>
|
|
<p>This week there was a bug at work, and I learned something new about
|
|
memory & swap & cgroups!</p>
|
|
<p>Understanding how memory usage works has been an ongoing project for a while – back in December, I
|
|
wrote <a href="/blog/2016/12/03/how-much-memory-is-my-process-using-/">How much memory is my process using?</a>
|
|
which explains how memory works on Linux.</p>
|
|
<p>So I felt surprised and worried yesterday when something was happening
|
|
with memory on Linux that I didn’t understand! Here’s the situation and
|
|
why I was confused:</p>
|
|
<p>We had some machines running builds. They were swapping. They had say
|
|
30GB of RAM in total. 15GB was being used by some processes and 15GB by the
|
|
filesystem cache. I was really confused about why these machines were
|
|
swapping, because – I know about memory! If 15GB of memory is being
|
|
used by the filesystem cache, the OS can always free that memory!
|
|
There’s no reason to swap!</p>
|
|
<p>I then learned about the “swappiness” setting, and that if “swappiness”
|
|
is high, then the OS is more likely to swap, even if it doesn’t
|
|
absolutely need to. We tried setting <code>sysctl vm.swappiness=1</code>, which
|
|
lets you tell the operating system “no, really, please don’t swap, just
|
|
take memory away from the filesystem cache instead”. The machine
|
|
continued to swap. I was confused.</p>
|
|
<p>After a while, we turned off swap and things got worse. Some processes
|
|
started being OOM killed. (the OOM killer on Linux will kill processes
|
|
if you run out of memory) But why? The boxes had free memory, didn’t
|
|
they? In my head I had an axiom “if a computer has free memory, there is
|
|
no reason the processes on it should get OOM killed”. Obviously there
|
|
was something I did not understand.</p>
|
|
<p>I finally looked at the output of <code>dmesg</code> (which is how you see messages the Linux
|
|
kernel prints about what it’s up to) to understand why the
|
|
processes were being OOM killed. And then there was this magic word:
|
|
<code>cgroups</code>. Everything became clear pretty quickly:</p>
|
|
<ul>
|
|
<li>the processes that were being killed were in a cgroup (which we talked
|
|
about back in this <a href="http://jvns.ca/blog/2016/10/10/what-even-is-a-container/">namespaces & groups</a> post)</li>
|
|
<li>cgroups can have <strong>memory limits</strong>, which are like “you are only
|
|
allowed to use 15GB of memory otherwise your processes will get
|
|
killed by the OOM killer”</li>
|
|
<li>and this memory limit was the reason the processes were being killed
|
|
even though there was free memory!</li>
|
|
</ul>
|
|
<h3 id="swap-cgroup-memory-limits-a-little-surprising" class="post-heading">
|
|
<a href="#swap-cgroup-memory-limits-a-little-surprising">
|
|
swap + cgroup memory limits = a little surprising
|
|
</a>
|
|
</h3>
|
|
<p>My model of memory limits on cgroups was always “if you use more than X
|
|
memory, you will get killed right away”. It turns out that that
|
|
assumptions was wrong! If you use more than X memory, you can still use
|
|
swap!</p>
|
|
<p>And apparently some kernels also support setting separate swap limits.
|
|
So you could set your memory limit to X and your swap limit to 0, which
|
|
would give you more predictable behavior. Swapping is weird and
|
|
confusing.</p>
|
|
<p>Anyway, we found out through all this that the processes in question had
|
|
recently started using much more memory for very understandable reasons,
|
|
and rolled back that change, and everything made sense again.</p>
|
|
<p>And more importantly than everything making sense, the build system was
|
|
happy again.</p>
|
|
<h3 id="does-swap-even-make-sense" class="post-heading">
|
|
<a href="#does-swap-even-make-sense">
|
|
does swap even make sense?
|
|
</a>
|
|
</h3>
|
|
<p>It’s not completely clear to me under what circumstances having swap on
|
|
a computer at all even makes sense. It seems like swap has some role on
|
|
desktop computers.</p>
|
|
<p>I am not sure though if any of the servers we operate benefit by having
|
|
swap enabled? This seems like it would be a good thing to understand.</p>
|
|
<p>Like I hear the advice “no, just always turn off swap, it will be better
|
|
overall” a lot and maybe that is the right thing! I think the reason
|
|
that swap is considered bad in production systems is that it has
|
|
weird/unpredictable effects and predictability is good.</p>
|
|
<p>Somewhat relatedly, this <a href="https://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/">swap insanity</a> article looks really interesting.</p>
|
|
<h3 id="understanding-memory-models-is-cool" class="post-heading">
|
|
<a href="#understanding-memory-models-is-cool">
|
|
understanding memory models is cool
|
|
</a>
|
|
</h3>
|
|
<p>I learned</p>
|
|
<ul>
|
|
<li>the vm.swappiness exists and you can use it to make a machine more or
|
|
less likely to swap</li>
|
|
<li>that when Linux OOM kills a process in a cgroup (“container”), it
|
|
actually prints a bunch of very useful stuff about the memory usage of
|
|
everything else in the cgroup at the time. I should remember to look
|
|
at dmesg earlier on!</li>
|
|
</ul>
|
|
<p>It’s really important to me to understand what’s happening on the
|
|
computers that I work with – when something happens like “this computer
|
|
is swapping and I don’t know WHY” it bothers me a lot.</p>
|
|
<p>Now if I ever see a process mysteriously swapping hopefully I will remember
|
|
about memory limits!</p>
|
|
|
|
</main>
|
|
|
|
<footer>
|
|
|
|
<style type="text/css">
|
|
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; display: inline;}
|
|
#mc_embed_signup {
|
|
display: inline;
|
|
}
|
|
#mc_embed_signup input.button {
|
|
background: #ff5e00;
|
|
display: inline;
|
|
color: white;
|
|
padding: 6px 12px;
|
|
}
|
|
</style>
|
|
<div class="sharing">
|
|
|
|
<style>
|
|
.form-inline {
|
|
display:flex; flex-flow: row wrap; justify-content: center;
|
|
}
|
|
.form-inline input, .form-inline span {
|
|
padding: 10px;
|
|
}
|
|
.form-inline input {
|
|
display:inline;
|
|
max-width:30%;
|
|
margin: 0 10px 0 0;
|
|
background-color: #fff;
|
|
border: 1px solid #ddd;
|
|
border-radius: 5px;
|
|
padding: 10px;
|
|
}
|
|
button {
|
|
background-color: #f50;
|
|
box-shadow: none;
|
|
border: 0;
|
|
border-radius: 5px;
|
|
color: white;
|
|
padding: 5px 10px;
|
|
}
|
|
@media (max-width: 800px) {
|
|
.form-inline input {
|
|
margin: 10px 0;
|
|
max-width:100% !important;
|
|
}
|
|
.form-inline {
|
|
flex-direction: column;
|
|
align-items: stretch;
|
|
}
|
|
}
|
|
</style>
|
|
|
|
<div align="center">
|
|
<form class="form-inline" action="https://app.convertkit.com/forms/1052396/subscriptions" method="post" data-uid="8884355abb" data-format="inline" data-version="5">
|
|
<span> Want a weekly digest of this blog?</span>
|
|
<input name="email_address" type="text" placeholder="Email address" />
|
|
<button type="submit" data-element="submit">Subscribe</button>
|
|
</form>
|
|
</div>
|
|
|
|
|
|
</div>
|
|
|
|
<p class="meta">
|
|
|
|
<a class="basic-alignment left" href="https://jvns.ca/blog/2017/02/16/hiring---opportunity/" title="Previous Post: Hiring & opportunity">Hiring & opportunity</a>
|
|
|
|
|
|
<a class="basic-alignment right" href="https://jvns.ca/blog/so-you-want-to-be-a-wizard/" title="Next Post: So you want to be a wizard">So you want to be a wizard</a>
|
|
|
|
</p>
|
|
</footer>
|
|
|
|
</article>
|
|
</div>
|
|
|
|
</div>
|
|
</div>
|
|
<nav role="navigation" class="footer-nav"> <a href="/">Archives</a>
|
|
</nav>
|
|
<footer role="contentinfo"><span class="credit">© Julia Evans. </span>
|
|
<span>If you like this, you may like <a href="https://web.archive.org/web/20181228051203/http://www.uliaea.ca/">Ulia Ea</a> or, more seriously, this list of <a href="https://jvns.ca/blogroll">blogs I love</a> or some <a href="https://jvns.ca/bookshelf">books I've read</a>. <br>
|
|
<p class="rc-scout__text"><i class="rc-scout__logo"></i>
|
|
You might also like the <a class="rc-scout__link" href="https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8">Recurse Center</a>, my very favorite programming community <a href="/categories/hackerschool/">(my posts about it)</a></p>
|
|
</span>
|
|
<style class="rc-scout__style" type="text/css">.rc-scout{display:block;padding:0;border:0;margin:0;}.rc-scout__text{display:block;padding:0;border:0;margin:0;height:100%;font-size:100%;}.rc-scout__logo{display:inline-block;padding:0;border:0;margin:0;width:0.85em;height:0.85em;background:no-repeat center url('data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2012%2015%22%3E%3Crect%20x%3D%220%22%20y%3D%220%22%20width%3D%2212%22%20height%3D%2210%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%221%22%20width%3D%2210%22%20height%3D%228%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%222%22%20width%3D%228%22%20height%3D%226%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%229%22%20width%3D%224%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%2211%22%20width%3D%2210%22%20height%3D%224%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%220%22%20y%3D%2212%22%20width%3D%2212%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%225%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%227%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%228%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%229%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3C%2Fsvg%3E');}.rc-scout__link:link,.rc-scout__link:visited{color:#61ae24;text-decoration:underline;}.rc-scout__link:hover,.rc-scout__link:active{color:#4e8b1d;}</style>
|
|
</footer>
|
|
<script type="text/rocketscript">
|
|
(function(){
|
|
var twitterWidgets = document.createElement('script');
|
|
twitterWidgets.type = 'text/javascript';
|
|
twitterWidgets.async = true;
|
|
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
|
|
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
|
|
})();
|
|
</script>
|
|
</div>
|
|
</body>
|
|
</html>
|
|
|