ishenwei/nexus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5523a7a1047aa4a04563bb842e7e60aa25908175
nexus/sreweekly/articles/217/07-why-strace-doesn-t-work-in-docker.html
admin af7633f9dc SRE weekly 所有文章
2026-09-12 17:23:01 +08:00

353 lines
19 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta charset="utf-8">
<title>Why strace doesn&#39;t work in Docker</title>
<meta name="author" content="Julia Evans">
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="description" content="Why strace doesn&#39;t work in Docker">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:title" content='Why strace doesn&#39;t work in Docker'>
<meta property="og:type" content="website" />
<meta property="og:url" content="https://jvns.ca/blog/2020/04/29/why-strace-doesnt-work-in-docker/" />
<meta property="og:site_name" content="Julia Evans" />
<link rel="canonical" href="https://jvns.ca/blog/2020/04/29/why-strace-doesnt-work-in-docker/">
<link href="/favicon.ico" rel="icon">
<link href="/stylesheets/screen.css" rel="preload" type="text/css" as="style">
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
<link href="/stylesheets/print.css" media="print" rel="stylesheet" type="text/css">
<link href="/atom.xml" rel="alternate" title="Julia Evans" type="application/atom+xml">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css" integrity="sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0" crossorigin="anonymous">
<script defer data-domain="jvns.ca" src="https://plausible.io/js/script.js"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js" integrity="sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4" crossorigin="anonymous"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
<script defer type="text/javascript">
window.heap=window.heap||[],heap.load=function(e,t){window.heap.appid=e,window.heap.config=t=t||{};var r=document.createElement("script");r.type="text/javascript",r.async=!0,r.src="https://cdn.heapanalytics.com/js/heap-"+e+".js";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(r,a);for(var n=function(e){return function(){heap.push([e].concat(Array.prototype.slice.call(arguments,0)))}},p=["addEventProperties","addUserProperties","clearEventProperties","identify","resetIdentity","removeEventProperty","setEventProperties","track","unsetEventProperty"],o=0;o<p.length;o++)heap[p[o]]=n(p[o])};
heap.load("2242143965");
</script>
</head>
<body>
<div id="skiptocontent">
<a href="#main">Skip to main content</a>
</div>
<div id="wrap">
<header role="banner">
<hgroup>
<h1><a href="/">Julia Evans</a></h1>
</hgroup>
<ul class="header-links">
<li><a href="/about">About</a></li>
<li><a href="/talks">Talks</a></li>
<li><a href="/projects/">Projects</a></li>
<li><a rel="me" href="https://social.jvns.ca/@b0rk">Mastodon</a></li>
<li><a href="https://bsky.app/profile/b0rk.jvns.ca">Bluesky</a></li>
<li><a href="https://github.com/jvns">Github</a></li>
</ul>
</header>
<nav role="navigation" class="header-nav"><ul class="main-navigation">
<li><a href="/categories/favorite/">Favorites</a></li>
<li><a href="/til/">TIL</a></li>
<li><a href="https://wizardzines.com">Zines</a></li>
<li class="subscription" data-subscription="rss"><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
</ul>
</nav>
<div id="main">
<div id="content">
<div>
<article class="hentry" role="article">
<header>
<h1 class="entry-title">Why strace doesn&#39;t work in Docker</h1>
<div class="post-tags">
</div>
<p class="meta sans">
<time class="date" datetime="2020-04-29T07:55:32" pubdate data-updated="true">
April 29, 2020
</time>
</p>
</header>
<main>
<p>While editing the capabilities page of the <a href="https://wizardzines.com/zines/containers">how containers work</a> zine, I found myself
trying to explain why <code>strace</code> doesn&rsquo;t work in a Docker container.</p>
<p>The problem here is &ndash; if I run <code>strace</code> in a Docker container on my laptop, this happens:</p>
<pre><code>$ docker run -it ubuntu:18.04 /bin/bash
$ # ... install strace ...
root@e27f594da870:/# strace ls
strace: ptrace(PTRACE_TRACEME, ...): Operation not permitted
</code></pre>
<p>strace works using the <code>ptrace</code> system call, so if <code>ptrace</code> isn&rsquo;t
allowed, it&rsquo;s definitely not gonna work! This is pretty easy to fix &ndash; on
my machine, this fixes it:</p>
<pre><code>docker run --cap-add=SYS_PTRACE -it ubuntu:18.04 /bin/bash
</code></pre>
<p>But I wasn&rsquo;t interested in fixing it, I wanted to know why it happens. So
why does strace not work, and why does <code>--cap-add=SYS_PTRACE</code> fix it?</p>
<h3 id="hypothesis-1-container-processes-are-missing-the-cap-sys-ptrace-capability" class="post-heading">
<a href="#hypothesis-1-container-processes-are-missing-the-cap-sys-ptrace-capability">
hypothesis 1: container processes are missing the <code>CAP_SYS_PTRACE</code> capability
</a>
</h3>
<p>I always thought the reason was that Docker container processes by
default didn&rsquo;t have the <code>CAP_SYS_PTRACE</code> capability. This is consistent
with it being fixed by <code>--cap-add=SYS_PTRACE</code>, right?</p>
<p>But this actually doesn&rsquo;t make sense for 2 reasons.</p>
<p><strong>Reason 1</strong>: Experimentally, as a regular user, I can strace on any process run by my
user. But if I check if my current process has the <code>CAP_SYS_PTRACE</code> capability, I don&rsquo;t:</p>
<pre><code>$ getpcaps $$
Capabilities for `11589': =
</code></pre>
<p><strong>Reason 2</strong>: <code>man capabilities</code> says this about <code>CAP_SYS_PTRACE</code>:</p>
<pre><code>CAP_SYS_PTRACE
* Trace arbitrary processes using ptrace(2);
</code></pre>
<p>So the point of <code>CAP_SYS_PTRACE</code> is to let you ptrace <strong>arbitrary</strong>
processes owned by any user, the way that root usually can. You shouldn&rsquo;t
need it to just ptrace a regular process owned by your user.</p>
<p>And I tested this a third way &ndash; I ran a Docker container with <code>docker run --cap-add=SYS_PTRACE -it ubuntu:18.04 /bin/bash</code>, dropped the
<code>CAP_SYS_PTRACE</code> capability, and I could still strace processes even
though I didn&rsquo;t have that capability anymore. What? Why?</p>
<h3 id="hypothesis-2-something-about-user-namespaces" class="post-heading">
<a href="#hypothesis-2-something-about-user-namespaces">
hypothesis 2: something about user namespaces???
</a>
</h3>
<p>My next (much less well-founded) hypothesis was something along the lines
of &ldquo;um, maybe the process is in a different user namespace and strace
doesn&rsquo;t work because of&hellip; reasons?&rdquo; This isn&rsquo;t really coherent but
here&rsquo;s what happened when I looked into it.</p>
<p>Is the container process in a different user namespace? Well, in the container:</p>
<pre><code>root@e27f594da870:/# ls /proc/$$/ns/user -l
... /proc/1/ns/user -&gt; 'user:[4026531837]'
</code></pre>
<p>On the host:</p>
<pre><code>bork@kiwi:~$ ls /proc/$$/ns/user -l
... /proc/12177/ns/user -&gt; 'user:[4026531837]'
</code></pre>
<p>Because the user namespace ID (<code>4026531837</code>) is the same, the root user
in the container is the exact same user as the root user on the host. So
there&rsquo;s definitely no reason it shouldn&rsquo;t be able to strace processes
that it created!</p>
<p>This hypothesis doesn&rsquo;t make much sense but I hadn&rsquo;t realized that the
root user in a Docker container is the same as the root user on the host,
so I thought that was interesting.</p>
<h3 id="hypothesis-3-the-ptrace-system-call-is-being-blocked-by-a-seccomp-bpf-rule" class="post-heading">
<a href="#hypothesis-3-the-ptrace-system-call-is-being-blocked-by-a-seccomp-bpf-rule">
hypothesis 3: the ptrace system call is being blocked by a seccomp-bpf rule
</a>
</h3>
<p>I also knew that Docker uses seccomp-bpf to stop container processes from
running a lot of system calls. And ptrace is in the <a href="https://docs.docker.com/engine/security/seccomp/">list of system calls
blocked by Docker&rsquo;s default seccomp
profile</a>! (actually the
list of allowed system calls is a whitelist, so it&rsquo;s just that ptrace is
not in the default whitelist. But it comes out to the same thing.)</p>
<p>That easily explains why strace wouldn&rsquo;t work in a Docker container &ndash; if
the <code>ptrace</code> system call is totally blocked, then of course you can&rsquo;t
call it at all and strace would fail.</p>
<p>Let&rsquo;s verify this hypothesis &ndash; if we disable all seccomp rules, can we
strace in a Docker container?</p>
<pre><code>$ docker run --security-opt seccomp=unconfined -it ubuntu:18.04 /bin/bash
$ strace ls
execve(&quot;/bin/ls&quot;, [&quot;ls&quot;], 0x7ffc69a65580 /* 8 vars */) = 0
... it works fine ...
</code></pre>
<p>Yes! It works! Great. Mystery solved, except&hellip;</p>
<h3 id="why-does-cap-add-sys-ptrace-fix-the-problem" class="post-heading">
<a href="#why-does-cap-add-sys-ptrace-fix-the-problem">
why does <code>--cap-add=SYS_PTRACE</code> fix the problem?
</a>
</h3>
<p>What we still haven&rsquo;t explained is: why does <code>--cap-add=SYS_PTRACE</code> would
fix the problem?</p>
<p>The man page for <code>docker run</code> explains the <code>--cap-add</code> argument this way:</p>
<pre><code>--cap-add=[]
Add Linux capabilities
</code></pre>
<p>That doesn&rsquo;t have anything to do with seccomp rules! What&rsquo;s going on?</p>
<h3 id="let-s-look-at-the-docker-source-code" class="post-heading">
<a href="#let-s-look-at-the-docker-source-code">
let&rsquo;s look at the Docker source code.
</a>
</h3>
<p>When the documentation doesn&rsquo;t help, the only thing to do is go look at
the source.</p>
<p>The nice thing about Go is, because dependencies are often vendored in a
Go repository, you can just grep the repository to figure out where the
code that does a thing is. So I cloned <code>github.com/moby/moby</code> and grepped
for some things, like <code>rg CAP_SYS_PTRACE</code>.</p>
<p>Here&rsquo;s what I think is going on. In containerd&rsquo;s seccomp implementation, in
<a href="https://github.com/containerd/containerd/blob/4be98fa28b62e8a012491d655a4d6818ef87b080/contrib/seccomp/seccomp_default.go#L527-L537">contrib/seccomp/seccomp_default.go</a>,
there&rsquo;s a bunch of code that makes sure that if a process has a
capability, then it&rsquo;s also given access (through a seccomp rule) to use
the system calls that go with that capability.</p>
<pre><code> case &quot;CAP_SYS_PTRACE&quot;:
s.Syscalls = append(s.Syscalls, specs.LinuxSyscall{
Names: []string{
&quot;kcmp&quot;,
&quot;process_vm_readv&quot;,
&quot;process_vm_writev&quot;,
&quot;ptrace&quot;,
},
Action: specs.ActAllow,
Args: []specs.LinuxSeccompArg{},
})
</code></pre>
<p>There&rsquo;s some other code that seems to do something very similar in
<a href="https://github.com/moby/moby/blob/cc0dfb6e7b22ad120c60a9ce770ea15415767cf9/profiles/seccomp/seccomp.go#L126-L132">profiles/seccomp/seccomp.go</a>
in moby and the <a href="https://github.com/moby/moby/blob/master/profiles/seccomp/default.json#L723-L739">default seccomp
profile</a>,
so it&rsquo;s possible that that&rsquo;s what&rsquo;s doing it instead.</p>
<p>So I think we have our answer!</p>
<h3 id="cap-add-in-docker-does-a-little-more-than-what-it-says" class="post-heading">
<a href="#cap-add-in-docker-does-a-little-more-than-what-it-says">
<code>--cap-add</code> in Docker does a little more than what it says
</a>
</h3>
<p>The upshot seems to be that <code>--cap-add</code> doesn&rsquo;t do exactly what it says
it does in the man page, it&rsquo;s more like
<code>--cap-add-and-also-whitelist-some-extra-system-calls-if-required</code>. Which makes
sense! If you have a capability like <code>CAP_SYS_PTRACE</code> which is supposed
to let you use the <code>process_vm_readv</code> system call but that system call is
blocked by a seccomp profile, that&rsquo;s not going to help you much!</p>
<p>So allowing the <code>process_vm_readv</code> and <code>ptrace</code> system calls when you
give the container <code>CAP_SYS_PTRACE</code> seems like a reasonable choice.</p>
<h3 id="strace-actually-does-work-in-newer-versions-of-docker" class="post-heading">
<a href="#strace-actually-does-work-in-newer-versions-of-docker">
strace actually does work in newer versions of Docker
</a>
</h3>
<p>As of <a href="https://github.com/moby/moby/commit/1124543ca8071074a537a15db251af46a5189907">this commit</a> (docker 19.03), Docker does actually allow the <code>ptrace</code> system calls for kernel versions newer than 4.8.</p>
<p>But the Docker version on my laptop is 18.09.7, so it predates that commit.</p>
<h3 id="that-s-all" class="post-heading">
<a href="#that-s-all">
that&rsquo;s all!
</a>
</h3>
<p>This was a fun small thing to investigate, and I think it&rsquo;s a nice
example of how containers are made of lots of moving pieces that work
together in not-completely-obvious ways.</p>
<p>If you liked this, you might like my new zine called <a href="https://wizardzines.com/zines/containers">How Containers Work</a> that explains the Linux kernel features that make containers work in 24 pages. You can read the pages on <a href="https://wizardzines.com/comics/capabilities/">capabilities</a> and <a href="https://wizardzines.com/comics/seccomp-bpf/">seccomp-bpf</a> from the zine.</p>
<div align="center">
<a href="https://wizardzines.com/zines/containers"><img width="300px" src="https://jvns.ca/images/containers-cover.jpg"></a>
</div>
</main>
<footer>
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; display: inline;}
#mc_embed_signup {
display: inline;
}
#mc_embed_signup input.button {
background: #ff5e00;
display: inline;
color: white;
padding: 6px 12px;
}
</style>
<div class="sharing">
<style>
.form-inline {
display:flex; flex-flow: row wrap; justify-content: center;
}
.form-inline input, .form-inline span {
padding: 10px;
}
.form-inline input {
display:inline;
max-width:30%;
margin: 0 10px 0 0;
background-color: #fff;
border: 1px solid #ddd;
border-radius: 5px;
padding: 10px;
}
button {
background-color: #f50;
box-shadow: none;
border: 0;
border-radius: 5px;
color: white;
padding: 5px 10px;
}
@media (max-width: 800px) {
.form-inline input {
margin: 10px 0;
max-width:100% !important;
}
.form-inline {
flex-direction: column;
align-items: stretch;
}
}
</style>
<div align="center">
<form class="form-inline" action="https://app.convertkit.com/forms/1052396/subscriptions" method="post" data-uid="8884355abb" data-format="inline" data-version="5">
<span> Want a weekly digest of this blog?</span>
<input name="email_address" type="text" placeholder="Email address" />
<button type="submit" data-element="submit">Subscribe</button>
</form>
</div>
</div>
<p class="meta">
<a class="basic-alignment left" href="https://jvns.ca/blog/2020/04/27/new-zine-how-containers-work/" title="Previous Post: New zine: How Containers Work!">New zine: How Containers Work!</a>
<a class="basic-alignment right" href="https://jvns.ca/blog/2020/05/08/metaphors-in-man-pages/" title="Next Post: Metaphors in man pages">Metaphors in man pages</a>
</p>
</footer>
</article>
</div>
</div>
</div>
<nav role="navigation" class="footer-nav"> <a href="/">Archives</a>
</nav>
<footer role="contentinfo"><span class="credit">&copy; Julia Evans. </span>
<span>If you like this, you may like <a href="https://web.archive.org/web/20181228051203/http://www.uliaea.ca/">Ulia Ea</a> or, more seriously, this list of <a href="https://jvns.ca/blogroll">blogs I love</a> or some <a href="https://jvns.ca/bookshelf">books I've read</a>. <br>
<p class="rc-scout__text"><i class="rc-scout__logo"></i>
You might also like the <a class="rc-scout__link" href="https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8">Recurse Center</a>, my very favorite programming community <a href="/categories/hackerschool/">(my posts about it)</a></p>
</span>
<style class="rc-scout__style" type="text/css">.rc-scout{display:block;padding:0;border:0;margin:0;}.rc-scout__text{display:block;padding:0;border:0;margin:0;height:100%;font-size:100%;}.rc-scout__logo{display:inline-block;padding:0;border:0;margin:0;width:0.85em;height:0.85em;background:no-repeat center url('data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2012%2015%22%3E%3Crect%20x%3D%220%22%20y%3D%220%22%20width%3D%2212%22%20height%3D%2210%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%221%22%20width%3D%2210%22%20height%3D%228%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%222%22%20width%3D%228%22%20height%3D%226%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%229%22%20width%3D%224%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%2211%22%20width%3D%2210%22%20height%3D%224%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%220%22%20y%3D%2212%22%20width%3D%2212%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%225%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%227%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%228%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%229%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3C%2Fsvg%3E');}.rc-scout__link:link,.rc-scout__link:visited{color:#61ae24;text-decoration:underline;}.rc-scout__link:hover,.rc-scout__link:active{color:#4e8b1d;}</style>
</footer>
<script type="text/rocketscript">
(function(){
var twitterWidgets = document.createElement('script');
twitterWidgets.type = 'text/javascript';
twitterWidgets.async = true;
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
})();
</script>
</div>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink