SRE weekly 所有文章
This commit is contained in:
211
sreweekly/markdown/217/07-why-strace-doesn-t-work-in-docker.md
Normal file
211
sreweekly/markdown/217/07-why-strace-doesn-t-work-in-docker.md
Normal file
@@ -0,0 +1,211 @@
|
||||
# Why strace doesn’t work in Docker
|
||||
|
||||
- **期号**: SRE Weekly Issue #217（2020-05-03）
|
||||
- **作者**: Julia Evans
|
||||
- **链接**: https://jvns.ca/blog/2020/04/29/why-strace-doesnt-work-in-docker/
|
||||
|
||||
## 简介
|
||||
|
||||
I love Julia Evans’s ability to teach me something new that I didn’t realize I didn’t know.
|
||||
|
||||
## 正文
|
||||
|
||||
# Why strace doesn't work in Docker
|
||||
|
||||
While editing the capabilities page of the [how containers work](https://wizardzines.com/zines/containers) zine, I found myself
|
||||
trying to explain why `strace` doesn’t work in a Docker container.
|
||||
|
||||
The problem here is – if I run `strace` in a Docker container on my laptop, this happens:
|
||||
|
||||
```
|
||||
$ docker run -it ubuntu:18.04 /bin/bash
|
||||
$ # ... install strace ...
|
||||
root@e27f594da870:/# strace ls
|
||||
strace: ptrace(PTRACE_TRACEME, ...): Operation not permitted
|
||||
```
|
||||
strace works using the `ptrace` system call, so if `ptrace` isn’t
|
||||
allowed, it’s definitely not gonna work! This is pretty easy to fix – on
|
||||
my machine, this fixes it:
|
||||
|
||||
```
|
||||
docker run --cap-add=SYS_PTRACE -it ubuntu:18.04 /bin/bash
|
||||
```
|
||||
But I wasn’t interested in fixing it, I wanted to know why it happens. So
|
||||
why does strace not work, and why does `--cap-add=SYS_PTRACE` fix it?
|
||||
|
||||
###
|
||||
|
||||
[hypothesis 1: container processes are missing the `CAP_SYS_PTRACE` capability](https://jvns.ca#hypothesis-1-container-processes-are-missing-the-cap-sys-ptrace-capability)
|
||||
|
||||
|
||||
`CAP_SYS_PTRACE` capability
|
||||
I always thought the reason was that Docker container processes by
|
||||
default didn’t have the `CAP_SYS_PTRACE` capability. This is consistent
|
||||
with it being fixed by `--cap-add=SYS_PTRACE`, right?
|
||||
|
||||
But this actually doesn’t make sense for 2 reasons.
|
||||
|
||||
**Reason 1**: Experimentally, as a regular user, I can strace on any process run by my
|
||||
user. But if I check if my current process has the `CAP_SYS_PTRACE` capability, I don’t:
|
||||
|
||||
```
|
||||
$ getpcaps $$
|
||||
Capabilities for `11589': =
|
||||
```
|
||||
**Reason 2**: `man capabilities` says this about `CAP_SYS_PTRACE`:
|
||||
|
||||
```
|
||||
CAP_SYS_PTRACE
|
||||
* Trace arbitrary processes using ptrace(2);
|
||||
```
|
||||
So the point of `CAP_SYS_PTRACE` is to let you ptrace **arbitrary**
|
||||
processes owned by any user, the way that root usually can. You shouldn’t
|
||||
need it to just ptrace a regular process owned by your user.
|
||||
|
||||
And I tested this a third way – I ran a Docker container with `docker run --cap-add=SYS_PTRACE -it ubuntu:18.04 /bin/bash`, dropped the
|
||||
`CAP_SYS_PTRACE` capability, and I could still strace processes even
|
||||
though I didn’t have that capability anymore. What? Why?
|
||||
|
||||
###
|
||||
|
||||
[hypothesis 2: something about user namespaces???](https://jvns.ca#hypothesis-2-something-about-user-namespaces)
|
||||
|
||||
|
||||
My next (much less well-founded) hypothesis was something along the lines of “um, maybe the process is in a different user namespace and strace doesn’t work because of… reasons?” This isn’t really coherent but here’s what happened when I looked into it.
|
||||
|
||||
Is the container process in a different user namespace? Well, in the container:
|
||||
|
||||
```
|
||||
root@e27f594da870:/# ls /proc/$$/ns/user -l
|
||||
... /proc/1/ns/user -> 'user:[4026531837]'
|
||||
```
|
||||
On the host:
|
||||
|
||||
```
|
||||
bork@kiwi:~$ ls /proc/$$/ns/user -l
|
||||
... /proc/12177/ns/user -> 'user:[4026531837]'
|
||||
```
|
||||
Because the user namespace ID (`4026531837`) is the same, the root user
|
||||
in the container is the exact same user as the root user on the host. So
|
||||
there’s definitely no reason it shouldn’t be able to strace processes
|
||||
that it created!
|
||||
|
||||
This hypothesis doesn’t make much sense but I hadn’t realized that the root user in a Docker container is the same as the root user on the host, so I thought that was interesting.
|
||||
|
||||
###
|
||||
|
||||
[hypothesis 3: the ptrace system call is being blocked by a seccomp-bpf rule](https://jvns.ca#hypothesis-3-the-ptrace-system-call-is-being-blocked-by-a-seccomp-bpf-rule)
|
||||
|
||||
|
||||
I also knew that Docker uses seccomp-bpf to stop container processes from
|
||||
running a lot of system calls. And ptrace is in the [list of system calls
|
||||
blocked by Docker’s default seccomp
|
||||
profile](https://docs.docker.com/engine/security/seccomp/)! (actually the
|
||||
list of allowed system calls is a whitelist, so it’s just that ptrace is
|
||||
not in the default whitelist. But it comes out to the same thing.)
|
||||
|
||||
That easily explains why strace wouldn’t work in a Docker container – if
|
||||
the `ptrace` system call is totally blocked, then of course you can’t
|
||||
call it at all and strace would fail.
|
||||
|
||||
Let’s verify this hypothesis – if we disable all seccomp rules, can we strace in a Docker container?
|
||||
|
||||
```
|
||||
$ docker run --security-opt seccomp=unconfined -it ubuntu:18.04 /bin/bash
|
||||
$ strace ls
|
||||
execve("/bin/ls", ["ls"], 0x7ffc69a65580 /* 8 vars */) = 0
|
||||
... it works fine ...
|
||||
```
|
||||
Yes! It works! Great. Mystery solved, except…
|
||||
|
||||
###
|
||||
|
||||
[why does `--cap-add=SYS_PTRACE` fix the problem?](https://jvns.ca#why-does-cap-add-sys-ptrace-fix-the-problem)
|
||||
|
||||
|
||||
`--cap-add=SYS_PTRACE` fix the problem?
|
||||
What we still haven’t explained is: why does `--cap-add=SYS_PTRACE` would
|
||||
fix the problem?
|
||||
|
||||
The man page for `docker run` explains the `--cap-add` argument this way:
|
||||
|
||||
```
|
||||
--cap-add=[]
|
||||
Add Linux capabilities
|
||||
```
|
||||
That doesn’t have anything to do with seccomp rules! What’s going on?
|
||||
|
||||
###
|
||||
|
||||
[let’s look at the Docker source code.](https://jvns.ca#let-s-look-at-the-docker-source-code)
|
||||
|
||||
|
||||
When the documentation doesn’t help, the only thing to do is go look at the source.
|
||||
|
||||
The nice thing about Go is, because dependencies are often vendored in a
|
||||
Go repository, you can just grep the repository to figure out where the
|
||||
code that does a thing is. So I cloned `github.com/moby/moby` and grepped
|
||||
for some things, like `rg CAP_SYS_PTRACE`.
|
||||
|
||||
Here’s what I think is going on. In containerd’s seccomp implementation, in
|
||||
[contrib/seccomp/seccomp_default.go](https://github.com/containerd/containerd/blob/4be98fa28b62e8a012491d655a4d6818ef87b080/contrib/seccomp/seccomp_default.go#L527-L537),
|
||||
there’s a bunch of code that makes sure that if a process has a
|
||||
capability, then it’s also given access (through a seccomp rule) to use
|
||||
the system calls that go with that capability.
|
||||
|
||||
```
|
||||
case "CAP_SYS_PTRACE":
|
||||
s.Syscalls = append(s.Syscalls, specs.LinuxSyscall{
|
||||
Names: []string{
|
||||
"kcmp",
|
||||
"process_vm_readv",
|
||||
"process_vm_writev",
|
||||
"ptrace",
|
||||
},
|
||||
Action: specs.ActAllow,
|
||||
Args: []specs.LinuxSeccompArg{},
|
||||
})
|
||||
```
|
||||
There’s some other code that seems to do something very similar in
|
||||
[profiles/seccomp/seccomp.go](https://github.com/moby/moby/blob/cc0dfb6e7b22ad120c60a9ce770ea15415767cf9/profiles/seccomp/seccomp.go#L126-L132)
|
||||
in moby and the [default seccomp
|
||||
profile](https://github.com/moby/moby/blob/master/profiles/seccomp/default.json#L723-L739),
|
||||
so it’s possible that that’s what’s doing it instead.
|
||||
|
||||
So I think we have our answer!
|
||||
|
||||
###
|
||||
|
||||
[`--cap-add` in Docker does a little more than what it says](https://jvns.ca#cap-add-in-docker-does-a-little-more-than-what-it-says)
|
||||
|
||||
|
||||
`--cap-add` in Docker does a little more than what it says
|
||||
The upshot seems to be that `--cap-add` doesn’t do exactly what it says
|
||||
it does in the man page, it’s more like
|
||||
`--cap-add-and-also-whitelist-some-extra-system-calls-if-required`. Which makes
|
||||
sense! If you have a capability like `CAP_SYS_PTRACE` which is supposed
|
||||
to let you use the `process_vm_readv` system call but that system call is
|
||||
blocked by a seccomp profile, that’s not going to help you much!
|
||||
|
||||
So allowing the `process_vm_readv` and `ptrace` system calls when you
|
||||
give the container `CAP_SYS_PTRACE` seems like a reasonable choice.
|
||||
|
||||
###
|
||||
|
||||
[strace actually does work in newer versions of Docker](https://jvns.ca#strace-actually-does-work-in-newer-versions-of-docker)
|
||||
|
||||
|
||||
As of [this commit](https://github.com/moby/moby/commit/1124543ca8071074a537a15db251af46a5189907) (docker 19.03), Docker does actually allow the `ptrace` system calls for kernel versions newer than 4.8.
|
||||
|
||||
But the Docker version on my laptop is 18.09.7, so it predates that commit.
|
||||
|
||||
###
|
||||
|
||||
[that’s all!](https://jvns.ca#that-s-all)
|
||||
|
||||
|
||||
This was a fun small thing to investigate, and I think it’s a nice example of how containers are made of lots of moving pieces that work together in not-completely-obvious ways.
|
||||
|
||||
If you liked this, you might like my new zine called [How Containers Work](https://wizardzines.com/zines/containers) that explains the Linux kernel features that make containers work in 24 pages. You can read the pages on [capabilities](https://wizardzines.com/comics/capabilities/) and [seccomp-bpf](https://wizardzines.com/comics/seccomp-bpf/) from the zine.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user