Files
nexus/sreweekly/articles/297/08-debugging-a-weird-file-not-found-error.html
2026-09-12 17:23:01 +08:00

378 lines
20 KiB
HTML

<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta charset="utf-8">
<title>Debugging a weird &#39;file not found&#39; error</title>
<meta name="author" content="Julia Evans">
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="description" content="Debugging a weird &#39;file not found&#39; error">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:title" content='Debugging a weird &#39;file not found&#39; error'>
<meta property="og:type" content="website" />
<meta property="og:url" content="https://jvns.ca/blog/2021/11/17/debugging-a-weird--file-not-found--error/" />
<meta property="og:site_name" content="Julia Evans" />
<link rel="canonical" href="https://jvns.ca/blog/2021/11/17/debugging-a-weird--file-not-found--error/">
<link href="/favicon.ico" rel="icon">
<link href="/stylesheets/screen.css" rel="preload" type="text/css" as="style">
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
<link href="/stylesheets/print.css" media="print" rel="stylesheet" type="text/css">
<link href="/atom.xml" rel="alternate" title="Julia Evans" type="application/atom+xml">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css" integrity="sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0" crossorigin="anonymous">
<script defer data-domain="jvns.ca" src="https://plausible.io/js/script.js"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js" integrity="sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4" crossorigin="anonymous"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
<script defer type="text/javascript">
window.heap=window.heap||[],heap.load=function(e,t){window.heap.appid=e,window.heap.config=t=t||{};var r=document.createElement("script");r.type="text/javascript",r.async=!0,r.src="https://cdn.heapanalytics.com/js/heap-"+e+".js";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(r,a);for(var n=function(e){return function(){heap.push([e].concat(Array.prototype.slice.call(arguments,0)))}},p=["addEventProperties","addUserProperties","clearEventProperties","identify","resetIdentity","removeEventProperty","setEventProperties","track","unsetEventProperty"],o=0;o<p.length;o++)heap[p[o]]=n(p[o])};
heap.load("2242143965");
</script>
</head>
<body>
<div id="skiptocontent">
<a href="#main">Skip to main content</a>
</div>
<div id="wrap">
<header role="banner">
<hgroup>
<h1><a href="/">Julia Evans</a></h1>
</hgroup>
<ul class="header-links">
<li><a href="/about">About</a></li>
<li><a href="/talks">Talks</a></li>
<li><a href="/projects/">Projects</a></li>
<li><a rel="me" href="https://social.jvns.ca/@b0rk">Mastodon</a></li>
<li><a href="https://bsky.app/profile/b0rk.jvns.ca">Bluesky</a></li>
<li><a href="https://github.com/jvns">Github</a></li>
</ul>
</header>
<nav role="navigation" class="header-nav"><ul class="main-navigation">
<li><a href="/categories/favorite/">Favorites</a></li>
<li><a href="/til/">TIL</a></li>
<li><a href="https://wizardzines.com">Zines</a></li>
<li class="subscription" data-subscription="rss"><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
</ul>
</nav>
<div id="main">
<div id="content">
<div>
<article class="hentry" role="article">
<header>
<h1 class="entry-title">Debugging a weird &#39;file not found&#39; error</h1>
<div class="post-tags">
</div>
<p class="meta sans">
<time class="date" datetime="2021-11-17T12:34:36" pubdate data-updated="true">
November 17, 2021
</time>
</p>
</header>
<main>
<p>Yesterday I ran into a weird error where I ran a program and got the error
&ldquo;file not found&rdquo; even though the program I was running existed. It&rsquo;s something
I&rsquo;ve run into before, but every time I&rsquo;m very surprised and confused by it
(what do you MEAN file not found, the file is RIGHT THERE???!!??)</p>
<p>So let&rsquo;s talk about what happened and why!</p>
<h3 id="the-error" class="post-heading">
<a href="#the-error">
the error
</a>
</h3>
<p>Let&rsquo;s start by showing the error message I got. I had a Go program called
<a href="https://gist.github.com/jvns/6147bc21fbb60b0090d543bb5e240134">serve.go</a>, and I was trying to bundle it into a Docker container with this
Dockerfile:</p>
<pre><code>FROM golang:1.17 AS go
ADD ./serve.go /app/serve.go
WORKDIR /app
RUN go build serve.go
FROM alpine:3.14
COPY --from=go /app/serve /app/serve
COPY ./static /app/static
WORKDIR /app/static
CMD [&quot;/app/serve&quot;]
</code></pre>
<p>This Dockerfile</p>
<ol>
<li>Builds the Go program</li>
<li>Copies the binary into an Alpine container</li>
</ol>
<p>Pretty simple. Seems like it should work, right?</p>
<p>But when I try to run <code>/app/serve</code>, this happens:</p>
<pre><code>$ docker build .
$ docker run -it broken-container:latest /app/serve
standard_init_linux.go:228: exec user process caused: no such file or directory
</code></pre>
<p>But the file definitely does exist:</p>
<pre><code>$ docker run -it broken-container:latest ls -l /app/serve
-rwxr-xr-x 1 root root 6220237 Nov 16 13:27 /app/serve
</code></pre>
<p>So what&rsquo;s going on?</p>
<h3 id="idea-1-permissions" class="post-heading">
<a href="#idea-1-permissions">
idea 1: permissions
</a>
</h3>
<p>At first I thought &ldquo;hmm, maybe the permissions are wrong?&rdquo;. But this can&rsquo;t be the problem, because:</p>
<ul>
<li>permission problems don&rsquo;t result in a &ldquo;no such file or directory&rdquo; error</li>
<li>in any case when we ran <code>ls -l</code>, we saw that the file was executable</li>
</ul>
<p>(I&rsquo;m including this even though it&rsquo;s &ldquo;obviously&rdquo; wrong just because I have
a lot of wrong thoughts when debugging, it&rsquo;s part of the process :) )</p>
<h3 id="idea-2-strace" class="post-heading">
<a href="#idea-2-strace">
idea 2: strace
</a>
</h3>
<p>Then I decided to use strace, as always. Let&rsquo;s see what stracing <code>/app/serve/</code> looks like</p>
<pre><code>$ docker run -it broken-container:latest /bin/sh
$ /app/static # apk add strace
(apk output omitted)
$ /app/static # strace /app/serve
execve(&quot;/app/serve&quot;, [&quot;/app/serve&quot;], 0x7ffdd08edd50 /* 6 vars */) = -1 ENOENT (No such file or directory)
strace: exec: No such file or directory
+++ exited with 1 +++
</code></pre>
<p>This is not that helpful, it just says &ldquo;No such file or directory&rdquo; again. But
at least we know that the error is being thrown right away when we run the
<code>evecve</code> system call, so that&rsquo;s good.</p>
<p>Interestingly though, this is different from what happens when we try to strace a nonexistent binary:</p>
<pre><code>$ strace /app/asdf
strace: Can't stat '/app/asdf': No such file or directory
</code></pre>
<h3 id="idea-3-google-enoent-but-file-exists-execve" class="post-heading">
<a href="#idea-3-google-enoent-but-file-exists-execve">
idea 3: google &ldquo;enoent but file exists execve&rdquo;
</a>
</h3>
<p>I vaguely remembered that there was some reason you could get an <code>ENOENT</code> error
when executing a program even if the file did exist, so I googled it. This led me
to <a href="https://superuser.com/a/507031">this stack overflow answer</a></p>
<p>which said, very helpfully:</p>
<blockquote>
<p>When execve() returns the error ENOENT, it can mean more than one thing: <br></p>
<ol>
<li>the program doesn&rsquo;t exist; <br></li>
<li>the program itself exists, but it requires an &ldquo;interpreter&rdquo; that doesn&rsquo;t exist.</li>
</ol>
<p>ELF executables can request to be loaded by another program, in a way very similar to <code>#!/bin/something</code> in shell scripts.</p>
</blockquote>
<p>That answer says that we can find the interpreter with <code>readelf -l $PROGRAM | grep interpreter</code>. So let&rsquo;s do that!</p>
<h3 id="step-4-use-readelf" class="post-heading">
<a href="#step-4-use-readelf">
step 4: use <code>readelf</code>
</a>
</h3>
<p>I didn&rsquo;t have <code>readelf</code> installed in the container and I wasn&rsquo;t sure how to
install it, so I ran <code>mount</code> to get the path to the container&rsquo;s filesystem and
then ran <code>readelf</code> from the host using that overlay directory.</p>
<p>(as an aside: this is kind of a weird way to do this, but as a result of writing
a <a href="https://wizardzines.com/zines/containers">containers zine</a> I&rsquo;m used to doing
weird things with containers and I think doing weird things is fun, so this way just seemed fastest to me at the time. That trick
won&rsquo;t work if you&rsquo;re on a Mac though, it only works on Linux)</p>
<pre><code>$ mount | grep docker
overlay on /var/lib/docker/overlay2/1ed587b302af7d3182135d02257f261fd491b7acf4648736d4c72f8382ecba0d/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/326ILTM2UXMVY64V7JFPCSDSKG:/var/lib/docker/overlay2/l/MGGPR357UOZZWXH3SH2AYHJL3E:/var/lib/docker/overlay2/l/EEEKSBSQ6VHGJ77YF224TBVMNV:/var/lib/docker/overlay2/l/RVKU36SQ3PXEQAGBRKSQRZFDGY,upperdir=/var/lib/docker/overlay2/1ed587b302af7d3182135d02257f261fd491b7acf4648736d4c72f8382ecba0d/diff,workdir=/var/lib/docker/overlay2/1ed587b302af7d3182135d02257f261fd491b7acf4648736d4c72f8382ecba0d/work,index=off)
$ # (then I copy and paste the &quot;merged&quot; directory from the output)
$ readelf -l /var/lib/docker/overlay2/1ed587b302af7d3182135d02257f261fd491b7acf4648736d4c72f8382ecba0d/merged/app/serve | grep interp
[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
01 .interp
03 .text .plt .interp .note.go.buildid
</code></pre>
<p>Okay, so the interpreter is <code>/lib64/ld-linux-x86-64.so.2</code>.</p>
<p>And sure enough, that file doesn&rsquo;t exist inside our Alpine container</p>
<pre><code>$ docker run -it broken-container:latest ls /lib64/ld-linux-x86-64.so.2
</code></pre>
<h3 id="step-5-victory" class="post-heading">
<a href="#step-5-victory">
step 5: victory!
</a>
</h3>
<p>Then I googled a little more and found out that there&rsquo;s a <code>golang:alpine</code>
container that&rsquo;s meant for doing Go builds targeted to be run in Alpine.</p>
<p>I switched to doing my build in the <code>golang:alpine</code> container and that fixed
everything.</p>
<h3 id="question-why-is-my-go-binary-dynamically-linked" class="post-heading">
<a href="#question-why-is-my-go-binary-dynamically-linked">
question: why is my Go binary dynamically linked?
</a>
</h3>
<p>The problem was with the program&rsquo;s interpreter. But I remembered that only
dynamically linked programs have interpreters, which is a bit weird &ndash; I
expected my Go binary to be statically linked! What&rsquo;s going on with that?</p>
<p>First, I double checked that the Go binary was actually dynamically linked using <code>file</code> and <code>ldd</code>: (<code>ldd</code> lists the dependencies of a dynamically linked executable! It&rsquo;s very useful!)</p>
<p>(I&rsquo;m using the docker overlay filesystem to get at the binary inside the container again)</p>
<pre><code>$ file /var/lib/docker/overlay2/1ed587b302af7d3182135d02257f261fd491b7acf4648736d4c72f8382ecba0d/merged/app/serve
/var/lib/docker/overlay2/1ed587b302af7d3182135d02257f261fd491b7acf4648736d4c72f8382ecba0d/merged/app/serve:
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked,
interpreter /lib64/ld-linux-x86-64.so.2, Go
BuildID=vd_DJvcyItRi4Q2RD0WL/z8P4ulttr6F6njfqx8CI/_odQWaUTR2e38bdHlD0-/ikjsOjlMbEOhj2qXv5AE,
not stripped
$ ldd /var/lib/docker/overlay2/1ed587b302af7d3182135d02257f261fd491b7acf4648736d4c72f8382ecba0d/merged/app/serve
linux-vdso.so.1 (0x00007ffe095a6000)
libpthread.so.0 =&gt; /usr/lib/libpthread.so.0 (0x00007f565a265000)
libc.so.6 =&gt; /usr/lib/libc.so.6 (0x00007f565a099000)
/lib64/ld-linux-x86-64.so.2 =&gt; /usr/lib64/ld-linux-x86-64.so.2 (0x00007f565a2b4000)
</code></pre>
<p>Now that I know it&rsquo;s dynamically linked, it&rsquo;s not that surprising that it
didn&rsquo;t work on a different system than it was compiled on.</p>
<p>Some Googling tells me that I can get Go to produce a statically linked binary by setting <code>CGO_ENABLED=0</code>. Let&rsquo;s see if that works.</p>
<pre><code>$ # first let's build it without that flag
$ go build serve.go
$ file ./serve
./serve: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=UGBmnMfFsuwMky4-k2Mt/RaNGsMI79eYC4-dcIiP4/J7v5rNGo3sNiJqdgNR12/eR_7mqqrsil_Lr6vt-rP, not stripped
$ ldd ./serve
linux-vdso.so.1 (0x00007fff679a6000)
libpthread.so.0 =&gt; /usr/lib/libpthread.so.0 (0x00007f659cb61000)
libc.so.6 =&gt; /usr/lib/libc.so.6 (0x00007f659c995000)
/lib64/ld-linux-x86-64.so.2 =&gt; /usr/lib64/ld-linux-x86-64.so.2 (0x00007f659cbb0000)
$ # and now with the CGO_ENABLED_0 flag
$ env CGO_ENABLED=0 go build serve.go
$ file ./serve
./serve: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=Kq392IB01ShfNVP5TugF/2q5hN74m5eLgfuzTZzR-/EatgRjlx5YYbpcroiE9q/0Fg3zUxJKY3lbsZ9Ufda, not stripped
$ ldd ./serve
not a dynamic executable
</code></pre>
<p>It works! I checked, and that&rsquo;s an alternative way to fix this bug &ndash; if I just set the
<code>CGO_ENABLED=0</code> environment variable in my build container, then I can build a
static binary and I don&rsquo;t need to switch to the <code>golang:alpine</code> container for my builds. I
kind of like that fix better.</p>
<p>And statically linking in this case doesn&rsquo;t even produce a bigger binary (for
some reason it seems to produce a slightly <em>smaller</em> binary?? I don&rsquo;t know why
that is)</p>
<p>I still don&rsquo;t understand <em>why</em> it&rsquo;s using cgo here, I ran <code>env | grep CGO</code> and I
definitely don&rsquo;t have <code>CGO_ENABLED=1</code> set in my environment, but I
don&rsquo;t feel like solving that mystery right now.</p>
<h3 id="that-was-a-fun-bug" class="post-heading">
<a href="#that-was-a-fun-bug">
that was a fun bug!
</a>
</h3>
<p>I thought this bug was a nice way to see how you can run into problems when
compiling a dynamically linked executable on one platform and running it on
another one! And to learn about the fact that ELF files have an interpreter!</p>
<p>I&rsquo;ve run into this &ldquo;file not found&rdquo; error a couple of times, and it feels kind
of mind bending because it initially seems impossible (BUT THE FILE IS THERE!!!
I SEE IT!!!). I hope this helps someone be less confused if you run into it!</p>
</main>
<footer>
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; display: inline;}
#mc_embed_signup {
display: inline;
}
#mc_embed_signup input.button {
background: #ff5e00;
display: inline;
color: white;
padding: 6px 12px;
}
</style>
<div class="sharing">
<style>
.form-inline {
display:flex; flex-flow: row wrap; justify-content: center;
}
.form-inline input, .form-inline span {
padding: 10px;
}
.form-inline input {
display:inline;
max-width:30%;
margin: 0 10px 0 0;
background-color: #fff;
border: 1px solid #ddd;
border-radius: 5px;
padding: 10px;
}
button {
background-color: #f50;
box-shadow: none;
border: 0;
border-radius: 5px;
color: white;
padding: 5px 10px;
}
@media (max-width: 800px) {
.form-inline input {
margin: 10px 0;
max-width:100% !important;
}
.form-inline {
flex-direction: column;
align-items: stretch;
}
}
</style>
<div align="center">
<form class="form-inline" action="https://app.convertkit.com/forms/1052396/subscriptions" method="post" data-uid="8884355abb" data-format="inline" data-version="5">
<span> Want a weekly digest of this blog?</span>
<input name="email_address" type="text" placeholder="Email address" />
<button type="submit" data-element="submit">Subscribe</button>
</form>
</div>
</div>
<p class="meta">
<a class="basic-alignment left" href="https://jvns.ca/blog/2021/11/15/esbuild-vue/" title="Previous Post: Some notes on using esbuild">Some notes on using esbuild</a>
<a class="basic-alignment right" href="https://jvns.ca/blog/2021/12/04/how-to-use-dig/" title="Next Post: How to use dig">How to use dig</a>
</p>
</footer>
</article>
</div>
</div>
</div>
<nav role="navigation" class="footer-nav"> <a href="/">Archives</a>
</nav>
<footer role="contentinfo"><span class="credit">&copy; Julia Evans. </span>
<span>If you like this, you may like <a href="https://web.archive.org/web/20181228051203/http://www.uliaea.ca/">Ulia Ea</a> or, more seriously, this list of <a href="https://jvns.ca/blogroll">blogs I love</a> or some <a href="https://jvns.ca/bookshelf">books I've read</a>. <br>
<p class="rc-scout__text"><i class="rc-scout__logo"></i>
You might also like the <a class="rc-scout__link" href="https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8">Recurse Center</a>, my very favorite programming community <a href="/categories/hackerschool/">(my posts about it)</a></p>
</span>
<style class="rc-scout__style" type="text/css">.rc-scout{display:block;padding:0;border:0;margin:0;}.rc-scout__text{display:block;padding:0;border:0;margin:0;height:100%;font-size:100%;}.rc-scout__logo{display:inline-block;padding:0;border:0;margin:0;width:0.85em;height:0.85em;background:no-repeat center url('data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2012%2015%22%3E%3Crect%20x%3D%220%22%20y%3D%220%22%20width%3D%2212%22%20height%3D%2210%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%221%22%20width%3D%2210%22%20height%3D%228%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%222%22%20width%3D%228%22%20height%3D%226%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%229%22%20width%3D%224%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%2211%22%20width%3D%2210%22%20height%3D%224%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%220%22%20y%3D%2212%22%20width%3D%2212%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%225%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%227%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%228%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%229%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3C%2Fsvg%3E');}.rc-scout__link:link,.rc-scout__link:visited{color:#61ae24;text-decoration:underline;}.rc-scout__link:hover,.rc-scout__link:active{color:#4e8b1d;}</style>
</footer>
<script type="text/rocketscript">
(function(){
var twitterWidgets = document.createElement('script');
twitterWidgets.type = 'text/javascript';
twitterWidgets.async = true;
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
})();
</script>
</div>
</body>
</html>