Files
nexus/sreweekly/articles/299/09-blog-dds-2021-11-27-rather-than-alchemy-methodical-troubleshooting.html
2026-09-12 17:23:01 +08:00

342 lines
21 KiB
HTML
Raw Permalink Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<html lang="en">
<!-- WARNING: Automatically generated file. Do not modify. Modify the blog directory entries instead. -->
<head>
<!-- vim: foldmethod=indent:spell:sw=2:smarttab
-->
<meta charSet="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>
<script async src="https://cse.google.com/cse.js?cx=001456668591254139637:te2uquaxidc"></script>
<script async defer crossorigin="anonymous" src="https://connect.facebook.net/en_GB/sdk.js#xfbml=1&version=v11.0" nonce="61tQy9ee"></script>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.7.1/css/all.min.css">
<title>blog dds: 2021-11-27 — Rather than alchemy, methodical troubleshooting</title>
<meta charset="utf-8">
<meta name="Author" content="Diomidis Spinellis">
<link rel="stylesheet" href="https://www.spinellis.gr/blog/style.css" type="text/css" />
<link href="../../a/pandoc-syntax-2.css" rel="stylesheet" type="text/css" />
<link rel="ToC" href="https://www.spinellis.gr/blog/contents.html" />
<link rev="Subdocument" href="https://www.spinellis.gr/blog" />
<link rel="me" href="https://twitter.com/CoolSWEng" />
<link rel="previous" href="https://www.spinellis.gr/blog/20210618" /><link rel="next" href="https://www.spinellis.gr/blog/20220310" />
<meta name="Generator" content="blog.pl 97cfdb2 2025-12-24 23:50:45 +0200">
<link rel="shortcut icon" href="https://www.spinellis.gr/favicon.ico" />
</head>
<body>
<div class="container">
<div class="row"> <!-- Logo row -->
<div class='col-sm-12 dds-bg'>
<span class="global-title-spinellis"><a href="https://www.spinellis.gr/blog">Diomidis Spinellis blog</a></span>
<img src="https://www.spinellis.gr/images/dds-logo.png" class="pull-right dds-logo-size" alt="dds logo" />
</div>
</div>
<hr class='dds-hr' />
<div class="row"> <!-- Title row -->
<div class="col-sm-9 blogtext">
<h1>Rather than alchemy, methodical troubleshooting</h1>
</div>
<div class="col-sm-2">
<a href="https://www.spinellis.gr"><i class="fa fa-home fa-lg"></i></a>
<a href="https://bsky.app/profile/CoolSWEng.bsky.social"><i class="fa-brands fa-bluesky fa-lg" style="color: #1185FE"></i></a>
<a href="https://mastodon.acm.org/"><i class="fa-brands fa-mastodon fa-lg" style="color: #595aff"></i></a>
<a href="https://www.facebook.com/diomidis.spinellis"><i class="fa-brands fa-facebook fa-lg" style="color: #3b5998"></i></a>
<a href="https://github.com/dspinellis/"><i class="fa-brands fa-github fa-lg" style="color: #444444"></i></a>
<a href="https://www.youtube.com/user/dspinellis/"><i class="fa-brands fa-youtube fa-lg" style="color: #ff0000"></i></a>
<a href="https://www.linkedin.com/in/dspinellis"><i class="fa-brands fa-linkedin fa-lg" style="color: #007bb6"></i></a>
</div>
<div class="col-sm-1"> <!-- Search button -->
<div class="btn-group pull-right">
<button class="btn btn-secondary btn-sm" data-toggle="modal" data-target="#modalSearch">
<span class="glyphicon glyphicon-search"></span>
<span class="hidden-sm hidden-md hidden-lg">Search</span>
</button>
</div>
<div class="pull-right">&nbsp;</div>
</div>
</div> <!-- Title row -->
<!-- Modal search window -->
<div id="modalSearch" class="modal fade" role="dialog">
<div class="modal-dialog">
<!-- Modal content-->
<div class="modal-content">
<div class="modal-header">
<h4 class="modal-title">Search the blog</h4>
</div>
<div class="modal-body">
<!-- See https://cse.google.com/cse/ -->
<div class='gcse-search'></div>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
</div>
</div> <!-- modal content -->
</div>
</div>
<div class="row"> <!-- Content / Contents row -->
<div class="col-sm-9 blogtext"> <!-- Left content -->
<p>I recently encountered a pesky problem while trying to
build a React Native project under Apple’s <em>Xcode</em>.
The build would fail with an error reporting:
<code>EMFILE: too many open files, watch</code>.
Frustratingly, all available advice on the web pointed to
different (often inexplicable) directions, none of which worked.
After tormenting myself with these, I decided to troubleshoot
the problem methodically, which allowed me to pinpoint it and
solve it with an uncommon and noteworthy application of the
<em>git bisect</em> command.
Here’s the story.</p>
<p>A Google search for
<a href="https://www.google.com/search?q=emfile+xcode">EMFILe xcode</a>
returns 49,000 results with many inexplicable and conflicting suggestions.
These <a href="https://github.com/facebook/watchman/issues/923">include the following</a>.</p>
<ul>
<li>Increasing the open files limit with <code>ulimit</code>, with <code>launchctl</code>,
by modifying <code>/etc/sysctl.conf</code>, or by creating a
<a href="https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c">plist file</a></li>
<li>Installing <em>watchman</em></li>
<li>Reinstalling <em>watchman</em></li>
<li>Upgrading <em>watchman</em></li>
<li>Installing watchman using <code>x86_64 brew</code></li>
<li>Reinstalling <em>brew</em></li>
<li>Running <em>watchman</em> under Rosetta 2</li>
<li>Not running <em>Xcode</em> under Rosetta</li>
<li>Clearing the build files</li>
<li>Removing and reinstalling the <code>node_modules</code> files (classic)</li>
<li>Removing the cached Cocoa Pods and reinstalling them,</li>
<li>Excluding the <code>arm64</code> architecture in the <em>Xcode</em> project’s build specifications</li>
<li>Using an earlier version of Node</li>
<li>Creating a symbolic link from <code>/usr/local/var/run/watchman</code></li>
</ul>
<p>Although advice from the web can often help us solve tough problems
in seconds, as the author of the book
<a href="https://www.spinellis.gr/debugging">Effective Debugging</a>,
I felt ashamed of wasting time by following increasingly nonsensical advice.
I therefore decided to troubleshoot the issue methodically, even though my
knowledge of <em>Xcode</em> and the macOS platform is very limited.
My reasoning was that the top-down troubleshooting advice I give in the book
can be applied to any problem.</p>
<p>My first step was to verify that the increased file limit I enforced
by creating a <code>/Library/LaunchDaemons/limit.maxfiles.plist</code> file
was visible to <em>Xcode</em>.
For this I created a one-line shell script file containing
<code>open -a Terminal</code>, and added this as a new <em>Xcode</em> “behavior”
(Xcode - Behaviors).
Running <code>ulimit -n</code> in the new terminal, showed me that <em>Xcode</em>’s
environment was indeed running with an increased limit on open files.
I further validated this by running a Perl command to create an
array of open files until it was no longer possible.</p>
<pre><code>perl -e &#39;for (;;) { open($a[++$i], &quot;&lt;&quot;, &quot;/dev/null&quot;) || die &quot;$!: $i&quot;; }&#39;
Too many open files in system: 61791 at -e line 1.</code></pre>
<p>Having excluded the possibility of a low open file limit, my second
step involved focusing on the failing command.
This was a React Native bundling step,
clearly visible on the <em>Xcode</em> build log.
I pasted the command on a terminal window, and, surprisingly, it worked
without a hitch.
I also pasted the command on the <em>Xcode</em> terminal window I had created,
and it worked there as well.
This demonstrated that the reason of the failure was the environment
in which the command was running.</p>
<p>My third step involved methodically determining how the command’s environment
was causing its failure.
The command was preceded by the setting of hundreds (488) of environment
variables, starting with <code>export ACTION=install</code> and ending with
<code>export variant=normal</code>.
Which of these was causing the command’s failure?</p>
<p>The failing command took 23 seconds to execute, so iteratively adding
each variable until the command would fail could take up to three hours.
Instead, I used the <code>git bisect</code> command to binary search for the offending
variable.
The <em>git bisect</em> command is normally used to find which of a series
of commits is responsible for a failure.</p>
<p>As I lacked such a series, I created a synthetic one for <em>git bisect</em>
to run on.
I started this process by exporting the <em>Xcode</em> log file, and editing
it into a shell script.
I then created a new Git branch in which I committed the script with all
variables commented out, e.g. <code>#export ACTION=install</code>.
This was the working commit.
I then run a shell loop to reintroduce each variable and commit each result,
thus creating a series of 488 commits (one for each introduced variable).</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb2-1"><a href="https://www.spinellis.gr/blog/20211127/#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co"># Iterate while there are commented export lines</span></span>
<span id="cb2-2"><a href="https://www.spinellis.gr/blog/20211127/#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="cf">while</span> <span class="fu">grep</span> <span class="at">-q</span> <span class="st">&#39;#export&#39;</span> xcode-bundle <span class="kw">;</span> <span class="cf">do</span></span>
<span id="cb2-3"><a href="https://www.spinellis.gr/blog/20211127/#cb2-3" aria-hidden="true" tabindex="-1"></a> <span class="co"># Uncomment the first commented line</span></span>
<span id="cb2-4"><a href="https://www.spinellis.gr/blog/20211127/#cb2-4" aria-hidden="true" tabindex="-1"></a> <span class="fu">sed</span> <span class="at">-i</span> .bak <span class="st">&#39;1,/#export/s/^#//&#39;</span> xcode-bundle</span>
<span id="cb2-5"><a href="https://www.spinellis.gr/blog/20211127/#cb2-5" aria-hidden="true" tabindex="-1"></a> <span class="co"># Commit the result</span></span>
<span id="cb2-6"><a href="https://www.spinellis.gr/blog/20211127/#cb2-6" aria-hidden="true" tabindex="-1"></a> <span class="fu">git</span> commit <span class="at">-m</span> <span class="st">&#39;Add export line&#39;</span> xcode-bundle</span>
<span id="cb2-7"><a href="https://www.spinellis.gr/blog/20211127/#cb2-7" aria-hidden="true" tabindex="-1"></a><span class="cf">done</span></span></code></pre></div>
<p>(If you want to find more about creating such commands,
consider following my free edX MOOC
<a href="https://www.spinellis.gr/unix?bl202101127">Unix Tools: Data, Software and Production Engineering</a>.)
At the end of the loop the script was in its failing form.</p>
<p>I then provided to <em>git bisect</em> the first correct and the first
failing commit, and left it running to find
the one that caused the failure.</p>
<pre><code>$ git bisect start
$ git bisect bad
$ git bisect good 6fec32ad0627ca1890a234982f08eb128ba9039a
Bisecting: 243 revisions left to test after this (roughly 8 steps)
$ git bisect run xcode-bundle
running xcode-bundle
[...]
Bisecting: 243 revisions left to test after this (roughly 8 steps)
Bisecting: 121 revisions left to test after this (roughly 7 steps)
Bisecting: 60 revisions left to test after this (roughly 6 steps)
Bisecting: 30 revisions left to test after this (roughly 5 steps)
Bisecting: 15 revisions left to test after this (roughly 4 steps)
Bisecting: 7 revisions left to test after this (roughly 3 steps)
Bisecting: 3 revisions left to test after this (roughly 2 steps)
Bisecting: 1 revision left to test after this (roughly 1 step)
Bisecting: 0 revisions left to test after this (roughly 0 steps)
879f283a7ea7cf92d9720c107aa34b90ee8f6451 is the first bad commit
bisect run success</code></pre>
<p>A look at the offending bad commit, showed me that the problematic
statement set the <code>PATH</code> environment variable.
Running the previously working bundling command with the set <code>PATH</code>
caused it to fail, validating the finding.
Thanks to the binary search strategy of <em>git bisect</em>
Git pinpointed the offending commit in just 9 rather than 488 steps,
which took less than four minutes rather than hours.</p>
<p>I examined the two <code>PATH</code> variables and saw that, among other differences,
the faulty <code>PATH</code> lacked the <em>brew</em> path <code>/opt/homebrew/bin</code>.
Adding the <em>brew</em> path to the faulty <code>PATH</code> variable fixed the
problem in the terminal window.</p>
<p>The last step involved fixing the issue in <em>Xcode</em>.
For that I modified the shell script associated with the
“Bundle React Native code and images” project’s build phase, adding
a line to fix the path: <code>export PATH="$PATH:/opt/homebrew/bin"</code>.</p>
<p>After wasting many hours with web-advice alchemy, I was able
to fix the problem in less than an hour of methodical troubleshooting.
Lesson learned!</p>
<p />
<!-- COMMENTS -->
<span><a class="btn btn-primary" href="https://www.spinellis.gr/cgi-bin/comment.pl?date=20211127#comments">Comments</a></span>
&nbsp;
<!-- Bluesky post -->
<a class="btn btn-primary"
href="https://bsky.app/intent/compose?text=Rather%20than%20alchemy%2C%20methodical%20troubleshooting%20https%3A%2F%2Fwww.spinellis.gr%2Fblog%2F20211127%20via%20%40CoolSWEng.bsky.social" target="_blank" style="background-color: #1185FE; color: white;"><i class="fa-brands fa-bluesky" style="margin-right: 6px;"></i>Post</a>
<!-- Mastodon post -->
<a class="btn btn-primary" href="https://toot.kytta.dev/?text=Rather%20than%20alchemy%2C%20methodical%20troubleshooting%20https%3A%2F%2Fwww.spinellis.gr%2Fblog%2F20211127%20via%20%40CoolSWEng%40mastodon.acm.org" target="_blank" style="background-color: #595aff"><i class="fa-brands fa-mastodon" style="margin-right: 6px;"></i> Toot!</a>
<!-- Twitter button -->
<span class="btn"><a href="https://twitter.com/share?ref_src=twsrc%5Etfw" class="twitter-share-button" data-show-count="false" data-size="large">Tweet</a><script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></span>
<!-- Facebook button -->
<span class="btn fb-share-button" data-size="large" data-href="https://www.spinellis.gr/blog/20211127/" data-layout="button" data-size="small"><a target="_blank" href="https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.spinellis.gr%2Fblog%2F20211127%2F&amp;src=sdkpreparse" class="fb-xfbml-parse-ignore">Share</a></span>
<p />
</div> <!-- Left content -->
<div class="col-sm-3"> <!-- Right content -->
<div class="panel panel-default">
<div class="panel-heading">Navigation</div>
<div class="panel-body">
<a href="https://www.spinellis.gr/blog/contents.html">blog contents</a> <br />
<a href="https://www.spinellis.gr/blog/index.html">dds blog</a> <br />
<a href="https://www.spinellis.gr/">dds home</a> <br />
<a href="https://www.spinellis.gr/cgi-bin/comment.pl?date=20211127#comments">comments</a><br />
<a href="https://www.spinellis.gr/blog/20210618/index.html">&laquo; The Evolution of the Unix System Architecture</a> <br />
<a href="https://www.spinellis.gr/blog/20220310/index.html">&raquo; The sorry state of software quality
</a> <br />
</div> <!-- panel body -->
</div> <!-- panel -->
<div class="panel panel-default">
<div class="panel-heading">Tagged as</div>
<div class="panel-body">
<a href="https://www.spinellis.gr/blog/Debugging.html" rel="tag">Debugging</a><br /> <a href="https://www.spinellis.gr/blog/Apple.html" rel="tag">Apple</a><br /> <a href="https://www.spinellis.gr/blog/JavaScript.html" rel="tag">JavaScript</a><br /> <a href="https://www.spinellis.gr/blog/Git.html" rel="tag">Git</a><br /> <a href="https://www.spinellis.gr/blog/Unix.html" rel="tag">Unix</a><br /> </div> <!-- panel body -->
</div> <!-- panel -->
<div class="panel panel-default">
<div class="panel-heading">Become a Unix command line wizard</div>
<div class="panel-body">
<a href="https://www.spinellis.gr/unix?source=blog-banner"><img src="/unix/blog-banner.png" class="img-responsive img-rounded" border="0" alt="edX MOOC on Unix Tools: Data, Software, and Production Engineering" /></a>
</div>
</div>
<div class="panel panel-default">
<div class="panel-heading">Debug like a master</div>
<div class="panel-body">
<a href="https://www.spinellis.gr/debugging/"><img src="/debugging/img/book/cover.jpg" class="img-responsive" alt="Book cover of Effective Debugging" /></a>
</div>
</div>
<div class="panel panel-default">
<div class="panel-heading">Compute with style</div>
<div class="panel-body">
<a href="https://www.spinellis.gr/computingstyle"><img src="/computingstyle/img/book/cover.jpg" class="img-responsive img-rounded" alt="Book cover of The Elements of Computing Style" /></a>
</div>
</div>
<div class="panel panel-default">
<div class="panel-heading">Syndication</div>
<div class="panel-body">
This blog is also available as an RSS feed: <a href="https://www.spinellis.gr/blog/dds-blog-rss.xml"><i class="fas fa-rss"></i></a><p />
</div> <!-- panel body -->
</div> <!-- panel -->
<div class="panel panel-default">
<div class="panel-heading">Recent posts</div>
<div class="panel-body">
<a href="https://www.spinellis.gr/blog/20251223/index.html">An initial analysis of the discovered Unix V4 tape
</a> (2025-12-23)<br />
<a href="https://www.spinellis.gr/blog/20250926/index.html">Why I Choose Email Over Messaging
</a> (2025-09-26)<br />
<a href="https://www.spinellis.gr/blog/20250626/index.html">Is it legal to use copyrighted works to train LLMs?
</a> (2025-06-26)<br />
<a href="https://www.spinellis.gr/blog/20250520/index.html">I'm removing the BSD advertising clause
</a> (2025-05-20)<br />
<a href="https://www.spinellis.gr/blog/20250411/index.html">The perils of GenAI student submissions
</a> (2025-04-11)<br />
<a href="https://www.spinellis.gr/blog/20241015/index.html">Unix make vs Apache Airflow
</a> (2024-10-15)<br />
<a href="https://www.spinellis.gr/blog/20240805/index.html">How (and how not) to present related work
</a> (2024-08-05)<br />
<a href="https://www.spinellis.gr/blog/20240205/index.html">An exception handling revelation
</a> (2024-02-05)<br />
<a href="https://www.spinellis.gr/blog/20230901/index.html">Extending the life of TomTom wearables
</a> (2023-09-01)<br />
<a href="https://www.spinellis.gr/blog/20230413/index.html">How AGI can conquer the world and what to do about it
</a> (2023-04-13)<br />
</div> <!-- panel body -->
</div> <!-- panel -->
</div> <!-- Right content -->
</div>
<div class='row'> <!-- HR -->
<div class='span12'>
<hr />
</div>
</div>
<div class='row'> <!-- Modification time -->
<div class='col-sm-10'>
<p class="small">
Last modified: Saturday, November 27, 2021 8:21 pm
</p>
</div>
</div>
<div class='row'> <!-- License BEGIN -->
<div class='col-sm-2'>
<!--Creative Commons License logo-->
<a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/"><img alt="Creative Commons Licence BY NC" style="border-width:0;height:5ex" src="/a/by-nc.eu.png" /></a>
</div>
<div class='col-sm-10'>
<p class="small">
Unless otherwise expressly stated, all original material on this page created by Diomidis Spinellis is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">Creative Commons Attribution-NonCommercial 4.0 International License</a>.
</p>
<br />
</div>
</div> <!-- License END -->
</div> <!-- container -->
</body>
</html>