342 lines
21 KiB
HTML
342 lines
21 KiB
HTML
<html lang="en">
|
||
<!-- WARNING: Automatically generated file. Do not modify. Modify the blog directory entries instead. -->
|
||
<head>
|
||
<!-- vim: foldmethod=indent:spell:sw=2:smarttab
|
||
-->
|
||
<meta charSet="utf-8"/>
|
||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
|
||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
|
||
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>
|
||
<script async src="https://cse.google.com/cse.js?cx=001456668591254139637:te2uquaxidc"></script>
|
||
<script async defer crossorigin="anonymous" src="https://connect.facebook.net/en_GB/sdk.js#xfbml=1&version=v11.0" nonce="61tQy9ee"></script>
|
||
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.7.1/css/all.min.css">
|
||
<title>blog dds: 2021-11-27 — Rather than alchemy, methodical troubleshooting</title>
|
||
<meta charset="utf-8">
|
||
<meta name="Author" content="Diomidis Spinellis">
|
||
<link rel="stylesheet" href="https://www.spinellis.gr/blog/style.css" type="text/css" />
|
||
<link href="../../a/pandoc-syntax-2.css" rel="stylesheet" type="text/css" />
|
||
|
||
|
||
<link rel="ToC" href="https://www.spinellis.gr/blog/contents.html" />
|
||
<link rev="Subdocument" href="https://www.spinellis.gr/blog" />
|
||
<link rel="me" href="https://twitter.com/CoolSWEng" />
|
||
<link rel="previous" href="https://www.spinellis.gr/blog/20210618" /><link rel="next" href="https://www.spinellis.gr/blog/20220310" />
|
||
<meta name="Generator" content="blog.pl 97cfdb2 2025-12-24 23:50:45 +0200">
|
||
|
||
<link rel="shortcut icon" href="https://www.spinellis.gr/favicon.ico" />
|
||
</head>
|
||
<body>
|
||
|
||
<div class="container">
|
||
<div class="row"> <!-- Logo row -->
|
||
<div class='col-sm-12 dds-bg'>
|
||
<span class="global-title-spinellis"><a href="https://www.spinellis.gr/blog">Diomidis Spinellis blog</a></span>
|
||
<img src="https://www.spinellis.gr/images/dds-logo.png" class="pull-right dds-logo-size" alt="dds logo" />
|
||
</div>
|
||
</div>
|
||
<hr class='dds-hr' />
|
||
|
||
<div class="row"> <!-- Title row -->
|
||
<div class="col-sm-9 blogtext">
|
||
<h1>Rather than alchemy, methodical troubleshooting</h1>
|
||
</div>
|
||
|
||
<div class="col-sm-2">
|
||
<a href="https://www.spinellis.gr"><i class="fa fa-home fa-lg"></i></a>
|
||
<a href="https://bsky.app/profile/CoolSWEng.bsky.social"><i class="fa-brands fa-bluesky fa-lg" style="color: #1185FE"></i></a>
|
||
<a href="https://mastodon.acm.org/"><i class="fa-brands fa-mastodon fa-lg" style="color: #595aff"></i></a>
|
||
<a href="https://www.facebook.com/diomidis.spinellis"><i class="fa-brands fa-facebook fa-lg" style="color: #3b5998"></i></a>
|
||
<a href="https://github.com/dspinellis/"><i class="fa-brands fa-github fa-lg" style="color: #444444"></i></a>
|
||
<a href="https://www.youtube.com/user/dspinellis/"><i class="fa-brands fa-youtube fa-lg" style="color: #ff0000"></i></a>
|
||
<a href="https://www.linkedin.com/in/dspinellis"><i class="fa-brands fa-linkedin fa-lg" style="color: #007bb6"></i></a>
|
||
</div>
|
||
|
||
<div class="col-sm-1"> <!-- Search button -->
|
||
<div class="btn-group pull-right">
|
||
<button class="btn btn-secondary btn-sm" data-toggle="modal" data-target="#modalSearch">
|
||
<span class="glyphicon glyphicon-search"></span>
|
||
<span class="hidden-sm hidden-md hidden-lg">Search</span>
|
||
</button>
|
||
</div>
|
||
<div class="pull-right"> </div>
|
||
|
||
</div>
|
||
</div> <!-- Title row -->
|
||
|
||
<!-- Modal search window -->
|
||
<div id="modalSearch" class="modal fade" role="dialog">
|
||
<div class="modal-dialog">
|
||
<!-- Modal content-->
|
||
<div class="modal-content">
|
||
<div class="modal-header">
|
||
<h4 class="modal-title">Search the blog</h4>
|
||
</div>
|
||
<div class="modal-body">
|
||
<!-- See https://cse.google.com/cse/ -->
|
||
<div class='gcse-search'></div>
|
||
</div>
|
||
<div class="modal-footer">
|
||
<button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
|
||
</div>
|
||
</div> <!-- modal content -->
|
||
</div>
|
||
</div>
|
||
|
||
<div class="row"> <!-- Content / Contents row -->
|
||
<div class="col-sm-9 blogtext"> <!-- Left content -->
|
||
<p>I recently encountered a pesky problem while trying to
|
||
build a React Native project under Apple’s <em>Xcode</em>.
|
||
The build would fail with an error reporting:
|
||
<code>EMFILE: too many open files, watch</code>.
|
||
Frustratingly, all available advice on the web pointed to
|
||
different (often inexplicable) directions, none of which worked.
|
||
After tormenting myself with these, I decided to troubleshoot
|
||
the problem methodically, which allowed me to pinpoint it and
|
||
solve it with an uncommon and noteworthy application of the
|
||
<em>git bisect</em> command.
|
||
Here’s the story.</p>
|
||
<p>A Google search for
|
||
<a href="https://www.google.com/search?q=emfile+xcode">EMFILe xcode</a>
|
||
returns 49,000 results with many inexplicable and conflicting suggestions.
|
||
These <a href="https://github.com/facebook/watchman/issues/923">include the following</a>.</p>
|
||
<ul>
|
||
<li>Increasing the open files limit with <code>ulimit</code>, with <code>launchctl</code>,
|
||
by modifying <code>/etc/sysctl.conf</code>, or by creating a
|
||
<a href="https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c">plist file</a></li>
|
||
<li>Installing <em>watchman</em></li>
|
||
<li>Reinstalling <em>watchman</em></li>
|
||
<li>Upgrading <em>watchman</em></li>
|
||
<li>Installing watchman using <code>x86_64 brew</code></li>
|
||
<li>Reinstalling <em>brew</em></li>
|
||
<li>Running <em>watchman</em> under Rosetta 2</li>
|
||
<li>Not running <em>Xcode</em> under Rosetta</li>
|
||
<li>Clearing the build files</li>
|
||
<li>Removing and reinstalling the <code>node_modules</code> files (classic)</li>
|
||
<li>Removing the cached Cocoa Pods and reinstalling them,</li>
|
||
<li>Excluding the <code>arm64</code> architecture in the <em>Xcode</em> project’s build specifications</li>
|
||
<li>Using an earlier version of Node</li>
|
||
<li>Creating a symbolic link from <code>/usr/local/var/run/watchman</code></li>
|
||
</ul>
|
||
<p>Although advice from the web can often help us solve tough problems
|
||
in seconds, as the author of the book
|
||
<a href="https://www.spinellis.gr/debugging">Effective Debugging</a>,
|
||
I felt ashamed of wasting time by following increasingly nonsensical advice.
|
||
I therefore decided to troubleshoot the issue methodically, even though my
|
||
knowledge of <em>Xcode</em> and the macOS platform is very limited.
|
||
My reasoning was that the top-down troubleshooting advice I give in the book
|
||
can be applied to any problem.</p>
|
||
<p>My first step was to verify that the increased file limit I enforced
|
||
by creating a <code>/Library/LaunchDaemons/limit.maxfiles.plist</code> file
|
||
was visible to <em>Xcode</em>.
|
||
For this I created a one-line shell script file containing
|
||
<code>open -a Terminal</code>, and added this as a new <em>Xcode</em> “behavior”
|
||
(Xcode - Behaviors).
|
||
Running <code>ulimit -n</code> in the new terminal, showed me that <em>Xcode</em>’s
|
||
environment was indeed running with an increased limit on open files.
|
||
I further validated this by running a Perl command to create an
|
||
array of open files until it was no longer possible.</p>
|
||
<pre><code>perl -e 'for (;;) { open($a[++$i], "<", "/dev/null") || die "$!: $i"; }'
|
||
Too many open files in system: 61791 at -e line 1.</code></pre>
|
||
<p>Having excluded the possibility of a low open file limit, my second
|
||
step involved focusing on the failing command.
|
||
This was a React Native bundling step,
|
||
clearly visible on the <em>Xcode</em> build log.
|
||
I pasted the command on a terminal window, and, surprisingly, it worked
|
||
without a hitch.
|
||
I also pasted the command on the <em>Xcode</em> terminal window I had created,
|
||
and it worked there as well.
|
||
This demonstrated that the reason of the failure was the environment
|
||
in which the command was running.</p>
|
||
<p>My third step involved methodically determining how the command’s environment
|
||
was causing its failure.
|
||
The command was preceded by the setting of hundreds (488) of environment
|
||
variables, starting with <code>export ACTION=install</code> and ending with
|
||
<code>export variant=normal</code>.
|
||
Which of these was causing the command’s failure?</p>
|
||
<p>The failing command took 23 seconds to execute, so iteratively adding
|
||
each variable until the command would fail could take up to three hours.
|
||
Instead, I used the <code>git bisect</code> command to binary search for the offending
|
||
variable.
|
||
The <em>git bisect</em> command is normally used to find which of a series
|
||
of commits is responsible for a failure.</p>
|
||
<p>As I lacked such a series, I created a synthetic one for <em>git bisect</em>
|
||
to run on.
|
||
I started this process by exporting the <em>Xcode</em> log file, and editing
|
||
it into a shell script.
|
||
I then created a new Git branch in which I committed the script with all
|
||
variables commented out, e.g. <code>#export ACTION=install</code>.
|
||
This was the working commit.
|
||
I then run a shell loop to reintroduce each variable and commit each result,
|
||
thus creating a series of 488 commits (one for each introduced variable).</p>
|
||
<div class="sourceCode" id="cb2"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb2-1"><a href="https://www.spinellis.gr/blog/20211127/#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co"># Iterate while there are commented export lines</span></span>
|
||
<span id="cb2-2"><a href="https://www.spinellis.gr/blog/20211127/#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="cf">while</span> <span class="fu">grep</span> <span class="at">-q</span> <span class="st">'#export'</span> xcode-bundle <span class="kw">;</span> <span class="cf">do</span></span>
|
||
<span id="cb2-3"><a href="https://www.spinellis.gr/blog/20211127/#cb2-3" aria-hidden="true" tabindex="-1"></a> <span class="co"># Uncomment the first commented line</span></span>
|
||
<span id="cb2-4"><a href="https://www.spinellis.gr/blog/20211127/#cb2-4" aria-hidden="true" tabindex="-1"></a> <span class="fu">sed</span> <span class="at">-i</span> .bak <span class="st">'1,/#export/s/^#//'</span> xcode-bundle</span>
|
||
<span id="cb2-5"><a href="https://www.spinellis.gr/blog/20211127/#cb2-5" aria-hidden="true" tabindex="-1"></a> <span class="co"># Commit the result</span></span>
|
||
<span id="cb2-6"><a href="https://www.spinellis.gr/blog/20211127/#cb2-6" aria-hidden="true" tabindex="-1"></a> <span class="fu">git</span> commit <span class="at">-m</span> <span class="st">'Add export line'</span> xcode-bundle</span>
|
||
<span id="cb2-7"><a href="https://www.spinellis.gr/blog/20211127/#cb2-7" aria-hidden="true" tabindex="-1"></a><span class="cf">done</span></span></code></pre></div>
|
||
<p>(If you want to find more about creating such commands,
|
||
consider following my free edX MOOC
|
||
<a href="https://www.spinellis.gr/unix?bl202101127">Unix Tools: Data, Software and Production Engineering</a>.)
|
||
At the end of the loop the script was in its failing form.</p>
|
||
<p>I then provided to <em>git bisect</em> the first correct and the first
|
||
failing commit, and left it running to find
|
||
the one that caused the failure.</p>
|
||
<pre><code>$ git bisect start
|
||
$ git bisect bad
|
||
$ git bisect good 6fec32ad0627ca1890a234982f08eb128ba9039a
|
||
Bisecting: 243 revisions left to test after this (roughly 8 steps)
|
||
$ git bisect run xcode-bundle
|
||
running xcode-bundle
|
||
[...]
|
||
Bisecting: 243 revisions left to test after this (roughly 8 steps)
|
||
Bisecting: 121 revisions left to test after this (roughly 7 steps)
|
||
Bisecting: 60 revisions left to test after this (roughly 6 steps)
|
||
Bisecting: 30 revisions left to test after this (roughly 5 steps)
|
||
Bisecting: 15 revisions left to test after this (roughly 4 steps)
|
||
Bisecting: 7 revisions left to test after this (roughly 3 steps)
|
||
Bisecting: 3 revisions left to test after this (roughly 2 steps)
|
||
Bisecting: 1 revision left to test after this (roughly 1 step)
|
||
Bisecting: 0 revisions left to test after this (roughly 0 steps)
|
||
|
||
879f283a7ea7cf92d9720c107aa34b90ee8f6451 is the first bad commit
|
||
bisect run success</code></pre>
|
||
<p>A look at the offending bad commit, showed me that the problematic
|
||
statement set the <code>PATH</code> environment variable.
|
||
Running the previously working bundling command with the set <code>PATH</code>
|
||
caused it to fail, validating the finding.
|
||
Thanks to the binary search strategy of <em>git bisect</em>
|
||
Git pinpointed the offending commit in just 9 rather than 488 steps,
|
||
which took less than four minutes rather than hours.</p>
|
||
<p>I examined the two <code>PATH</code> variables and saw that, among other differences,
|
||
the faulty <code>PATH</code> lacked the <em>brew</em> path <code>/opt/homebrew/bin</code>.
|
||
Adding the <em>brew</em> path to the faulty <code>PATH</code> variable fixed the
|
||
problem in the terminal window.</p>
|
||
<p>The last step involved fixing the issue in <em>Xcode</em>.
|
||
For that I modified the shell script associated with the
|
||
“Bundle React Native code and images” project’s build phase, adding
|
||
a line to fix the path: <code>export PATH="$PATH:/opt/homebrew/bin"</code>.</p>
|
||
<p>After wasting many hours with web-advice alchemy, I was able
|
||
to fix the problem in less than an hour of methodical troubleshooting.
|
||
Lesson learned!</p>
|
||
|
||
<p />
|
||
<!-- COMMENTS -->
|
||
<span><a class="btn btn-primary" href="https://www.spinellis.gr/cgi-bin/comment.pl?date=20211127#comments">Comments</a></span>
|
||
|
||
<!-- Bluesky post -->
|
||
|
||
<a class="btn btn-primary"
|
||
href="https://bsky.app/intent/compose?text=Rather%20than%20alchemy%2C%20methodical%20troubleshooting%20https%3A%2F%2Fwww.spinellis.gr%2Fblog%2F20211127%20via%20%40CoolSWEng.bsky.social" target="_blank" style="background-color: #1185FE; color: white;"><i class="fa-brands fa-bluesky" style="margin-right: 6px;"></i>Post</a>
|
||
|
||
<!-- Mastodon post -->
|
||
<a class="btn btn-primary" href="https://toot.kytta.dev/?text=Rather%20than%20alchemy%2C%20methodical%20troubleshooting%20https%3A%2F%2Fwww.spinellis.gr%2Fblog%2F20211127%20via%20%40CoolSWEng%40mastodon.acm.org" target="_blank" style="background-color: #595aff"><i class="fa-brands fa-mastodon" style="margin-right: 6px;"></i> Toot!</a>
|
||
<!-- Twitter button -->
|
||
<span class="btn"><a href="https://twitter.com/share?ref_src=twsrc%5Etfw" class="twitter-share-button" data-show-count="false" data-size="large">Tweet</a><script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></span>
|
||
|
||
<!-- Facebook button -->
|
||
<span class="btn fb-share-button" data-size="large" data-href="https://www.spinellis.gr/blog/20211127/" data-layout="button" data-size="small"><a target="_blank" href="https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.spinellis.gr%2Fblog%2F20211127%2F&src=sdkpreparse" class="fb-xfbml-parse-ignore">Share</a></span>
|
||
<p />
|
||
|
||
</div> <!-- Left content -->
|
||
|
||
<div class="col-sm-3"> <!-- Right content -->
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Navigation</div>
|
||
<div class="panel-body">
|
||
|
||
<a href="https://www.spinellis.gr/blog/contents.html">blog contents</a> <br />
|
||
<a href="https://www.spinellis.gr/blog/index.html">dds blog</a> <br />
|
||
<a href="https://www.spinellis.gr/">dds home</a> <br />
|
||
<a href="https://www.spinellis.gr/cgi-bin/comment.pl?date=20211127#comments">comments</a><br />
|
||
<a href="https://www.spinellis.gr/blog/20210618/index.html">« The Evolution of the Unix System Architecture</a> <br />
|
||
<a href="https://www.spinellis.gr/blog/20220310/index.html">» The sorry state of software quality
|
||
</a> <br />
|
||
</div> <!-- panel body -->
|
||
</div> <!-- panel -->
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Tagged as</div>
|
||
<div class="panel-body">
|
||
<a href="https://www.spinellis.gr/blog/Debugging.html" rel="tag">Debugging</a><br /> <a href="https://www.spinellis.gr/blog/Apple.html" rel="tag">Apple</a><br /> <a href="https://www.spinellis.gr/blog/JavaScript.html" rel="tag">JavaScript</a><br /> <a href="https://www.spinellis.gr/blog/Git.html" rel="tag">Git</a><br /> <a href="https://www.spinellis.gr/blog/Unix.html" rel="tag">Unix</a><br /> </div> <!-- panel body -->
|
||
</div> <!-- panel -->
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Become a Unix command line wizard</div>
|
||
<div class="panel-body">
|
||
<a href="https://www.spinellis.gr/unix?source=blog-banner"><img src="/unix/blog-banner.png" class="img-responsive img-rounded" border="0" alt="edX MOOC on Unix Tools: Data, Software, and Production Engineering" /></a>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Debug like a master</div>
|
||
<div class="panel-body">
|
||
<a href="https://www.spinellis.gr/debugging/"><img src="/debugging/img/book/cover.jpg" class="img-responsive" alt="Book cover of Effective Debugging" /></a>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Compute with style</div>
|
||
<div class="panel-body">
|
||
<a href="https://www.spinellis.gr/computingstyle"><img src="/computingstyle/img/book/cover.jpg" class="img-responsive img-rounded" alt="Book cover of The Elements of Computing Style" /></a>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Syndication</div>
|
||
<div class="panel-body">
|
||
|
||
This blog is also available as an RSS feed: <a href="https://www.spinellis.gr/blog/dds-blog-rss.xml"><i class="fas fa-rss"></i></a><p />
|
||
</div> <!-- panel body -->
|
||
</div> <!-- panel -->
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Recent posts</div>
|
||
<div class="panel-body">
|
||
<a href="https://www.spinellis.gr/blog/20251223/index.html">An initial analysis of the discovered Unix V4 tape
|
||
</a> (2025-12-23)<br />
|
||
<a href="https://www.spinellis.gr/blog/20250926/index.html">Why I Choose Email Over Messaging
|
||
</a> (2025-09-26)<br />
|
||
<a href="https://www.spinellis.gr/blog/20250626/index.html">Is it legal to use copyrighted works to train LLMs?
|
||
</a> (2025-06-26)<br />
|
||
<a href="https://www.spinellis.gr/blog/20250520/index.html">I'm removing the BSD advertising clause
|
||
</a> (2025-05-20)<br />
|
||
<a href="https://www.spinellis.gr/blog/20250411/index.html">The perils of GenAI student submissions
|
||
</a> (2025-04-11)<br />
|
||
<a href="https://www.spinellis.gr/blog/20241015/index.html">Unix make vs Apache Airflow
|
||
</a> (2024-10-15)<br />
|
||
<a href="https://www.spinellis.gr/blog/20240805/index.html">How (and how not) to present related work
|
||
</a> (2024-08-05)<br />
|
||
<a href="https://www.spinellis.gr/blog/20240205/index.html">An exception handling revelation
|
||
</a> (2024-02-05)<br />
|
||
<a href="https://www.spinellis.gr/blog/20230901/index.html">Extending the life of TomTom wearables
|
||
</a> (2023-09-01)<br />
|
||
<a href="https://www.spinellis.gr/blog/20230413/index.html">How AGI can conquer the world and what to do about it
|
||
</a> (2023-04-13)<br />
|
||
</div> <!-- panel body -->
|
||
</div> <!-- panel -->
|
||
|
||
</div> <!-- Right content -->
|
||
</div>
|
||
|
||
<div class='row'> <!-- HR -->
|
||
<div class='span12'>
|
||
<hr />
|
||
</div>
|
||
</div>
|
||
<div class='row'> <!-- Modification time -->
|
||
<div class='col-sm-10'>
|
||
<p class="small">
|
||
Last modified: Saturday, November 27, 2021 8:21 pm
|
||
</p>
|
||
</div>
|
||
</div>
|
||
<div class='row'> <!-- License BEGIN -->
|
||
|
||
<div class='col-sm-2'>
|
||
<!--Creative Commons License logo-->
|
||
<a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/"><img alt="Creative Commons Licence BY NC" style="border-width:0;height:5ex" src="/a/by-nc.eu.png" /></a>
|
||
</div>
|
||
<div class='col-sm-10'>
|
||
<p class="small">
|
||
Unless otherwise expressly stated, all original material on this page created by Diomidis Spinellis is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">Creative Commons Attribution-NonCommercial 4.0 International License</a>.
|
||
</p>
|
||
<br />
|
||
</div>
|
||
|
||
</div> <!-- License END -->
|
||
</div> <!-- container -->
|
||
</body>
|
||
</html>
|