917 lines
92 KiB
HTML
917 lines
92 KiB
HTML
<!--
|
|
This file has been auto-generated by main.rs from a markdown file of the same name.
|
|
Do not edit it by hand.
|
|
-->
|
|
<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<title>Way too many ways to wait on a child process with a timeout</title>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
<link type="application/atom+xml" href="/blog/feed.xml" rel="self">
|
|
<link rel="shortcut icon" type="image/ico" href="/blog/favicon.ico">
|
|
<link rel="stylesheet" type="text/css" href="main.css">
|
|
<link rel="stylesheet" href="https://unpkg.com/@highlightjs/cdn-assets@11.8.0/styles/default.min.css">
|
|
<script type="module" src="main.js" async></script>
|
|
<script type="module" src="search.js" async></script>
|
|
</head>
|
|
<body>
|
|
|
|
<div id="banner">
|
|
<div id="name">
|
|
<img id="me" src="me.jpeg">
|
|
<span>Philippe Gaultier</span>
|
|
</div>
|
|
<input id="search" placeholder="🔎 Search" autocomplete=off>
|
|
<ul>
|
|
<li> <button id="dark-light-mode">Dark/Light</button> </li>
|
|
<li> <a href="/blog/body_of_work.html">Body of work</a> </li>
|
|
<li> <a href="/blog/articles-by-tag.html">Tags</a> </li>
|
|
<li> <a href="https://github.com/gaultier/resume/raw/master/Philippe_Gaultier_resume_en.pdf">
|
|
Resume
|
|
</a> </li>
|
|
|
|
<li> <a href="/blog/feed.xml">
|
|
<svg viewBox="0 0 24 24" fill="currentColor" xmlns="http://www.w3.org/2000/svg">
|
|
<path fill-rule="evenodd" clip-rule="evenodd" d="M5.5 3.5C4.39543 3.5 3.5 4.39543 3.5 5.5V18.5C3.5 19.6046 4.39543 20.5 5.5 20.5H18.5C19.6046 20.5 20.5 19.6046 20.5 18.5V5.5C20.5 4.39543 19.6046 3.5 18.5 3.5H5.5ZM7 19C8.10457 19 9 18.1046 9 17C9 15.8954 8.10457 15 7 15C5.89543 15 5 15.8954 5 17C5 18.1046 5.89543 19 7 19ZM6.14863 10.5052C6.14863 10.0379 6.52746 9.65906 6.99478 9.65906C7.95949 9.65906 8.91476 9.84908 9.80603 10.2183C10.6973 10.5874 11.5071 11.1285 12.1893 11.8107C12.8715 12.4929 13.4126 13.3027 13.7817 14.194C14.1509 15.0852 14.3409 16.0405 14.3409 17.0052C14.3409 17.4725 13.9621 17.8514 13.4948 17.8514C13.0275 17.8514 12.6486 17.4725 12.6486 17.0052C12.6486 16.2627 12.5024 15.5275 12.2183 14.8416C11.9341 14.1556 11.5177 13.5324 10.9927 13.0073C10.4676 12.4823 9.84437 12.0659 9.15842 11.7817C8.47246 11.4976 7.73726 11.3514 6.99478 11.3514C6.52746 11.3514 6.14863 10.9725 6.14863 10.5052ZM7 5.15385C6.53268 5.15385 6.15385 5.53268 6.15385 6C6.15385 6.46732 6.53268 6.84615 7 6.84615C8.33342 6.84615 9.65379 7.10879 10.8857 7.61907C12.1176 8.12935 13.237 8.87728 14.1799 9.82015C15.1227 10.763 15.8707 11.8824 16.3809 13.1143C16.8912 14.3462 17.1538 15.6666 17.1538 17C17.1538 17.4673 17.5327 17.8462 18 17.8462C18.4673 17.8462 18.8462 17.4673 18.8462 17C18.8462 15.4443 18.5397 13.9039 17.9444 12.4667C17.3491 11.0294 16.4765 9.72352 15.3765 8.6235C14.2765 7.52349 12.9706 6.65091 11.5333 6.05558C10.0961 5.46026 8.55566 5.15385 7 5.15385Z" fill="currentColor"/>
|
|
</svg>
|
|
</a> </li>
|
|
|
|
<li> <a href="https://www.linkedin.com/in/philippegaultier/">
|
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" data-supported-dps="24x24" fill="currentColor" width="24" height="24" focusable="false">
|
|
<path d="M20.5 2h-17A1.5 1.5 0 002 3.5v17A1.5 1.5 0 003.5 22h17a1.5 1.5 0 001.5-1.5v-17A1.5 1.5 0 0020.5 2zM8 19H5v-9h3zM6.5 8.25A1.75 1.75 0 118.3 6.5a1.78 1.78 0 01-1.8 1.75zM19 19h-3v-4.74c0-1.42-.6-1.93-1.38-1.93A1.74 1.74 0 0013 14.19a.66.66 0 000 .14V19h-3v-9h2.9v1.3a3.11 3.11 0 012.7-1.4c1.55 0 3.36.86 3.36 3.66z"/>
|
|
</svg>
|
|
</a> </li>
|
|
<li> <a href="https://github.com/gaultier">
|
|
<svg height="32" aria-hidden="true" viewBox="0 0 24 24" version="1.1" width="32" data-view-component="true" fill="currentColor">
|
|
<path d="M12.5.75C6.146.75 1 5.896 1 12.25c0 5.089 3.292 9.387 7.863 10.91.575.101.79-.244.79-.546 0-.273-.014-1.178-.014-2.142-2.889.532-3.636-.704-3.866-1.35-.13-.331-.69-1.352-1.18-1.625-.402-.216-.977-.748-.014-.762.906-.014 1.553.834 1.769 1.179 1.035 1.74 2.688 1.25 3.349.948.1-.747.402-1.25.733-1.538-2.559-.287-5.232-1.279-5.232-5.678 0-1.25.445-2.285 1.178-3.09-.115-.288-.517-1.467.115-3.048 0 0 .963-.302 3.163 1.179.92-.259 1.897-.388 2.875-.388.977 0 1.955.13 2.875.388 2.2-1.495 3.162-1.179 3.162-1.179.633 1.581.23 2.76.115 3.048.733.805 1.179 1.825 1.179 3.09 0 4.413-2.688 5.39-5.247 5.678.417.36.776 1.05.776 2.128 0 1.538-.014 2.774-.014 3.162 0 .302.216.662.79.547C20.709 21.637 24 17.324 24 12.25 24 5.896 18.854.75 12.5.75Z"/>
|
|
</svg>
|
|
</a> </li>
|
|
<li> <a href="https://hachyderm.io/@pg">
|
|
<svg width="75" height="79" viewBox="0 0 75 79" xmlns="http://www.w3.org/2000/svg" fill="currentColor">
|
|
<path d="M73.8393 17.4898C72.6973 9.00165 65.2994 2.31235 56.5296 1.01614C55.05 0.797115 49.4441 0 36.4582 0H36.3612C23.3717 0 20.585 0.797115 19.1054 1.01614C10.5798 2.27644 2.79399 8.28712 0.904997 16.8758C-0.00358524 21.1056 -0.100549 25.7949 0.0682394 30.0965C0.308852 36.2651 0.355538 42.423 0.91577 48.5665C1.30307 52.6474 1.97872 56.6957 2.93763 60.6812C4.73325 68.042 12.0019 74.1676 19.1233 76.6666C26.7478 79.2728 34.9474 79.7055 42.8039 77.9162C43.6682 77.7151 44.5217 77.4817 45.3645 77.216C47.275 76.6092 49.5123 75.9305 51.1571 74.7385C51.1797 74.7217 51.1982 74.7001 51.2112 74.6753C51.2243 74.6504 51.2316 74.6229 51.2325 74.5948V68.6416C51.2321 68.6154 51.2259 68.5896 51.2142 68.5661C51.2025 68.5426 51.1858 68.522 51.1651 68.5058C51.1444 68.4896 51.1204 68.4783 51.0948 68.4726C51.0692 68.4669 51.0426 68.467 51.0171 68.4729C45.9835 69.675 40.8254 70.2777 35.6502 70.2682C26.7439 70.2682 24.3486 66.042 23.6626 64.2826C23.1113 62.762 22.7612 61.1759 22.6212 59.5646C22.6197 59.5375 22.6247 59.5105 22.6357 59.4857C22.6466 59.4609 22.6633 59.4391 22.6843 59.422C22.7053 59.4048 22.73 59.3929 22.7565 59.3871C22.783 59.3813 22.8104 59.3818 22.8367 59.3886C27.7864 60.5826 32.8604 61.1853 37.9522 61.1839C39.1768 61.1839 40.3978 61.1839 41.6224 61.1516C46.7435 61.008 52.1411 60.7459 57.1796 59.7621C57.3053 59.7369 57.431 59.7154 57.5387 59.6831C65.4861 58.157 73.0493 53.3672 73.8178 41.2381C73.8465 40.7606 73.9184 36.2364 73.9184 35.7409C73.9219 34.0569 74.4606 23.7949 73.8393 17.4898Z"/>
|
|
<path d="M61.2484 27.0263V48.114H52.8916V27.6475C52.8916 23.3388 51.096 21.1413 47.4437 21.1413C43.4287 21.1413 41.4177 23.7409 41.4177 28.8755V40.0782H33.1111V28.8755C33.1111 23.7409 31.0965 21.1413 27.0815 21.1413C23.4507 21.1413 21.6371 23.3388 21.6371 27.6475V48.114H13.2839V27.0263C13.2839 22.7176 14.384 19.2946 16.5843 16.7572C18.8539 14.2258 21.8311 12.926 25.5264 12.926C29.8036 12.926 33.0357 14.5705 35.1905 17.8559L37.2698 21.346L39.3527 17.8559C41.5074 14.5705 44.7395 12.926 49.0095 12.926C52.7013 12.926 55.6784 14.2258 57.9553 16.7572C60.1531 19.2922 61.2508 22.7152 61.2484 27.0263Z" fill="#928374" />
|
|
<defs>
|
|
<linearGradient id="paint0_linear_549_34" x1="37.0692" y1="0" x2="37.0692" y2="79" gradientUnits="userSpaceOnUse">
|
|
<stop stop-color="#6364FF"/>
|
|
<stop offset="1" stop-color="#563ACC"/>
|
|
</linearGradient>
|
|
</defs>
|
|
</svg>
|
|
</a> </li>
|
|
<li> <a href="https://bsky.app/profile/pgaultier.bsky.social">
|
|
<svg fill="currentColor" viewBox="0 0 64 57" width="32" style="width: 32px; height: 28.5px;"><path d="M13.873 3.805C21.21 9.332 29.103 20.537 32 26.55v15.882c0-.338-.13.044-.41.867-1.512 4.456-7.418 21.847-20.923 7.944-7.111-7.32-3.819-14.64 9.125-16.85-7.405 1.264-15.73-.825-18.014-9.015C1.12 23.022 0 8.51 0 6.55 0-3.268 8.579-.182 13.873 3.805ZM50.127 3.805C42.79 9.332 34.897 20.537 32 26.55v15.882c0-.338.13.044.41.867 1.512 4.456 7.418 21.847 20.923 7.944 7.111-7.32 3.819-14.64-9.125-16.85 7.405 1.264 15.73-.825 18.014-9.015C62.88 23.022 64 8.51 64 6.55c0-9.818-8.578-6.732-13.873-2.745Z"/></svg>
|
|
</a> </li>
|
|
</ul>
|
|
</div>
|
|
<div id="search-matches" hidden>
|
|
</div>
|
|
<div id="pseudo-body">
|
|
|
|
<div class="article-prelude">
|
|
<p><a href="/blog"> ⏴ Back to all articles</a></p>
|
|
|
|
<p class="publication-date">Published on 2024-11-09. Last modified on 2026-03-04.</p>
|
|
</div>
|
|
<div class="article-title">
|
|
<h1>Way too many ways to wait on a child process with a timeout</h1>
|
|
<div class="tags"> <a href="/blog/articles-by-tag.html#unix" class="tag">Unix</a> <a href="/blog/articles-by-tag.html#signals" class="tag">Signals</a> <a href="/blog/articles-by-tag.html#c" class="tag">C</a> <a href="/blog/articles-by-tag.html#linux" class="tag">Linux</a> <a href="/blog/articles-by-tag.html#freebsd" class="tag">FreeBSD</a> <a href="/blog/articles-by-tag.html#illumos" class="tag">Illumos</a> <a href="/blog/articles-by-tag.html#macos" class="tag">MacOS</a> </div>
|
|
</div>
|
|
<details class="toc"><summary>Table of contents</summary>
|
|
<ul>
|
|
|
|
<li>
|
|
<a href="#what-are-we-building">What are we building?</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="#first-approach-old-school-sigsuspend">First approach: old-school sigsuspend</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="#second-approach-sigtimedwait">Second approach: sigtimedwait</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="#third-approach-self-pipe-trick">Third approach: Self-pipe trick</a>
|
|
<ul>
|
|
|
|
<li>
|
|
<a href="#a-simpler-self-pipe-trick">A simpler self-pipe trick</a>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="#fourth-approach-linux-s-signalfd">Fourth approach: Linux's signalfd</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="#fifth-approach-process-descriptors">Fifth approach: process descriptors</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="#sixth-approach-macos-s-and-bsd-s-kqueue">Sixth approach: MacOS's and BSD's kqueue</a>
|
|
<ul>
|
|
|
|
<li>
|
|
<a href="#a-parenthesis-libkqueue">A parenthesis: libkqueue</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="#another-parenthesis-solaris-illumos-s-ports">Another parenthesis: Solaris/illumos's ports</a>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="#seventh-approach-linux-s-io-uring">Seventh approach: Linux's io_uring</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="#eigth-approach-threads">Eigth approach: Threads</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="#ninth-approach-active-polling">Ninth approach: Active polling.</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="#conclusion">Conclusion</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="#addendum-the-code">Addendum: The code</a>
|
|
</li>
|
|
</ul>
|
|
</details>
|
|
|
|
<p><em>Windows is not covered at all in this article.</em></p>
|
|
<p><em>Discussions: <a href="https://old.reddit.com/r/programming/comments/1godk0x/way_too_many_ways_to_wait_on_a_child_process_with/">/r/programming</a>, <a href="https://news.ycombinator.com/item?id=42103200">HN</a>, <a href="https://lobste.rs/s/2awfwc/way_too_many_ways_wait_on_child_process">Lobsters</a></em></p>
|
|
<p>I often need to launch a program in the terminal in a retry loop. Maybe because it's flaky, or because it tries to contact a remote service that is not available. A few scenarios:</p>
|
|
<ul>
|
|
<li>ssh to a (re)starting machine.</li>
|
|
<li><code>psql</code> to a (re)starting database.</li>
|
|
<li>Ensuring that a network service started fine with <code>netcat</code>.</li>
|
|
<li>File system commands over NFS.</li>
|
|
</ul>
|
|
<p>It's a common problem, so much so that there are two utilities that I usually reach for:</p>
|
|
<ul>
|
|
<li><a href="https://www.gnu.org/software/coreutils/manual/html_node/timeout-invocation.html">timeout</a> from GNU coreutils, which launches a command with a timeout (useful if the command itself does not have a <code>--timeout</code> option).</li>
|
|
<li><a href="https://github.com/rye/eb">eb</a> which runs a command with a certain number of times with an exponential backoff. That's useful to avoid hammering a server with connection attempts for example.</li>
|
|
</ul>
|
|
<p>This will all sound familiar to people who develop distributed systems: they have long known that this is <a href="https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/">best practice</a> to retry an operation:</p>
|
|
<ul>
|
|
<li>With a timeout (either constant or adaptive).</li>
|
|
<li>A bounded number of times e.g. 10.</li>
|
|
<li>With a waiting time between each retry, either a constant one or an increasing one e.g. with exponential backoff.</li>
|
|
<li>With jitter, although this point also seemed the least important since most of us use non real-time operating systems which introduce some jitter anytime we sleep or wait on something with a timeout. The AWS article makes a point that in highly contended systems, the jitter parameter is very important, but for the scope of this article I'll leave it out.</li>
|
|
</ul>
|
|
<p>This is best practice in distributed systems, and we often need to do the same on the command line. But the two aforementioned tools only do that partially:</p>
|
|
<ul>
|
|
<li><code>timeout</code> does not retry.</li>
|
|
<li><code>eb</code> does not have a timeout.</li>
|
|
</ul>
|
|
<p>So let's implement our own that does both! As we'll see, it's much less straightforward, and thus more interesting, than I thought. It's a whirlwind tour through Unix deeps. If you're interested in systems programming, Operating Systems, multiplexed I/O, data races, weird historical APIs, and all the ways you can shoot yourself in the foot with just a few system calls, you're in the right place!</p>
|
|
<h2 id="what-are-we-building">
|
|
<a class="title" href="#what-are-we-building">What are we building?</a>
|
|
<a class="hash-anchor" href="#what-are-we-building" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h2>
|
|
<p>I call the tool we are building <code>ueb</code> for: micro exponential backoff. It does up to 10 retries, with a waiting period in between that starts at an arbitrary 128 ms and doubles every retry. The timeout for the subprocess is the same as the sleep time, so that it's adaptive and we give the subprocess a longer and longer time to finish successfully. These numbers would probably be exposed as command line options in a real polished program, but there's no time, we have to demo it:</p>
|
|
<pre>
|
|
<div class="code-header">
|
|
<span>Shell</span>
|
|
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
|
|
</div>
|
|
<code class="language-shell"><span class="line-number"></span><span class="code-hl"># This returns immediately since it succeeds on the first try.</span>
|
|
<span class="line-number"></span><span class="code-hl">$ ueb true</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"># This retries 10 times since the command always fails, waiting more and more time between each try, and finally returns the last exit code of the command (1).</span>
|
|
<span class="line-number"></span><span class="code-hl">$ ueb false</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"># This retries a few times (~ 4 times), until the waiting time exceeds the duration of the sub-program. It exits with `0` since from the POV of our program, the sub-program finally finished in its alloted time.</span>
|
|
<span class="line-number"></span><span class="code-hl">$ ueb sleep 1</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"># Run a program that prints the date and time, and exits with a random status code, to see how it works.</span>
|
|
<span class="line-number"></span><span class="code-hl">$ ueb sh -c 'date --iso-8601=ns; export R=$(($RANDOM % 5)); echo $R; exit $R'</span>
|
|
<span class="line-number"></span><span class="code-hl">2024-11-10T15:48:49,499172093+01:00</span>
|
|
<span class="line-number"></span><span class="code-hl">4</span>
|
|
<span class="line-number"></span><span class="code-hl">2024-11-10T15:48:49,628818472+01:00</span>
|
|
<span class="line-number"></span><span class="code-hl">3</span>
|
|
<span class="line-number"></span><span class="code-hl">2024-11-10T15:48:49,886557676+01:00</span>
|
|
<span class="line-number"></span><span class="code-hl">4</span>
|
|
<span class="line-number"></span><span class="code-hl">2024-11-10T15:48:50,400199626+01:00</span>
|
|
<span class="line-number"></span><span class="code-hl">3</span>
|
|
<span class="line-number"></span><span class="code-hl">2024-11-10T15:48:51,425937132+01:00</span>
|
|
<span class="line-number"></span><span class="code-hl">2</span>
|
|
<span class="line-number"></span><span class="code-hl">2024-11-10T15:48:53,475565645+01:00</span>
|
|
<span class="line-number"></span><span class="code-hl">2</span>
|
|
<span class="line-number"></span><span class="code-hl">2024-11-10T15:48:57,573278508+01:00</span>
|
|
<span class="line-number"></span><span class="code-hl">1</span>
|
|
<span class="line-number"></span><span class="code-hl">2024-11-10T15:49:05,767338611+01:00</span>
|
|
<span class="line-number"></span><span class="code-hl">0</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"># Some more practical examples.</span>
|
|
<span class="line-number"></span><span class="code-hl">$ ueb ssh <some_ip></span>
|
|
<span class="line-number"></span><span class="code-hl">$ ueb createdb my_great_database -h 0.0.0.0 -U postgres</span>
|
|
</code></pre>
|
|
<p>If you want to monitor the retries and the sleeps, you can use <code>strace</code> or <code>dtrace</code>:</p>
|
|
<pre>
|
|
<div class="code-header">
|
|
<span>Shell</span>
|
|
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
|
|
</div>
|
|
<code class="language-shell"><span class="line-number"></span><span class="code-hl">$ strace ueb sleep 1</span>
|
|
</code></pre>
|
|
<p>Note that the sub-command should be idempotent, otherwise we might create a given resource twice, or the command might have succeeded right after our timeout triggered but also right before we killed it, so our program thinks it timed out and thus needs to be retried. There is this small data race window, which is completely fine if the command is idempotent but will erroneously retry the command to the bitter end otherwise. There is also the case where the sub-command does stuff over the network for example creating a resource, it succeeds, but the ACK is never received due to network issues. The sub-command will think it failed and retry. Again, fairly standard stuff in distributed systems but I thought it was worth mentioning.</p>
|
|
<p>So how do we implement it?</p>
|
|
<p>Immediately, we notice something: even though there are a bazillion ways to wait on a child process to finish (<code>wait</code>, <code>wait3</code>, <code>wait4</code>, <code>waitid</code>, <code>waitpid</code>), none of them take a timeout as an argument. This has sparked numerous questions online (<a href="https://stackoverflow.com/questions/18542089/how-to-wait-on-child-process-to-finish-with-time-limit">1</a>, <a href="https://stackoverflow.com/questions/18476138/is-there-a-version-of-the-wait-system-call-that-sets-a-timeout">2</a>), with in my opinion unsatisfactory answers. So let's explore this rabbit hole.</p>
|
|
<p>We'd like the pseudo-code to be something like:</p>
|
|
<pre>
|
|
<div class="code-header">
|
|
<span>Plaintext</span>
|
|
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
|
|
</div>
|
|
<code class="language-plaintext"><span class="line-number"></span><span class="code-hl">wait_ms := 128</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl">for retry in 0..<10:</span>
|
|
<span class="line-number"></span><span class="code-hl"> child_pid := run_command_in_subprocess(cmd)</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> ret := wait_for_process_to_finish_with_timeout_ms(child_pid, wait_ms)</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (did_process_finish_successfully(ret)):</span>
|
|
<span class="line-number"></span><span class="code-hl"> exit(0)</span>
|
|
<span class="line-number"></span><span class="code-hl"> </span>
|
|
<span class="line-number"></span><span class="code-hl"> // In case of a timeout, we need to kill the child process and retry.</span>
|
|
<span class="line-number"></span><span class="code-hl"> kill(child_pid, SIGKILL)</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> // Reap zombie process to avoid a resource leak.</span>
|
|
<span class="line-number"></span><span class="code-hl"> waitpid(child_pid)</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> sleep_ms(wait_ms);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> wait_ms *= 2;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl">// All retries exhausted, exit with an error code.</span>
|
|
<span class="line-number"></span><span class="code-hl">exit(1)</span>
|
|
</code></pre>
|
|
<p><em>There is a degenerate case where the give command to run is wrong (e.g. typo in the parameters) or the executable does not exist, and our program will happily retry it to the bitter end. But there is solace: this is bounded by the number of retries (10). That's why we do not retry forever.</em></p>
|
|
<h2 id="first-approach-old-school-sigsuspend">
|
|
<a class="title" href="#first-approach-old-school-sigsuspend">First approach: old-school sigsuspend</a>
|
|
<a class="hash-anchor" href="#first-approach-old-school-sigsuspend" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h2>
|
|
<p>That's how <code>timeout</code> from coreutils <a href="https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/timeout.c;h=5600ce42957dcf117785f6a361ef72ac9c2df352;hb=HEAD">implements</a> it. This is quite simple on paper:</p>
|
|
<ol>
|
|
<li>We opt-in to receive a <code>SIGCHLD</code> signal when the child processes finishes with: <code>signal(SIGCHLD, on_chld_signal)</code> where <code>on_chld_signal</code> is a function pointer we provide. Even if the signal handler does not do anything in this case.</li>
|
|
<li>We schedule a <code>SIGALARM</code> signal with <code>alarm</code> or more preferably <code>setitimer</code> which can take a duration in microseconds whereas <code>alarm</code> can only handle seconds. There's also <code>timer_create/timer_settime</code> which handles nanoseconds. It depends what the OS and hardware support.</li>
|
|
<li>We wait for either signal with <code>sigsuspend</code> which suspends the program until a given set of signals arrive.</li>
|
|
<li>We should not forget to <code>wait</code> on the child process to avoid leaving zombie processes behind.</li>
|
|
</ol>
|
|
<p>The reality is grimmer, looking through the <code>timeout</code> implementation:</p>
|
|
<ul>
|
|
<li>We could have inherited any signal mask from our parent so we need to explicitly unblock the signals we are interested in.</li>
|
|
<li>Signals can be sent to a process group we need to handle that case.</li>
|
|
<li>We have to avoid entering a 'signal loop'.</li>
|
|
<li>Our process can be implicitly multi-threaded due to some <code>timer_settime</code> implementations, therefore a <code>SIGALRM</code> signal sent to a process group, can result in the signal being sent multiple times to a process (I am directly quoting the code comments from the <code>timeout</code> program here).</li>
|
|
<li>When using <code>timer_create</code>, we need to take care of cleaning it up with <code>timer_delete</code>, lest we have a resource leak when retrying.</li>
|
|
<li>The signal handler may be called concurrently and we have to be aware of that.</li>
|
|
<li>Depending on the timer implementation we chose, we are susceptible to clock adjustments for example going back. E.g. <code>setitimer</code> only offers the <code>CLOCK_REALTIME</code> clock option for counting time, which is just the wall clock. We'd like something like <code>CLOCK_MONOTONIC</code> or <code>CLOCK_MONOTONIC_RAW</code> (the latter being Linux specific).</li>
|
|
</ul>
|
|
<p>So... I don't <em>love</em> this approach:</p>
|
|
<ul>
|
|
<li>I find signals hard. It's basically a global <code>goto</code> to a completely different location.</li>
|
|
<li>A signal handler is forced to use global mutable state, which is better avoided if possible, and it does not play nice with threads.</li>
|
|
<li>Lots of functions are not 'signal-safe', and that has led to security vulnerabilities in the past e.g. in <a href="https://www.qualys.com/2024/07/01/cve-2024-6387/regresshion.txt">ssh</a>. In short, non-atomic operations are not signal safe because they might be suspended in the middle, thus leaving an inconsistent state behind. So, we have to read documentation very carefully to ensure that we only call signal safe functions in our signal handler, and cherry on the cake, that varies from platform to platform, or even between libc versions on the same platform.</li>
|
|
<li>Signals do not compose well with other Unix entities such as file descriptors and sockets. For example, we cannot <code>poll</code> on signals. There are platform specific solutions though, keep on reading.</li>
|
|
<li>Different signals have different default behaviors, and this gets inherited in child processes, so you cannot assume anything in your program and have to be very defensive. Who knows what the parent process, e.g. the shell, set as the signal mask? If you read through the whole implementation of the <code>timeout</code> program, a lot of the code is dedicated to setting signal masks in the parent, forking, immediately changing the signal mask in the child and the parent, etc. Now, I believe modern Unices offer more control than <code>fork()</code> about what signal mask the child should be created with, so maybe it got better. Still, it's a lot of stuff to know.</li>
|
|
<li>They are many libc functions and system calls relating to signals and that's a lot to learn. A non-exhaustive list e.g. on Linux: <code>kill(1), alarm(2), kill(2), pause(2), sigaction(2), signalfd(2), sigpending(2), sigprocmask(2), sigsuspend(2), bsd_signal(3), killpg(3), raise(3), siginterrupt(3), sigqueue(3), sigsetops(3), sigvec(3), sysv_signal(3), signal(7)</code>. Oh wait, I forgot <code>sigemptyset(3)</code> and <code>sigaddset(3)</code>. And I'm sure I overlooked a few more!</li>
|
|
</ul>
|
|
<p>So, let's stick with signals for a bit but simplify our current approach.</p>
|
|
<h2 id="second-approach-sigtimedwait">
|
|
<a class="title" href="#second-approach-sigtimedwait">Second approach: sigtimedwait</a>
|
|
<a class="hash-anchor" href="#second-approach-sigtimedwait" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h2>
|
|
<p>Wouldn't it be great if we could wait on a signal, say, <code>SIGCHLD</code>, with a timeout? Oh look, a system call that does exactly that <em>and</em> is standardized by POSIX 2001. Cool! I am not quite sure why the <code>timeout</code> program does not use it, but we sure as hell can. My only guess would be that they want to support old Unices pre 2001, or non POSIX systems.</p>
|
|
<p><em>A knowledgeable reader has <a href="https://github.com/gaultier/blog/issues/22">pointed out</a> that <code>sigtimedwait</code> was optional in POSIX 2001 and as such not implemented in some operating systems. It was made mandatory in POSIX 2008 but the adoption was slow.</em></p>
|
|
<p>Anyways, here's a very straightforward implementation:</p>
|
|
<pre>
|
|
<div class="code-header">
|
|
<span>C</span>
|
|
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
|
|
</div>
|
|
<code class="language-c"><span class="line-number"></span><span class="code-hl">#define _GNU_SOURCE</span>
|
|
<span class="line-number"></span><span class="code-hl">#include <errno.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <signal.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <stdint.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <sys/wait.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <unistd.h></span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl">void on_sigchld(int sig) { (void)sig; }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl">int main(int argc, char *argv[]) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> (void)argc;</span>
|
|
<span class="line-number"></span><span class="code-hl"> signal(SIGCHLD, on_sigchld);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> uint32_t wait_ms = 128;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> for (int retry = 0; retry < 10; retry += 1) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> int child_pid = fork();</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == child_pid) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> if (0 == child_pid) { // Child</span>
|
|
<span class="line-number"></span><span class="code-hl"> argv += 1;</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == execvp(argv[0], argv)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> __builtin_unreachable();</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> sigset_t sigset = {0};</span>
|
|
<span class="line-number"></span><span class="code-hl"> sigemptyset(&sigset);</span>
|
|
<span class="line-number"></span><span class="code-hl"> sigaddset(&sigset, SIGCHLD);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> siginfo_t siginfo = {0};</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> struct timespec timeout = {</span>
|
|
<span class="line-number"></span><span class="code-hl"> .tv_sec = wait_ms / 1000,</span>
|
|
<span class="line-number"></span><span class="code-hl"> .tv_nsec = (wait_ms % 1000) * 1000 * 1000,</span>
|
|
<span class="line-number"></span><span class="code-hl"> };</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> int sig = sigtimedwait(&sigset, &siginfo, &timeout);</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == sig && EAGAIN != errno) { // Error</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 != sig) { // Child finished.</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (WIFEXITED(siginfo.si_status) && 0 == WEXITSTATUS(siginfo.si_status)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return 0;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == kill(child_pid, SIGKILL)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == wait(NULL)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> usleep(wait_ms * 1000);</span>
|
|
<span class="line-number"></span><span class="code-hl"> wait_ms *= 2;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> return 1;</span>
|
|
<span class="line-number"></span><span class="code-hl">}</span>
|
|
</code></pre>
|
|
<p>I like this implementation. It's pretty easy to convince ourselves looking at the code that it is obviously correct, and that's a very important factor for me.</p>
|
|
<p>We still have to deal with signals though. Could we reduce their imprint on our code?</p>
|
|
<h2 id="third-approach-self-pipe-trick">
|
|
<a class="title" href="#third-approach-self-pipe-trick">Third approach: Self-pipe trick</a>
|
|
<a class="hash-anchor" href="#third-approach-self-pipe-trick" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h2>
|
|
<p>This is a really nifty, quite well known <a href="https://cr.yp.to/docs/selfpipe.html">trick</a> at this point, where we bridge the world of signals with the world of file descriptors with the <code>pipe(2)</code> system call.</p>
|
|
<p>Usually, pipes are a form of inter-process communication, and here we do not want to communicate with the child process (since it could be any program, and most programs do not get chatty with their parent process). What we do is: in the signal handler for <code>SIGCHLD</code>, we simply write (anything) to our own pipe. We know this is signal-safe so it's good.</p>
|
|
<p>And you know what's cool with pipes? They are simply a file descriptor which we can <code>poll</code>. With a timeout. Nice! Here goes:</p>
|
|
<pre>
|
|
<div class="code-header">
|
|
<span>C</span>
|
|
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
|
|
</div>
|
|
<code class="language-c"><span class="line-number"></span><span class="code-hl">#define _GNU_SOURCE</span>
|
|
<span class="line-number"></span><span class="code-hl">#include <errno.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <poll.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <signal.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <stdint.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <sys/wait.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <unistd.h></span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl">static int pipe_fd[2] = {0};</span>
|
|
<span class="line-number"></span><span class="code-hl">void on_sigchld(int sig) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> (void)sig;</span>
|
|
<span class="line-number"></span><span class="code-hl"> char dummy = 0;</span>
|
|
<span class="line-number"></span><span class="code-hl"> write(pipe_fd[1], &dummy, 1);</span>
|
|
<span class="line-number"></span><span class="code-hl">}</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl">int main(int argc, char *argv[]) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> (void)argc;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == pipe(pipe_fd)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> signal(SIGCHLD, on_sigchld);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> uint32_t wait_ms = 128;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> for (int retry = 0; retry < 10; retry += 1) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> int child_pid = fork();</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == child_pid) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> if (0 == child_pid) { // Child</span>
|
|
<span class="line-number"></span><span class="code-hl"> argv += 1;</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == execvp(argv[0], argv)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> __builtin_unreachable();</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> struct pollfd poll_fd = {</span>
|
|
<span class="line-number"></span><span class="code-hl"> .fd = pipe_fd[0],</span>
|
|
<span class="line-number"></span><span class="code-hl"> .events = POLLIN,</span>
|
|
<span class="line-number"></span><span class="code-hl"> };</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> // Wait for the child to finish with a timeout.</span>
|
|
<span class="line-number"></span><span class="code-hl"> poll(&poll_fd, 1, (int)wait_ms);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> kill(child_pid, SIGKILL);</span>
|
|
<span class="line-number"></span><span class="code-hl"> int status = 0;</span>
|
|
<span class="line-number"></span><span class="code-hl"> wait(&status);</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (WIFEXITED(status) && 0 == WEXITSTATUS(status)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return 0;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> char dummy = 0;</span>
|
|
<span class="line-number"></span><span class="code-hl"> read(pipe_fd[0], &dummy, 1);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> usleep(wait_ms * 1000);</span>
|
|
<span class="line-number"></span><span class="code-hl"> wait_ms *= 2;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> return 1;</span>
|
|
<span class="line-number"></span><span class="code-hl">}</span>
|
|
</code></pre>
|
|
<p>So we still have one signal handler but the rest of our program does not deal with signals in any way (well, except to kill the child when the timeout triggers, but that's invisible).</p>
|
|
<p>There are a few catches with this implementation:</p>
|
|
<ul>
|
|
<li>Contrary to <code>sigtimedwait</code>, <code>poll</code> does not give us the exit status of the child, we have to get it with <code>wait</code>. Which is fine.</li>
|
|
<li>In the case that the timeout fired, we <code>kill</code> the child process. However, the child process, being forcefully ended, will result in a <code>SIGCHLD</code> signal being sent to our program. Which will then trigger our signal handler, which will then write a value to the pipe. So we need to unconditionally read from the pipe after killing the child and before retrying. If we only read from the pipe if the child ended by itself, that will result in the pipe and the child process being desynced.</li>
|
|
<li><p>In some complex programs, we'd have to use <code>ppoll</code> instead of <code>poll</code>. <code>ppoll</code> prevents a set of signals from interrupting the polling. That's to avoid some data races (again, more data races!). Quoting from the man page for <code>pselect</code> which is analogous to <code>ppoll</code>:</p>
|
|
<blockquote>
|
|
<p>The reason that pselect() is needed is that if one wants to wait for either a signal
|
|
or for a file descriptor to become ready, then an atomic test is needed to prevent
|
|
race conditions. (Suppose the signal handler sets a global flag and returns. Then a
|
|
test of this global flag followed by a call of select() could hang indefinitely if the
|
|
signal arrived just after the test but just before the call. By contrast, pselect()
|
|
allows one to first block signals, handle the signals that have come in, then call pselect()
|
|
with the desired sigmask, avoiding the race.)</p>
|
|
</blockquote>
|
|
</li>
|
|
</ul>
|
|
<p>So, this trick is clever, but wouldn't it be nice if we could avoid signals <em>entirely</em>?</p>
|
|
<h3 id="a-simpler-self-pipe-trick">
|
|
<a class="title" href="#a-simpler-self-pipe-trick">A simpler self-pipe trick</a>
|
|
<a class="hash-anchor" href="#a-simpler-self-pipe-trick" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h3>
|
|
<p>An astute reader <a href="https://hachyderm.io/@markd/113461301892152667">pointed out</a> that this trick can be simplified to not deal with signals at all and instead leverage two facts:</p>
|
|
<ul>
|
|
<li>A child inherits the open file descriptors of the parent (including the ones from a pipe)</li>
|
|
<li>When a process exits, the OS automatically closes its file descriptors</li>
|
|
</ul>
|
|
<p>Behind the scenes, at the OS level, there is a reference count for a file descriptor shared by multiple processes. It gets decremented when doing <code>close(fd)</code> or by a process terminating. When this count reaches 0, it is closed for real. And you know what system call can watch for a file descriptor closing? Good old <code>poll</code>!</p>
|
|
<p>So the improved approach is as follows:</p>
|
|
<ol>
|
|
<li>Each retry, we create a new pipe.</li>
|
|
<li>We fork.</li>
|
|
<li>The parent closes the write end pipe and the child closes the read end pipe. Effectively, the parent owns the read end and the child owns the write end.</li>
|
|
<li>The parent polls on the read end.</li>
|
|
<li>When the child finishes, it automatically closes the write end which in turn triggers an event in <code>poll</code>.</li>
|
|
<li>We cleanup before retrying (if needed)</li>
|
|
</ol>
|
|
<p>So in a way, it's not really a <em>self</em>-pipe, it's more precisely a pipe between the parent and the child, and nothing gets written or read, it's just used by the child to signal it's done when it closes its end. Which is a useful approach for many cases outside of our little program.</p>
|
|
<p>Here is the code:</p>
|
|
<pre>
|
|
<div class="code-header">
|
|
<span>C</span>
|
|
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
|
|
</div>
|
|
<code class="language-c"><span class="line-number"></span><span class="code-hl">#define _GNU_SOURCE</span>
|
|
<span class="line-number"></span><span class="code-hl">#include <errno.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <poll.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <stdint.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <sys/wait.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <unistd.h></span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl">int main(int argc, char *argv[]) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> (void)argc;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> uint32_t wait_ms = 128;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> for (int retry = 0; retry < 10; retry += 1) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> int pipe_fd[2] = {0};</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == pipe(pipe_fd)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> int child_pid = fork();</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == child_pid) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> if (0 == child_pid) { // Child</span>
|
|
<span class="line-number"></span><span class="code-hl"> // Close the read end of the pipe.</span>
|
|
<span class="line-number"></span><span class="code-hl"> close(pipe_fd[0]);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> argv += 1;</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == execvp(argv[0], argv)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> __builtin_unreachable();</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> // Close the write end of the pipe.</span>
|
|
<span class="line-number"></span><span class="code-hl"> close(pipe_fd[1]);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> struct pollfd poll_fd = {</span>
|
|
<span class="line-number"></span><span class="code-hl"> .fd = pipe_fd[0],</span>
|
|
<span class="line-number"></span><span class="code-hl"> .events = POLLHUP | POLLIN,</span>
|
|
<span class="line-number"></span><span class="code-hl"> };</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> // Wait for the child to finish with a timeout.</span>
|
|
<span class="line-number"></span><span class="code-hl"> poll(&poll_fd, 1, (int)wait_ms);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> kill(child_pid, SIGKILL);</span>
|
|
<span class="line-number"></span><span class="code-hl"> int status = 0;</span>
|
|
<span class="line-number"></span><span class="code-hl"> wait(&status);</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (WIFEXITED(status) && 0 == WEXITSTATUS(status)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return 0;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> close(pipe_fd[0]);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> usleep(wait_ms * 1000);</span>
|
|
<span class="line-number"></span><span class="code-hl"> wait_ms *= 2;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> return 1;</span>
|
|
<span class="line-number"></span><span class="code-hl">}</span>
|
|
</code></pre>
|
|
<p>Voila, no signals and no global state!</p>
|
|
<h2 id="fourth-approach-linux-s-signalfd">
|
|
<a class="title" href="#fourth-approach-linux-s-signalfd">Fourth approach: Linux's signalfd</a>
|
|
<a class="hash-anchor" href="#fourth-approach-linux-s-signalfd" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h2>
|
|
<p>This is a short one: on Linux, there is a system call that does exactly the same as the self-pipe trick: from a signal, it gives us a file descriptor that we can <code>poll</code>. So, we can entirely remove our pipe and signal handler and instead <code>poll</code> the file descriptor that <code>signalfd</code> gives us.</p>
|
|
<p>Cool, but also....Was it really necessary to introduce a system call for that? I guess the advantage is clarity.</p>
|
|
<p>I would prefer extending <code>poll</code> to support things other than file descriptors, instead of converting everything a file descriptor to be able to use <code>poll</code>.</p>
|
|
<p>Ok, next!</p>
|
|
<h2 id="fifth-approach-process-descriptors">
|
|
<a class="title" href="#fifth-approach-process-descriptors">Fifth approach: process descriptors</a>
|
|
<a class="hash-anchor" href="#fifth-approach-process-descriptors" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h2>
|
|
<p><em>Recommended reading about this topic: <a href="https://lwn.net/Articles/801319/">1</a> and <a href="https://lwn.net/Articles/794707/">2</a>.</em></p>
|
|
<p>In the recent years (starting with Linux 5.3 and FreeBSD 9), people realized that process identifiers (<code>pid</code>s) have a number of problems:</p>
|
|
<ul>
|
|
<li>PIDs are recycled and the space is small, so collisions will happen. Typically, a process spawns a child process, some work happens, and then the parent decides to send a signal to the PID of the child. But it turns out that the child already terminated (unbeknownst to the parent) and another process took its place with the same PID. So now the parent is sending signals, or communicating with, a process that it thinks is its original child but is in fact something completely different. Chaos and security issues ensue. Now, in our very simple case, that would not really happen, but perhaps the root user is running our program, or, imagine that you are implementing the init process with PID 1, e.g. systemd: you can kill any process on the machine! Or think of the case of re-parenting a process. Or sending a certain PID to another process and they send a signal to it at some point in the future. It becomes hairy and it's a very real problem.</li>
|
|
<li>Data races are hard to escape (see the previous point).</li>
|
|
<li><p>It's easy to accidentally send a signal to all processes with <code>kill(0, SIGKILL)</code> or <code>kill(-1, SIGKILL)</code> if the developer has not checked that all previous operations succeeded. This is a classic mistake:</p>
|
|
<pre>
|
|
<div class="code-header">
|
|
<span>C</span>
|
|
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
|
|
</div>
|
|
<code class="language-c"><span class="line-number"></span><span class="code-hl">int child_pid = fork(); // This fork fails and returns -1.</span>
|
|
<span class="line-number"></span><span class="code-hl">... // (do not check that fork succeeded);</span>
|
|
<span class="line-number"></span><span class="code-hl">kill(child_pid, SIGKILL); // Effectively: kill(-1, SIGKILL)</span>
|
|
</code></pre>
|
|
</li>
|
|
</ul>
|
|
<p>And the kernel developers have worked hard to introduce a better concept: process descriptors, which are (almost) bog-standard file descriptors, like files or sockets. After all, that's what sparked our whole investigation: we wanted to use <code>poll</code> and it did not work on a PID. PIDs and signals do not compose well, but file descriptors do. Also, just like file descriptors, process descriptors are per-process. If I open a file with <code>open()</code> and get the file descriptor <code>3</code>, it is scoped to my process. Another process can <code>close(3)</code> and it will refer to their own file descriptor, and not affect my file descriptor. That's great, we get isolation, so bugs in our code do not affect other processes.</p>
|
|
<p>So, Linux and FreeBSD have introduced the same concepts but with slightly different APIs (unfortunately), and I have no idea about other OSes:</p>
|
|
<ul>
|
|
<li>A child process can be created with <code>clone3(..., CLONE_PIDFD)</code> (Linux) or <code>pdfork()</code> (FreeBSD) which returns a process descriptor which is almost like a normal file descriptor. On Linux, a process descriptor can also be obtained from a PID with <code>pidfd_open(pid)</code> e.g. after a normal <code>fork</code> was done (but there is a risk of a data race in some cases!). Once we have the process descriptor, we do not need the PID anymore.</li>
|
|
<li>We wait on the process descriptor with <code>poll(..., timeout)</code> (or <code>select</code>, or <code>epoll</code>, etc).</li>
|
|
<li>We kill the child process using the process descriptor with <code>pidfd_send_signal</code> (Linux) or <code>close</code> (FreeBSD) or <code>pdkill</code> (FreeBSD).</li>
|
|
<li>We wait on the zombie child process again using the process descriptor to get its exit status.</li>
|
|
</ul>
|
|
<p>And voila, no signals! Isolation! Composability! (Almost) No PIDs in our program! Life can be nice sometimes. It's just unfortunate that there isn't a cross-platform API for that.</p>
|
|
<p>Here's the Linux implementation:</p>
|
|
<pre>
|
|
<div class="code-header">
|
|
<span>C</span>
|
|
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
|
|
</div>
|
|
<code class="language-c"><span class="line-number"></span><span class="code-hl">#define _GNU_SOURCE</span>
|
|
<span class="line-number"></span><span class="code-hl">#include <errno.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <poll.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <stdint.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <sys/syscall.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <sys/wait.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <unistd.h></span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl">int main(int argc, char *argv[]) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> (void)argc;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> uint32_t wait_ms = 128;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> for (int retry = 0; retry < 10; retry += 1) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> int child_pid = fork();</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == child_pid) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> if (0 == child_pid) { // Child</span>
|
|
<span class="line-number"></span><span class="code-hl"> argv += 1;</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == execvp(argv[0], argv)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> __builtin_unreachable();</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> // Parent.</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> int child_fd = (int)syscall(SYS_pidfd_open, child_pid, 0);</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == child_fd) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> struct pollfd poll_fd = {</span>
|
|
<span class="line-number"></span><span class="code-hl"> .fd = child_fd,</span>
|
|
<span class="line-number"></span><span class="code-hl"> .events = POLLHUP | POLLIN,</span>
|
|
<span class="line-number"></span><span class="code-hl"> };</span>
|
|
<span class="line-number"></span><span class="code-hl"> // Wait for the child to finish with a timeout.</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == poll(&poll_fd, 1, (int)wait_ms)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == syscall(SYS_pidfd_send_signal, child_fd, SIGKILL, NULL, 0)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> siginfo_t siginfo = {0};</span>
|
|
<span class="line-number"></span><span class="code-hl"> // Get exit status of child & reap zombie.</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == waitid(P_PIDFD, (id_t)child_fd, &siginfo, WEXITED)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> if (WIFEXITED(siginfo.si_status) && 0 == WEXITSTATUS(siginfo.si_status)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return 0;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> wait_ms *= 2;</span>
|
|
<span class="line-number"></span><span class="code-hl"> usleep(wait_ms * 1000);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> close(child_fd);</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl">}</span>
|
|
</code></pre>
|
|
<p>A small note: To <code>poll</code> a process descriptor, Linux wants us to use <code>POLLIN</code> whereas FreeBSD wants us to use <code>POLLHUP</code>. So we use <code>POLLHUP | POLLIN</code> since there are no side-effects to use both.</p>
|
|
<p>Another small note: a process descriptor, just like a file descriptor, takes up resources on the kernel side and we can reach some system limits (or even the memory limit), so it's good practice to <code>close</code> it as soon as possible to free up resources. For us, that's right before retrying. On FreeBSD, closing the process descriptor also kills the process, so it's very short, just one system call. On Linux, we need to do both.</p>
|
|
<h2 id="sixth-approach-macos-s-and-bsd-s-kqueue">
|
|
<a class="title" href="#sixth-approach-macos-s-and-bsd-s-kqueue">Sixth approach: MacOS's and BSD's kqueue</a>
|
|
<a class="hash-anchor" href="#sixth-approach-macos-s-and-bsd-s-kqueue" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h2>
|
|
<p>It feels like cheating, but MacOS and the BSDs have had <code>kqueue</code> for decades which works out of the box with PIDs. It's a bit similar to <code>poll</code> or <code>epoll</code> on Linux:</p>
|
|
<pre>
|
|
<div class="code-header">
|
|
<span>C</span>
|
|
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
|
|
</div>
|
|
<code class="language-c"><span class="line-number"></span><span class="code-hl">#include <errno.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <signal.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <stdint.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <sys/event.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <sys/wait.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <unistd.h></span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl">int main(int argc, char *argv[]) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> (void)argc;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> uint32_t wait_ms = 128;</span>
|
|
<span class="line-number"></span><span class="code-hl"> int queue = kqueuex(KQUEUE_CLOEXEC);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> for (int retry = 0; retry < 10; retry += 1) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> int child_pid = fork();</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == child_pid) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> if (0 == child_pid) { // Child</span>
|
|
<span class="line-number"></span><span class="code-hl"> argv += 1;</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == execvp(argv[0], argv)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> __builtin_unreachable();</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> struct kevent change_list = {</span>
|
|
<span class="line-number"></span><span class="code-hl"> .ident = child_pid,</span>
|
|
<span class="line-number"></span><span class="code-hl"> .filter = EVFILT_PROC,</span>
|
|
<span class="line-number"></span><span class="code-hl"> .fflags = NOTE_EXIT,</span>
|
|
<span class="line-number"></span><span class="code-hl"> .flags = EV_ADD | EV_CLEAR,</span>
|
|
<span class="line-number"></span><span class="code-hl"> };</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> struct kevent event_list = {0};</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> struct timespec timeout = {</span>
|
|
<span class="line-number"></span><span class="code-hl"> .tv_sec = wait_ms / 1000,</span>
|
|
<span class="line-number"></span><span class="code-hl"> .tv_nsec = (wait_ms % 1000) * 1000 * 1000,</span>
|
|
<span class="line-number"></span><span class="code-hl"> };</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> int ret = kevent(queue, &change_list, 1, &event_list, 1, &timeout);</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == ret) { // Error</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (1 == ret) { // Child finished.</span>
|
|
<span class="line-number"></span><span class="code-hl"> int status = 0;</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == wait(&status)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (WIFEXITED(status) && 0 == WEXITSTATUS(status)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return 0;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> kill(child_pid, SIGKILL);</span>
|
|
<span class="line-number"></span><span class="code-hl"> wait(NULL);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> change_list = (struct kevent){</span>
|
|
<span class="line-number"></span><span class="code-hl"> .ident = child_pid,</span>
|
|
<span class="line-number"></span><span class="code-hl"> .filter = EVFILT_PROC,</span>
|
|
<span class="line-number"></span><span class="code-hl"> .fflags = NOTE_EXIT,</span>
|
|
<span class="line-number"></span><span class="code-hl"> .flags = EV_DELETE,</span>
|
|
<span class="line-number"></span><span class="code-hl"> };</span>
|
|
<span class="line-number"></span><span class="code-hl"> kevent(queue, &change_list, 1, NULL, 0, NULL);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> usleep(wait_ms * 1000);</span>
|
|
<span class="line-number"></span><span class="code-hl"> wait_ms *= 2;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> return 1;</span>
|
|
<span class="line-number"></span><span class="code-hl">}</span>
|
|
</code></pre>
|
|
<p>The only surprising thing, perhaps, is that a <code>kqueue</code> is stateful, so once the child process exited by itself or was killed, we have to remove the watcher on its PID, since the next time we spawn a child process, the PID will very likely be different. <code>kqueue</code> offers the flag <code>EV_ONESHOT</code>, which automatically deletes the event from the queue once it has been consumed by us. However, it would not help in all cases: if the timeout triggers, no event was consumed, and we have to kill the child process, which creates an event in the queue! So we have to always consume/delete the event from the queue right before we retry, with a second <code>kevent</code> call. That's the same situation as with the self-pipe approach where we unconditionally <code>read</code> from the pipe to 'clear' it before retrying.</p>
|
|
<p>I love that <code>kqueue</code> works with every kind of Unix entity: file descriptor, pipes, PIDs, Vnodes, sockets, etc. Even signals! However, I am not sure that I love its statefulness. I find the <code>poll</code> API simpler, since it's stateless. But perhaps this behavior is necessary for some corner cases or for performance to avoid the linear scanning that <code>poll</code> entails? It's interesting to observe that Linux's <code>epoll</code> went the same route as <code>kqueue</code> with a similar API, however, <code>epoll</code> can only watch plain file descriptors.</p>
|
|
<h3 id="a-parenthesis-libkqueue">
|
|
<a class="title" href="#a-parenthesis-libkqueue">A parenthesis: libkqueue</a>
|
|
<a class="hash-anchor" href="#a-parenthesis-libkqueue" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h3>
|
|
<p><code>kqueue</code> is only for MacOS and BSDs....Or is it?</p>
|
|
<p>There is this library, <a href="https://github.com/mheily/libkqueue">libkqueue</a>, that acts as a compatibility layer to be able to use <code>kqueue</code> on all major operating systems, mainly Windows, Linux, and even Solaris/illumos!</p>
|
|
<p>So...How do they do it then? How can we, on an OS like Linux, watch a PID with the <code>kqueue</code> API, when the OS does not support that functionality (neither with <code>poll</code> or <code>epoll</code>)? Well, the solution is actually very simple:</p>
|
|
<ul>
|
|
<li>On Linux 5.3+, they use <code>pidfd_open</code> + <code>poll/epoll</code>. Hey, we just did that a few sections above!</li>
|
|
<li><p>On older versions of Linux, they handle the signals, like GNU's <code>timeout</code>. It has a number of known shortcomings which is testament to the hardships of using signals. To just quote one piece:</p>
|
|
<blockquote>
|
|
<p>Because the Linux kernel coalesces SIGCHLD (and other signals), the only way to reliably determine if a monitored process has exited, is to loop through all PIDs registered by any kqueue when we receive a SIGCHLD. This involves many calls to waitid(2) and may have a negative performance impact.</p>
|
|
</blockquote>
|
|
</li>
|
|
</ul>
|
|
<h3 id="another-parenthesis-solaris-illumos-s-ports">
|
|
<a class="title" href="#another-parenthesis-solaris-illumos-s-ports">Another parenthesis: Solaris/illumos's ports</a>
|
|
<a class="hash-anchor" href="#another-parenthesis-solaris-illumos-s-ports" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h3>
|
|
<p>So, if it was not enough that each major OS has its own way to watch many different kinds of entities (Windows has its own thing called <a href="https://learn.microsoft.com/en-us/windows/win32/fileio/i-o-completion-ports">I/O completion ports</a>, MacOS & BSDs have <code>kqueue</code>, Linux has <code>epoll</code>), Solaris/illumos shows up and says: Watch me do my own thing. Well actually I do not know the chronology, they might in fact have been first, and some illumos kernel developers (namely Brian Cantrill in the fabulous <a href="https://www.youtube.com/watch?v=wTVfAMRj-7E">Cantrillogy</a>) have admitted that it would have been better for everyone if they also had adopted <code>kqueue</code>.</p>
|
|
<p>Anyways, their own system is called <a href="https://www.illumos.org/man/3C/port_create">port</a> (or is it ports?) and it looks so similar to <code>kqueue</code> it's almost painful. And weirdly, they support all the different kinds of entities that <code>kqueue</code> supports <em>except</em> PIDs! And I am not sure that they support process descriptors either e.g. <code>pidfd_open</code>. However, they have an extensive compatibility layer for Linux so perhaps they do there.</p>
|
|
<p><em>EDIT: illumos has <a href="https://illumos.org/man/3PROC/Pctlfd">Pctlfd</a> which seems to give a file descriptor for a given process, and this file descriptor could then be used <code>port_create</code> or <code>poll</code>.</em></p>
|
|
<h2 id="seventh-approach-linux-s-io-uring">
|
|
<a class="title" href="#seventh-approach-linux-s-io-uring">Seventh approach: Linux's io_uring</a>
|
|
<a class="hash-anchor" href="#seventh-approach-linux-s-io-uring" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h2>
|
|
<p><code>io_uring</code> is the last candidate to enter the already packed ring (eh) of different-yet-similar ways to do 'I/O multiplexing', meaning to wait with a timeout on various kinds of entities to do interesting 'stuff'. We queue a system call e.g. <code>wait</code>, as well as a timeout, and we wait for either to complete. If <code>wait</code> completed first and the exit status is a success, we exit. Otherwise, we retry. Familiar stuff at this point. <code>io_uring</code> essentially makes every system call asynchronous with a uniform API. That's exactly what we want! <code>io_uring</code> only exposes <code>waitid</code> and only in very recent versions, which is completely fine.</p>
|
|
<p>Incidentally, this approach is exactly what <code>liburing</code> does in a <a href="https://github.com/axboe/liburing/blob/fd3e498/test/waitid.c#L58">unit test</a>.</p>
|
|
<p>Alternatively, we can only queue the <code>waitid</code> and use <code>io_uring_wait_cqe_timeout</code> to mimick <code>poll(..., timeout)</code>:</p>
|
|
<pre>
|
|
<div class="code-header">
|
|
<span>C</span>
|
|
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
|
|
</div>
|
|
<code class="language-c"><span class="line-number"></span><span class="code-hl">#define _DEFAULT_SOURCE</span>
|
|
<span class="line-number"></span><span class="code-hl">#include <liburing.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <sys/wait.h></span>
|
|
<span class="line-number"></span><span class="code-hl">#include <unistd.h></span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl">int main(int argc, char *argv[]) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> (void)argc;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> struct io_uring ring = {0};</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (io_uring_queue_init(2, &ring,</span>
|
|
<span class="line-number"></span><span class="code-hl"> IORING_SETUP_SINGLE_ISSUER |</span>
|
|
<span class="line-number"></span><span class="code-hl"> IORING_SETUP_DEFER_TASKRUN) < 0) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return 1;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> uint32_t wait_ms = 128;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> for (int retry = 0; retry < 10; retry += 1) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> int child_pid = fork();</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == child_pid) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> if (0 == child_pid) { // Child</span>
|
|
<span class="line-number"></span><span class="code-hl"> argv += 1;</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (-1 == execvp(argv[0], argv)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return errno;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> __builtin_unreachable();</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> struct io_uring_sqe *sqe = NULL;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> // Queue `waitid`.</span>
|
|
<span class="line-number"></span><span class="code-hl"> sqe = io_uring_get_sqe(&ring);</span>
|
|
<span class="line-number"></span><span class="code-hl"> siginfo_t si = {0};</span>
|
|
<span class="line-number"></span><span class="code-hl"> io_uring_prep_waitid(sqe, P_PID, (id_t)child_pid, &si, WEXITED, 0);</span>
|
|
<span class="line-number"></span><span class="code-hl"> sqe->user_data = 1;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> io_uring_submit(&ring);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> struct __kernel_timespec ts = {</span>
|
|
<span class="line-number"></span><span class="code-hl"> .tv_sec = wait_ms / 1000,</span>
|
|
<span class="line-number"></span><span class="code-hl"> .tv_nsec = (wait_ms % 1000) * 1000 * 1000,</span>
|
|
<span class="line-number"></span><span class="code-hl"> };</span>
|
|
<span class="line-number"></span><span class="code-hl"> struct io_uring_cqe *cqe = NULL;</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> int ret = io_uring_wait_cqe_timeout(&ring, &cqe, &ts);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> // If child exited successfully: the end.</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (ret == 0 && cqe->res >= 0 && cqe->user_data == 1 &&</span>
|
|
<span class="line-number"></span><span class="code-hl"> WIFEXITED(si.si_status) && 0 == WEXITSTATUS(si.si_status)) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> return 0;</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> if (ret == 0) {</span>
|
|
<span class="line-number"></span><span class="code-hl"> io_uring_cqe_seen(&ring, cqe);</span>
|
|
<span class="line-number"></span><span class="code-hl"> } else {</span>
|
|
<span class="line-number"></span><span class="code-hl"> kill(child_pid, SIGKILL);</span>
|
|
<span class="line-number"></span><span class="code-hl"> // Drain the CQE.</span>
|
|
<span class="line-number"></span><span class="code-hl"> ret = io_uring_wait_cqe(&ring, &cqe);</span>
|
|
<span class="line-number"></span><span class="code-hl"> io_uring_cqe_seen(&ring, cqe);</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> wait(NULL);</span>
|
|
<span class="line-number"></span><span class="code-hl"></span>
|
|
<span class="line-number"></span><span class="code-hl"> wait_ms *= 2;</span>
|
|
<span class="line-number"></span><span class="code-hl"> usleep(wait_ms * 1000);</span>
|
|
<span class="line-number"></span><span class="code-hl"> }</span>
|
|
<span class="line-number"></span><span class="code-hl"> return 1;</span>
|
|
<span class="line-number"></span><span class="code-hl">}</span>
|
|
</code></pre>
|
|
<p>The only difficulty here is in case of timeout: we kill the child directly, and we need to consume and discard the <code>waitid</code> entry in the completion queue. Just like <code>kqueue</code>.</p>
|
|
<p>One caveat for io_uring: it's only supported on modern kernels (5.1+).</p>
|
|
<p>Another caveat: some cloud providers e.g. Google Cloud disable <code>io_uring</code> due to security concerns when running untrusted code. So it's not ubiquitous.</p>
|
|
<h2 id="eigth-approach-threads">
|
|
<a class="title" href="#eigth-approach-threads">Eigth approach: Threads</a>
|
|
<a class="hash-anchor" href="#eigth-approach-threads" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h2>
|
|
<p>Readers have <a href="https://news.ycombinator.com/vote?id=42107420&how=up&auth=20ac3216e63a60ca250d82b6a051d7dfaa9f18c9&goto=item%3Fid%3D42103200#42107420">pointed out</a> that threads are also a solution, albeit a suboptimal one. Here's the approach:</p>
|
|
<ol>
|
|
<li>Spawn a thread, it will be in charge of spawning the child process, storing the child PID in a global thread-safe variable (e.g. protected by a mutex). It then <code>wait</code>s on the child in a blocking way.</li>
|
|
<li>If the child exits, <code>wait</code> will return the status, which is also written in a global thread-safe variable, and the thread ends.</li>
|
|
<li>In the main thread, wait on the other thread with a timeout, e.g. with <code>pthread_timedjoin_np</code>.</li>
|
|
<li>If the child did not exit successfully, this is the same as usual: kill, wait, sleep, and retry.</li>
|
|
</ol>
|
|
<p>If the threads library supports returning a value from a thread, like <code>pthread</code> or C11 threads do, that could be used to return the exit status of the child to simplify the code a bit.</p>
|
|
<p>Also, we could make the thread spawning logic a bit more efficient by not spawning a new thread for each retry, if we wanted to. Instead, we communicate with the other thread with a queue or such to instruct it to spawn the child again. It's more complex though.</p>
|
|
<p>Now, this approach works but is kind of cumbersome (as noted by the readers), because threads interact in surprising ways with signals (yay, another thing to watch out for!) so we may have to set up signal masks to block/ignore some, and we must take care of not introducing data-races due to the global variables.</p>
|
|
<p>Unless the problem is embarassingly parallel and the threads share nothing (e.g.: dividing an array into pieces and each thread gets its own piece to work on), I am reminded of the adage: "You had two problems. You reach out for X. You now have 3 problems". And threads are often the X.</p>
|
|
<p>Still, it's a useful tool in the toolbox.</p>
|
|
<h2 id="ninth-approach-active-polling">
|
|
<a class="title" href="#ninth-approach-active-polling">Ninth approach: Active polling.</a>
|
|
<a class="hash-anchor" href="#ninth-approach-active-polling" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h2>
|
|
<p>That's looping in user code with micro-sleeping to actively poll on the child status in a non-blocking way, for example using <code>wait(..., WNOHANG)</code>. Unless you have a very bizzare use case and you know what you are doing, please do not do this. This is unnecessary, bad for power consumption, and all we achieve is noticing late that the child ended. This approach is just here for completeness.</p>
|
|
<h2 id="conclusion">
|
|
<a class="title" href="#conclusion">Conclusion</a>
|
|
<a class="hash-anchor" href="#conclusion" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h2>
|
|
<p>I find signals and spawning child process to be the hardest parts of Unix. Evidently this is not a rare opinion, looking at the development in these areas: process descriptors, the various expansions to the venerable <code>fork</code> with <code>vfork</code>, <code>clone</code>, <code>clone3</code>, <code>clone6</code>, a bazillion different ways to do I/O multiplexing, etc.</p>
|
|
<p>So what's the best approach then in a complex program? Let's recap:</p>
|
|
<ul>
|
|
<li>If you need maximum portability and are a Unix wizard, you can use <code>sigsuspend</code>.</li>
|
|
<li>If you are not afraid of signals, want a simpler API that still widely supported, and the use case is very specific (like ours), you can use <code>sigtimedwait</code>.</li>
|
|
<li>If you favor correctness and work with recent Linux and FreeBSD versions, you can use process descriptors with shims to get the same API on both OSes. That's probably my favorite option if it's applicable.</li>
|
|
<li>If you only care about MacOS and BSDs (or accept to use <code>libkqueue</code> on Linux), you can use <code>kqueue</code> because it works out of the box with PIDs, you avoid signals completely, and it's used in all the big libraries out of there e.g. <code>libuv</code>.</li>
|
|
<li>If you only care about bleeding edge Linux, are already using <code>io_uring</code> in your code, you can use <code>io_uring</code>.</li>
|
|
<li>If you only care about Linux and are afraid of using <code>io_uring</code>, you can use <code>signalfd</code> + <code>poll</code>.</li>
|
|
</ul>
|
|
<p>I often look at complex code and think: what are the chances that this is correct? What are the chances that I missed something? Is there a way to make it so simple that it is obviously correct? And how can I limit the blast of a bug I wrote? Will I understand this code in 3 months? When dealing with signals, I was constantly finding weird corner cases and timing issues leading to data races. You would not believe how many times I got my system completely frozen while writing this article, because I accidentally fork-bombed myself or simply forgot to reap zombie processes.</p>
|
|
<p>And to be fair to the OS developers that have to implement them: I do not think they did a bad job! I am sure it's super hard to implement! It's just that the whole concept and the available APIs are very easy to misuse. It's a good illustration of how a good API, the right abstraction, can enable great programs, and a poor API, the wrong abstraction, can be the root cause of various bugs in many programs for decades.</p>
|
|
<p>And OS developers have noticed and are working on new, better abstractions!</p>
|
|
<p>Process descriptors seem to me so straightforward, so obviously correct, that I would definitely favor them over signals. They simply remove entire classes of bugs. If these are not available to me, I would perhaps use <code>kqueue</code> instead (with <code>libkqueue</code> emulation when necessary), because it means my program can be extended easily to watch for over types of entities and I like that the API is very straightforward: one call to create the queue and one call to use it.</p>
|
|
<p>Finally, I regret that there is so much fragmentation across all operating systems. Perhaps <code>io_uring</code> will become more than a Linuxism and spread to Windows, MacOS, the BSDs, and illumos in the future?</p>
|
|
<h2 id="addendum-the-code">
|
|
<a class="title" href="#addendum-the-code">Addendum: The code</a>
|
|
<a class="hash-anchor" href="#addendum-the-code" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
|
|
</h2>
|
|
<p>The code is available <a href="https://github.com/gaultier/c/tree/master/ueb">here</a>. It does not have any dependencies except libc (well, and libkqueue for <code>kqueue.c</code> on Linux). All of these programs are in the worst case 27 KiB in size, with debug symbols enabled and linking statically to musl. They do not allocate any memory themselves.
|
|
For comparison, <a href="https://github.com/rye/eb">eb</a> has 24 dependencies and is 1.2 MiB! That's roughly 50x times more.</p>
|
|
<p><a href="/blog"> ⏴ Back to all articles</a></p>
|
|
|
|
<div id="donate">
|
|
<em>
|
|
<p>If you enjoy what you're reading, you want to support me, and can afford it: <a href="https://paypal.me/philigaultier?country.x=DE&locale.x=en_US">Support me</a>. That allows me to write more cool articles!</p>
|
|
|
|
<p>
|
|
This blog is <a href="https://github.com/gaultier/blog">open-source</a>!
|
|
If you find a problem, please open a Github issue.
|
|
The content of this blog as well as the code snippets are under the <a href="https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_(%22BSD_License_2.0%22,_%22Revised_BSD_License%22,_%22New_BSD_License%22,_or_%22Modified_BSD_License%22)">BSD-3 License</a> which I also usually use for all my personal projects. It's basically free for every use but you have to mention me as the original author.
|
|
</p>
|
|
<p>
|
|
My views are my own and not those of my employer.
|
|
</p>
|
|
</em>
|
|
</div>
|
|
|
|
</div>
|
|
</body>
|
|
</html>
|