Files
nexus/sreweekly/articles/441/05-what-s-the-big-deal-about-deterministic-simulation-testing.html
2026-09-12 17:23:01 +08:00

565 lines
47 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="canonical" href="https://notes.eatonphil.com/2024-08-20-deterministic-simulation-testing.html">
<title>What's the big deal about Deterministic Simulation Testing? | notes.eatonphil.com</title>
<meta name="description" content="What's the big deal about Deterministic Simulation Testing?" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<link rel="stylesheet" type="text/css" href="/style.css" />
<link rel="alternate" type="application/rss+xml" href="/rss.xml" />
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=IBM+Plex+Mono">
<script async onload="loadGA()" src="https://www.googletagmanager.com/gtag/js?id=UA-58109156-2"></script>
<script>
function loadGA() {
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-58109156-2');
}
</script>
<script defer src="https://cdn.usefathom.com/script.js" data-site="CEPUOLOQ"></script>
</head>
<body>
<header>
<div class="lfw">
<div class="container">
<div class="row">
<a href="https://eatonphil.com/2025-holiday-fundraiser.html">2025 Holiday Fundraiser</a>
</div>
</div>
</div>
<div class="container">
<div>
<div class="row">
<div>
<a href="https://eatonphil.com" class="sm-link">
Home
</a>
<a href="/" class="sm-link">
Blog
</a>
<a href="/rss.xml" class="sm-link">
RSS
</a>
</div>
<div class="subscribe">
<!-- <a href="https://eatonphil.com/hire-me.html">Hire me</a> -->
<!--
Hardcode link to home page because some pages don't
have a subscribe box. Some pages hide the subscribe.
-->
<a href="https://eatonphil.com/subscribe.html">
Subscribe
</a>
</div>
</div>
<hr />
<div class="">
<h2>August 20, 2024</h2>
<h1>What's the big deal about Deterministic Simulation Testing?</h1>
<div class="row" style="padding-bottom: 5px">
<div class="tags"><a href="/tags/testing.html" class="tag">testing</a><a href="/tags/dst.html" class="tag">dst</a><a href="/tags/databases.html" class="tag">databases</a><a href="/tags/distsys.html" class="tag">distsys</a></div>
</div>
</div>
</div>
</div>
</header>
<div class="container">
<div class="col-6">
<div class="post">
<p>Bugs in distributed systems are hard to find, largely because systems
interact in chaotic ways. And even once you've found a bug, it can be
anywhere from simple to impossible to reproduce it. It's about as far
away as you can get from the ideal test environment: property testing
a pure function.</p>
<p>But what if we could write our code in a way that we can isolate the
chaotic aspects of our distributed system during <i>testing</i>: run
multiple systems communicating with each other on a <i>single
thread</i> and control all randomness in each system? And property
test this single-threaded version of the distributed system with
controlled randomness, all the while injecting faults (fancy term for
unhappy path behavior like errors and latency) we might see in the
real-world?</p>
<p>Crazy as it sounds, people actually do this. It's called Deterministic
Simulation Testing (DST). And it's become more and more popular with
startups like FoundationDB, Antithesis, TigerBeetle, Polar Signals,
and WarpStream; as well as folks like Tyler Neely and Pekka Enberg,
talking about and making use of this technique.</p>
<p>It has become so popular to talk about DST in my corner of the world
that I worry it risks coming off sounding too magical and maybe a
little hyped. It's worth getting a better understanding of both the
benefits and the limitations.</p>
<p>Thank you to <a href="https://www.linkedin.com/in/alexmillerdb/">Alex Miller</a>
and <a href="https://www.linkedin.com/in/will-wilson-330276112/">Will Wilson</a>
for reviewing a version of this post.</p>
<h3 id="randomness-and-time">Randomness and time</h3><p>A big source of non-determinism in business logic is the use of random
numbers—in your code or your transitive dependencies or your language
runtime or your operating system.</p>
<p>Crucially, DST does not imply you can't have randomness! DST merely
assumes that you have a global seed for all randomness in your program
and that the simulator controls the seed. The seed may change across
runs of the simulator.</p>
<p>Once you observe a bad state as a result of running the simulation on
a random seed, you allow the user to enter the same seed again. This
allows the user to recreate the entire program run that led to that
observed bad state. Allows the user to debug the program trivially.</p>
<p>Another big source of non-determinism is being dependent on time. As
with randomness, DST does not mean you can't depend on time. DST means
you must be able to control the clock during the simulation.</p>
<p>To "control" randomness or time basically means you support dependency
injection, or the old-school alternative to dependency injection
called <i>passing the dependency as an explicit parameter</i>. Rather
than referring to a global clock or a global seed, you need to be able
to receive a clock or a seed from someone.</p>
<p>For example we might separate the operation of an application into the
language's <code>main()</code> entrypoint and an actual application <code>start()</code>
entrypoint.</p>
<div class="highlight"><pre><span></span><span class="c1"># app.pseudocode</span>
<span class="k">def</span> <span class="nf">start</span><span class="p">(</span><span class="n">clock</span><span class="p">,</span> <span class="n">seed</span><span class="p">):</span>
<span class="c1"># lots of business logic that might depend on time or do random things</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">:</span>
<span class="n">clock</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">clock</span><span class="p">()</span>
<span class="n">seed</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
<span class="n">app</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="n">clock</span><span class="p">,</span> <span class="n">seed</span><span class="p">)</span>
</pre></div>
<p>The application entrypoint is where we must be able to
swap out a real clock or real random seed for one controlled by our
simulator:</p>
<div class="highlight"><pre><span></span><span class="c1"># sim.pseudocode</span>
<span class="kn">import</span> <span class="s2">&quot;app.pseudocode&quot;</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">:</span>
<span class="n">sim_clock</span> <span class="o">=</span> <span class="n">make_sim_clock</span><span class="p">()</span>
<span class="n">sim_seed</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">DST_SEED</span> <span class="ow">or</span> <span class="n">time</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">app</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="n">sim_clock</span><span class="p">,</span> <span class="n">sim_seed</span><span class="p">)</span>
<span class="n">catch</span><span class="p">(</span><span class="n">e</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Bad execution at seed: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">sim_seed</span><span class="p">)</span>
<span class="n">throw</span> <span class="n">e</span>
</pre></div>
<p>Let's look at another example.</p>
<h3 id="converting-an-existing-function">Converting an existing function</h3><p>Let's say that we had a helper method that kept calling a function
until it succeeded, with backoff.</p>
<div class="highlight"><pre><span></span><span class="c1"># retry.pseudocode</span>
<span class="k">class</span> <span class="nc">Backoff</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">init</span><span class="p">:</span>
<span class="n">this</span><span class="o">.</span><span class="n">rnd</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">seed</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">now</span><span class="p">())</span>
<span class="n">this</span><span class="o">.</span><span class="n">tries</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">retry_backoff</span><span class="p">(</span><span class="n">f</span><span class="p">):</span>
<span class="k">while</span> <span class="n">this</span><span class="o">.</span><span class="n">tries</span> <span class="o">&lt;</span> <span class="mi">3</span><span class="p">:</span>
<span class="k">if</span> <span class="n">f</span><span class="p">():</span>
<span class="k">return</span>
<span class="k">await</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">this</span><span class="o">.</span><span class="n">rnd</span><span class="o">.</span><span class="n">gen</span><span class="p">())</span>
<span class="n">this</span><span class="o">.</span><span class="n">tries</span><span class="o">++</span>
</pre></div>
<p>There is a single source of nondeterminism here and it's where we
generate a seed. We could parameterize the seed, but since we want to
call <code>time.sleep()</code> and since in DST we control the time, we can just
parameterize <code>time</code>.</p>
<div class="highlight"><pre><span></span><span class="c1"># retry.psuedocode</span>
<span class="k">class</span> <span class="nc">Backoff</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">init</span><span class="p">(</span><span class="n">this</span><span class="p">,</span> <span class="n">time</span><span class="p">):</span>
<span class="n">this</span><span class="o">.</span><span class="n">time</span> <span class="o">=</span> <span class="n">time</span>
<span class="n">this</span><span class="o">.</span><span class="n">rnd</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">seed</span> <span class="o">=</span> <span class="n">this</span><span class="o">.</span><span class="n">time</span><span class="o">.</span><span class="n">now</span><span class="p">())</span>
<span class="n">this</span><span class="o">.</span><span class="n">tries</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">retry_backoff</span><span class="p">(</span><span class="n">this</span><span class="p">,</span> <span class="n">f</span><span class="p">):</span>
<span class="k">while</span> <span class="n">this</span><span class="o">.</span><span class="n">tries</span> <span class="o">&lt;</span> <span class="mi">3</span><span class="p">:</span>
<span class="k">if</span> <span class="n">f</span><span class="p">():</span>
<span class="k">return</span>
<span class="k">await</span> <span class="n">this</span><span class="o">.</span><span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">this</span><span class="o">.</span><span class="n">rnd</span><span class="o">.</span><span class="n">gen</span><span class="p">())</span>
<span class="n">this</span><span class="o">.</span><span class="n">tries</span><span class="o">++</span>
</pre></div>
<p>Now we can write a little simulator to test this:</p>
<div class="highlight"><pre><span></span><span class="c1"># sim.psuedocode</span>
<span class="kn">import</span> <span class="s2">&quot;retry.pseudocode&quot;</span>
<span class="n">sim_time</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">now</span><span class="p">:</span> <span class="mi">0</span>
<span class="n">sleep</span><span class="p">:</span> <span class="p">(</span><span class="n">ms</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
<span class="k">await</span> <span class="n">future</span><span class="o">.</span><span class="n">wait</span><span class="p">(</span><span class="n">ms</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">tick</span><span class="p">:</span> <span class="p">(</span><span class="n">ms</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="n">now</span> <span class="o">+=</span> <span class="n">ms</span>
<span class="p">}</span>
<span class="n">backoff</span> <span class="o">=</span> <span class="n">Backoff</span><span class="p">(</span><span class="n">sim_time</span><span class="p">)</span>
<span class="k">while</span> <span class="n">true</span><span class="p">:</span>
<span class="n">failures</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">f</span> <span class="o">=</span> <span class="p">()</span> <span class="o">=&gt;</span> <span class="p">{</span>
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">&gt;</span> <span class="mf">0.5</span><span class="p">:</span>
<span class="n">failures</span><span class="o">++</span>
<span class="k">return</span> <span class="n">false</span>
<span class="k">return</span> <span class="n">true</span>
<span class="p">}</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">while</span> <span class="n">sim_time</span><span class="o">.</span><span class="n">now</span> <span class="o">&lt;</span> <span class="mi">60</span><span class="nb">min</span><span class="p">:</span>
<span class="n">promise</span> <span class="o">=</span> <span class="n">backoff</span><span class="o">.</span><span class="n">retry_backoff</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="n">sim_time</span><span class="o">.</span><span class="n">tick</span><span class="p">(</span><span class="mi">1</span><span class="n">ms</span><span class="p">)</span>
<span class="k">if</span> <span class="n">promise</span><span class="o">.</span><span class="n">read</span><span class="p">():</span>
<span class="k">break</span>
<span class="n">assert_expect_failure_and_expected_time_elapse</span><span class="p">(</span><span class="n">sim_time</span><span class="p">,</span> <span class="n">failures</span><span class="p">)</span>
<span class="n">catch</span><span class="p">(</span><span class="n">e</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Found logical error with seed: </span><span class="si">%d</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">seed</span><span class="p">)</span>
<span class="n">throw</span> <span class="n">e</span>
</pre></div>
<p>This demonstrates a few critical aspects of DST. First, the simulator
itself depends on randomness. But allows the user to provide a seed so
they can replay a simulation that discovers a bug. The controlled
randomness in the simulator is what lets us do property testing.</p>
<p>Second, the simulation workload must be written by the user. Even when
you've got a platform like Antithesis that gives you an environment
for DST, it's up to you to exercise the application.</p>
<p>Now let's get a little more complex.</p>
<h3 id="a-single-thread-and-asynchronous-io">A single thread and asynchronous IO</h3><p>The determinism of multiple threads can only be controlled at the
operating system or emulator or hypervisor layer. Realistically, that
would require third-party systems like Antithesis or
<a href="https://github.com/facebookexperimental/hermit">Hermit</a> (which, don't
get excited, is not actively developed and hasn't worked on any
interesting program of mine) or <a href="https://rr-project.org/">rr</a>.</p>
<p>These systems transparently transform multi-threaded code into single
threaded code. But also note that Hermit and rr have only limited
ability to do fault injection which, in addition to deterministic
execution, is a goal of ours. And you can't run them on a mac. And
<a href="https://github.com/rr-debugger/rr/issues/1373">can't</a>
<a href="https://github.com/facebookexperimental/hermit?tab=readme-ov-file#support">run</a>
them on ARM.</p>
<p>But we can, and would like, to write a simulator without writing a new
operating system or emulator or hypervisor, and without a third-party
system. So we must limit ourselves to writing code that can be
collapsed into a single thread. Significantly, since using blocking IO
would mean an entire class of concurrency bugs could not be discovered
while running the simulator in a single thread, we must limit
ourselves to asynchronous IO.</p>
<p>Single threaded and asynchronous IO. These are already two big limitations.</p>
<p>Some languages like Go are entirely built around transparent
multi-threading and blocking IO. Polar Signals
<a href="https://www.polarsignals.com/blog/posts/2024/05/28/mostly-dst-in-go">solved</a>
this for DST by compiling their application to WASM where it would run
on a single thread. But that wasn't enough. Even on a single thread,
the Go runtime intentionally schedules goroutines randomly. So Polar
Signals forked the Go runtime to control this randomness with an
environment variable. That's kind of crazy. Resonate took <a href="https://github.com/resonatehq/resonate/blob/268c588e302f13187309e4b37636d19595d42fa1/internal/kernel/scheduler/coroutine.go">another
approach</a>
that also looks cumbersome. I'm not going to attempt to describe
it. Go seems like a difficult choice of a language if you want to do
DST.</p>
<p>Like Go, Rust has no builtin async IO. The most mature async IO
library is tokio. The tokio folks attempted to provide a
tokio-compatible <a href="https://github.com/tokio-rs/simulator">simulator</a>
implementation with all sources of nondeterminism removed. From what I
can tell, they did not at any point fully
<a href="https://github.com/tokio-rs/tokio/issues/1845">succeed</a>. That repo
has now been replaced with a "this is very experimental" tokio-rs
project called <a href="https://github.com/tokio-rs/turmoil">turmoil</a> that
provides deterministic execution plus network fault injection. (But
not disk fault injection. More on that later.) It isn't surprising
that it is difficult to provide deterministic execution for an IO
library that was not designed for it. tokio is a large project with
many transitive dependencies. They must all be combed for
non-determinism.</p>
<p>On the other hand, Pekka has <a href="https://github.com/penberg/hiisi/blob/main/hiisi-server/src/io/generic.rs">already
demonstrated</a>
for us how we might build a simpler Rust async IO library that is
designed to be simulation tested. He modeled this on the TigerBeetle
design King and I
<a href="https://tigerbeetle.com/blog/a-friendly-abstraction-over-iouring-and-kqueue">wrote</a>
about two years ago.</p>
<p>So let's sketch out a program that does buggy IO and let's look at how
we can apply DST to it.</p>
<div class="highlight"><pre><span></span><span class="c1"># readfile.pseudocode</span>
<span class="k">def</span> <span class="nf">read_file</span><span class="p">(</span><span class="n">io</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">into_buffer</span><span class="p">):</span>
<span class="n">f</span> <span class="o">=</span> <span class="k">await</span> <span class="n">io</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="n">read_buffer</span> <span class="o">=</span> <span class="p">[</span><span class="mi">4096</span><span class="p">]</span><span class="n">u8</span><span class="p">{}</span>
<span class="k">while</span> <span class="n">true</span><span class="p">:</span>
<span class="n">err</span><span class="p">,</span> <span class="n">n_read</span> <span class="o">=</span> <span class="k">await</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="o">&amp;</span><span class="n">read_buffer</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">==</span> <span class="n">io</span><span class="o">.</span><span class="n">EOF</span><span class="p">:</span>
<span class="n">into_buffer</span><span class="o">.</span><span class="n">copy_maybe_allocate</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">sizeof</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">)])</span>
<span class="k">return</span>
<span class="k">if</span> <span class="n">err</span><span class="p">:</span>
<span class="n">throw</span> <span class="n">err</span>
<span class="n">into_buffer</span><span class="o">.</span><span class="n">copy_maybe_allocate</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">sizeof</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">)])</span>
</pre></div>
<p>In our simulator, we will provide a mocked out IO system and we will
randomly inject various errors while asserting pre- and
post-conditions.</p>
<div class="highlight"><pre><span></span><span class="c1"># sim.psuedocode</span>
<span class="kn">import</span> <span class="s2">&quot;readfile.pseudocode&quot;</span>
<span class="n">seed</span> <span class="o">=</span> <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">DST_SEED</span> <span class="err">?</span> <span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">DST_SEED</span><span class="p">)</span> <span class="p">:</span> <span class="n">time</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
<span class="n">rnd</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">seed</span><span class="p">)</span>
<span class="k">while</span> <span class="n">true</span><span class="p">:</span>
<span class="n">sim_disk_data</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand_bytes</span><span class="p">(</span><span class="mi">10</span><span class="n">MB</span><span class="p">)</span>
<span class="n">sim_fd</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">pos</span><span class="p">:</span> <span class="mi">0</span>
<span class="n">EOF</span><span class="p">:</span> <span class="n">Error</span><span class="p">(</span><span class="s2">&quot;eof&quot;</span><span class="p">)</span>
<span class="n">read</span><span class="p">:</span> <span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="n">buf</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
<span class="n">partial_read</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand_in_range_inclusive</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">sizeof</span><span class="p">(</span><span class="n">buf</span><span class="p">))</span>
<span class="n">memcpy</span><span class="p">(</span><span class="n">sim_disk_data</span><span class="p">,</span> <span class="n">buf</span><span class="p">,</span> <span class="n">fd</span><span class="o">.</span><span class="n">pos</span><span class="p">,</span> <span class="n">partial_read</span><span class="p">)</span>
<span class="n">fd</span><span class="o">.</span><span class="n">pos</span> <span class="o">+=</span> <span class="n">partial_read</span>
<span class="k">if</span> <span class="n">fd</span><span class="o">.</span><span class="n">pos</span> <span class="o">==</span> <span class="n">sizeof</span><span class="p">(</span><span class="n">sim_disk_data</span><span class="p">):</span>
<span class="k">return</span> <span class="n">io</span><span class="o">.</span><span class="n">EOF</span><span class="p">,</span> <span class="n">partial_read</span>
<span class="k">return</span> <span class="n">partial_read</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">sim_io</span> <span class="o">=</span> <span class="p">{</span>
<span class="nb">open</span><span class="p">:</span> <span class="p">(</span><span class="n">filename</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="n">sim_fd</span>
<span class="p">}</span>
<span class="n">out_buf</span> <span class="o">=</span> <span class="n">Vector</span><span class="o">&lt;</span><span class="n">u8</span><span class="o">&gt;.</span><span class="n">new</span><span class="p">()</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">read_file</span><span class="p">(</span><span class="n">sim_io</span><span class="p">,</span> <span class="s2">&quot;somefile&quot;</span><span class="p">,</span> <span class="n">out_buf</span><span class="p">)</span>
<span class="n">assert_bytes_equal</span><span class="p">(</span><span class="n">out_buf</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">sim_disk_data</span><span class="p">)</span>
<span class="n">catch</span> <span class="p">(</span><span class="n">e</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Found logical error with seed: </span><span class="si">%d</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">seed</span><span class="p">)</span>
<span class="n">throw</span> <span class="n">e</span>
</pre></div>
<p>And with this simulator we would have eventually caught our partial
read bug! In our original program when we wrote:</p>
<div class="highlight"><pre><span></span> <span class="n">into_buffer</span><span class="o">.</span><span class="n">copy_maybe_allocate</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">sizeof</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">)])</span>
</pre></div>
<p>We should have written:</p>
<div class="highlight"><pre><span></span> <span class="n">into_buffer</span><span class="o">.</span><span class="n">copy_maybe_allocate</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">n_read</span><span class="p">])</span>
</pre></div>
<p>Great! Let's get a little more complex.</p>
<h3 id="a-distributed-system">A distributed system</h3><p>I already mentioned in the beginning that the gist of deterministic
simulation testing a distributed system is that you get all of the
nodes in the system to run in the same process. This would be
basically impossible if you wanted to test a system that involved your
application plus Kafka plus Postgres plus Redis. But if your system is
a self-contained distributed system, such as one that embeds a Raft
library for high availability of your application, you can actually
run multiple nodes into the same process!</p>
<p>For a system like this, our simulator might look like:</p>
<div class="highlight"><pre><span></span><span class="c1"># sim.pseudocode</span>
<span class="kn">import</span> <span class="s2">&quot;distsys-node.pseudocode&quot;</span>
<span class="n">seed</span> <span class="o">=</span> <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">DST_SEED</span> <span class="err">?</span> <span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">DST_SEED</span><span class="p">)</span> <span class="p">:</span> <span class="n">time</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
<span class="n">rnd</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">seed</span><span class="p">)</span>
<span class="k">while</span> <span class="n">true</span><span class="p">:</span>
<span class="n">sim_fd</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">send</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="n">buf</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
<span class="c1"># Inject random failure.</span>
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">&gt;</span> <span class="mf">.5</span><span class="p">:</span>
<span class="n">throw</span> <span class="n">Error</span><span class="p">(</span><span class="s1">&#39;bad write&#39;</span><span class="p">)</span>
<span class="c1"># Inject random latency.</span>
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">&gt;</span> <span class="mf">.5</span><span class="p">:</span>
<span class="k">await</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">())</span>
<span class="n">n_written</span> <span class="o">=</span> <span class="n">assert_ok</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">fd</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">buf</span><span class="p">))</span>
<span class="k">return</span> <span class="n">n_written</span>
<span class="p">},</span>
<span class="n">recv</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="n">buf</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
<span class="c1"># Inject random failure.</span>
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">&gt;</span> <span class="mf">.5</span><span class="p">:</span>
<span class="n">throw</span> <span class="n">Error</span><span class="p">(</span><span class="s1">&#39;bad read&#39;</span><span class="p">)</span>
<span class="c1"># Inject random latency.</span>
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">&gt;</span> <span class="mf">.5</span><span class="p">:</span>
<span class="k">await</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">())</span>
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">fd</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">sim_io</span> <span class="o">=</span> <span class="p">{</span>
<span class="nb">open</span><span class="p">:</span> <span class="p">(</span><span class="n">filename</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
<span class="c1"># Inject random failure.</span>
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">&gt;</span> <span class="mf">.5</span><span class="p">:</span>
<span class="n">throw</span> <span class="n">Error</span><span class="p">(</span><span class="s1">&#39;bad open&#39;</span><span class="p">)</span>
<span class="c1"># Inject random latency.</span>
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">&gt;</span> <span class="mf">.5</span><span class="p">:</span>
<span class="k">await</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">())</span>
<span class="k">return</span> <span class="n">sim_fd</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">all_ports</span> <span class="o">=</span> <span class="p">[</span><span class="mi">6000</span><span class="p">,</span> <span class="mi">6001</span><span class="p">,</span> <span class="mi">6002</span><span class="p">]</span>
<span class="n">nodes</span> <span class="o">=</span> <span class="p">[</span>
<span class="k">await</span> <span class="n">distsys</span><span class="o">-</span><span class="n">node</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="n">sim_io</span><span class="p">,</span> <span class="n">all_ports</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">all_ports</span><span class="p">),</span>
<span class="k">await</span> <span class="n">distsys</span><span class="o">-</span><span class="n">node</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="n">sim_io</span><span class="p">,</span> <span class="n">all_ports</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">all_ports</span><span class="p">),</span>
<span class="k">await</span> <span class="n">distsys</span><span class="o">-</span><span class="n">node</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="n">sim_io</span><span class="p">,</span> <span class="n">all_ports</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span> <span class="n">all_ports</span><span class="p">),</span>
<span class="p">]</span>
<span class="n">history</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand_bytes</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="n">value</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand_bytes</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="n">nodes</span><span class="p">[</span><span class="n">rnd</span><span class="o">.</span><span class="n">rand_in_range_inclusive</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">nodes</span><span class="p">)]</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="n">history</span><span class="o">.</span><span class="n">add</span><span class="p">((</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">))</span>
<span class="n">assert_valid_history</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">history</span><span class="p">)</span>
<span class="c1"># Crash a process every so often</span>
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">&gt;</span> <span class="mf">0.75</span><span class="p">:</span>
<span class="n">node</span> <span class="o">=</span> <span class="n">nodes</span><span class="p">[</span><span class="n">rnd</span><span class="o">.</span><span class="n">rand_in_range_inclusive</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">)]</span>
<span class="n">node</span><span class="o">.</span><span class="n">restart</span><span class="p">()</span>
<span class="n">catch</span> <span class="p">(</span><span class="n">e</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Found logical error with seed: </span><span class="si">%d</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">seed</span><span class="p">)</span>
<span class="n">throw</span> <span class="n">e</span>
</pre></div>
<p>I'm completely hand waving here to demonstrate the broader point and
not any specific testing strategy for a specific distributed
system. The important points are that these three nodes run in the
same process, on different ports.</p>
<p>We control disk IO. We control network IO. We control how time
elapses. We run a deterministic simulated workload against the three
node system while injecting disk, network, and process faults.</p>
<p>And we are constantly checking for an invalid state. When we get the
invalid state, we can be sure the user can easily recreate this
invalid state.</p>
<h3 id="other-sources-of-non-determinism">Other sources of non-determinism</h3><p>Within some error margin, most CPU instructions and most CPU behavior are
considered to be deterministic. There are, however, certain CPU
instructions that are <a href="https://cs.stackexchange.com/questions/132842/under-which-conditions-a-given-program-is-deterministic-on-x86-64-machines/132856#132856">definitely
not</a>. Unfortunately
that might
<a href="https://github.com/facebookexperimental/hermit/issues/34">include</a>
system calls. It might also
<a href="https://stackoverflow.com/a/8171032">include</a> malloc. There is very
little to trust.</p>
<p>If we <a href="https://antithesis.com/blog/deterministic_hypervisor/">ignore</a>
Antithesis, people doing DST seem not to worry about these smaller
bits of nondeterminism. Yet it's generally agreed that DST is still
worthwhile anyway. The intuition here is that every bit of
non-determinism you can eliminate makes it that much easier to
reproduce bugs when you find them.</p>
<p>Put another way: determinism, even among DST practitioners, remains a spectrum.</p>
<h3 id="considerations">Considerations</h3><p>As you may have noticed already from some of the pseudocode, DST is not a panacea.</p>
<h4 id="consideration-1:-edges">Consideration 1: Edges</h4><p>First, because you must swap out non-deterministic parts of your code,
you are not actually testing the entirety of your code. You are
certainly encouraged to keep the deterministic kernel large. But there
will always be the non-deterministic edges.</p>
<p>Without a system like Antithesis which gives you an entire
deterministic machine, you can't test your whole program.</p>
<p>But even with Antithesis you cannot test the <i>integration</i> between your
system and external systems. You must mock out the external systems.</p>
<p>It's also worth noting that there are many areas where you could
inject simulation. You could do it at a high-level RPC and storage
layer. This would be simpler and easier to understand. But then you'd
be omitting testing and error-handling of lower-level errors.</p>
<h4 id="consideration-2:-your-workload(s)">Consideration 2: Your workload(s)</h4><p>DST is dependent on your creativity and thoroughness of your workload
as much as any other type of test or benchmark.</p>
<p>Just as you wouldn't depend on one single benchmark to qualify your
application, you may not want to depend on a single simulated
workload.</p>
<p>Or as Will Wilson put it for me:</p>
<blockquote><p>The biggest challenge of DST in my experience is that tuning all the
random distributions, the parameters of your system, the workload,
the fault injection, etc. so that it produces interesting behavior
is very challenging and very labor intensive. As with fuzzing or
PBT, it's terrifyingly easy to build a DST system that appears to be
doing a ton of testing, but actually never explores very much of the
state space of your system. At FoundationDB, the vast majority of
the work we put into the simulator was an iterative process of
hunting for what wasn't being covered by our tests and then figuring
out how to make the tests better. This process often resembles
science more than it does engineering.</p>
<p>Unfortunately, unlike with fuzzing, mere branch coverage in your
code is usually a pretty poor signal for the kinds of systems you
want to test with DST. At Antithesis we handle this with <a
href="https://antithesis.com/docs/best_practices/sometimes_assertions.html">Sometimes
assertions</a>, at FDB we did something pretty similar, and I assume
TigerBeetle and others have their own version of this. But of course
the ultimate figure of merit is whether your DST system is finding
100% of your bugs. It's quite difficult to get to the point that it
does. The truly ambitious part of Antithesis isn't the hypervisor,
but the fact that we also aim to solve the much harder "is my DST
working?" problem with minimal human guidance or supervision.</p>
</blockquote>
<h4 id="consideration-3:-your-knowledge-of-what-you-mocked">Consideration 3: Your knowledge of what you mocked</h4><p>When you mock out the behavior of disk or network IO, the benefits of
DST are tied to your understanding of the spectrum of behavior that
may happen in the real world.</p>
<p>What are all possible error conditions? What are the extreme latency
bounds of the original method? What about corruption or misdirected
IO?</p>
<p>The flipside here is that only in deterministic simulation testing can
you configure these crazy scenarios to happen at a <i>configurable
regularity</i>. You can kick off a set of runs that have especially
high IO latency or especially high corrupt reads/writes. Joran and I
<a href="https://tigerbeetle.com/blog/2023-07-11-we-put-a-distributed-database-in-the-browser">wrote</a>
a year ago about how the TigerBeetle simulator does exactly this.</p>
<h4 id="consideration-4:-non-reproducible-seeds-as-code-changes">Consideration 4: Non-reproducible seeds as code changes</h4><p>Critically, the reproducibility of DST only helps so long as your <i>code
doesn't change</i>. As soon as your code changes, the seed may no longer
even get you to the state where the bug was exhibited. So the
reproducibility of DST means more that it may help you convert the
seed simulation run into an integration test that describes the
precise scenario even as the code changes.</p>
<h4 id="consideration-5:-time-and-compute">Consideration 5: Time and compute</h4><p>Because of Consideration 4, you need to keep rerunning the simulator
not just to keep finding new seeds and new histories but because the
new seeds and new histories may change every time you make changes to
code.</p>
<h3 id="what-about-jepsen?">What about Jepsen?</h3><p>Jepsen does limited process and network fault injection while testing
for linearizability. It's a fantastic project.</p>
<p>However, it represents only a subset of what is possible with
Deterministic Simulation Testing (if you actually put in the effort
described above to get there).</p>
<p>But even more importantly, Jepsen has nothing to do with deterministic
execution. If Jepsen finds a bug and your system can't do
deterministic execution, you may or may not be able to reproduce that
Jepsen bug.</p>
<p>Here's another Will Wilson
<a href="https://antithesis.com/blog/is_something_bugging_you/">quote</a> for you
on Jepsen and FoundationDB:</p>
<blockquote><p>Anyway, we did [Deterministic Simulation Testing] for a while and
found all of the bugs in the database. I know, I know, that’s an
insane thing to say. It’s kind of true though. In the entire history
of the company, I think we only ever had one or two bugs reported by
a customer. Ever. Kyle Kingsbury aka “aphyr” didn’t even bother
testing it with Jepsen, because he didn’t think he’d find anything.</p>
</blockquote>
<h3 id="conclusion">Conclusion</h3><p>The degree to which you can place faith in DST alone, and not time
spent in production, has limits. However, it certainly does no harm to
employ DST. And, barring the considerations described above, will
likely make the kernel of your product significantly more
stable. Furthermore, everyone who uses DST knows about these
considerations. But I think it's worthwhile to list them out to help
folks who do not know DST to build an intuition for what it's
excellent at.</p>
<p>Further reading:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=4fFDFbi3toc">"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson</a></li>
<li><a href="https://www.polarsignals.com/blog/posts/2024/05/28/mostly-dst-in-go">(Mostly) Deterministic Simulation Testing in Go</a></li>
<li><a href="https://github.com/madsim-rs/madsim">Magical Deterministic Simulator for distributed systems in Rust</a></li>
</ul>
<p><blockquote class="twitter-tweet"><p lang="en" dir="ltr">I wrote a new post talking through the basics, considerations, and limitations of Deterministic Simulation Testing.<a href="https://t.co/9Fp5ytL7Wz">https://t.co/9Fp5ytL7Wz</a> <a href="https://t.co/xRE6FOwc0P">pic.twitter.com/xRE6FOwc0P</a></p>&mdash; Phil Eaton (@eatonphil) <a href="https://twitter.com/eatonphil/status/1825851204632445377?ref_src=twsrc%5Etfw">August 20, 2024</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></p>
<style>.feedback{display:initial;}</style>
</div>
<div class="feedback">
<h4>Feedback</h4>
<p>As always,
please <a href="mailto:phil@eatonphil.com">email</a>
or <a href="https://twitter.com/eatonphil">tweet me</a>
with questions, corrections, or ideas!</p>
</div>
</div>
</div>
<footer>
<div class="container">
<div>
<div id="subscribe">
<iframe frameBorder="0" src="https://cdn.forms-content-1.sg-form.com/e8ed4c3b-dd46-11f0-b6e6-ce075f2c5f80"></iframe>
</div>
</div>
</div>
</footer>
</body>
</html>