565 lines
47 KiB
HTML
565 lines
47 KiB
HTML
<!DOCTYPE html>
|
||
<html>
|
||
<head>
|
||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
||
<link rel="canonical" href="https://notes.eatonphil.com/2024-08-20-deterministic-simulation-testing.html">
|
||
<title>What's the big deal about Deterministic Simulation Testing? | notes.eatonphil.com</title>
|
||
<meta name="description" content="What's the big deal about Deterministic Simulation Testing?" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||
<link rel="stylesheet" type="text/css" href="/style.css" />
|
||
<link rel="alternate" type="application/rss+xml" href="/rss.xml" />
|
||
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=IBM+Plex+Mono">
|
||
|
||
<script async onload="loadGA()" src="https://www.googletagmanager.com/gtag/js?id=UA-58109156-2"></script>
|
||
<script>
|
||
function loadGA() {
|
||
window.dataLayer = window.dataLayer || [];
|
||
function gtag(){dataLayer.push(arguments);}
|
||
gtag('js', new Date());
|
||
gtag('config', 'UA-58109156-2');
|
||
}
|
||
</script>
|
||
<script defer src="https://cdn.usefathom.com/script.js" data-site="CEPUOLOQ"></script>
|
||
</head>
|
||
<body>
|
||
<header>
|
||
<div class="lfw">
|
||
<div class="container">
|
||
<div class="row">
|
||
<a href="https://eatonphil.com/2025-holiday-fundraiser.html">2025 Holiday Fundraiser</a>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="container">
|
||
<div>
|
||
<div class="row">
|
||
<div>
|
||
<a href="https://eatonphil.com" class="sm-link">
|
||
Home
|
||
</a>
|
||
<a href="/" class="sm-link">
|
||
Blog
|
||
</a>
|
||
<a href="/rss.xml" class="sm-link">
|
||
RSS
|
||
</a>
|
||
</div>
|
||
|
||
<div class="subscribe">
|
||
<!-- <a href="https://eatonphil.com/hire-me.html">Hire me</a> -->
|
||
<!--
|
||
Hardcode link to home page because some pages don't
|
||
have a subscribe box. Some pages hide the subscribe.
|
||
-->
|
||
<a href="https://eatonphil.com/subscribe.html">
|
||
Subscribe
|
||
</a>
|
||
</div>
|
||
</div>
|
||
<hr />
|
||
<div class="">
|
||
<h2>August 20, 2024</h2>
|
||
<h1>What's the big deal about Deterministic Simulation Testing?</h1>
|
||
|
||
<div class="row" style="padding-bottom: 5px">
|
||
<div class="tags"><a href="/tags/testing.html" class="tag">testing</a><a href="/tags/dst.html" class="tag">dst</a><a href="/tags/databases.html" class="tag">databases</a><a href="/tags/distsys.html" class="tag">distsys</a></div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</header>
|
||
<div class="container">
|
||
<div class="col-6">
|
||
<div class="post">
|
||
<p>Bugs in distributed systems are hard to find, largely because systems
|
||
interact in chaotic ways. And even once you've found a bug, it can be
|
||
anywhere from simple to impossible to reproduce it. It's about as far
|
||
away as you can get from the ideal test environment: property testing
|
||
a pure function.</p>
|
||
<p>But what if we could write our code in a way that we can isolate the
|
||
chaotic aspects of our distributed system during <i>testing</i>: run
|
||
multiple systems communicating with each other on a <i>single
|
||
thread</i> and control all randomness in each system? And property
|
||
test this single-threaded version of the distributed system with
|
||
controlled randomness, all the while injecting faults (fancy term for
|
||
unhappy path behavior like errors and latency) we might see in the
|
||
real-world?</p>
|
||
<p>Crazy as it sounds, people actually do this. It's called Deterministic
|
||
Simulation Testing (DST). And it's become more and more popular with
|
||
startups like FoundationDB, Antithesis, TigerBeetle, Polar Signals,
|
||
and WarpStream; as well as folks like Tyler Neely and Pekka Enberg,
|
||
talking about and making use of this technique.</p>
|
||
<p>It has become so popular to talk about DST in my corner of the world
|
||
that I worry it risks coming off sounding too magical and maybe a
|
||
little hyped. It's worth getting a better understanding of both the
|
||
benefits and the limitations.</p>
|
||
<p>Thank you to <a href="https://www.linkedin.com/in/alexmillerdb/">Alex Miller</a>
|
||
and <a href="https://www.linkedin.com/in/will-wilson-330276112/">Will Wilson</a>
|
||
for reviewing a version of this post.</p>
|
||
<h3 id="randomness-and-time">Randomness and time</h3><p>A big source of non-determinism in business logic is the use of random
|
||
numbers—in your code or your transitive dependencies or your language
|
||
runtime or your operating system.</p>
|
||
<p>Crucially, DST does not imply you can't have randomness! DST merely
|
||
assumes that you have a global seed for all randomness in your program
|
||
and that the simulator controls the seed. The seed may change across
|
||
runs of the simulator.</p>
|
||
<p>Once you observe a bad state as a result of running the simulation on
|
||
a random seed, you allow the user to enter the same seed again. This
|
||
allows the user to recreate the entire program run that led to that
|
||
observed bad state. Allows the user to debug the program trivially.</p>
|
||
<p>Another big source of non-determinism is being dependent on time. As
|
||
with randomness, DST does not mean you can't depend on time. DST means
|
||
you must be able to control the clock during the simulation.</p>
|
||
<p>To "control" randomness or time basically means you support dependency
|
||
injection, or the old-school alternative to dependency injection
|
||
called <i>passing the dependency as an explicit parameter</i>. Rather
|
||
than referring to a global clock or a global seed, you need to be able
|
||
to receive a clock or a seed from someone.</p>
|
||
<p>For example we might separate the operation of an application into the
|
||
language's <code>main()</code> entrypoint and an actual application <code>start()</code>
|
||
entrypoint.</p>
|
||
<div class="highlight"><pre><span></span><span class="c1"># app.pseudocode</span>
|
||
|
||
<span class="k">def</span> <span class="nf">start</span><span class="p">(</span><span class="n">clock</span><span class="p">,</span> <span class="n">seed</span><span class="p">):</span>
|
||
<span class="c1"># lots of business logic that might depend on time or do random things</span>
|
||
|
||
<span class="k">def</span> <span class="nf">main</span><span class="p">:</span>
|
||
<span class="n">clock</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">clock</span><span class="p">()</span>
|
||
<span class="n">seed</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
|
||
<span class="n">app</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="n">clock</span><span class="p">,</span> <span class="n">seed</span><span class="p">)</span>
|
||
</pre></div>
|
||
<p>The application entrypoint is where we must be able to
|
||
swap out a real clock or real random seed for one controlled by our
|
||
simulator:</p>
|
||
<div class="highlight"><pre><span></span><span class="c1"># sim.pseudocode</span>
|
||
|
||
<span class="kn">import</span> <span class="s2">"app.pseudocode"</span>
|
||
|
||
<span class="k">def</span> <span class="nf">main</span><span class="p">:</span>
|
||
<span class="n">sim_clock</span> <span class="o">=</span> <span class="n">make_sim_clock</span><span class="p">()</span>
|
||
<span class="n">sim_seed</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">DST_SEED</span> <span class="ow">or</span> <span class="n">time</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="n">app</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="n">sim_clock</span><span class="p">,</span> <span class="n">sim_seed</span><span class="p">)</span>
|
||
<span class="n">catch</span><span class="p">(</span><span class="n">e</span><span class="p">):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Bad execution at seed: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">sim_seed</span><span class="p">)</span>
|
||
<span class="n">throw</span> <span class="n">e</span>
|
||
</pre></div>
|
||
<p>Let's look at another example.</p>
|
||
<h3 id="converting-an-existing-function">Converting an existing function</h3><p>Let's say that we had a helper method that kept calling a function
|
||
until it succeeded, with backoff.</p>
|
||
<div class="highlight"><pre><span></span><span class="c1"># retry.pseudocode</span>
|
||
<span class="k">class</span> <span class="nc">Backoff</span><span class="p">:</span>
|
||
<span class="k">def</span> <span class="nf">init</span><span class="p">:</span>
|
||
<span class="n">this</span><span class="o">.</span><span class="n">rnd</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">seed</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">now</span><span class="p">())</span>
|
||
<span class="n">this</span><span class="o">.</span><span class="n">tries</span> <span class="o">=</span> <span class="mi">0</span>
|
||
|
||
<span class="k">async</span> <span class="k">def</span> <span class="nf">retry_backoff</span><span class="p">(</span><span class="n">f</span><span class="p">):</span>
|
||
<span class="k">while</span> <span class="n">this</span><span class="o">.</span><span class="n">tries</span> <span class="o"><</span> <span class="mi">3</span><span class="p">:</span>
|
||
<span class="k">if</span> <span class="n">f</span><span class="p">():</span>
|
||
<span class="k">return</span>
|
||
|
||
<span class="k">await</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">this</span><span class="o">.</span><span class="n">rnd</span><span class="o">.</span><span class="n">gen</span><span class="p">())</span>
|
||
<span class="n">this</span><span class="o">.</span><span class="n">tries</span><span class="o">++</span>
|
||
</pre></div>
|
||
<p>There is a single source of nondeterminism here and it's where we
|
||
generate a seed. We could parameterize the seed, but since we want to
|
||
call <code>time.sleep()</code> and since in DST we control the time, we can just
|
||
parameterize <code>time</code>.</p>
|
||
<div class="highlight"><pre><span></span><span class="c1"># retry.psuedocode</span>
|
||
<span class="k">class</span> <span class="nc">Backoff</span><span class="p">:</span>
|
||
<span class="k">def</span> <span class="nf">init</span><span class="p">(</span><span class="n">this</span><span class="p">,</span> <span class="n">time</span><span class="p">):</span>
|
||
<span class="n">this</span><span class="o">.</span><span class="n">time</span> <span class="o">=</span> <span class="n">time</span>
|
||
<span class="n">this</span><span class="o">.</span><span class="n">rnd</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">seed</span> <span class="o">=</span> <span class="n">this</span><span class="o">.</span><span class="n">time</span><span class="o">.</span><span class="n">now</span><span class="p">())</span>
|
||
<span class="n">this</span><span class="o">.</span><span class="n">tries</span> <span class="o">=</span> <span class="mi">0</span>
|
||
|
||
<span class="k">async</span> <span class="k">def</span> <span class="nf">retry_backoff</span><span class="p">(</span><span class="n">this</span><span class="p">,</span> <span class="n">f</span><span class="p">):</span>
|
||
<span class="k">while</span> <span class="n">this</span><span class="o">.</span><span class="n">tries</span> <span class="o"><</span> <span class="mi">3</span><span class="p">:</span>
|
||
<span class="k">if</span> <span class="n">f</span><span class="p">():</span>
|
||
<span class="k">return</span>
|
||
|
||
<span class="k">await</span> <span class="n">this</span><span class="o">.</span><span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">this</span><span class="o">.</span><span class="n">rnd</span><span class="o">.</span><span class="n">gen</span><span class="p">())</span>
|
||
<span class="n">this</span><span class="o">.</span><span class="n">tries</span><span class="o">++</span>
|
||
</pre></div>
|
||
<p>Now we can write a little simulator to test this:</p>
|
||
<div class="highlight"><pre><span></span><span class="c1"># sim.psuedocode</span>
|
||
<span class="kn">import</span> <span class="s2">"retry.pseudocode"</span>
|
||
|
||
<span class="n">sim_time</span> <span class="o">=</span> <span class="p">{</span>
|
||
<span class="n">now</span><span class="p">:</span> <span class="mi">0</span>
|
||
<span class="n">sleep</span><span class="p">:</span> <span class="p">(</span><span class="n">ms</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
|
||
<span class="k">await</span> <span class="n">future</span><span class="o">.</span><span class="n">wait</span><span class="p">(</span><span class="n">ms</span><span class="p">)</span>
|
||
<span class="p">}</span>
|
||
<span class="n">tick</span><span class="p">:</span> <span class="p">(</span><span class="n">ms</span><span class="p">)</span> <span class="o">=></span> <span class="n">now</span> <span class="o">+=</span> <span class="n">ms</span>
|
||
<span class="p">}</span>
|
||
|
||
<span class="n">backoff</span> <span class="o">=</span> <span class="n">Backoff</span><span class="p">(</span><span class="n">sim_time</span><span class="p">)</span>
|
||
|
||
<span class="k">while</span> <span class="n">true</span><span class="p">:</span>
|
||
<span class="n">failures</span> <span class="o">=</span> <span class="mi">0</span>
|
||
<span class="n">f</span> <span class="o">=</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
|
||
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">></span> <span class="mf">0.5</span><span class="p">:</span>
|
||
<span class="n">failures</span><span class="o">++</span>
|
||
<span class="k">return</span> <span class="n">false</span>
|
||
|
||
<span class="k">return</span> <span class="n">true</span>
|
||
<span class="p">}</span>
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="k">while</span> <span class="n">sim_time</span><span class="o">.</span><span class="n">now</span> <span class="o"><</span> <span class="mi">60</span><span class="nb">min</span><span class="p">:</span>
|
||
<span class="n">promise</span> <span class="o">=</span> <span class="n">backoff</span><span class="o">.</span><span class="n">retry_backoff</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
|
||
<span class="n">sim_time</span><span class="o">.</span><span class="n">tick</span><span class="p">(</span><span class="mi">1</span><span class="n">ms</span><span class="p">)</span>
|
||
<span class="k">if</span> <span class="n">promise</span><span class="o">.</span><span class="n">read</span><span class="p">():</span>
|
||
<span class="k">break</span>
|
||
|
||
<span class="n">assert_expect_failure_and_expected_time_elapse</span><span class="p">(</span><span class="n">sim_time</span><span class="p">,</span> <span class="n">failures</span><span class="p">)</span>
|
||
<span class="n">catch</span><span class="p">(</span><span class="n">e</span><span class="p">):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Found logical error with seed: </span><span class="si">%d</span><span class="s2">"</span><span class="p">,</span> <span class="n">seed</span><span class="p">)</span>
|
||
<span class="n">throw</span> <span class="n">e</span>
|
||
</pre></div>
|
||
<p>This demonstrates a few critical aspects of DST. First, the simulator
|
||
itself depends on randomness. But allows the user to provide a seed so
|
||
they can replay a simulation that discovers a bug. The controlled
|
||
randomness in the simulator is what lets us do property testing.</p>
|
||
<p>Second, the simulation workload must be written by the user. Even when
|
||
you've got a platform like Antithesis that gives you an environment
|
||
for DST, it's up to you to exercise the application.</p>
|
||
<p>Now let's get a little more complex.</p>
|
||
<h3 id="a-single-thread-and-asynchronous-io">A single thread and asynchronous IO</h3><p>The determinism of multiple threads can only be controlled at the
|
||
operating system or emulator or hypervisor layer. Realistically, that
|
||
would require third-party systems like Antithesis or
|
||
<a href="https://github.com/facebookexperimental/hermit">Hermit</a> (which, don't
|
||
get excited, is not actively developed and hasn't worked on any
|
||
interesting program of mine) or <a href="https://rr-project.org/">rr</a>.</p>
|
||
<p>These systems transparently transform multi-threaded code into single
|
||
threaded code. But also note that Hermit and rr have only limited
|
||
ability to do fault injection which, in addition to deterministic
|
||
execution, is a goal of ours. And you can't run them on a mac. And
|
||
<a href="https://github.com/rr-debugger/rr/issues/1373">can't</a>
|
||
<a href="https://github.com/facebookexperimental/hermit?tab=readme-ov-file#support">run</a>
|
||
them on ARM.</p>
|
||
<p>But we can, and would like, to write a simulator without writing a new
|
||
operating system or emulator or hypervisor, and without a third-party
|
||
system. So we must limit ourselves to writing code that can be
|
||
collapsed into a single thread. Significantly, since using blocking IO
|
||
would mean an entire class of concurrency bugs could not be discovered
|
||
while running the simulator in a single thread, we must limit
|
||
ourselves to asynchronous IO.</p>
|
||
<p>Single threaded and asynchronous IO. These are already two big limitations.</p>
|
||
<p>Some languages like Go are entirely built around transparent
|
||
multi-threading and blocking IO. Polar Signals
|
||
<a href="https://www.polarsignals.com/blog/posts/2024/05/28/mostly-dst-in-go">solved</a>
|
||
this for DST by compiling their application to WASM where it would run
|
||
on a single thread. But that wasn't enough. Even on a single thread,
|
||
the Go runtime intentionally schedules goroutines randomly. So Polar
|
||
Signals forked the Go runtime to control this randomness with an
|
||
environment variable. That's kind of crazy. Resonate took <a href="https://github.com/resonatehq/resonate/blob/268c588e302f13187309e4b37636d19595d42fa1/internal/kernel/scheduler/coroutine.go">another
|
||
approach</a>
|
||
that also looks cumbersome. I'm not going to attempt to describe
|
||
it. Go seems like a difficult choice of a language if you want to do
|
||
DST.</p>
|
||
<p>Like Go, Rust has no builtin async IO. The most mature async IO
|
||
library is tokio. The tokio folks attempted to provide a
|
||
tokio-compatible <a href="https://github.com/tokio-rs/simulator">simulator</a>
|
||
implementation with all sources of nondeterminism removed. From what I
|
||
can tell, they did not at any point fully
|
||
<a href="https://github.com/tokio-rs/tokio/issues/1845">succeed</a>. That repo
|
||
has now been replaced with a "this is very experimental" tokio-rs
|
||
project called <a href="https://github.com/tokio-rs/turmoil">turmoil</a> that
|
||
provides deterministic execution plus network fault injection. (But
|
||
not disk fault injection. More on that later.) It isn't surprising
|
||
that it is difficult to provide deterministic execution for an IO
|
||
library that was not designed for it. tokio is a large project with
|
||
many transitive dependencies. They must all be combed for
|
||
non-determinism.</p>
|
||
<p>On the other hand, Pekka has <a href="https://github.com/penberg/hiisi/blob/main/hiisi-server/src/io/generic.rs">already
|
||
demonstrated</a>
|
||
for us how we might build a simpler Rust async IO library that is
|
||
designed to be simulation tested. He modeled this on the TigerBeetle
|
||
design King and I
|
||
<a href="https://tigerbeetle.com/blog/a-friendly-abstraction-over-iouring-and-kqueue">wrote</a>
|
||
about two years ago.</p>
|
||
<p>So let's sketch out a program that does buggy IO and let's look at how
|
||
we can apply DST to it.</p>
|
||
<div class="highlight"><pre><span></span><span class="c1"># readfile.pseudocode</span>
|
||
<span class="k">def</span> <span class="nf">read_file</span><span class="p">(</span><span class="n">io</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">into_buffer</span><span class="p">):</span>
|
||
<span class="n">f</span> <span class="o">=</span> <span class="k">await</span> <span class="n">io</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
|
||
<span class="n">read_buffer</span> <span class="o">=</span> <span class="p">[</span><span class="mi">4096</span><span class="p">]</span><span class="n">u8</span><span class="p">{}</span>
|
||
<span class="k">while</span> <span class="n">true</span><span class="p">:</span>
|
||
<span class="n">err</span><span class="p">,</span> <span class="n">n_read</span> <span class="o">=</span> <span class="k">await</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="o">&</span><span class="n">read_buffer</span><span class="p">)</span>
|
||
<span class="k">if</span> <span class="n">err</span> <span class="o">==</span> <span class="n">io</span><span class="o">.</span><span class="n">EOF</span><span class="p">:</span>
|
||
<span class="n">into_buffer</span><span class="o">.</span><span class="n">copy_maybe_allocate</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">sizeof</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">)])</span>
|
||
<span class="k">return</span>
|
||
|
||
<span class="k">if</span> <span class="n">err</span><span class="p">:</span>
|
||
<span class="n">throw</span> <span class="n">err</span>
|
||
|
||
<span class="n">into_buffer</span><span class="o">.</span><span class="n">copy_maybe_allocate</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">sizeof</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">)])</span>
|
||
</pre></div>
|
||
<p>In our simulator, we will provide a mocked out IO system and we will
|
||
randomly inject various errors while asserting pre- and
|
||
post-conditions.</p>
|
||
<div class="highlight"><pre><span></span><span class="c1"># sim.psuedocode</span>
|
||
<span class="kn">import</span> <span class="s2">"readfile.pseudocode"</span>
|
||
|
||
<span class="n">seed</span> <span class="o">=</span> <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">DST_SEED</span> <span class="err">?</span> <span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">DST_SEED</span><span class="p">)</span> <span class="p">:</span> <span class="n">time</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
|
||
<span class="n">rnd</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">seed</span><span class="p">)</span>
|
||
|
||
<span class="k">while</span> <span class="n">true</span><span class="p">:</span>
|
||
<span class="n">sim_disk_data</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand_bytes</span><span class="p">(</span><span class="mi">10</span><span class="n">MB</span><span class="p">)</span>
|
||
<span class="n">sim_fd</span> <span class="o">=</span> <span class="p">{</span>
|
||
<span class="n">pos</span><span class="p">:</span> <span class="mi">0</span>
|
||
<span class="n">EOF</span><span class="p">:</span> <span class="n">Error</span><span class="p">(</span><span class="s2">"eof"</span><span class="p">)</span>
|
||
<span class="n">read</span><span class="p">:</span> <span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="n">buf</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
|
||
<span class="n">partial_read</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand_in_range_inclusive</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">sizeof</span><span class="p">(</span><span class="n">buf</span><span class="p">))</span>
|
||
<span class="n">memcpy</span><span class="p">(</span><span class="n">sim_disk_data</span><span class="p">,</span> <span class="n">buf</span><span class="p">,</span> <span class="n">fd</span><span class="o">.</span><span class="n">pos</span><span class="p">,</span> <span class="n">partial_read</span><span class="p">)</span>
|
||
<span class="n">fd</span><span class="o">.</span><span class="n">pos</span> <span class="o">+=</span> <span class="n">partial_read</span>
|
||
<span class="k">if</span> <span class="n">fd</span><span class="o">.</span><span class="n">pos</span> <span class="o">==</span> <span class="n">sizeof</span><span class="p">(</span><span class="n">sim_disk_data</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="n">io</span><span class="o">.</span><span class="n">EOF</span><span class="p">,</span> <span class="n">partial_read</span>
|
||
<span class="k">return</span> <span class="n">partial_read</span>
|
||
<span class="p">}</span>
|
||
<span class="p">}</span>
|
||
<span class="n">sim_io</span> <span class="o">=</span> <span class="p">{</span>
|
||
<span class="nb">open</span><span class="p">:</span> <span class="p">(</span><span class="n">filename</span><span class="p">)</span> <span class="o">=></span> <span class="n">sim_fd</span>
|
||
<span class="p">}</span>
|
||
|
||
<span class="n">out_buf</span> <span class="o">=</span> <span class="n">Vector</span><span class="o"><</span><span class="n">u8</span><span class="o">>.</span><span class="n">new</span><span class="p">()</span>
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="n">read_file</span><span class="p">(</span><span class="n">sim_io</span><span class="p">,</span> <span class="s2">"somefile"</span><span class="p">,</span> <span class="n">out_buf</span><span class="p">)</span>
|
||
<span class="n">assert_bytes_equal</span><span class="p">(</span><span class="n">out_buf</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">sim_disk_data</span><span class="p">)</span>
|
||
<span class="n">catch</span> <span class="p">(</span><span class="n">e</span><span class="p">):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Found logical error with seed: </span><span class="si">%d</span><span class="s2">"</span><span class="p">,</span> <span class="n">seed</span><span class="p">)</span>
|
||
<span class="n">throw</span> <span class="n">e</span>
|
||
</pre></div>
|
||
<p>And with this simulator we would have eventually caught our partial
|
||
read bug! In our original program when we wrote:</p>
|
||
<div class="highlight"><pre><span></span> <span class="n">into_buffer</span><span class="o">.</span><span class="n">copy_maybe_allocate</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">sizeof</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">)])</span>
|
||
</pre></div>
|
||
<p>We should have written:</p>
|
||
<div class="highlight"><pre><span></span> <span class="n">into_buffer</span><span class="o">.</span><span class="n">copy_maybe_allocate</span><span class="p">(</span><span class="n">read_buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">n_read</span><span class="p">])</span>
|
||
</pre></div>
|
||
<p>Great! Let's get a little more complex.</p>
|
||
<h3 id="a-distributed-system">A distributed system</h3><p>I already mentioned in the beginning that the gist of deterministic
|
||
simulation testing a distributed system is that you get all of the
|
||
nodes in the system to run in the same process. This would be
|
||
basically impossible if you wanted to test a system that involved your
|
||
application plus Kafka plus Postgres plus Redis. But if your system is
|
||
a self-contained distributed system, such as one that embeds a Raft
|
||
library for high availability of your application, you can actually
|
||
run multiple nodes into the same process!</p>
|
||
<p>For a system like this, our simulator might look like:</p>
|
||
<div class="highlight"><pre><span></span><span class="c1"># sim.pseudocode</span>
|
||
<span class="kn">import</span> <span class="s2">"distsys-node.pseudocode"</span>
|
||
|
||
<span class="n">seed</span> <span class="o">=</span> <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">DST_SEED</span> <span class="err">?</span> <span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">DST_SEED</span><span class="p">)</span> <span class="p">:</span> <span class="n">time</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
|
||
<span class="n">rnd</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">seed</span><span class="p">)</span>
|
||
|
||
<span class="k">while</span> <span class="n">true</span><span class="p">:</span>
|
||
<span class="n">sim_fd</span> <span class="o">=</span> <span class="p">{</span>
|
||
<span class="n">send</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="n">buf</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
|
||
<span class="c1"># Inject random failure.</span>
|
||
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">></span> <span class="mf">.5</span><span class="p">:</span>
|
||
<span class="n">throw</span> <span class="n">Error</span><span class="p">(</span><span class="s1">'bad write'</span><span class="p">)</span>
|
||
|
||
<span class="c1"># Inject random latency.</span>
|
||
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">></span> <span class="mf">.5</span><span class="p">:</span>
|
||
<span class="k">await</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">())</span>
|
||
|
||
<span class="n">n_written</span> <span class="o">=</span> <span class="n">assert_ok</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">fd</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">buf</span><span class="p">))</span>
|
||
<span class="k">return</span> <span class="n">n_written</span>
|
||
<span class="p">},</span>
|
||
<span class="n">recv</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="n">buf</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
|
||
<span class="c1"># Inject random failure.</span>
|
||
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">></span> <span class="mf">.5</span><span class="p">:</span>
|
||
<span class="n">throw</span> <span class="n">Error</span><span class="p">(</span><span class="s1">'bad read'</span><span class="p">)</span>
|
||
|
||
<span class="c1"># Inject random latency.</span>
|
||
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">></span> <span class="mf">.5</span><span class="p">:</span>
|
||
<span class="k">await</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">())</span>
|
||
|
||
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">fd</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span>
|
||
<span class="p">}</span>
|
||
<span class="p">}</span>
|
||
<span class="n">sim_io</span> <span class="o">=</span> <span class="p">{</span>
|
||
<span class="nb">open</span><span class="p">:</span> <span class="p">(</span><span class="n">filename</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
|
||
<span class="c1"># Inject random failure.</span>
|
||
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">></span> <span class="mf">.5</span><span class="p">:</span>
|
||
<span class="n">throw</span> <span class="n">Error</span><span class="p">(</span><span class="s1">'bad open'</span><span class="p">)</span>
|
||
|
||
<span class="c1"># Inject random latency.</span>
|
||
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">></span> <span class="mf">.5</span><span class="p">:</span>
|
||
<span class="k">await</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">())</span>
|
||
|
||
<span class="k">return</span> <span class="n">sim_fd</span>
|
||
<span class="p">}</span>
|
||
<span class="p">}</span>
|
||
|
||
<span class="n">all_ports</span> <span class="o">=</span> <span class="p">[</span><span class="mi">6000</span><span class="p">,</span> <span class="mi">6001</span><span class="p">,</span> <span class="mi">6002</span><span class="p">]</span>
|
||
<span class="n">nodes</span> <span class="o">=</span> <span class="p">[</span>
|
||
<span class="k">await</span> <span class="n">distsys</span><span class="o">-</span><span class="n">node</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="n">sim_io</span><span class="p">,</span> <span class="n">all_ports</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">all_ports</span><span class="p">),</span>
|
||
<span class="k">await</span> <span class="n">distsys</span><span class="o">-</span><span class="n">node</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="n">sim_io</span><span class="p">,</span> <span class="n">all_ports</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">all_ports</span><span class="p">),</span>
|
||
<span class="k">await</span> <span class="n">distsys</span><span class="o">-</span><span class="n">node</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="n">sim_io</span><span class="p">,</span> <span class="n">all_ports</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span> <span class="n">all_ports</span><span class="p">),</span>
|
||
<span class="p">]</span>
|
||
<span class="n">history</span> <span class="o">=</span> <span class="p">[]</span>
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="n">key</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand_bytes</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
|
||
<span class="n">value</span> <span class="o">=</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand_bytes</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
|
||
<span class="n">nodes</span><span class="p">[</span><span class="n">rnd</span><span class="o">.</span><span class="n">rand_in_range_inclusive</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">nodes</span><span class="p">)]</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
|
||
<span class="n">history</span><span class="o">.</span><span class="n">add</span><span class="p">((</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">))</span>
|
||
<span class="n">assert_valid_history</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">history</span><span class="p">)</span>
|
||
|
||
<span class="c1"># Crash a process every so often</span>
|
||
<span class="k">if</span> <span class="n">rnd</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">></span> <span class="mf">0.75</span><span class="p">:</span>
|
||
<span class="n">node</span> <span class="o">=</span> <span class="n">nodes</span><span class="p">[</span><span class="n">rnd</span><span class="o">.</span><span class="n">rand_in_range_inclusive</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">)]</span>
|
||
<span class="n">node</span><span class="o">.</span><span class="n">restart</span><span class="p">()</span>
|
||
<span class="n">catch</span> <span class="p">(</span><span class="n">e</span><span class="p">):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Found logical error with seed: </span><span class="si">%d</span><span class="s2">"</span><span class="p">,</span> <span class="n">seed</span><span class="p">)</span>
|
||
<span class="n">throw</span> <span class="n">e</span>
|
||
</pre></div>
|
||
<p>I'm completely hand waving here to demonstrate the broader point and
|
||
not any specific testing strategy for a specific distributed
|
||
system. The important points are that these three nodes run in the
|
||
same process, on different ports.</p>
|
||
<p>We control disk IO. We control network IO. We control how time
|
||
elapses. We run a deterministic simulated workload against the three
|
||
node system while injecting disk, network, and process faults.</p>
|
||
<p>And we are constantly checking for an invalid state. When we get the
|
||
invalid state, we can be sure the user can easily recreate this
|
||
invalid state.</p>
|
||
<h3 id="other-sources-of-non-determinism">Other sources of non-determinism</h3><p>Within some error margin, most CPU instructions and most CPU behavior are
|
||
considered to be deterministic. There are, however, certain CPU
|
||
instructions that are <a href="https://cs.stackexchange.com/questions/132842/under-which-conditions-a-given-program-is-deterministic-on-x86-64-machines/132856#132856">definitely
|
||
not</a>. Unfortunately
|
||
that might
|
||
<a href="https://github.com/facebookexperimental/hermit/issues/34">include</a>
|
||
system calls. It might also
|
||
<a href="https://stackoverflow.com/a/8171032">include</a> malloc. There is very
|
||
little to trust.</p>
|
||
<p>If we <a href="https://antithesis.com/blog/deterministic_hypervisor/">ignore</a>
|
||
Antithesis, people doing DST seem not to worry about these smaller
|
||
bits of nondeterminism. Yet it's generally agreed that DST is still
|
||
worthwhile anyway. The intuition here is that every bit of
|
||
non-determinism you can eliminate makes it that much easier to
|
||
reproduce bugs when you find them.</p>
|
||
<p>Put another way: determinism, even among DST practitioners, remains a spectrum.</p>
|
||
<h3 id="considerations">Considerations</h3><p>As you may have noticed already from some of the pseudocode, DST is not a panacea.</p>
|
||
<h4 id="consideration-1:-edges">Consideration 1: Edges</h4><p>First, because you must swap out non-deterministic parts of your code,
|
||
you are not actually testing the entirety of your code. You are
|
||
certainly encouraged to keep the deterministic kernel large. But there
|
||
will always be the non-deterministic edges.</p>
|
||
<p>Without a system like Antithesis which gives you an entire
|
||
deterministic machine, you can't test your whole program.</p>
|
||
<p>But even with Antithesis you cannot test the <i>integration</i> between your
|
||
system and external systems. You must mock out the external systems.</p>
|
||
<p>It's also worth noting that there are many areas where you could
|
||
inject simulation. You could do it at a high-level RPC and storage
|
||
layer. This would be simpler and easier to understand. But then you'd
|
||
be omitting testing and error-handling of lower-level errors.</p>
|
||
<h4 id="consideration-2:-your-workload(s)">Consideration 2: Your workload(s)</h4><p>DST is dependent on your creativity and thoroughness of your workload
|
||
as much as any other type of test or benchmark.</p>
|
||
<p>Just as you wouldn't depend on one single benchmark to qualify your
|
||
application, you may not want to depend on a single simulated
|
||
workload.</p>
|
||
<p>Or as Will Wilson put it for me:</p>
|
||
<blockquote><p>The biggest challenge of DST in my experience is that tuning all the
|
||
random distributions, the parameters of your system, the workload,
|
||
the fault injection, etc. so that it produces interesting behavior
|
||
is very challenging and very labor intensive. As with fuzzing or
|
||
PBT, it's terrifyingly easy to build a DST system that appears to be
|
||
doing a ton of testing, but actually never explores very much of the
|
||
state space of your system. At FoundationDB, the vast majority of
|
||
the work we put into the simulator was an iterative process of
|
||
hunting for what wasn't being covered by our tests and then figuring
|
||
out how to make the tests better. This process often resembles
|
||
science more than it does engineering.</p>
|
||
<p>Unfortunately, unlike with fuzzing, mere branch coverage in your
|
||
code is usually a pretty poor signal for the kinds of systems you
|
||
want to test with DST. At Antithesis we handle this with <a
|
||
href="https://antithesis.com/docs/best_practices/sometimes_assertions.html">Sometimes
|
||
assertions</a>, at FDB we did something pretty similar, and I assume
|
||
TigerBeetle and others have their own version of this. But of course
|
||
the ultimate figure of merit is whether your DST system is finding
|
||
100% of your bugs. It's quite difficult to get to the point that it
|
||
does. The truly ambitious part of Antithesis isn't the hypervisor,
|
||
but the fact that we also aim to solve the much harder "is my DST
|
||
working?" problem with minimal human guidance or supervision.</p>
|
||
</blockquote>
|
||
<h4 id="consideration-3:-your-knowledge-of-what-you-mocked">Consideration 3: Your knowledge of what you mocked</h4><p>When you mock out the behavior of disk or network IO, the benefits of
|
||
DST are tied to your understanding of the spectrum of behavior that
|
||
may happen in the real world.</p>
|
||
<p>What are all possible error conditions? What are the extreme latency
|
||
bounds of the original method? What about corruption or misdirected
|
||
IO?</p>
|
||
<p>The flipside here is that only in deterministic simulation testing can
|
||
you configure these crazy scenarios to happen at a <i>configurable
|
||
regularity</i>. You can kick off a set of runs that have especially
|
||
high IO latency or especially high corrupt reads/writes. Joran and I
|
||
<a href="https://tigerbeetle.com/blog/2023-07-11-we-put-a-distributed-database-in-the-browser">wrote</a>
|
||
a year ago about how the TigerBeetle simulator does exactly this.</p>
|
||
<h4 id="consideration-4:-non-reproducible-seeds-as-code-changes">Consideration 4: Non-reproducible seeds as code changes</h4><p>Critically, the reproducibility of DST only helps so long as your <i>code
|
||
doesn't change</i>. As soon as your code changes, the seed may no longer
|
||
even get you to the state where the bug was exhibited. So the
|
||
reproducibility of DST means more that it may help you convert the
|
||
seed simulation run into an integration test that describes the
|
||
precise scenario even as the code changes.</p>
|
||
<h4 id="consideration-5:-time-and-compute">Consideration 5: Time and compute</h4><p>Because of Consideration 4, you need to keep rerunning the simulator
|
||
not just to keep finding new seeds and new histories but because the
|
||
new seeds and new histories may change every time you make changes to
|
||
code.</p>
|
||
<h3 id="what-about-jepsen?">What about Jepsen?</h3><p>Jepsen does limited process and network fault injection while testing
|
||
for linearizability. It's a fantastic project.</p>
|
||
<p>However, it represents only a subset of what is possible with
|
||
Deterministic Simulation Testing (if you actually put in the effort
|
||
described above to get there).</p>
|
||
<p>But even more importantly, Jepsen has nothing to do with deterministic
|
||
execution. If Jepsen finds a bug and your system can't do
|
||
deterministic execution, you may or may not be able to reproduce that
|
||
Jepsen bug.</p>
|
||
<p>Here's another Will Wilson
|
||
<a href="https://antithesis.com/blog/is_something_bugging_you/">quote</a> for you
|
||
on Jepsen and FoundationDB:</p>
|
||
<blockquote><p>Anyway, we did [Deterministic Simulation Testing] for a while and
|
||
found all of the bugs in the database. I know, I know, that’s an
|
||
insane thing to say. It’s kind of true though. In the entire history
|
||
of the company, I think we only ever had one or two bugs reported by
|
||
a customer. Ever. Kyle Kingsbury aka “aphyr” didn’t even bother
|
||
testing it with Jepsen, because he didn’t think he’d find anything.</p>
|
||
</blockquote>
|
||
<h3 id="conclusion">Conclusion</h3><p>The degree to which you can place faith in DST alone, and not time
|
||
spent in production, has limits. However, it certainly does no harm to
|
||
employ DST. And, barring the considerations described above, will
|
||
likely make the kernel of your product significantly more
|
||
stable. Furthermore, everyone who uses DST knows about these
|
||
considerations. But I think it's worthwhile to list them out to help
|
||
folks who do not know DST to build an intuition for what it's
|
||
excellent at.</p>
|
||
<p>Further reading:</p>
|
||
<ul>
|
||
<li><a href="https://www.youtube.com/watch?v=4fFDFbi3toc">"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson</a></li>
|
||
<li><a href="https://www.polarsignals.com/blog/posts/2024/05/28/mostly-dst-in-go">(Mostly) Deterministic Simulation Testing in Go</a></li>
|
||
<li><a href="https://github.com/madsim-rs/madsim">Magical Deterministic Simulator for distributed systems in Rust</a></li>
|
||
</ul>
|
||
<p><blockquote class="twitter-tweet"><p lang="en" dir="ltr">I wrote a new post talking through the basics, considerations, and limitations of Deterministic Simulation Testing.<a href="https://t.co/9Fp5ytL7Wz">https://t.co/9Fp5ytL7Wz</a> <a href="https://t.co/xRE6FOwc0P">pic.twitter.com/xRE6FOwc0P</a></p>— Phil Eaton (@eatonphil) <a href="https://twitter.com/eatonphil/status/1825851204632445377?ref_src=twsrc%5Etfw">August 20, 2024</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></p>
|
||
<style>.feedback{display:initial;}</style>
|
||
</div>
|
||
<div class="feedback">
|
||
<h4>Feedback</h4>
|
||
<p>As always,
|
||
please <a href="mailto:phil@eatonphil.com">email</a>
|
||
or <a href="https://twitter.com/eatonphil">tweet me</a>
|
||
with questions, corrections, or ideas!</p>
|
||
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<footer>
|
||
<div class="container">
|
||
<div>
|
||
<div id="subscribe">
|
||
<iframe frameBorder="0" src="https://cdn.forms-content-1.sg-form.com/e8ed4c3b-dd46-11f0-b6e6-ce075f2c5f80"></iframe>
|
||
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</footer>
|
||
</body>
|
||
</html>
|