1065 lines
142 KiB
HTML
1065 lines
142 KiB
HTML
<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
|
<link rel="canonical" href="https://notes.eatonphil.com/2024-09-29-build-a-serverless-acid-database-with-this-one-neat-trick.html">
|
|
<title>Build a serverless ACID database with this one neat trick (atomic PutIfAbsent) | notes.eatonphil.com</title>
|
|
<meta name="description" content="Build a serverless ACID database with this one neat trick (atomic PutIfAbsent)" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
|
<link rel="stylesheet" type="text/css" href="/style.css" />
|
|
<link rel="alternate" type="application/rss+xml" href="/rss.xml" />
|
|
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=IBM+Plex+Mono">
|
|
|
|
<script async onload="loadGA()" src="https://www.googletagmanager.com/gtag/js?id=UA-58109156-2"></script>
|
|
<script>
|
|
function loadGA() {
|
|
window.dataLayer = window.dataLayer || [];
|
|
function gtag(){dataLayer.push(arguments);}
|
|
gtag('js', new Date());
|
|
gtag('config', 'UA-58109156-2');
|
|
}
|
|
</script>
|
|
<script defer src="https://cdn.usefathom.com/script.js" data-site="CEPUOLOQ"></script>
|
|
</head>
|
|
<body>
|
|
<header>
|
|
<div class="lfw">
|
|
<div class="container">
|
|
<div class="row">
|
|
<a href="https://eatonphil.com/2025-holiday-fundraiser.html">2025 Holiday Fundraiser</a>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div class="container">
|
|
<div>
|
|
<div class="row">
|
|
<div>
|
|
<a href="https://eatonphil.com" class="sm-link">
|
|
Home
|
|
</a>
|
|
<a href="/" class="sm-link">
|
|
Blog
|
|
</a>
|
|
<a href="/rss.xml" class="sm-link">
|
|
RSS
|
|
</a>
|
|
</div>
|
|
|
|
<div class="subscribe">
|
|
<!-- <a href="https://eatonphil.com/hire-me.html">Hire me</a> -->
|
|
<!--
|
|
Hardcode link to home page because some pages don't
|
|
have a subscribe box. Some pages hide the subscribe.
|
|
-->
|
|
<a href="https://eatonphil.com/subscribe.html">
|
|
Subscribe
|
|
</a>
|
|
</div>
|
|
</div>
|
|
<hr />
|
|
<div class="">
|
|
<h2>September 29, 2024</h2>
|
|
<h1>Build a serverless ACID database with this one neat trick (atomic PutIfAbsent)</h1>
|
|
|
|
<div class="row" style="padding-bottom: 5px">
|
|
<div class="tags"><a href="/tags/transactions.html" class="tag">transactions</a><a href="/tags/databases.html" class="tag">databases</a><a href="/tags/go.html" class="tag">go</a><a href="/tags/delta-lake.html" class="tag">delta-lake</a></div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</header>
|
|
<div class="container">
|
|
<div class="col-6">
|
|
<div class="post">
|
|
<p>Delta Lake is an open protocol for serverless ACID databases. Due to
|
|
its simplicity, scalability, and the number of open-source
|
|
implementations, it's quickly becoming the DuckDB of serverless
|
|
transactional databases for analytics workloads. Iceberg is a
|
|
contender too, and is similar in many ways. But since Delta Lake is
|
|
simpler (simple != better) that's where we'll focus in this post.</p>
|
|
<p>Delta Lake has one of the most accessible database papers I've read
|
|
(<a href="https://www.vldb.org/pvldb/vol13/p3411-armbrust.pdf">link</a>). It's
|
|
kind of like the
|
|
<a href="https://github.com/xoreaxeaxeax/movfuscator">movfuscator</a> of
|
|
databases.</p>
|
|
<p>Thanks to its simplicity, in this post we'll implement a Delta
|
|
Lake-inspired serverless ACID database in 500 lines of Go code with
|
|
zero dependencies. It will support creating tables, inserting rows
|
|
into a table, and scanning all rows in a table. All while allowing
|
|
concurrent readers and writers and achieving <a href="https://jepsen.io/consistency">snapshot
|
|
isolation</a>.</p>
|
|
<p>There are other critical parts of Delta Lake we'll ignore: updating
|
|
rows, deleting rows, checkpointing the transaction metadata log,
|
|
compaction, and probably much more I'm not aware of. We must start
|
|
somewhere.</p>
|
|
<p>All code for this post is <a href="https://github.com/eatonphil/otf">available on GitHub</a>.</p>
|
|
<h3 id="delta-lake-basics">Delta Lake basics</h3><p>Delta Lake writes immutable data files to blob storage. It stores the
|
|
names of new data files for a transaction in a metadata file. It
|
|
handles concurrency (i.e. achieves snapshot isolation) with an atomic
|
|
PutIfAbsent operation on the metadata file for the transaction.</p>
|
|
<p>This method of concurrency control works because the metadata files
|
|
follow a naming scheme that includes the transaction id in the file
|
|
name. When a new transaction starts, it finds all existing metadata
|
|
files and picks its own transaction id by adding 1 to the largest
|
|
transaction id it sees.</p>
|
|
<p>When a transaction goes to commit, writing the metadata file will
|
|
fail if another transaction has already picked the same transaction
|
|
id.</p>
|
|
<p>If a transaction does no writes and creates no tables, the transaction
|
|
does not attempt to write any metadata file. Snapshot isolation!</p>
|
|
<p>Let's dig into the implementation.</p>
|
|
<h3 id="boilerplate">Boilerplate</h3><p>Let's give ourselves some nice assertion methods, a debug method, and
|
|
a uuid generator. In <code>main.go</code>:</p>
|
|
<div class="highlight"><pre><span></span><span class="kn">package</span><span class="w"> </span><span class="nx">main</span>
|
|
|
|
<span class="kn">import</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="s">"encoding/json"</span>
|
|
<span class="w"> </span><span class="s">"fmt"</span>
|
|
<span class="w"> </span><span class="s">"io"</span>
|
|
<span class="w"> </span><span class="s">"os"</span>
|
|
<span class="w"> </span><span class="s">"path"</span>
|
|
<span class="w"> </span><span class="s">"slices"</span>
|
|
<span class="w"> </span><span class="s">"strings"</span>
|
|
<span class="p">)</span>
|
|
|
|
<span class="kd">func</span><span class="w"> </span><span class="nx">assert</span><span class="p">(</span><span class="nx">b</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span><span class="w"> </span><span class="nx">msg</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">!</span><span class="nx">b</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">msg</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="kd">func</span><span class="w"> </span><span class="nx">assertEq</span><span class="p">[</span><span class="nx">C</span><span class="w"> </span><span class="kt">comparable</span><span class="p">](</span><span class="nx">a</span><span class="w"> </span><span class="nx">C</span><span class="p">,</span><span class="w"> </span><span class="nx">b</span><span class="w"> </span><span class="nx">C</span><span class="p">,</span><span class="w"> </span><span class="nx">prefix</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">a</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="nx">b</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"%s '%v' != '%v'"</span><span class="p">,</span><span class="w"> </span><span class="nx">prefix</span><span class="p">,</span><span class="w"> </span><span class="nx">a</span><span class="p">,</span><span class="w"> </span><span class="nx">b</span><span class="p">))</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="kd">var</span><span class="w"> </span><span class="nx">DEBUG</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">slices</span><span class="p">.</span><span class="nx">Contains</span><span class="p">(</span><span class="nx">os</span><span class="p">.</span><span class="nx">Args</span><span class="p">,</span><span class="w"> </span><span class="s">"--debug"</span><span class="p">)</span>
|
|
|
|
<span class="kd">func</span><span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="nx">a</span><span class="w"> </span><span class="o">...</span><span class="kt">any</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">!</span><span class="nx">DEBUG</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">args</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">append</span><span class="p">([]</span><span class="kt">any</span><span class="p">{</span><span class="s">"[DEBUG]"</span><span class="p">},</span><span class="w"> </span><span class="nx">a</span><span class="o">...</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="nx">args</span><span class="o">...</span><span class="p">)</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="c1">// https://datatracker.ietf.org/doc/html/rfc4122#section-4.4</span>
|
|
<span class="kd">func</span><span class="w"> </span><span class="nx">uuidv4</span><span class="p">()</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">f</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Open</span><span class="p">(</span><span class="s">"/dev/random"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">assert</span><span class="p">(</span><span class="nx">err</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"could not open /dev/random: %s"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">))</span>
|
|
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
|
|
|
|
<span class="w"> </span><span class="nx">buf</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span><span class="w"> </span><span class="mi">16</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">n</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nx">Read</span><span class="p">(</span><span class="nx">buf</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">assert</span><span class="p">(</span><span class="nx">err</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"could not read 16 bytes from /dev/random: %s"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">))</span>
|
|
<span class="w"> </span><span class="nx">assert</span><span class="p">(</span><span class="nx">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">buf</span><span class="p">),</span><span class="w"> </span><span class="s">"expected 16 bytes from /dev/random"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Set bit 6 to 0</span>
|
|
<span class="w"> </span><span class="nx">buf</span><span class="p">[</span><span class="mi">8</span><span class="p">]</span><span class="w"> </span><span class="o">&=</span><span class="w"> </span><span class="p">^(</span><span class="nb">byte</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="mi">6</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="c1">// Set bit 7 to 1</span>
|
|
<span class="w"> </span><span class="nx">buf</span><span class="p">[</span><span class="mi">8</span><span class="p">]</span><span class="w"> </span><span class="o">|=</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="mi">7</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Set version</span>
|
|
<span class="w"> </span><span class="nx">buf</span><span class="p">[</span><span class="mi">6</span><span class="p">]</span><span class="w"> </span><span class="o">&=</span><span class="w"> </span><span class="p">^(</span><span class="nb">byte</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="mi">4</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">buf</span><span class="p">[</span><span class="mi">6</span><span class="p">]</span><span class="w"> </span><span class="o">&=</span><span class="w"> </span><span class="p">^(</span><span class="nb">byte</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="mi">5</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">buf</span><span class="p">[</span><span class="mi">6</span><span class="p">]</span><span class="w"> </span><span class="o">|=</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="mi">6</span>
|
|
<span class="w"> </span><span class="nx">buf</span><span class="p">[</span><span class="mi">6</span><span class="p">]</span><span class="w"> </span><span class="o">&=</span><span class="w"> </span><span class="p">^(</span><span class="nb">byte</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="mi">7</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"%x-%x-%x-%x-%x"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="nx">buf</span><span class="p">[:</span><span class="mi">4</span><span class="p">],</span>
|
|
<span class="w"> </span><span class="nx">buf</span><span class="p">[</span><span class="mi">4</span><span class="p">:</span><span class="mi">6</span><span class="p">],</span>
|
|
<span class="w"> </span><span class="nx">buf</span><span class="p">[</span><span class="mi">6</span><span class="p">:</span><span class="mi">8</span><span class="p">],</span>
|
|
<span class="w"> </span><span class="nx">buf</span><span class="p">[</span><span class="mi">8</span><span class="p">:</span><span class="mi">10</span><span class="p">],</span>
|
|
<span class="w"> </span><span class="nx">buf</span><span class="p">[</span><span class="mi">10</span><span class="p">:</span><span class="mi">16</span><span class="p">])</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>Is that uuid method correct? Hopefully. Efficient? No. But it's
|
|
preferable to avoid dependencies in pedagogical projects.</p>
|
|
<p>Moving on.</p>
|
|
<h3 id="blob-storage-requirements">Blob storage requirements</h3><p>As mentioned above, the basic requirement is that we support
|
|
atomically writing some bytes to a location if the location doesn't
|
|
already exist.</p>
|
|
<p>On top of that we also need the ability to list locations by prefix,
|
|
and the ability to read the bytes at some location.</p>
|
|
<p class="note">
|
|
We'll diverge from Delta Lake in how we name files on disk. For one,
|
|
we'll keep all files in the same directory with a fixed prefix for
|
|
metadata and another table name prefix for each data file. This
|
|
simplifies the implementation of <code>listPrefix</code> a bit.
|
|
<br />
|
|
<br />
|
|
However, this also diverges from Delta Lake in that transactions
|
|
will represent all tables. In Delta Lake that is not so. Delta Lake
|
|
has a per-table transaction log. Only transactions that read and
|
|
write the same table in Delta Lake achieve snapshot isolation.
|
|
</p><p>So let's set up an interface to describe these requirements:</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">type</span><span class="w"> </span><span class="nx">objectStorage</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="c1">// Must be atomic.</span>
|
|
<span class="w"> </span><span class="nx">putIfAbsent</span><span class="p">(</span><span class="nx">name</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">bytes</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span>
|
|
<span class="w"> </span><span class="nx">listPrefix</span><span class="p">(</span><span class="nx">prefix</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">([]</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">read</span><span class="p">(</span><span class="nx">name</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>And this is literally all we need to get ACID transactions. That's crazy!</p>
|
|
<h4 id="atomic-put-and-cloud-blob-storage">Atomic Put and cloud blob storage</h4><p>We could implement the atomic <code>putIfAbsent</code> part of this interface in
|
|
2024 using <a href="https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/">conditional
|
|
writes</a>
|
|
on S3. Or we could implement this interface with the <code>If-None-Match</code>
|
|
<a href="https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-conditional-headers-for-blob-service-operations">header</a>
|
|
on Azure Cloud Storage. Or we could implement this interface with the
|
|
<code>x-goog-if-generation-match</code>
|
|
<a href="https://cloud.google.com/storage/docs/xml-api/put-object">header</a> on
|
|
Google Cloud Storage.</p>
|
|
<p>Indeed a good exercise for the reader would be to implement this
|
|
interface for other blob storage providers and see your serverless
|
|
cloud database in action!</p>
|
|
<p>But the simplest method of all is to implement it on the filesystem,
|
|
which is what we'll do next.</p>
|
|
<h3 id="a-filesystem-blob-store">A filesystem blob store</h3><p>If we had a server we could implement atomic <code>putIfAbsent</code> with a
|
|
mutex. But we're serverless baby. Thankfully, POSIX <a href="https://rcrowley.org/2010/01/06/things-unix-can-do-atomically.html">supports atomic
|
|
link</a>
|
|
which will fail if the new name is already a file.</p>
|
|
<p>So we'll just create a temporary file and write out all
|
|
bytes. Finally, we link the temporary file to the permanent name we
|
|
intended. For cleanliness (not correctness), if there is an error at
|
|
any point, we'll remove the temporary file.</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">type</span><span class="w"> </span><span class="nx">fileObjectStorage</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">basedir</span><span class="w"> </span><span class="kt">string</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="kd">func</span><span class="w"> </span><span class="nx">newFileObjectStorage</span><span class="p">(</span><span class="nx">basedir</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">fileObjectStorage</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="o">&</span><span class="nx">fileObjectStorage</span><span class="p">{</span><span class="nx">basedir</span><span class="p">}</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">fos</span><span class="w"> </span><span class="o">*</span><span class="nx">fileObjectStorage</span><span class="p">)</span><span class="w"> </span><span class="nx">putIfAbsent</span><span class="p">(</span><span class="nx">name</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">bytes</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">tmpfilename</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">path</span><span class="p">.</span><span class="nx">Join</span><span class="p">(</span><span class="nx">fos</span><span class="p">.</span><span class="nx">basedir</span><span class="p">,</span><span class="w"> </span><span class="nx">uuidv4</span><span class="p">())</span>
|
|
<span class="w"> </span><span class="nx">f</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">OpenFile</span><span class="p">(</span><span class="nx">tmpfilename</span><span class="p">,</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">O_WRONLY</span><span class="o">|</span><span class="nx">os</span><span class="p">.</span><span class="nx">O_CREATE</span><span class="p">,</span><span class="w"> </span><span class="mo">0644</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">written</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">0</span>
|
|
<span class="w"> </span><span class="nx">bufSize</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">16</span>
|
|
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">written</span><span class="w"> </span><span class="p"><</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">bytes</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">toWrite</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">min</span><span class="p">(</span><span class="nx">written</span><span class="o">+</span><span class="nx">bufSize</span><span class="p">,</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">bytes</span><span class="p">))</span>
|
|
<span class="w"> </span><span class="nx">n</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nx">Write</span><span class="p">(</span><span class="nx">bytes</span><span class="p">[</span><span class="nx">written</span><span class="p">:</span><span class="nx">toWrite</span><span class="p">])</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">removeErr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Remove</span><span class="p">(</span><span class="nx">tmpfilename</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">assert</span><span class="p">(</span><span class="nx">removeErr</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not remove"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">written</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="nx">n</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nx">Sync</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">removeErr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Remove</span><span class="p">(</span><span class="nx">tmpfilename</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">assert</span><span class="p">(</span><span class="nx">removeErr</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not remove"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">removeErr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Remove</span><span class="p">(</span><span class="nx">tmpfilename</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">assert</span><span class="p">(</span><span class="nx">removeErr</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not remove"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">filename</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">path</span><span class="p">.</span><span class="nx">Join</span><span class="p">(</span><span class="nx">fos</span><span class="p">.</span><span class="nx">basedir</span><span class="p">,</span><span class="w"> </span><span class="nx">name</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Link</span><span class="p">(</span><span class="nx">tmpfilename</span><span class="p">,</span><span class="w"> </span><span class="nx">filename</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">removeErr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Remove</span><span class="p">(</span><span class="nx">tmpfilename</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">assert</span><span class="p">(</span><span class="nx">removeErr</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not remove"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p class="note">
|
|
<a
|
|
href="https://news.ycombinator.com/item?id=41702593">yencabulator</a>
|
|
on HN pointed out that an earlier version of this post had a buggy
|
|
implementation of <code>putIfAbsent</code> (that attempted to manage
|
|
atomicity solely via <code>O_EXCL | O_CREAT</code>) would leave
|
|
around potentially bad metadata files if the <code>os.Remove</code>
|
|
call ever failed.
|
|
<br />
|
|
<br />
|
|
The <code>link</code> approach works around that because the file is
|
|
already fully and correctly written by the time we do the link.
|
|
</p><p><code>listPrefix</code> and <code>read</code> are minimal wrappers around filesystem APIs:</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">fos</span><span class="w"> </span><span class="o">*</span><span class="nx">fileObjectStorage</span><span class="p">)</span><span class="w"> </span><span class="nx">listPrefix</span><span class="p">(</span><span class="nx">prefix</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">([]</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">dir</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">path</span><span class="p">.</span><span class="nx">Join</span><span class="p">(</span><span class="nx">fos</span><span class="p">.</span><span class="nx">basedir</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">f</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Open</span><span class="p">(</span><span class="nx">dir</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">files</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span>
|
|
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">EOF</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">names</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span>
|
|
<span class="w"> </span><span class="nx">names</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nx">Readdirnames</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">EOF</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">n</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">names</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">prefix</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">""</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="nx">strings</span><span class="p">.</span><span class="nx">HasPrefix</span><span class="p">(</span><span class="nx">n</span><span class="p">,</span><span class="w"> </span><span class="nx">prefix</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">files</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">files</span><span class="p">,</span><span class="w"> </span><span class="nx">n</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">files</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">fos</span><span class="w"> </span><span class="o">*</span><span class="nx">fileObjectStorage</span><span class="p">)</span><span class="w"> </span><span class="nx">read</span><span class="p">(</span><span class="nx">name</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">filename</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">path</span><span class="p">.</span><span class="nx">Join</span><span class="p">(</span><span class="nx">fos</span><span class="p">.</span><span class="nx">basedir</span><span class="p">,</span><span class="w"> </span><span class="nx">name</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">ReadFile</span><span class="p">(</span><span class="nx">filename</span><span class="p">)</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>It is worth talking a bit about reading a directory though. Go doesn't
|
|
provide a nice iterator API for us and I didn't want to implement this
|
|
as callbacks with
|
|
<a href="https://pkg.go.dev/path/filepath#WalkDir"><code>path/filepath.WalkDir</code></a>.</p>
|
|
<p>We could use <a href="https://pkg.go.dev/os#File.ReadDir"><code>os.File.ReadDir</code></a>
|
|
but it allocates for all files in the directory. Sure, in a
|
|
pedagogical project we don't worry about millions of files. But the
|
|
<code>ReadDir</code> API, the error cases in particular, also isn't much simpler
|
|
than <a href="https://pkg.go.dev/os#File.Readdirnames"><code>Readdirnames</code></a>.</p>
|
|
<p class="note">
|
|
What's more, even though we iterated through batches of directory
|
|
entries, and did prefix filtering before accumulating, we still could
|
|
have considered returning an iterator here ourselves. It seems
|
|
possible and likely that the number of data files grows quite large in
|
|
a production system. But I was lazy.
|
|
</p><p>It would be nice if Go introduced an actual iterator API for
|
|
reading a directory. :)</p>
|
|
<h4 id="delta-lake-and-stale-reads">Delta Lake and stale reads</h4><p>In any case the ACID properties of Delta Lake (and Iceberg) don't
|
|
depend on being able to read up-to-date data.</p>
|
|
<p>This is because concurrent (or stale) transactions that <em>write</em> will
|
|
<em>fail on commit</em>. And also because all files written (even metadata
|
|
files) are immutable.</p>
|
|
<p>Since all data is immutable, we will always be able to read at least a
|
|
consistent snapshot of data. But we will never be able to get
|
|
SERIALIZABLE <strong>read-only</strong> transactions. This is just how Delta Lake
|
|
and Iceberg work. And it is a <a href="https://jepsen.io/consistency">similar</a>
|
|
or better consistency level to what any major SQL database <a href="https://github.com/ept/hermitage">gives you
|
|
by default</a>.</p>
|
|
<p>You'll see what I mean later on when we implement transaction commits.</p>
|
|
<h3 id="transaction-boilerplate">Transaction boilerplate</h3><p>Now that we've got a blob storage abstraction and a filesystem
|
|
implementation of it, let's start sketching out what a client and what
|
|
a transaction looks like.</p>
|
|
<p>In Delta Lake, a transaction consists of a list of actions. An action
|
|
might be to define a table's schema, or to add a data file, or to
|
|
remove a data file, etc. In this post we'll only implement the first
|
|
two actions.</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">type</span><span class="w"> </span><span class="nx">DataobjectAction</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">Name</span><span class="w"> </span><span class="kt">string</span>
|
|
<span class="w"> </span><span class="nx">Table</span><span class="w"> </span><span class="kt">string</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="kd">type</span><span class="w"> </span><span class="nx">ChangeMetadataAction</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">Table</span><span class="w"> </span><span class="kt">string</span>
|
|
<span class="w"> </span><span class="nx">Columns</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="c1">// an enum, only one field will be non-nil</span>
|
|
<span class="kd">type</span><span class="w"> </span><span class="nx">Action</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">AddDataobject</span><span class="w"> </span><span class="o">*</span><span class="nx">DataobjectAction</span>
|
|
<span class="w"> </span><span class="nx">ChangeMetadata</span><span class="w"> </span><span class="o">*</span><span class="nx">ChangeMetadataAction</span>
|
|
<span class="w"> </span><span class="c1">// TODO: Support object removal.</span>
|
|
<span class="w"> </span><span class="c1">// DeleteDataobject *DataobjectAction</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>These fields are all exported (i.e. capitalized, if you're not
|
|
familiar with Go) because we will be writing them to disk when the
|
|
transaction commits as the transaction's metadata.</p>
|
|
<p>In fact <code>Action</code>s and the transaction's id will be the only parts of
|
|
the transaction we write to disk. Everything else will be in-memory
|
|
state.</p>
|
|
<p>For our convenience we will track in memory a history of all previous
|
|
actions, a mapping of table columns, and a mapping of unflushed data
|
|
by table.</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">type</span><span class="w"> </span><span class="nx">transaction</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">Id</span><span class="w"> </span><span class="kt">int</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Both are mapping table name to a list of actions on the table.</span>
|
|
<span class="w"> </span><span class="nx">previousActions</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">][]</span><span class="nx">Action</span>
|
|
<span class="w"> </span><span class="nx">Actions</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">][]</span><span class="nx">Action</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Mapping tables to column names.</span>
|
|
<span class="w"> </span><span class="nx">tables</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">][]</span><span class="kt">string</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Mapping table name to unflushed/in-memory rows. When rows</span>
|
|
<span class="w"> </span><span class="c1">// are flushed, the dataobject that contains them is added to</span>
|
|
<span class="w"> </span><span class="c1">// `tx.actions` above and `tx.unflushedDataPointer[table]` is</span>
|
|
<span class="w"> </span><span class="c1">// reset to `0`.</span>
|
|
<span class="w"> </span><span class="nx">unflushedData</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="o">*</span><span class="p">[</span><span class="nx">DATAOBJECT_SIZE</span><span class="p">][]</span><span class="kt">any</span>
|
|
<span class="w"> </span><span class="nx">unflushedDataPointer</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">int</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>Only the current <code>transaction</code> will ever have
|
|
<code>transaction.previousActions</code> filled out. <code>transaction.tables</code> will be
|
|
populated when the transaction starts by reading through
|
|
<code>transaction.previousActions</code> for <code>ChangeMetadataAction</code>s, and we will
|
|
also add onto it when we create a table in the current transaction.</p>
|
|
<p>We will append to <code>transaction.Actions</code> every time we write a new data
|
|
file and every time we create a new table.</p>
|
|
<p>We will add rows to <code>transaction.unflushedData</code> for a table until
|
|
<code>transaction.unflushedDataPointer</code> for that table reaches
|
|
<code>DATAOBJECT_SIZE</code> upon which time we will write that data to disk and
|
|
add a <code>DataobjectAction</code> entry to <code>transaction.Actions</code>.</p>
|
|
<h3 id="client-boilerplate">Client boilerplate</h3><p>A <code>client</code> will consist of an <code>objectStorage</code> implementation and a
|
|
possibly empty <code>*transaction</code>. Empty meaning there is no current
|
|
transaction.</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">type</span><span class="w"> </span><span class="nx">client</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">os</span><span class="w"> </span><span class="nx">objectStorage</span>
|
|
<span class="w"> </span><span class="c1">// Current transaction, if any. Only one transaction per</span>
|
|
<span class="w"> </span><span class="c1">// client at a time. All reads and writes must be within a</span>
|
|
<span class="w"> </span><span class="c1">// transaction.</span>
|
|
<span class="w"> </span><span class="nx">tx</span><span class="w"> </span><span class="o">*</span><span class="nx">transaction</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="kd">func</span><span class="w"> </span><span class="nx">newClient</span><span class="p">(</span><span class="nx">os</span><span class="w"> </span><span class="nx">objectStorage</span><span class="p">)</span><span class="w"> </span><span class="nx">client</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">client</span><span class="p">{</span><span class="nx">os</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">}</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="kd">var</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="nx">errExistingTx</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Errorf</span><span class="p">(</span><span class="s">"Existing Transaction"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">errNoTx</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Errorf</span><span class="p">(</span><span class="s">"No Transaction"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">errTableExists</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Errorf</span><span class="p">(</span><span class="s">"Table Exists"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">errNoTable</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Errorf</span><span class="p">(</span><span class="s">"No Such Table"</span><span class="p">)</span>
|
|
<span class="p">)</span>
|
|
</pre></div>
|
|
<h4 id="client-or-database?">Client or database?</h4><p>In a previous version of my code I named this <code>client</code> struct
|
|
<code>database</code>. But that's misleading. There is no central database. There
|
|
is just the client and the blob storage.</p>
|
|
<p>Clients work with transactions directly and only when attempting to
|
|
commit does the blob storage abstraction let the client know if the
|
|
transaction succeeded or not.</p>
|
|
<h3 id="starting-a-transaction">Starting a transaction</h3><p>When we start a transaction, we will first read all existing
|
|
transactions from disk and accumulate the actions from each prior
|
|
transaction.</p>
|
|
<p>We will interpret <code>ChangeMetadataAction</code>s and materialize them into a
|
|
current view of all tables.</p>
|
|
<p>And we will assign a transaction ID to this transaction to be 1
|
|
greater than the largest existing transaction ID we see.</p>
|
|
<p>Again it doesn't matter if the <code>listPrefix</code> call we use returns an
|
|
up-to-date list. Notably on blob storage there are few guarantees
|
|
about LIST operations recency. The Delta Lake paper mentions this too.</p>
|
|
<p>Out-of-date transactions attempting to write will be caught when we go
|
|
to commit the transaction. Out-of-date transactions attempting only to
|
|
read will still read a consistent snapshot.</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">d</span><span class="w"> </span><span class="o">*</span><span class="nx">client</span><span class="p">)</span><span class="w"> </span><span class="nx">newTx</span><span class="p">()</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">errExistingTx</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">logPrefix</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="s">"_log_"</span>
|
|
<span class="w"> </span><span class="nx">txLogFilenames</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">os</span><span class="p">.</span><span class="nx">listPrefix</span><span class="p">(</span><span class="nx">logPrefix</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">tx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&</span><span class="nx">transaction</span><span class="p">{}</span>
|
|
<span class="w"> </span><span class="nx">tx</span><span class="p">.</span><span class="nx">previousActions</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">][]</span><span class="nx">Action</span><span class="p">{}</span>
|
|
<span class="w"> </span><span class="nx">tx</span><span class="p">.</span><span class="nx">Actions</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">][]</span><span class="nx">Action</span><span class="p">{}</span>
|
|
<span class="w"> </span><span class="nx">tx</span><span class="p">.</span><span class="nx">tables</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">][]</span><span class="kt">string</span><span class="p">{}</span>
|
|
<span class="w"> </span><span class="nx">tx</span><span class="p">.</span><span class="nx">unflushedData</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="o">*</span><span class="p">[</span><span class="nx">DATAOBJECT_SIZE</span><span class="p">][]</span><span class="kt">any</span><span class="p">{}</span>
|
|
<span class="w"> </span><span class="nx">tx</span><span class="p">.</span><span class="nx">unflushedDataPointer</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">int</span><span class="p">{}</span>
|
|
|
|
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">txLogFilename</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">txLogFilenames</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">bytes</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">os</span><span class="p">.</span><span class="nx">read</span><span class="p">(</span><span class="nx">txLogFilename</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">oldTx</span><span class="w"> </span><span class="nx">transaction</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">json</span><span class="p">.</span><span class="nx">Unmarshal</span><span class="p">(</span><span class="nx">bytes</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">oldTx</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="c1">// Transaction metadata files are sorted</span>
|
|
<span class="w"> </span><span class="c1">// lexicographically so that the most recent</span>
|
|
<span class="w"> </span><span class="c1">// transaction (i.e. the one with the largest</span>
|
|
<span class="w"> </span><span class="c1">// transaction id) will be last and tx.Id will end up</span>
|
|
<span class="w"> </span><span class="c1">// 1 greater than the most recent transaction ID we</span>
|
|
<span class="w"> </span><span class="c1">// see on disk.</span>
|
|
<span class="w"> </span><span class="nx">tx</span><span class="p">.</span><span class="nx">Id</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">oldTx</span><span class="p">.</span><span class="nx">Id</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span>
|
|
|
|
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">table</span><span class="p">,</span><span class="w"> </span><span class="nx">actions</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">oldTx</span><span class="p">.</span><span class="nx">Actions</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">action</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">actions</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">action</span><span class="p">.</span><span class="nx">AddDataobject</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">tx</span><span class="p">.</span><span class="nx">previousActions</span><span class="p">[</span><span class="nx">table</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">tx</span><span class="p">.</span><span class="nx">previousActions</span><span class="p">[</span><span class="nx">table</span><span class="p">],</span><span class="w"> </span><span class="nx">action</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">action</span><span class="p">.</span><span class="nx">ChangeMetadata</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="c1">// Store the latest version of</span>
|
|
<span class="w"> </span><span class="c1">// each table in memory for</span>
|
|
<span class="w"> </span><span class="c1">// easy lookup.</span>
|
|
<span class="w"> </span><span class="nx">mtd</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">action</span><span class="p">.</span><span class="nx">ChangeMetadata</span>
|
|
<span class="w"> </span><span class="nx">tx</span><span class="p">.</span><span class="nx">tables</span><span class="p">[</span><span class="nx">table</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">mtd</span><span class="p">.</span><span class="nx">Columns</span>
|
|
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"unsupported action: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">action</span><span class="p">))</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">tx</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>And we're set.</p>
|
|
<h3 id="creating-a-table">Creating a table</h3><p>When we create a table, we need to add a <code>ChangeMetadataAction</code> to the
|
|
transactions <code>Actions</code>. And we also want to add the table info to the
|
|
in-memory <code>transaction.tables</code> field.</p>
|
|
<p>We don't do any of this durably. The change here will be written to
|
|
disk on commit (if the transaction succeeds).</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">d</span><span class="w"> </span><span class="o">*</span><span class="nx">client</span><span class="p">)</span><span class="w"> </span><span class="nx">createTable</span><span class="p">(</span><span class="nx">table</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">columns</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">errNoTx</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">exists</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">tables</span><span class="p">[</span><span class="nx">table</span><span class="p">];</span><span class="w"> </span><span class="nx">exists</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">errTableExists</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Store it in the in-memory mapping.</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">tables</span><span class="p">[</span><span class="nx">table</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">columns</span>
|
|
|
|
<span class="w"> </span><span class="c1">// And also add it to the action history for future transactions.</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">Actions</span><span class="p">[</span><span class="nx">table</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">Actions</span><span class="p">[</span><span class="nx">table</span><span class="p">],</span><span class="w"> </span><span class="nx">Action</span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">ChangeMetadata</span><span class="p">:</span><span class="w"> </span><span class="o">&</span><span class="nx">ChangeMetadataAction</span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">Table</span><span class="p">:</span><span class="w"> </span><span class="nx">table</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="nx">Columns</span><span class="p">:</span><span class="w"> </span><span class="nx">columns</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="p">},</span>
|
|
<span class="w"> </span><span class="p">})</span>
|
|
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>Easy peasy. Now for the fun part, writing data!</p>
|
|
<h3 id="writing-a-row">Writing a row</h3><p>This is the next area where we'll diverge from Delta Lake. For the
|
|
sake of zero dependencies we are going to store data in-memory as an
|
|
array of array of <code>any</code>. And when we later write rows to disk we'll
|
|
write them as JSON. A real Delta Lake implementation would store data
|
|
in-memory in Apache Arrow format, and write to disk as Parquet.</p>
|
|
<p>In line with Delta Lake though we will buffer data in memory until we
|
|
get 64K rows. When we get 64K rows for a particular table we will
|
|
flush all those rows to disk. (When we go to commit a transaction we
|
|
will flush any outstanding rows.)</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">d</span><span class="w"> </span><span class="o">*</span><span class="nx">client</span><span class="p">)</span><span class="w"> </span><span class="nx">writeRow</span><span class="p">(</span><span class="nx">table</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">row</span><span class="w"> </span><span class="p">[]</span><span class="kt">any</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">errNoTx</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">tables</span><span class="p">[</span><span class="nx">table</span><span class="p">];</span><span class="w"> </span><span class="p">!</span><span class="nx">ok</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">errNoTable</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Try to find an unflushed/in-memory dataobject for this table</span>
|
|
<span class="w"> </span><span class="nx">pointer</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">unflushedDataPointer</span><span class="p">[</span><span class="nx">table</span><span class="p">]</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">!</span><span class="nx">ok</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">unflushedDataPointer</span><span class="p">[</span><span class="nx">table</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mi">0</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">unflushedData</span><span class="p">[</span><span class="nx">table</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="o">&</span><span class="p">[</span><span class="nx">DATAOBJECT_SIZE</span><span class="p">][]</span><span class="kt">any</span><span class="p">{}</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">pointer</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nx">DATAOBJECT_SIZE</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">flushRows</span><span class="p">(</span><span class="nx">table</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">pointer</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mi">0</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">unflushedData</span><span class="p">[</span><span class="nx">table</span><span class="p">][</span><span class="nx">pointer</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">row</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">unflushedDataPointer</span><span class="p">[</span><span class="nx">table</span><span class="p">]</span><span class="o">++</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>Now let's implement flushing.</p>
|
|
<h3 id="flushing-a-data-object">Flushing a data object</h3><p>Recall that data objects in Delta Lake (and Iceberg) are
|
|
immutable. Once we've got enough data to write a data object, we give
|
|
it a unique name, write it to disk, and add a <code>AddObjectAction</code> to the
|
|
transaction's list of <code>Actions</code>.</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">type</span><span class="w"> </span><span class="nx">dataobject</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">Table</span><span class="w"> </span><span class="kt">string</span>
|
|
<span class="w"> </span><span class="nx">Name</span><span class="w"> </span><span class="kt">string</span>
|
|
<span class="w"> </span><span class="nx">Data</span><span class="w"> </span><span class="p">[</span><span class="nx">DATAOBJECT_SIZE</span><span class="p">][]</span><span class="kt">any</span>
|
|
<span class="w"> </span><span class="nx">Len</span><span class="w"> </span><span class="kt">int</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">d</span><span class="w"> </span><span class="o">*</span><span class="nx">client</span><span class="p">)</span><span class="w"> </span><span class="nx">flushRows</span><span class="p">(</span><span class="nx">table</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">errNoTx</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="c1">// First write out dataobject if there is anything to write out.</span>
|
|
<span class="w"> </span><span class="nx">pointer</span><span class="p">,</span><span class="w"> </span><span class="nx">exists</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">unflushedDataPointer</span><span class="p">[</span><span class="nx">table</span><span class="p">]</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">!</span><span class="nx">exists</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="nx">pointer</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">df</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">dataobject</span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">Table</span><span class="p">:</span><span class="w"> </span><span class="nx">table</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="nx">Name</span><span class="p">:</span><span class="w"> </span><span class="nx">uuidv4</span><span class="p">(),</span>
|
|
<span class="w"> </span><span class="nx">Data</span><span class="p">:</span><span class="w"> </span><span class="o">*</span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">unflushedData</span><span class="p">[</span><span class="nx">table</span><span class="p">],</span>
|
|
<span class="w"> </span><span class="nx">Len</span><span class="p">:</span><span class="w"> </span><span class="nx">pointer</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="nx">bytes</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">json</span><span class="p">.</span><span class="nx">Marshal</span><span class="p">(</span><span class="nx">df</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">os</span><span class="p">.</span><span class="nx">putIfAbsent</span><span class="p">(</span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"_table_%s_%s"</span><span class="p">,</span><span class="w"> </span><span class="nx">table</span><span class="p">,</span><span class="w"> </span><span class="nx">df</span><span class="p">.</span><span class="nx">Name</span><span class="p">),</span><span class="w"> </span><span class="nx">bytes</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Then record the newly written data file.</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">Actions</span><span class="p">[</span><span class="nx">table</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">Actions</span><span class="p">[</span><span class="nx">table</span><span class="p">],</span><span class="w"> </span><span class="nx">Action</span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">AddDataobject</span><span class="p">:</span><span class="w"> </span><span class="o">&</span><span class="nx">DataobjectAction</span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">Table</span><span class="p">:</span><span class="w"> </span><span class="nx">table</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="nx">Name</span><span class="p">:</span><span class="w"> </span><span class="nx">df</span><span class="p">.</span><span class="nx">Name</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="p">},</span>
|
|
<span class="w"> </span><span class="p">})</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Reset in-memory pointer.</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">unflushedDataPointer</span><span class="p">[</span><span class="nx">table</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mi">0</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>That's it for writing data! Let's now look at reading data.</p>
|
|
<h3 id="scanning-a-table">Scanning a table</h3><p>We're going to make scanning mildly more complicated than it needed to
|
|
be in pedagogical code because we'll have <code>client.scan()</code> return an
|
|
iterator rather than an array with all rows.</p>
|
|
<p>The <code>scanIterator</code> will first read from in-memory (unflushed)
|
|
data. And then it will read through every data object for the table
|
|
that is still a part of this transaction. We will know which data
|
|
objects are still a part of this transaction by reading through all
|
|
<code>AddDataobject</code> actions. A future version of this project would also
|
|
eliminate data object files from the list by observing
|
|
<code>DeleteDataobject</code> actions. But we don't do that in this post.</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">d</span><span class="w"> </span><span class="o">*</span><span class="nx">client</span><span class="p">)</span><span class="w"> </span><span class="nx">scan</span><span class="p">(</span><span class="nx">table</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">scanIterator</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="nx">errNoTx</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">dataobjects</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span>
|
|
<span class="w"> </span><span class="nx">allActions</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">previousActions</span><span class="p">[</span><span class="nx">table</span><span class="p">],</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">Actions</span><span class="p">[</span><span class="nx">table</span><span class="p">]</span><span class="o">...</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">action</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">allActions</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">action</span><span class="p">.</span><span class="nx">AddDataobject</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">dataobjects</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">dataobjects</span><span class="p">,</span><span class="w"> </span><span class="nx">action</span><span class="p">.</span><span class="nx">AddDataobject</span><span class="p">.</span><span class="nx">Name</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">unflushedRows</span><span class="w"> </span><span class="p">[</span><span class="nx">DATAOBJECT_SIZE</span><span class="p">][]</span><span class="kt">any</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">data</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">unflushedData</span><span class="p">[</span><span class="nx">table</span><span class="p">];</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">unflushedRows</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="o">*</span><span class="nx">data</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="o">&</span><span class="nx">scanIterator</span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">unflushedRows</span><span class="p">:</span><span class="w"> </span><span class="nx">unflushedRows</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="nx">unflushedRowsLen</span><span class="p">:</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">unflushedDataPointer</span><span class="p">[</span><span class="nx">table</span><span class="p">],</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">:</span><span class="w"> </span><span class="nx">d</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="nx">table</span><span class="p">:</span><span class="w"> </span><span class="nx">table</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="nx">dataobjects</span><span class="p">:</span><span class="w"> </span><span class="nx">dataobjects</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>The <code>scanIterator</code> needs to track where we are in in-memory rows, in
|
|
data objects, and within a particular data object.</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">type</span><span class="w"> </span><span class="nx">scanIterator</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="w"> </span><span class="o">*</span><span class="nx">client</span>
|
|
<span class="w"> </span><span class="nx">table</span><span class="w"> </span><span class="kt">string</span>
|
|
|
|
<span class="w"> </span><span class="c1">// First we iterate through unflushed rows.</span>
|
|
<span class="w"> </span><span class="nx">unflushedRows</span><span class="w"> </span><span class="p">[</span><span class="nx">DATAOBJECT_SIZE</span><span class="p">][]</span><span class="kt">any</span>
|
|
<span class="w"> </span><span class="nx">unflushedRowsLen</span><span class="w"> </span><span class="kt">int</span>
|
|
<span class="w"> </span><span class="nx">unflushedRowPointer</span><span class="w"> </span><span class="kt">int</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Then we move through each dataobject.</span>
|
|
<span class="w"> </span><span class="nx">dataobjects</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span>
|
|
<span class="w"> </span><span class="nx">dataobjectsPointer</span><span class="w"> </span><span class="kt">int</span>
|
|
|
|
<span class="w"> </span><span class="c1">// And within each dataobject we iterate through rows.</span>
|
|
<span class="w"> </span><span class="nx">dataobject</span><span class="w"> </span><span class="o">*</span><span class="nx">dataobject</span>
|
|
<span class="w"> </span><span class="nx">dataobjectRowPointer</span><span class="w"> </span><span class="kt">int</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>And the <code>scanIterator</code> will be driven by a <code>next()</code> method that goes
|
|
through in-memory data first and then through what's on disk.</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">d</span><span class="w"> </span><span class="o">*</span><span class="nx">client</span><span class="p">)</span><span class="w"> </span><span class="nx">readDataobject</span><span class="p">(</span><span class="nx">table</span><span class="p">,</span><span class="w"> </span><span class="nx">name</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">dataobject</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">bytes</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">os</span><span class="p">.</span><span class="nx">read</span><span class="p">(</span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"_table_%s_%s"</span><span class="p">,</span><span class="w"> </span><span class="nx">table</span><span class="p">,</span><span class="w"> </span><span class="nx">name</span><span class="p">))</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">do</span><span class="w"> </span><span class="nx">dataobject</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">json</span><span class="p">.</span><span class="nx">Unmarshal</span><span class="p">(</span><span class="nx">bytes</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">do</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="o">&</span><span class="nx">do</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="c1">// returns (nil, nil) when done</span>
|
|
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">si</span><span class="w"> </span><span class="o">*</span><span class="nx">scanIterator</span><span class="p">)</span><span class="w"> </span><span class="nx">next</span><span class="p">()</span><span class="w"> </span><span class="p">([]</span><span class="kt">any</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="c1">// Iterate through in-memory rows first.</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">unflushedRowPointer</span><span class="w"> </span><span class="p"><</span><span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">unflushedRowsLen</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">row</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">unflushedRows</span><span class="p">[</span><span class="nx">si</span><span class="p">.</span><span class="nx">unflushedRowPointer</span><span class="p">]</span>
|
|
<span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">unflushedRowPointer</span><span class="o">++</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">row</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="c1">// If we've gotten through all dataobjects on disk we're done.</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobjectsPointer</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobjects</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobject</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">name</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobjects</span><span class="p">[</span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobjectsPointer</span><span class="p">]</span>
|
|
<span class="w"> </span><span class="nx">o</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">d</span><span class="p">.</span><span class="nx">readDataobject</span><span class="p">(</span><span class="nx">si</span><span class="p">.</span><span class="nx">table</span><span class="p">,</span><span class="w"> </span><span class="nx">name</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobject</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">o</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobjectRowPointer</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobject</span><span class="p">.</span><span class="nx">Len</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobjectsPointer</span><span class="o">++</span>
|
|
<span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobject</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobjectRowPointer</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mi">0</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">next</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">row</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobject</span><span class="p">.</span><span class="nx">Data</span><span class="p">[</span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobjectRowPointer</span><span class="p">]</span>
|
|
<span class="w"> </span><span class="nx">si</span><span class="p">.</span><span class="nx">dataobjectRowPointer</span><span class="o">++</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">row</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>That's it for scanning a table! The final piece of the puzzle is
|
|
committing a transaction.</p>
|
|
<h3 id="committing-a-transaction">Committing a transaction</h3><p>When we commit a transaction we must flush any remaining data. A
|
|
read-only transaction (one which has no <code>Actions</code>) is immediately
|
|
done. There is no concurrency check.</p>
|
|
<p>Otherwise we will serialize transaction state and attempt to
|
|
atomically <code>putIfAbsent</code>.</p>
|
|
<p>The only way this will fail is if there is another concurrent writer.</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">d</span><span class="w"> </span><span class="o">*</span><span class="nx">client</span><span class="p">)</span><span class="w"> </span><span class="nx">commitTx</span><span class="p">()</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">errNoTx</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Flush any outstanding data</span>
|
|
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">table</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">tables</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">flushRows</span><span class="p">(</span><span class="nx">table</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">wrote</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="kc">false</span>
|
|
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">actions</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">Actions</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">actions</span><span class="p">)</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">wrote</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">true</span>
|
|
<span class="w"> </span><span class="k">break</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="c1">// Read-only transaction, no need to do a concurrency check.</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">!</span><span class="nx">wrote</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">filename</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"_log_%020d"</span><span class="p">,</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">Id</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="c1">// We won't store previous actions, they will be recovered on</span>
|
|
<span class="w"> </span><span class="c1">// new transactions. So unset them. Honestly not totally</span>
|
|
<span class="w"> </span><span class="c1">// clear why.</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">.</span><span class="nx">previousActions</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="w"> </span><span class="nx">bytes</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">json</span><span class="p">.</span><span class="nx">Marshal</span><span class="p">(</span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">os</span><span class="p">.</span><span class="nx">putIfAbsent</span><span class="p">(</span><span class="nx">filename</span><span class="p">,</span><span class="w"> </span><span class="nx">bytes</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">tx</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">nil</span>
|
|
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span>
|
|
<span class="p">}</span>
|
|
|
|
<span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="s">"unimplemented"</span><span class="p">)</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>This is the crux of Delta Lake. It's simple. And honestly it's a bit
|
|
shocking. Real Delta Lake does support automatic retries in some
|
|
cases. But primarily you are limited to a single writer per table,
|
|
even if the writers are writing non-conflicting rows. Iceberg is
|
|
basically the same here, it's just how metadata is tracked that
|
|
differs.</p>
|
|
<p class="note">
|
|
As mentioned in another note above, our implementation is actually
|
|
stricter than Delta Lake since it manages all table transaction logs
|
|
together. This means you can get snapshot isolation across all
|
|
tables (which Delta Lake doesn't support) but it will mean
|
|
significantly more contention and failed write transactions.
|
|
</p><p>The Delta Lake and Iceberg folks apparently wanted to avoid
|
|
FoundationDB (i.e. the Snowflake architecture, which is mentioned in
|
|
the Delta Lake paper) so much that they'd give up row-level
|
|
concurrency to be mostly serverless.</p>
|
|
<p>Is it worth it? Dunno. Delta Lake and Iceberg are getting massive
|
|
adoption. Many very smart people have worked, and continue to work, on
|
|
both. Moreover it is apparently what the market wants. Every
|
|
database-like product is implementing, or is planning to implement,
|
|
Delta Lake or Iceberg.</p>
|
|
<h3 id="trying-it-out">Trying it out</h3><p>Let's add a test in <code>main_test.go</code> to see what happens with concurrent
|
|
writers. Follow the comments and debug logs for details:</p>
|
|
<div class="highlight"><pre><span></span><span class="kn">package</span><span class="w"> </span><span class="nx">main</span>
|
|
|
|
<span class="kn">import</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="s">"os"</span>
|
|
<span class="w"> </span><span class="s">"testing"</span>
|
|
<span class="p">)</span>
|
|
|
|
<span class="kd">func</span><span class="w"> </span><span class="nx">TestConcurrentTableWriters</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">dir</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">MkdirTemp</span><span class="p">(</span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"test-database"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Remove</span><span class="p">(</span><span class="nx">dir</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="nx">fos</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">newFileObjectStorage</span><span class="p">(</span><span class="nx">dir</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">c1Writer</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">newClient</span><span class="p">(</span><span class="nx">fos</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">c2Writer</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">newClient</span><span class="p">(</span><span class="nx">fos</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Have c2Writer start up a transaction.</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c2Writer</span><span class="p">.</span><span class="nx">newTx</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not start first c2 tx"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c2] new tx"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="c1">// But then have c1Writer start a transaction and commit it first.</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">newTx</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not start first c1 tx"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1] new tx"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">createTable</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">"a"</span><span class="p">,</span><span class="w"> </span><span class="s">"b"</span><span class="p">})</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not create x"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1] Created table"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">writeRow</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="kt">any</span><span class="p">{</span><span class="s">"Joey"</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">})</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not write first row"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1] Wrote row"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">writeRow</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="kt">any</span><span class="p">{</span><span class="s">"Yue"</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">})</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not write second row"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1] Wrote row"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">commitTx</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not commit tx"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1] Committed tx"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Now go back to c2 and write data.</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c2Writer</span><span class="p">.</span><span class="nx">createTable</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">"a"</span><span class="p">,</span><span class="w"> </span><span class="s">"b"</span><span class="p">})</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not create x"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c2] Created table"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c2Writer</span><span class="p">.</span><span class="nx">writeRow</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="kt">any</span><span class="p">{</span><span class="s">"Holly"</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">})</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not write first row"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c2] Wrote row"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c2Writer</span><span class="p">.</span><span class="nx">commitTx</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="nx">assert</span><span class="p">(</span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"concurrent commit must fail"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c2] tx not committed"</span><span class="p">)</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>Try it out:</p>
|
|
<div class="highlight"><pre><span></span><span class="gp">$ </span>go<span class="w"> </span>mod<span class="w"> </span>init<span class="w"> </span>otf
|
|
<span class="gp">$ </span>go<span class="w"> </span>mod<span class="w"> </span>tidy
|
|
<span class="gp">$ </span>go<span class="w"> </span><span class="nb">test</span><span class="w"> </span>-run<span class="w"> </span>TestConcurrentTableWriters<span class="w"> </span>--<span class="w"> </span>--debug
|
|
<span class="go">[DEBUG] [c2] new tx</span>
|
|
<span class="go">[DEBUG] [c1] new tx</span>
|
|
<span class="go">[DEBUG] [c1] Created table</span>
|
|
<span class="go">[DEBUG] [c1] Wrote row</span>
|
|
<span class="go">[DEBUG] [c1] Wrote row</span>
|
|
<span class="go">[DEBUG] [c1] Committed tx</span>
|
|
<span class="go">[DEBUG] [c2] Created table</span>
|
|
<span class="go">[DEBUG] [c2] Wrote row</span>
|
|
<span class="go">[DEBUG] [c2] tx not committed</span>
|
|
<span class="go">PASS</span>
|
|
<span class="go">ok otf 0.311s</span>
|
|
</pre></div>
|
|
<p>That's pretty cool.</p>
|
|
<p>And what about a reader and concurrent writer? Observe that the reader
|
|
always reads a snapshot. Follow the comments again for detail:</p>
|
|
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">TestConcurrentReaderWithWriterReadsSnapshot</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">dir</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">MkdirTemp</span><span class="p">(</span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"test-database"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Remove</span><span class="p">(</span><span class="nx">dir</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="nx">fos</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">newFileObjectStorage</span><span class="p">(</span><span class="nx">dir</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">c1Writer</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">newClient</span><span class="p">(</span><span class="nx">fos</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">c2Reader</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">newClient</span><span class="p">(</span><span class="nx">fos</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="c1">// First create some data and commit the transaction.</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">newTx</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not start first c1 tx"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1Writer] Started tx"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">createTable</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">"a"</span><span class="p">,</span><span class="w"> </span><span class="s">"b"</span><span class="p">})</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not create x"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1Writer] Created table"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">writeRow</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="kt">any</span><span class="p">{</span><span class="s">"Joey"</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">})</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not write first row"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1Writer] Wrote row"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">writeRow</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="kt">any</span><span class="p">{</span><span class="s">"Yue"</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">})</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not write second row"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1Writer] Wrote row"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">commitTx</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not commit tx"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1Writer] Committed tx"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Now start a new transaction for more edits.</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">newTx</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not start second c1 tx"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1Writer] Starting new write tx"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Before we commit this second write-transaction, start a</span>
|
|
<span class="w"> </span><span class="c1">// read transaction.</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c2Reader</span><span class="p">.</span><span class="nx">newTx</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not start c2 tx"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c2Reader] Started tx"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Write and commit rows in c1.</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">writeRow</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="kt">any</span><span class="p">{</span><span class="s">"Ada"</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">})</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not write third row"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1Writer] Wrote third row"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Scan x in read-only transaction</span>
|
|
<span class="w"> </span><span class="nx">it</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">c2Reader</span><span class="p">.</span><span class="nx">scan</span><span class="p">(</span><span class="s">"x"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not scan x"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c2Reader] Started scanning"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">seen</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">0</span>
|
|
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">row</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">it</span><span class="p">.</span><span class="nx">next</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not iterate x scan"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">row</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c2Reader] Done scanning"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">break</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c2Reader] Got row in reader tx"</span><span class="p">,</span><span class="w"> </span><span class="nx">row</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">seen</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">row</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="w"> </span><span class="s">"Joey"</span><span class="p">,</span><span class="w"> </span><span class="s">"row mismatch in c1"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">row</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w"> </span><span class="s">"row mismatch in c1"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">row</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="w"> </span><span class="s">"Yue"</span><span class="p">,</span><span class="w"> </span><span class="s">"row mismatch in c1"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">row</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span><span class="w"> </span><span class="mf">2.0</span><span class="p">,</span><span class="w"> </span><span class="s">"row mismatch in c1"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">seen</span><span class="o">++</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">seen</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="s">"expected two rows"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Scan x in c1 write transaction</span>
|
|
<span class="w"> </span><span class="nx">it</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">scan</span><span class="p">(</span><span class="s">"x"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not scan x in c1"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1Writer] Started scanning"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">seen</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mi">0</span>
|
|
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">row</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">it</span><span class="p">.</span><span class="nx">next</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not iterate x scan in c1"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">row</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1Writer] Done scanning"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">break</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1Writer] Got row in tx"</span><span class="p">,</span><span class="w"> </span><span class="nx">row</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">seen</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">row</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="w"> </span><span class="s">"Ada"</span><span class="p">,</span><span class="w"> </span><span class="s">"row mismatch in c1"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="c1">// Since this hasn't been serialized to JSON, it's still an int not a float.</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">row</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="s">"row mismatch in c1"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">seen</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">row</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="w"> </span><span class="s">"Joey"</span><span class="p">,</span><span class="w"> </span><span class="s">"row mismatch in c1"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">row</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w"> </span><span class="s">"row mismatch in c1"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">row</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="w"> </span><span class="s">"Yue"</span><span class="p">,</span><span class="w"> </span><span class="s">"row mismatch in c1"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">row</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span><span class="w"> </span><span class="mf">2.0</span><span class="p">,</span><span class="w"> </span><span class="s">"row mismatch in c1"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
|
|
<span class="w"> </span><span class="nx">seen</span><span class="o">++</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">seen</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="s">"expected three rows"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Writer committing should succeed.</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c1Writer</span><span class="p">.</span><span class="nx">commitTx</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not commit second tx"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c1Writer] Committed tx"</span><span class="p">)</span>
|
|
|
|
<span class="w"> </span><span class="c1">// Reader committing should succeed.</span>
|
|
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c2Reader</span><span class="p">.</span><span class="nx">commitTx</span><span class="p">()</span>
|
|
<span class="w"> </span><span class="nx">assertEq</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="s">"could not commit read-only tx"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="nx">debug</span><span class="p">(</span><span class="s">"[c2Reader] Committed tx"</span><span class="p">)</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
<p>Run it:</p>
|
|
<div class="highlight"><pre><span></span><span class="gp">$ </span>go<span class="w"> </span><span class="nb">test</span><span class="w"> </span>-run<span class="w"> </span>TestConcurrentReaderWithWriterReadsSnapshot<span class="w"> </span>--<span class="w"> </span>--debug
|
|
<span class="go">[DEBUG] [c1Writer] Started tx</span>
|
|
<span class="go">[DEBUG] [c1Writer] Created table</span>
|
|
<span class="go">[DEBUG] [c1Writer] Wrote row</span>
|
|
<span class="go">[DEBUG] [c1Writer] Wrote row</span>
|
|
<span class="go">[DEBUG] [c1Writer] Committed tx</span>
|
|
<span class="go">[DEBUG] [c1Writer] Starting new write tx</span>
|
|
<span class="go">[DEBUG] [c2Reader] Started tx</span>
|
|
<span class="go">[DEBUG] [c1Writer] Wrote third row</span>
|
|
<span class="go">[DEBUG] [c2Reader] Started scanning</span>
|
|
<span class="go">[DEBUG] [c2Reader] Got row in reader tx [Joey 1]</span>
|
|
<span class="go">[DEBUG] [c2Reader] Got row in reader tx [Yue 2]</span>
|
|
<span class="go">[DEBUG] [c2Reader] Done scanning</span>
|
|
<span class="go">[DEBUG] [c1Writer] Started scanning</span>
|
|
<span class="go">[DEBUG] [c1Writer] Got row in tx [Ada 3]</span>
|
|
<span class="go">[DEBUG] [c1Writer] Got row in tx [Joey 1]</span>
|
|
<span class="go">[DEBUG] [c1Writer] Got row in tx [Yue 2]</span>
|
|
<span class="go">[DEBUG] [c1Writer] Done scanning</span>
|
|
<span class="go">[DEBUG] [c1Writer] Committed tx</span>
|
|
<span class="go">[DEBUG] [c2Reader] Committed tx</span>
|
|
<span class="go">PASS</span>
|
|
<span class="go">ok otf 0.252s</span>
|
|
</pre></div>
|
|
<p>Sweet.</p>
|
|
<h3 id="what's-next?">What's next?</h3><p>As mentioned, we didn't touch a lot of things. Handling updates and
|
|
deletes, transaction log checkpoints, data object compaction, etc.</p>
|
|
<p>Take a close look at the <a href="https://www.vldb.org/pvldb/vol13/p3411-armbrust.pdf">Delta Lake
|
|
paper</a> and the
|
|
<a href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md">Delta Lake
|
|
Spec</a> and
|
|
see what you can do!</p>
|
|
<p><blockquote class="twitter-tweet"><p lang="en" dir="ltr">Build a serverless ACID database with this one neat trick.<br><br>(New blog post)<a href="https://t.co/rHgfKSPY6q">https://t.co/rHgfKSPY6q</a> <a href="https://t.co/1hmjsxIk6w">pic.twitter.com/1hmjsxIk6w</a></p>— Phil Eaton (@eatonphil) <a href="https://twitter.com/eatonphil/status/1840474893491560777?ref_src=twsrc%5Etfw">September 29, 2024</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></p>
|
|
<style>.feedback{display:initial;}</style>
|
|
</div>
|
|
<div class="feedback">
|
|
<h4>Feedback</h4>
|
|
<p>As always,
|
|
please <a href="mailto:phil@eatonphil.com">email</a>
|
|
or <a href="https://twitter.com/eatonphil">tweet me</a>
|
|
with questions, corrections, or ideas!</p>
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<footer>
|
|
<div class="container">
|
|
<div>
|
|
<div id="subscribe">
|
|
<iframe frameBorder="0" src="https://cdn.forms-content-1.sg-form.com/e8ed4c3b-dd46-11f0-b6e6-ce075f2c5f80"></iframe>
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</footer>
|
|
</body>
|
|
</html>
|