Files
nexus/sreweekly/articles/95/03-moving-half-a-million-database-tables-to-aws-aurora-part-1.html
2026-09-12 17:23:01 +08:00

330 lines
20 KiB
HTML

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta http-equiv="X-UA-Compatible" content="ie=edge" />
<title>Moving Half a Million Database Tables to AWS Aurora (Part 1) - Pressbooks</title>
<link rel="canonical" href="https://pressbooks.org/blog/2017/10/19/moving-half-a-million-database-tables-to-aws-aurora-part-1/" />
<meta property="og:site_name" content="Pressbooks" />
<meta property="og:title" content="Moving Half a Million Database Tables to AWS Aurora (Part 1) - Pressbooks" />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://pressbooks.org/blog/2017/10/19/moving-half-a-million-database-tables-to-aws-aurora-part-1/" />
<link rel="icon" href="/favicon.ico" sizes="any">
<link rel="icon" href="/icon.svg" type="image/svg+xml">
<link rel="apple-touch-icon" href="/apple-touch-icon.png">
<link rel="manifest" href="/manifest.webmanifest">
<link rel="alternate" type="application/rss+xml" href="https://pressbooks.org/feed.xml" />
<link rel="stylesheet" href="/css/main.css">
<link rel="stylesheet" href="https://unpkg.com/prismjs@^1.24/themes/prism-okaidia.css">
<link rel="stylesheet" href="https://unpkg.com/prismjs@^1.24/plugins/treeview/prism-treeview.css">
<!-- Google tag (gtag.js) -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-1KE8PMYEWF"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-1KE8PMYEWF');
</script></head>
<body>
<a class="[ skip-link ] [ button ]" href="#main-content">Skip to content</a>
<header role="banner" class="site-head">
<div class="wrapper">
<div class="site-head__inner">
<a href="/" aria-label="Pressbooks - home" class="site-head__brand">
<img src="/images/pressbooks-logo.svg" alt="Pressbooks logo" />
</a>
<p class="site-head__subhead">Open publishing. Open web. Open source.</p>
<nav
class="[ nav ] [ site-head__nav ]"
aria-label="Primary navigaton"
>
<ul class="nav__list cluster">
<li>
<a href="/"
>Home</a
>
</li>
<li>
<a href="/blog/" data-state="active"
>Blog</a
>
</li>
<li>
<a href="/dev-guides/"
>Developer Guides</a
>
</li>
<li>
<a href="/user-guides/"
>User Guides</a
>
</li>
<li>
<a href="/get-involved/"
>Get Involved</a
>
</li>
</ul>
</nav>
</div>
</div>
</header>
<main tabindex="-1" id="main-content" >
<article>
<div class="[ post ] [ flow wrapper ]">
<header>
<h1>Moving Half a Million Database Tables to AWS Aurora (Part 1)</h1>
<p class="meta">
<p class="author vcard"><span class="visually-hidden">Posted by </span><span class="fn n">Dac Chartrand</span></p>
<time datetime="2017-10-19T12:00:00.000Z">October 19th, 2017</time>
</p>
<div class="tags">
<svg xmlns="http://www.w3.org/2000/svg" class="tags-icon" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2">
<path stroke-linecap="round" stroke-linejoin="round" d="M7 7h.01M7 3h5c.512 0 1.024.195 1.414.586l7 7a2 2 0 010 2.828l-7 7a2 2 0 01-2.828 0l-7-7A1.994 1.994 0 013 12V7a4 4 0 014-4z" />
</svg>
<ul class="tags-list" aria-label="Tags">
<li>
<a href="/tag/core/">Core</a>
</li>
</ul>
</div>
</header>
<p>This post is about migrating Pressbooks.com to AWS.</p>
<h2 id="does-it-scale" tabindex="-1">Does It Scale?</h2>
<p>At <a href="/">Pressbooks</a> we use
<a href="https://codex.wordpress.org/Create_A_Network">WordPress Multisite</a> as a development
platform. <a href="https://github.com/pressbooks/pressbooks">Pressbooks changes WordPress</a> and
makes every blog a book.</p>
<p>The prevailing wisdom of the day is that a relational database should have a manageable
set of tables with lots of rows of data in them ...and then there's WordPress.</p>
<p>WP Multisite creates
<a href="https://codex.wordpress.org/Database_Description#Site_Specific_Tables">10 database tables</a>
for every blog <em>(or in our case: for every book.)</em> Tables share IDs but they are not
enforced at the database level. There are no foreign key constraints, no triggers, nor
routines. It's simple no frills MySQL.</p>
<p>Pressbooks dot com is currently running on a single bare metal server. <em>(Our hosted
instances are already on AWS)</em> This server has a single MariaDB database with 60,000 books
in it. When we do the math that's over 600,000 tables in one database.
<a href="https://www.askbjoernhansen.com/2008/02/14/10000_tables_in_one_mysql_database.html">Are you nuts?!</a>
Unusual? Horrible? Yet
<a href="https://dev.mysql.com/doc/refman/5.7/en/database-count-limit.html">entirely possible</a>,
<a href="https://www.percona.com/blog/2017/10/01/one-million-tables-mysql-8-0/">even plausible</a>.</p>
<p>I've met people who worked at Automattic and
<a href="https://www.meetup.com/wp-mtl/events/240377606/">from the stories, I heard</a> WordPress dot
com uses the same WP Multisite technology but instead of half a million tables it's over a
billion tables <em>(probably not all in the same database though, more on that later)</em>.</p>
<p>I call this the Schrödinger's Cat of database design because so long as we don't look it's
alive?</p>
<h2 id="prerequisites" tabindex="-1">Prerequisites</h2>
<p>As I said, we already host hundreds of WP Multisite networks on AWS. We build, manage, and
deploy to our infrastructure using <a href="https://www.terraform.io/">Terraform</a>,
<a href="https://www.ansible.com/">Ansible</a>, <a href="https://wp-cli.org/">wp-cli</a> and
<a href="https://github.com/roots/trellis">all things competent</a>. We simply just, sort of, well,
neglected to move our freemium site over because we were too busy.</p>
<p>The time has come.</p>
<h2 id="research" tabindex="-1">Research</h2>
<p>We tried <code>mysqldump</code>. It was too slow. Our tests showed that a dump would take days.</p>
<p>Some colleagues recommended <a href="https://aws.amazon.com/dms/">AWS DMS</a>. It did not work. Some
reasons:</p>
<ul>
<li><a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MySQL.html#CHAP_Source.MySQL.Limitations">The <code>AUTO_INCREMENT</code> attribute is not migrated.</a></li>
<li>WP Multisite tables are prefixed <code>wp_1_, wp_2_, wp_3_, ...</code> MySQL considers the
underscore a
<a href="https://stackoverflow.com/questions/8236818/why-does-underscore-match-hyphen">one character wildcard</a>
when used in LIKE queries, DMS provided no way to escape it for table filters.</li>
<li>Unexplained crashing on anything less than a <code>c4.large</code> <em>(failed last Error The task
stopped abnormally Stop Reason RECOVERABLE_ERROR Error Level RECOVERABLE)</em></li>
<li>Mostly, the fastest migration we could get going, running 4 tasks in parallel, was an
ETA of 10 days</li>
</ul>
<p>We asked for help. Ned had a conference call with a reputable consulting firm and they gave us a quote: $34K USD + travel &amp; on-site
expenses.</p>
<p><img src="/images/coffee-spitting.gif" alt="Coffee spitting "></p>
<p>Next, our research led us to <a href="https://github.com/maxbube/mydumper">mydumper</a>. From the
README:</p>
<blockquote>
<p>== What is mydumper? Why? ==</p>
<ul>
<li>Parallelism (hence, speed) and performance (avoids expensive character set conversion
routines, efficient code overall)</li>
<li>Easier to manage output (separate files for tables, dump metadata, etc, easy to
view/parse data)</li>
<li>Consistency - maintains snapshot across all threads, provides accurate master and
slave log positions, etc</li>
<li>Manageability - supports PCRE for specifying database and tables inclusions and
exclusions</li>
</ul>
</blockquote>
<p>So far so good...</p>
<blockquote>
<p>== How to build it? ==</p>
</blockquote>
<p><img src="/images/jerry-seinfeld-leaving.gif" alt="Jerry Seinfeld leaving"></p>
<p>Just kidding. It turns out we don't have to build mydumper. On
Ubuntu <code>sudo apt install mydumper</code> works fine. Similar command using <code>yum</code> on CentOS.</p>
<p>Our tests conclude that mydumper finishes in hours instead of days.</p>
<h2 id="plan" tabindex="-1">Plan</h2>
<p>It is our opinion that this kind of problem is better suited for a
<a href="https://en.wikipedia.org/wiki/Document-oriented_database">document-oriented database</a>.
Given that this is the database design we inherited, there's not much we can do about it,
so we'll try our best with what we've got. ¯\<em>(ツ)</em>/¯</p>
<p>At a billion tables, Automattic has already established its own internal best practices
with plugins like <a href="https://github.com/Automattic/hyperdb">HypderDB</a>. Unfortunately HyperDB
doesn't have <a href="https://getcomposer.org/">Composer</a> support and doesn't look
maintained. <a href="https://github.com/stuttter/ludicrousdb">LudicrousDB</a>, a Composer compatible
drop-in that works with our <a href="https://github.com/roots/bedrock/">existing tech stack</a>, to
the rescue.</p>
<blockquote>
<p>LudicrousDB is an advanced database interface for WordPress that supports replication,
fail-over, load balancing, and partitioning, based on Automattic's HyperDB drop-in.</p>
</blockquote>
<p>With LudicrousDB tested and working, we are moving towards a 101 slice approach. 1 slice
for core tables and 100 slices for books.</p>
<p>The idea is to use the last <strong>two digits</strong> of a book ID to pick one of 100 slices. If this
becomes unmanageable in the future <em>(important to remember that we already have over half
a million</em> tables <em>in 1 database and
<a href="https://knowyourmeme.com/memes/this-is-fine">things are fine</a>)</em>, we can change the
splitting algorithm by adding a condition to use the last <strong>X digits</strong> on books with IDs
bigger than <strong>Y</strong>.</p>
<h2 id="code" tabindex="-1">Code</h2>
<p><em>For informational purposes only. Read the snippets and reason about them. Copy/paste at
your own peril.</em></p>
<h2 id="ludicrous-db-callback" tabindex="-1">LudicrousDB Callback</h2>
<pre class="language-php"><code class="language-php"><span class="token comment">/**
* Slices
*
* We can predict what slice a blog is in by looking
* at the last two digits of the id. Examples:
*
* + blog_id: 9, in db09
* + blog_id: 74, in db74
* + blog_id: 999989, in db89
* + blog_id: 9200, in db00
*
* @param $query
* @param \LudicrousDB $wpdb
*
* @return string
*/</span>
<span class="token keyword">function</span> <span class="token function-definition function">pb_db_callback</span><span class="token punctuation">(</span> <span class="token variable">$query</span><span class="token punctuation">,</span> <span class="token variable">$wpdb</span> <span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span> <span class="token function">preg_match</span><span class="token punctuation">(</span> <span class="token string double-quoted-string">"/^<span class="token interpolation"><span class="token punctuation">{</span><span class="token variable">$wpdb</span><span class="token operator">-></span><span class="token property">base_prefix</span><span class="token punctuation">}</span></span>\d+_/i"</span><span class="token punctuation">,</span> <span class="token variable">$wpdb</span><span class="token operator">-></span><span class="token property">table</span> <span class="token punctuation">)</span> <span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token variable">$last_two_digits</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token keyword type-casting">int</span><span class="token punctuation">)</span> <span class="token function">substr</span><span class="token punctuation">(</span> <span class="token variable">$wpdb</span><span class="token operator">-></span><span class="token property">blogid</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">2</span> <span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token variable">$db</span> <span class="token operator">=</span> <span class="token function">sprintf</span><span class="token punctuation">(</span> <span class="token string single-quoted-string">'db%02d'</span><span class="token punctuation">,</span> <span class="token variable">$last_two_digits</span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// db00, db01, db02, ..., db99</span>
<span class="token keyword">return</span> <span class="token variable">$db</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token string single-quoted-string">'global'</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token variable">$wpdb</span><span class="token operator">-></span><span class="token function">add_callback</span><span class="token punctuation">(</span> <span class="token string single-quoted-string">'pb_db_callback'</span> <span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h3 id="export-db-into-101-slices" tabindex="-1">Export DB Into 101 Slices:</h3>
<pre class="language-bash"><code class="language-bash"><span class="token shebang important">#!/bin/bash</span>
<span class="token comment"># This script will CREATE 101 directories in current</span>
<span class="token comment"># working directory, you have been warned!</span>
<span class="token assign-left variable">db</span><span class="token operator">=</span><span class="token string">'old_database_name'</span>
<span class="token function">sudo</span> mydumper <span class="token parameter variable">--regex</span><span class="token operator">=</span><span class="token string">"^<span class="token variable">${db}</span>\.wp_[a-zA-Z]+.*"</span> <span class="token parameter variable">--database</span><span class="token operator">=</span><span class="token string">"<span class="token variable">${db}</span>"</span> <span class="token parameter variable">--outputdir</span><span class="token operator">=</span><span class="token string">"core"</span> --build-empty-files
<span class="token keyword">for</span> <span class="token variable"><span class="token punctuation">((</span>i<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">;</span> i<span class="token operator">&lt;=</span><span class="token number">99</span><span class="token punctuation">;</span> i<span class="token operator">++</span><span class="token punctuation">))</span></span><span class="token punctuation">;</span> <span class="token keyword">do</span>
<span class="token assign-left variable">ii</span><span class="token operator">=</span><span class="token variable"><span class="token variable">`</span><span class="token builtin class-name">printf</span> %02d $i<span class="token variable">`</span></span>
<span class="token function">sudo</span> mydumper <span class="token parameter variable">--regex</span><span class="token operator">=</span><span class="token string">"^<span class="token variable">${db}</span>\.(wp_<span class="token variable">${i}</span>_|wp_\d+<span class="token variable">${ii}</span>_).*"</span> <span class="token parameter variable">--database</span><span class="token operator">=</span><span class="token string">"<span class="token variable">${db}</span>"</span> <span class="token parameter variable">--outputdir</span><span class="token operator">=</span><span class="token string">"<span class="token variable">${ii}</span>"</span> --build-empty-files
<span class="token keyword">done</span></code></pre>
<h3 id="import-101-slices" tabindex="-1">Import 101 Slices:</h3>
<pre class="language-bash"><code class="language-bash"><span class="token shebang important">#!/bin/bash</span>
<span class="token comment"># This script will READ 101 directories in current</span>
<span class="token comment"># working directory</span>
<span class="token assign-left variable">db</span><span class="token operator">=</span><span class="token string">'new_database_name'</span>
<span class="token function">sudo</span> myloader <span class="token parameter variable">--directory</span><span class="token operator">=</span><span class="token string">"core"</span> <span class="token parameter variable">--database</span><span class="token operator">=</span><span class="token string">"<span class="token variable">${db}</span>"</span> --overwrite-tables
<span class="token keyword">for</span> <span class="token variable"><span class="token punctuation">((</span>i<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">;</span> i<span class="token operator">&lt;=</span><span class="token number">99</span><span class="token punctuation">;</span> i<span class="token operator">++</span><span class="token punctuation">))</span></span><span class="token punctuation">;</span> <span class="token keyword">do</span>
<span class="token assign-left variable">ii</span><span class="token operator">=</span><span class="token variable"><span class="token variable">`</span><span class="token builtin class-name">printf</span> %02d $i<span class="token variable">`</span></span>
<span class="token function">sudo</span> myloader <span class="token parameter variable">--directory</span><span class="token operator">=</span><span class="token string">"<span class="token variable">${ii}</span>"</span> <span class="token parameter variable">--database</span><span class="token operator">=</span><span class="token string">"<span class="token variable">${db}</span>_<span class="token variable">${ii}</span>"</span> --overwrite-tables
<span class="token keyword">done</span></code></pre>
<h2 id="did-it-work" tabindex="-1">Did it work?</h2>
<p><a href="https://youtu.be/RA5WCpFY0PE">https://youtu.be/RA5WCpFY0PE</a></p>
<p>To be continued in Part 2...</p>
<h2 id="bonus-tips" tabindex="-1">Bonus tips:</h2>
<p>Because Pressbooks has so many MySQL tables, the Clients I use are always getting stuck or
freezing. Here are some tricks I use to keep sane:</p>
<ul>
<li>Don't let MySQL Workbench load the table schemas. Set up your GUI so that schemas are in
a separate tab, disable autoloading, autocomplete, etc. <em>(Edit ⇨ Preferences ⇨ SQL
Editor)</em></li>
<li>Disable MySQL CLI auto-completion
with <a href="https://dev.mysql.com/doc/refman/5.7/en/mysql-command-options.html#option_mysql_auto-rehash"><code>--disable-auto-rehash</code></a></li>
</ul>
</div>
<footer class="pagination">
<div class="wrapper">
<nav class="pagination__inner" aria-label="Pagination links">
<a href="/blog/2017/09/27/our-new-roadmap/" data-direction="backwards">
<span>&larr; Previous</span>
</a>
<a href="/blog/2017/11/07/moving-half-a-million-database-tables-to-aws-aurora-part-2/" data-direction="forwards">
<span>Next &rarr;</span>
</a>
</nav>
</div>
</footer>
</article>
</main>
<footer class="[ wrapper ]">
<p>&copy; 2025 Pressbooks.</p>
</footer>
</body>
</html>