330 lines
20 KiB
HTML
330 lines
20 KiB
HTML
|
|
|
|
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="UTF-8" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
|
<meta http-equiv="X-UA-Compatible" content="ie=edge" />
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<title>Moving Half a Million Database Tables to AWS Aurora (Part 1) - Pressbooks</title>
|
|
<link rel="canonical" href="https://pressbooks.org/blog/2017/10/19/moving-half-a-million-database-tables-to-aws-aurora-part-1/" />
|
|
|
|
<meta property="og:site_name" content="Pressbooks" />
|
|
<meta property="og:title" content="Moving Half a Million Database Tables to AWS Aurora (Part 1) - Pressbooks" />
|
|
<meta property="og:type" content="website" />
|
|
<meta property="og:url" content="https://pressbooks.org/blog/2017/10/19/moving-half-a-million-database-tables-to-aws-aurora-part-1/" />
|
|
|
|
|
|
|
|
|
|
|
|
<link rel="icon" href="/favicon.ico" sizes="any">
|
|
<link rel="icon" href="/icon.svg" type="image/svg+xml">
|
|
<link rel="apple-touch-icon" href="/apple-touch-icon.png">
|
|
<link rel="manifest" href="/manifest.webmanifest">
|
|
<link rel="alternate" type="application/rss+xml" href="https://pressbooks.org/feed.xml" />
|
|
<link rel="stylesheet" href="/css/main.css">
|
|
<link rel="stylesheet" href="https://unpkg.com/prismjs@^1.24/themes/prism-okaidia.css">
|
|
<link rel="stylesheet" href="https://unpkg.com/prismjs@^1.24/plugins/treeview/prism-treeview.css">
|
|
|
|
<!-- Google tag (gtag.js) -->
|
|
<script async src="https://www.googletagmanager.com/gtag/js?id=G-1KE8PMYEWF"></script>
|
|
<script>
|
|
window.dataLayer = window.dataLayer || [];
|
|
function gtag(){dataLayer.push(arguments);}
|
|
gtag('js', new Date());
|
|
|
|
gtag('config', 'G-1KE8PMYEWF');
|
|
</script></head>
|
|
<body>
|
|
<a class="[ skip-link ] [ button ]" href="#main-content">Skip to content</a>
|
|
<header role="banner" class="site-head">
|
|
<div class="wrapper">
|
|
<div class="site-head__inner">
|
|
<a href="/" aria-label="Pressbooks - home" class="site-head__brand">
|
|
<img src="/images/pressbooks-logo.svg" alt="Pressbooks logo" />
|
|
</a>
|
|
<p class="site-head__subhead">Open publishing. Open web. Open source.</p>
|
|
<nav
|
|
class="[ nav ] [ site-head__nav ]"
|
|
aria-label="Primary navigaton"
|
|
>
|
|
<ul class="nav__list cluster">
|
|
|
|
<li>
|
|
<a href="/"
|
|
>Home</a
|
|
>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="/blog/" data-state="active"
|
|
>Blog</a
|
|
>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="/dev-guides/"
|
|
>Developer Guides</a
|
|
>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="/user-guides/"
|
|
>User Guides</a
|
|
>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="/get-involved/"
|
|
>Get Involved</a
|
|
>
|
|
</li>
|
|
|
|
</ul>
|
|
</nav>
|
|
</div>
|
|
</div>
|
|
</header>
|
|
|
|
|
|
<main tabindex="-1" id="main-content" >
|
|
|
|
<article>
|
|
<div class="[ post ] [ flow wrapper ]">
|
|
<header>
|
|
<h1>Moving Half a Million Database Tables to AWS Aurora (Part 1)</h1>
|
|
<p class="meta">
|
|
<p class="author vcard"><span class="visually-hidden">Posted by </span><span class="fn n">Dac Chartrand</span></p>
|
|
<time datetime="2017-10-19T12:00:00.000Z">October 19th, 2017</time>
|
|
</p>
|
|
|
|
<div class="tags">
|
|
<svg xmlns="http://www.w3.org/2000/svg" class="tags-icon" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2">
|
|
<path stroke-linecap="round" stroke-linejoin="round" d="M7 7h.01M7 3h5c.512 0 1.024.195 1.414.586l7 7a2 2 0 010 2.828l-7 7a2 2 0 01-2.828 0l-7-7A1.994 1.994 0 013 12V7a4 4 0 014-4z" />
|
|
</svg>
|
|
<ul class="tags-list" aria-label="Tags">
|
|
|
|
<li>
|
|
<a href="/tag/core/">Core</a>
|
|
</li>
|
|
|
|
</ul>
|
|
</div>
|
|
|
|
</header>
|
|
|
|
<p>This post is about migrating Pressbooks.com to AWS.</p>
|
|
<h2 id="does-it-scale" tabindex="-1">Does It Scale?</h2>
|
|
<p>At <a href="/">Pressbooks</a> we use
|
|
<a href="https://codex.wordpress.org/Create_A_Network">WordPress Multisite</a> as a development
|
|
platform. <a href="https://github.com/pressbooks/pressbooks">Pressbooks changes WordPress</a> and
|
|
makes every blog a book.</p>
|
|
<p>The prevailing wisdom of the day is that a relational database should have a manageable
|
|
set of tables with lots of rows of data in them ...and then there's WordPress.</p>
|
|
<p>WP Multisite creates
|
|
<a href="https://codex.wordpress.org/Database_Description#Site_Specific_Tables">10 database tables</a>
|
|
for every blog <em>(or in our case: for every book.)</em> Tables share IDs but they are not
|
|
enforced at the database level. There are no foreign key constraints, no triggers, nor
|
|
routines. It's simple no frills MySQL.</p>
|
|
<p>Pressbooks dot com is currently running on a single bare metal server. <em>(Our hosted
|
|
instances are already on AWS)</em> This server has a single MariaDB database with 60,000 books
|
|
in it. When we do the math that's over 600,000 tables in one database.
|
|
<a href="https://www.askbjoernhansen.com/2008/02/14/10000_tables_in_one_mysql_database.html">Are you nuts?!</a>
|
|
Unusual? Horrible? Yet
|
|
<a href="https://dev.mysql.com/doc/refman/5.7/en/database-count-limit.html">entirely possible</a>,
|
|
<a href="https://www.percona.com/blog/2017/10/01/one-million-tables-mysql-8-0/">even plausible</a>.</p>
|
|
<p>I've met people who worked at Automattic and
|
|
<a href="https://www.meetup.com/wp-mtl/events/240377606/">from the stories, I heard</a> WordPress dot
|
|
com uses the same WP Multisite technology but instead of half a million tables it's over a
|
|
billion tables <em>(probably not all in the same database though, more on that later)</em>.</p>
|
|
<p>I call this the Schrödinger's Cat of database design because so long as we don't look it's
|
|
alive?</p>
|
|
<h2 id="prerequisites" tabindex="-1">Prerequisites</h2>
|
|
<p>As I said, we already host hundreds of WP Multisite networks on AWS. We build, manage, and
|
|
deploy to our infrastructure using <a href="https://www.terraform.io/">Terraform</a>,
|
|
<a href="https://www.ansible.com/">Ansible</a>, <a href="https://wp-cli.org/">wp-cli</a> and
|
|
<a href="https://github.com/roots/trellis">all things competent</a>. We simply just, sort of, well,
|
|
neglected to move our freemium site over because we were too busy.</p>
|
|
<p>The time has come.</p>
|
|
<h2 id="research" tabindex="-1">Research</h2>
|
|
<p>We tried <code>mysqldump</code>. It was too slow. Our tests showed that a dump would take days.</p>
|
|
<p>Some colleagues recommended <a href="https://aws.amazon.com/dms/">AWS DMS</a>. It did not work. Some
|
|
reasons:</p>
|
|
<ul>
|
|
<li><a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MySQL.html#CHAP_Source.MySQL.Limitations">The <code>AUTO_INCREMENT</code> attribute is not migrated.</a></li>
|
|
<li>WP Multisite tables are prefixed <code>wp_1_, wp_2_, wp_3_, ...</code> MySQL considers the
|
|
underscore a
|
|
<a href="https://stackoverflow.com/questions/8236818/why-does-underscore-match-hyphen">one character wildcard</a>
|
|
when used in LIKE queries, DMS provided no way to escape it for table filters.</li>
|
|
<li>Unexplained crashing on anything less than a <code>c4.large</code> <em>(failed last Error The task
|
|
stopped abnormally Stop Reason RECOVERABLE_ERROR Error Level RECOVERABLE)</em></li>
|
|
<li>Mostly, the fastest migration we could get going, running 4 tasks in parallel, was an
|
|
ETA of 10 days</li>
|
|
</ul>
|
|
<p>We asked for help. Ned had a conference call with a reputable consulting firm and they gave us a quote: $34K USD + travel & on-site
|
|
expenses.</p>
|
|
<p><img src="/images/coffee-spitting.gif" alt="Coffee spitting "></p>
|
|
<p>Next, our research led us to <a href="https://github.com/maxbube/mydumper">mydumper</a>. From the
|
|
README:</p>
|
|
<blockquote>
|
|
<p>== What is mydumper? Why? ==</p>
|
|
<ul>
|
|
<li>Parallelism (hence, speed) and performance (avoids expensive character set conversion
|
|
routines, efficient code overall)</li>
|
|
<li>Easier to manage output (separate files for tables, dump metadata, etc, easy to
|
|
view/parse data)</li>
|
|
<li>Consistency - maintains snapshot across all threads, provides accurate master and
|
|
slave log positions, etc</li>
|
|
<li>Manageability - supports PCRE for specifying database and tables inclusions and
|
|
exclusions</li>
|
|
</ul>
|
|
</blockquote>
|
|
<p>So far so good...</p>
|
|
<blockquote>
|
|
<p>== How to build it? ==</p>
|
|
</blockquote>
|
|
<p><img src="/images/jerry-seinfeld-leaving.gif" alt="Jerry Seinfeld leaving"></p>
|
|
<p>Just kidding. It turns out we don't have to build mydumper. On
|
|
Ubuntu <code>sudo apt install mydumper</code> works fine. Similar command using <code>yum</code> on CentOS.</p>
|
|
<p>Our tests conclude that mydumper finishes in hours instead of days.</p>
|
|
<h2 id="plan" tabindex="-1">Plan</h2>
|
|
<p>It is our opinion that this kind of problem is better suited for a
|
|
<a href="https://en.wikipedia.org/wiki/Document-oriented_database">document-oriented database</a>.
|
|
Given that this is the database design we inherited, there's not much we can do about it,
|
|
so we'll try our best with what we've got. ¯\<em>(ツ)</em>/¯</p>
|
|
<p>At a billion tables, Automattic has already established its own internal best practices
|
|
with plugins like <a href="https://github.com/Automattic/hyperdb">HypderDB</a>. Unfortunately HyperDB
|
|
doesn't have <a href="https://getcomposer.org/">Composer</a> support and doesn't look
|
|
maintained. <a href="https://github.com/stuttter/ludicrousdb">LudicrousDB</a>, a Composer compatible
|
|
drop-in that works with our <a href="https://github.com/roots/bedrock/">existing tech stack</a>, to
|
|
the rescue.</p>
|
|
<blockquote>
|
|
<p>LudicrousDB is an advanced database interface for WordPress that supports replication,
|
|
fail-over, load balancing, and partitioning, based on Automattic's HyperDB drop-in.</p>
|
|
</blockquote>
|
|
<p>With LudicrousDB tested and working, we are moving towards a 101 slice approach. 1 slice
|
|
for core tables and 100 slices for books.</p>
|
|
<p>The idea is to use the last <strong>two digits</strong> of a book ID to pick one of 100 slices. If this
|
|
becomes unmanageable in the future <em>(important to remember that we already have over half
|
|
a million</em> tables <em>in 1 database and
|
|
<a href="https://knowyourmeme.com/memes/this-is-fine">things are fine</a>)</em>, we can change the
|
|
splitting algorithm by adding a condition to use the last <strong>X digits</strong> on books with IDs
|
|
bigger than <strong>Y</strong>.</p>
|
|
<h2 id="code" tabindex="-1">Code</h2>
|
|
<p><em>For informational purposes only. Read the snippets and reason about them. Copy/paste at
|
|
your own peril.</em></p>
|
|
<h2 id="ludicrous-db-callback" tabindex="-1">LudicrousDB Callback</h2>
|
|
<pre class="language-php"><code class="language-php"><span class="token comment">/**
|
|
* Slices
|
|
*
|
|
* We can predict what slice a blog is in by looking
|
|
* at the last two digits of the id. Examples:
|
|
*
|
|
* + blog_id: 9, in db09
|
|
* + blog_id: 74, in db74
|
|
* + blog_id: 999989, in db89
|
|
* + blog_id: 9200, in db00
|
|
*
|
|
* @param $query
|
|
* @param \LudicrousDB $wpdb
|
|
*
|
|
* @return string
|
|
*/</span>
|
|
<span class="token keyword">function</span> <span class="token function-definition function">pb_db_callback</span><span class="token punctuation">(</span> <span class="token variable">$query</span><span class="token punctuation">,</span> <span class="token variable">$wpdb</span> <span class="token punctuation">)</span> <span class="token punctuation">{</span>
|
|
<span class="token keyword">if</span> <span class="token punctuation">(</span> <span class="token function">preg_match</span><span class="token punctuation">(</span> <span class="token string double-quoted-string">"/^<span class="token interpolation"><span class="token punctuation">{</span><span class="token variable">$wpdb</span><span class="token operator">-></span><span class="token property">base_prefix</span><span class="token punctuation">}</span></span>\d+_/i"</span><span class="token punctuation">,</span> <span class="token variable">$wpdb</span><span class="token operator">-></span><span class="token property">table</span> <span class="token punctuation">)</span> <span class="token punctuation">)</span> <span class="token punctuation">{</span>
|
|
<span class="token variable">$last_two_digits</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token keyword type-casting">int</span><span class="token punctuation">)</span> <span class="token function">substr</span><span class="token punctuation">(</span> <span class="token variable">$wpdb</span><span class="token operator">-></span><span class="token property">blogid</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">2</span> <span class="token punctuation">)</span><span class="token punctuation">;</span>
|
|
<span class="token variable">$db</span> <span class="token operator">=</span> <span class="token function">sprintf</span><span class="token punctuation">(</span> <span class="token string single-quoted-string">'db%02d'</span><span class="token punctuation">,</span> <span class="token variable">$last_two_digits</span> <span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// db00, db01, db02, ..., db99</span>
|
|
<span class="token keyword">return</span> <span class="token variable">$db</span><span class="token punctuation">;</span>
|
|
<span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>
|
|
<span class="token keyword">return</span> <span class="token string single-quoted-string">'global'</span><span class="token punctuation">;</span>
|
|
<span class="token punctuation">}</span>
|
|
<span class="token punctuation">}</span>
|
|
<span class="token variable">$wpdb</span><span class="token operator">-></span><span class="token function">add_callback</span><span class="token punctuation">(</span> <span class="token string single-quoted-string">'pb_db_callback'</span> <span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
|
|
<h3 id="export-db-into-101-slices" tabindex="-1">Export DB Into 101 Slices:</h3>
|
|
<pre class="language-bash"><code class="language-bash"><span class="token shebang important">#!/bin/bash</span>
|
|
|
|
<span class="token comment"># This script will CREATE 101 directories in current</span>
|
|
<span class="token comment"># working directory, you have been warned!</span>
|
|
|
|
<span class="token assign-left variable">db</span><span class="token operator">=</span><span class="token string">'old_database_name'</span>
|
|
|
|
<span class="token function">sudo</span> mydumper <span class="token parameter variable">--regex</span><span class="token operator">=</span><span class="token string">"^<span class="token variable">${db}</span>\.wp_[a-zA-Z]+.*"</span> <span class="token parameter variable">--database</span><span class="token operator">=</span><span class="token string">"<span class="token variable">${db}</span>"</span> <span class="token parameter variable">--outputdir</span><span class="token operator">=</span><span class="token string">"core"</span> --build-empty-files
|
|
<span class="token keyword">for</span> <span class="token variable"><span class="token punctuation">((</span>i<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">;</span> i<span class="token operator"><=</span><span class="token number">99</span><span class="token punctuation">;</span> i<span class="token operator">++</span><span class="token punctuation">))</span></span><span class="token punctuation">;</span> <span class="token keyword">do</span>
|
|
<span class="token assign-left variable">ii</span><span class="token operator">=</span><span class="token variable"><span class="token variable">`</span><span class="token builtin class-name">printf</span> %02d $i<span class="token variable">`</span></span>
|
|
<span class="token function">sudo</span> mydumper <span class="token parameter variable">--regex</span><span class="token operator">=</span><span class="token string">"^<span class="token variable">${db}</span>\.(wp_<span class="token variable">${i}</span>_|wp_\d+<span class="token variable">${ii}</span>_).*"</span> <span class="token parameter variable">--database</span><span class="token operator">=</span><span class="token string">"<span class="token variable">${db}</span>"</span> <span class="token parameter variable">--outputdir</span><span class="token operator">=</span><span class="token string">"<span class="token variable">${ii}</span>"</span> --build-empty-files
|
|
<span class="token keyword">done</span></code></pre>
|
|
<h3 id="import-101-slices" tabindex="-1">Import 101 Slices:</h3>
|
|
<pre class="language-bash"><code class="language-bash"><span class="token shebang important">#!/bin/bash</span>
|
|
|
|
<span class="token comment"># This script will READ 101 directories in current</span>
|
|
<span class="token comment"># working directory</span>
|
|
|
|
<span class="token assign-left variable">db</span><span class="token operator">=</span><span class="token string">'new_database_name'</span>
|
|
|
|
<span class="token function">sudo</span> myloader <span class="token parameter variable">--directory</span><span class="token operator">=</span><span class="token string">"core"</span> <span class="token parameter variable">--database</span><span class="token operator">=</span><span class="token string">"<span class="token variable">${db}</span>"</span> --overwrite-tables
|
|
<span class="token keyword">for</span> <span class="token variable"><span class="token punctuation">((</span>i<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">;</span> i<span class="token operator"><=</span><span class="token number">99</span><span class="token punctuation">;</span> i<span class="token operator">++</span><span class="token punctuation">))</span></span><span class="token punctuation">;</span> <span class="token keyword">do</span>
|
|
<span class="token assign-left variable">ii</span><span class="token operator">=</span><span class="token variable"><span class="token variable">`</span><span class="token builtin class-name">printf</span> %02d $i<span class="token variable">`</span></span>
|
|
<span class="token function">sudo</span> myloader <span class="token parameter variable">--directory</span><span class="token operator">=</span><span class="token string">"<span class="token variable">${ii}</span>"</span> <span class="token parameter variable">--database</span><span class="token operator">=</span><span class="token string">"<span class="token variable">${db}</span>_<span class="token variable">${ii}</span>"</span> --overwrite-tables
|
|
<span class="token keyword">done</span></code></pre>
|
|
<h2 id="did-it-work" tabindex="-1">Did it work?</h2>
|
|
<p><a href="https://youtu.be/RA5WCpFY0PE">https://youtu.be/RA5WCpFY0PE</a></p>
|
|
<p>To be continued in Part 2...</p>
|
|
<h2 id="bonus-tips" tabindex="-1">Bonus tips:</h2>
|
|
<p>Because Pressbooks has so many MySQL tables, the Clients I use are always getting stuck or
|
|
freezing. Here are some tricks I use to keep sane:</p>
|
|
<ul>
|
|
<li>Don't let MySQL Workbench load the table schemas. Set up your GUI so that schemas are in
|
|
a separate tab, disable autoloading, autocomplete, etc. <em>(Edit ⇨ Preferences ⇨ SQL
|
|
Editor)</em></li>
|
|
<li>Disable MySQL CLI auto-completion
|
|
with <a href="https://dev.mysql.com/doc/refman/5.7/en/mysql-command-options.html#option_mysql_auto-rehash"><code>--disable-auto-rehash</code></a></li>
|
|
</ul>
|
|
|
|
</div>
|
|
|
|
|
|
|
|
<footer class="pagination">
|
|
<div class="wrapper">
|
|
<nav class="pagination__inner" aria-label="Pagination links">
|
|
|
|
<a href="/blog/2017/09/27/our-new-roadmap/" data-direction="backwards">
|
|
<span>← Previous</span>
|
|
</a>
|
|
|
|
|
|
<a href="/blog/2017/11/07/moving-half-a-million-database-tables-to-aws-aurora-part-2/" data-direction="forwards">
|
|
<span>Next →</span>
|
|
</a>
|
|
|
|
</nav>
|
|
</div>
|
|
</footer>
|
|
|
|
</article>
|
|
|
|
</main>
|
|
|
|
<footer class="[ wrapper ]">
|
|
<p>© 2025 Pressbooks.</p>
|
|
</footer>
|
|
</body>
|
|
</html>
|