13 lines
17 KiB
HTML
13 lines
17 KiB
HTML
<!doctype html><html lang=en-us><head><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge,chrome=1"><title>Migrations: the sole scalable fix to tech debt. | Irrational Exuberance</title><meta name=viewport content="width=device-width,minimum-scale=1"><meta name=description content="Migrations are both essential and frustratingly frequent as your codebase ages and your business grows: most tools and processes only support about one order magnitude of growth before becoming ineffective, so rapid growth makes them a way of life. This post takes a look at why migrations are so important, and also how to run them effectively."><meta name=generator content="Hugo 0.164.0"><meta name=ROBOTS content="INDEX, FOLLOW"><link rel=stylesheet href=/ananke/dist/main.css_5c99d70a7725bacd4c701e995b969fea.css><link rel=stylesheet href=/static/pygments.css><meta property="og:title" content="Migrations: the sole scalable fix to tech debt."><meta property="og:description" content="Migrations are both essential and frustratingly frequent as your codebase ages and your business grows: most tools and processes only support about one order magnitude of growth before becoming ineffective, so rapid growth makes them a way of life. This post takes a look at why migrations are so important, and also how to run them effectively."><meta property="og:type" content="article"><meta property="og:url" content="https://lethain.com/migrations/"><meta property="og:image" content="https://lethain.com/static/blog/2018/migrations-hero.png"><meta property="article:section" content="posts"><meta property="article:published_time" content="2018-04-15T06:00:00-07:00"><meta property="article:modified_time" content="2018-04-15T06:00:00-07:00"><meta itemprop=name content="Migrations: the sole scalable fix to tech debt."><meta itemprop=description content="Migrations are both essential and frustratingly frequent as your codebase ages and your business grows: most tools and processes only support about one order magnitude of growth before becoming ineffective, so rapid growth makes them a way of life. This post takes a look at why migrations are so important, and also how to run them effectively."><meta itemprop=datePublished content="2018-04-15T06:00:00-07:00"><meta itemprop=dateModified content="2018-04-15T06:00:00-07:00"><meta itemprop=wordCount content="1222"><meta itemprop=image content="https://lethain.com/static/author.png"><meta itemprop=keywords content="Infrastructure,Management"><meta name=twitter:card content="summary"><meta name=twitter:image content="https://lethain.com/static/blog/2018/migrations-hero.png"><meta name=twitter:title content="Migrations: the sole scalable fix to tech debt."><meta name=twitter:description content="Migrations are both essential and frustratingly frequent as your codebase ages and your business grows: most tools and processes only support about one order magnitude of growth before becoming ineffective, so rapid growth makes them a way of life. This post takes a look at why migrations are so important, and also how to run them effectively."><script defer data-domain=lethain.com src=https://plausible.io/js/script.js></script></head><body class="ma0 avenir bg-white production"><header><div class=bg-white><nav class="pv3 ph3 ph4-ns" role=navigation><div class="flex-l justify-between items-center center"><a href=/ class="f3 fw2 no-underline black-90 dib">Irrational Exuberance</a><div class="flex-l items-center"><ul class="pl0 mr3"><li class="list f5 f4-ns fw4 dib pr3"><a class="no-underline black-90" href=/featured/ title="Popular page">Popular</a></li><li class="list f5 f4-ns fw4 dib pr3"><a class="no-underline black-90" href=/tags/ title="Tags page">Tags</a></li><li class="list f5 f4-ns fw4 dib pr3"><a class="no-underline black-90" href=/newsletter/ title="Newsletter page">Newsletter</a></li><li class="list f5 f4-ns fw4 dib pr3"><a class="no-underline black-90" href=/feeds.xml title="RSS page">RSS</a></li><li class="list f5 f4-ns fw4 dib pr3"><a class="no-underline black-90" href=/about title="About page">About</a></li></ul></div></div></nav></div></header><main class=pb7 role=main><article class="flex-l flex-wrap justify-between mw8 center ph3"><header class="mt4 w-100"><img class=w-100 src=https://lethain.com/static/blog/2018/migrations-hero.png><h1 class="f1 athelas mt3 mb1">Migrations: the sole scalable fix to tech debt.</h1><div class=mt4><a href=/tags/infrastructure class="link f5 mid-gray no-underline sans-serif">infrastructure (57),
|
|
</a><a href=/tags/management class="link f5 mid-gray no-underline sans-serif">management (220)</a></div></header><div class="nested-copy-line-height lh-copy serif f5 nested-links nested-img mid-gray pr4-l w-two-thirds-l"><p>The most interesting migration I ever participated in was Uber’s migration from Puppet-managed services to a fully self-service provisioning model where any engineer at the company could spin up a new service in two clicks. Not only could they, they did, provisioning multiple services each day by the time the service was complete, and every newly hired engineer spinning up a service from scratch their first day.</p><p>What made this migration so interesting was the volume. When we started, provisioning a new service took about two weeks of clock time and about two days of engineering time, and we were falling further behind each day. At the time it was a more-than-just-a-little stressful, but it was also a perfect laboratory to learn how to run large-scale software migrations: it was large enough to see even small shifts, and long enough that we got to experiment with a number of approaches.</p><p>Migrations are both essential and frustratingly frequent as your codebase ages and your business grows: most tools and processes <a href=https://lethain.com/productivity-in-the-age-of-hypergrowth/>only support about one order of magnitude of growth</a> before becoming ineffective, so rapid growth makes them a way of life. This isn’t because they’re bad processes or poor tools, quite the opposite: the fact that something stops working at significantly increased scale is a sign that it was designed appropriately to the previous constraints <a href=https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it>rather than being over designed</a>.</p><p>As a result you switch tools a lot, and your ability to migrate to new software can easily become the defining constraint for your overall velocity. Given their importance, we don’t talk about running migrations very often; let’s remedy that!</p><h2 id=why-migrations-matter>Why migrations matter</h2><p>Migrations matter because they are usually the only available avenue to make meaningful progress on technical debt.</p><p>Engineers hate technical debt. If there is an easy project they can personally do to reduce tech debt, they’ll take it on themselves. Engineering managers hate technical debt, too. If there is an easy project their team can execute in isolation, they’ll get it scheduled. In aggregate, this leads to a dynamic where there is very little low-hanging fruit to reduce technical debt, and most remaining options require many teams working together to implement them: migrations.</p><p>Each migration aims to create technical leverage (“your indexes no longer have to fit on a single server!”) or reduce technical debt (“your acknowledged writes are guaranteed to persist a master failover”). They occupy the awkward territory of reduced immediate contribution today in exchange for more capacity tomorrow. This makes them controversial to schedule, and as your systems become larger, they become more expensive.</p><p>Lore tells us that Googlers have a phrase, “Running to stand still”, to describe a team whose entire capacity is consumed in upgrading dependencies and patterns, such that it can’t make forward progress on the product/system they own. Spending <em>all</em> your time on migrations is extreme, but every mid-sized company has a long queue of migrations it can’t staff: moving from VMs to containers, rolling out circuit-breaking, moving to the new build tool; the list extends effortlessly into the sunset.</p><p>Migrations are the only mechanism to effectively manage technical debt as your company and code grow. If you don’t get effective at software and system migrations, you’ll end up languishing in technical debt. (And still have to do one later anyway, it’s just that it’ll probably be a full rewrite.)</p><h2 id=running-good-migrations>Running good migrations</h2><p>The good news is that while migrations are hard, there is a pretty standard playbook that works remarkably well: Derisk, Enable, and then Finish.</p><h3 id=derisk>Derisk</h3><p>The first phase of a migration is <strong>derisking</strong> it, and to do so as quickly and cheaply as possible. Write a <strong>design document</strong> and shop it with the teams that you believe will have the hardest time migrating. Iterate. Shop it with teams who have atypical patterns and edge cases. Iterate. Test it against the next six to twelve months of roadmap. Iterate.</p><p>After you’ve evolved the design, the next step is to <strong>embed into the most challenging one or two teams</strong>, and work side by side with those teams to build, evolve and migrate to the new system. Don’t start with the easiest migrations, which can lead to a false sense of security.</p><p>Effective derisking is essential, because <strong>each team that endorses a migration is making a bet on you</strong> that you’re going to get this damn thing done, and not leave them with a migration to an abandoned system that they have to revert. If you leave one migration partially finished, folks will be exceedingly suspicious of participating in the next.</p><h3 id=enable>Enable</h3><p>Once you’ve validated the solution solves the intended problem, it’s time to start sharpening your tools. Many folks start migrations by generating tracking tickets for teams to implement, but it’s better to slow down and build tooling to <strong><a href=https://lethain.com/refactoring-programmatically/>programmatically migrate the easy ninety-percent</a></strong>. This radically reduces the migration’s cost to the broader organization, which increases their success rate and creates more future opportunities to migrate.</p><p>Once you’ve handled as much of the migration programmatically as possible, figure out the <strong>self-service tooling and documentation</strong> you can provide to allow folks to make the necessary changes without getting stuck. The best migration tools are incremental and reversible: folks should be able to immediately return to previous behavior if something goes wrong, and have the necessary expressiveness to derisk their particular migration path.</p><p>Documentation and self-service tooling are products, and thrive under the same regime: sit down with some teams and watch them follow your instructions, then improve them. Find another team, repeat. Spending an extra two days intentionally making your documentation clean and tools intuitive can save years in large migrations. Do it!</p><h3 id=finish>Finish</h3><p>The last phase of a migration is deprecating the legacy system you’ve replaced. This requires getting to 100% adoption, and that can be quite challenging.</p><p>Start by <strong>stopping the bleeding</strong>, which is ensuring that all newly written code uses the new approach. That can be installing a ratchet in your linters, or updating your documentation and self-service tooling. This is always the first step, because it turns time into your friend. Instead of falling behind by default, you’re now making progress by default.</p><p>Ok, now you should start <strong>generating tracking tickets</strong>, and a mechanism which <strong>pushes migration status</strong> to teams that need to migrate and to the general management structure. It’s important to give wider management context around migrations because they are the folks who need to prioritize the migrations; if a team isn’t working on a migration, it’s typically because their leadership has not prioritized it.</p><p>At this point you’re pretty close to complete, but have the long tail of weird or unstaffed. Your tool now is <strong>finish it yourself</strong>. It’s not necessarily fun, but getting to 100% is going to require the team leading the migration to dig into the nooks and crannies themselves.</p><p>My final tip for finishing migrations is around recognition. It’s important to celebrate migrations while they’re ongoing, but the majority of the celebration and <strong>recognition should be reserved for its successful completion</strong>. In particular, starting but not finishing migrations often incurs significant technical debt, so your incentives and recognition structure should be careful to avoid perverse incentives.</p><hr><p>What have you seen make migrations more effective? What are some of the anti-patterns you’ve experienced?</p><p>Thanks to <a href=https://twitter.com/ingridavendano>Ingrid</a>, <a href=https://twitter.com/JorgeO>Jorge</a> and <a href=https://twitter.com/b0rk>Julia</a> for shaping this post.</p><div class="mt4 w-100"><time class="f6 mv4 dib tracked" datetime=2018-04-15T06:00:00-07:00>Published on April 15, 2018.</time></div><div class="mt6 instapaper_ignoref"></div></div><aside class="w-30-l mt1-l"><div class="bg-light-gray pa3 cf"><p class="f5 mb3">Hi folks. I'm <a href=/about>Will Larson</a>.</p><p class=f5>If you're looking to reach out to me, here are <a href=/ways-i-help/>ways I help</a>.</p><p class="f5 b mb2">Books</p><p class=f5>I wrote
|
|
<a href=https://www.amazon.com/Elegant-Puzzle-Systems-Engineering-Management/dp/1732265186>An Elegant Puzzle</a>,
|
|
<a href=https://staffeng.com/book>Staff Engineer</a>,
|
|
<a href=https://www.amazon.com/Engineering-Executives-Primer-Impactful-Leadership/dp/1098149483/>The Engineering Executive's Primer</a>, and
|
|
<a href=https://craftingengstrategy.com/>Crafting Engineering Strategy</a>.</p><p class=cf><div class="fl w-25"><a href=https://www.amazon.com/Elegant-Puzzle-Systems-Engineering-Management/dp/1732265186><img src=/static/blog/2019/aep-small-lq.jpg></a></div><div class="fl w-25"><a href=https://staffeng.com/book><img src=/static/blog/staffeng/StaffEngBookMed.jpg></a></div><div class="fl w-25"><a href=https://www.amazon.com/Engineering-Executives-Primer-Impactful-Leadership/dp/1098149483/><img src=/static/blog/2023/primer-cover-small.jpg></a></div><div class="fl w-25"><a href=https://craftingengstrategy.com/><img src=/static/ces_cover_small.png></a></div></p></div><div class="bg-light-gray pa3 nested-list-reset nested-copy-line-height nested-links"><div class=pb3><p class="f5 b mb2">Newsletter</p><p class="f6 mt0 mb3">If you'd like to get get updates, <a href=/newsletter/>subscribe</a> for weekly emails, or follow my <a href=/feeds.xml>RSS feed</a>.</p><form action=https://buttondown.com/api/emails/embed-subscribe/lethain method=post><input type=email name=email placeholder=you@example.com required class="input-reset ba b--black-20 pa2 w-100 br2 mb2">
|
|
<input type=hidden name=tag value=lethain.com>
|
|
<input type=hidden name=embed value=1>
|
|
<input type=submit value=Subscribe class="b ph3 pv2 input-reset ba b--black bg-white black pointer f6 br2 grow"></form></div><div class=pb3><p class="f5 b mb3">Popular</p><ul class="pa0 list f5"><li class=mb2><a href=/agents-series/>Building internal agents</a></li><li class=mb2><a href=/good-eng-mgmt-is-a-fad/>"Good engineering management" is a fad</a></li><li class=mb2><a href=/orchestration-heavy-leadership-heavy/>Moving from an orchestration-heavy to leadership-heavy management role.</a></li><li class=mb2><a href=/engineering-cost-model/>Eng org seniority-mix model.</a></li><li class=mb2><a href=/quality/>How to create software quality.</a></li></ul></div><div class=pb3><p class="f5 b mb3">Recent</p><ul class="pa0 list f5"><li class=mb2><a href=/decisions-not-dates/>Roadmap decisions rather than dates.</a></li><li class=mb2><a href=/middle-management-roles-were-also-a-trap/>Middle management roles are also a trap.</a></li><li class=mb2><a href=/generated-demand/>Generated and suppressed demand.</a></li><li class=mb2><a href=/make-no-assumptions/>Make no assumptions.</a></li><li class=mb2><a href=/revised-rules-of-engineering-leadership/>Revised rules of engineering leadership.</a></li></ul></div><div><p class="f5 b mb3">Related</p><ul class="pa0 list f5"><li class=mb2><a href=/product-management-infra-engineering/>Product management in infrastructure eng.</a></li><li class=mb2><a href=/things-learned-in-2017/>Engineering management stuff I learned in 2017.</a></li><li class=mb2><a href=/infrastructure-between-cost-center-and-before-ego-trip/>Infrastructure between cost center and ego trip</a></li><li class=mb2><a href=/redis-protocol/>The Redis Protocol is pretty great.</a></li><li class=mb2><a href=/some-of-my-favorite-technical-papers/>Some of my favorite technical papers.</a></li></ul></div></div></aside></article></main><footer class="bg-grey bottom-0 pa3 tc" role=contentinfo><div class="flex justify-between"><a class="f6 fw4 no-underline black-50 dn dib-ns pv2 ph3" href=https://lethain.com/>© Will Larson 2026
|
|
</a><a class="f5 fw4 no-underline black-50 dn dib-ns pv2 ph4" href=/tags/>Tags
|
|
</a><a class="f5 fw4 no-underline black-50 dn dib-ns pv2 ph4" href=/newsletter/>Newsletter
|
|
</a><a class="f5 fw4 no-underline black-50 dn dib-ns pv2 ph4" href=/feeds.xml>RSS
|
|
</a><a class="f5 fw4 no-underline black-50 dn dib-ns pv2 ph4" href=/about/>About</a></div></footer></body></html> |