Files
nexus/sreweekly/articles/405/06-zero-downtime-postgres-upgrades.html
2026-09-12 17:23:01 +08:00

625 lines
285 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html><html lang="en" data-scroll-behavior="smooth"><head><meta charSet="utf-8" data-next-head=""/><meta name="viewport" content="width=device-width" data-next-head=""/><title data-next-head="">Zero downtime Postgres upgrades | Knock</title><meta name="robots" content="index,follow" data-next-head=""/><meta name="description" content="How we upgraded from Postgres 11.9 to 15.3 with zero downtime" data-next-head=""/><meta name="twitter:card" content="summary_large_image" data-next-head=""/><meta name="twitter:site" content="@knocklabs" data-next-head=""/><meta name="twitter:creator" content="@knocklabs" data-next-head=""/><meta property="og:title" content="Zero downtime Postgres upgrades | Knock" data-next-head=""/><meta property="og:description" content="How we upgraded from Postgres 11.9 to 15.3 with zero downtime" data-next-head=""/><meta property="og:url" content="https://knock.app/blog/zero-downtime-postgres-upgrades" data-next-head=""/><meta property="og:type" content="website" data-next-head=""/><meta property="og:image" content="https://knock.app/assets/blog/preview/zero-downtime-postgres-upgrades-og.png" data-next-head=""/><meta property="og:image:alt" content="Zero downtime Postgres upgrades | Knock" data-next-head=""/><meta property="og:image:width" content="1200" data-next-head=""/><meta property="og:image:height" content="630" data-next-head=""/><meta property="og:site_name" content="Knock" data-next-head=""/><link rel="canonical" href="https://knock.app/blog/zero-downtime-postgres-upgrades" data-next-head=""/><meta name="theme-color" content="#000" data-next-head=""/><meta name="application-name" content="Knock" data-next-head=""/><meta name="msapplication-TileColor" content="#000000" data-next-head=""/><meta name="msapplication-config" content="/favicon/browserconfig.xml" data-next-head=""/><link rel="icon" href="/favicon/favicon-dark.svg" data-next-head=""/><link rel="mask-icon" href="/favicon/favicon.svg" color="#000" data-next-head=""/><link rel="preload" href="/fonts/open-sauce/OpenSauceOne-Bold.woff" as="woff" data-next-head=""/><link rel="preload" href="/fonts/open-sauce/OpenSauceOne-Bold.woff2" as="woff2" data-next-head=""/><link rel="preload" href="/fonts/open-sauce/OpenSauceOne-Medium.woff" as="woff" data-next-head=""/><link rel="preload" href="/fonts/open-sauce/OpenSauceOne-Medium.woff2" as="woff2" data-next-head=""/><link rel="preload" href="/assets/product-menu/workflows-default-noGrid.png" as="image" data-next-head=""/><link rel="preload" href="/assets/product-menu/workflows-hover-noGrid.png" as="image" data-next-head=""/><link rel="preload" href="/assets/product-menu/guides-default-noGrid.png" as="image" data-next-head=""/><link rel="preload" href="/assets/product-menu/guides-hover-noGrid.png" as="image" data-next-head=""/><link rel="preload" href="/assets/product-menu/broadcasts-default-noGrid.png" as="image" data-next-head=""/><link rel="preload" href="/assets/product-menu/broadcasts-hover-noGrid.png" as="image" data-next-head=""/><link rel="preload" href="/assets/product-menu/agents-default-noGrid.png" as="image" data-next-head=""/><link rel="preload" href="/assets/product-menu/agents-hover-noGrid.png" as="image" data-next-head=""/><link rel="preload" href="/assets/blog/how-nellis-auction-doubled-engagement/cover-image.png" as="image" data-next-head=""/><link rel="preload" href="/_next/static/immutable/media/83afe278b6a6bb3c-s.p.45535valc9rzk.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/_next/static/immutable/media/c50f3c9c65fbdb75-s.p.18_orl2af6obj.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/_next/static/immutable/media/OpenSauceOne_Regular-s.p.00k21ulu5-xem.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/_next/static/immutable/media/OpenSauceOne_Medium-s.p.0la99f4snd655.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/_next/static/immutable/media/OpenSauceOne_Bold-s.p.0la020ehtwap5.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/_next/static/immutable/media/PPNeueMontreal_Medium-s.p.3g0kl44yumdnh.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/_next/static/immutable/media/PPNeueMontreal_Bold-s.p.09hir3szn1_sq.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/_next/static/immutable/chunks/3ld5x6_gysw72.css" as="style"/><link rel="preload" href="/_next/static/immutable/chunks/09kkgfji_b4dk.css" as="style"/><link rel="preload" href="/_next/static/immutable/chunks/1rxs0tdjm4sjw.css" as="style"/><link rel="preload" href="/_next/static/immutable/chunks/1jrowddgarsos.css" as="style"/><script type="application/ld+json" data-next-head="">{"@context":"https://schema.org","@type":"WebSite","name":"Knock","url":"https://knock.app","description":"How we upgraded from Postgres 11.9 to 15.3 with zero downtime"}</script><script type="application/ld+json" data-next-head="">{"@context":"https://schema.org","@type":"Article","headline":"Zero downtime Postgres upgrades","image":["https://knock.app/assets/blog/preview/zero-downtime-postgres-upgrades-og.png"],"datePublished":"2023-12-12T12:00:00.000Z","dateModified":"2023-12-12T12:00:00.000Z","author":[{"@type":"Person","name":"Brent Anderson"},{"@type":"Organization","name":"Knock","url":"https://knock.app"}]}</script><script>
window.GRAVITY_PIXEL_CONFIG = {
proxyEndpoint: "/gr/"
};
</script><script>
!function(w,d,t,u,n,a,m){w['GravityPixelObject']=n;w[n]=w[n]||function(){
(w[n].q=w[n].q||[]).push(arguments)},w[n].l=1*new Date();a=d.createElement(t),
m=d.getElementsByTagName(t)[0];a.async=1;a.src=u;m.parentNode.insertBefore(a,m)
}(window,document,'script','https://code.trygravity.ai/gr-pix.js','gravity');
gravity('init', '6e23cfea-7ed5-4adc-8ec8-ef0f13c288e1');
</script><link rel="stylesheet" href="/_next/static/immutable/chunks/3ld5x6_gysw72.css" data-n-g=""/><link rel="stylesheet" href="/_next/static/immutable/chunks/09kkgfji_b4dk.css" data-n-g=""/><link rel="stylesheet" href="/_next/static/immutable/chunks/1rxs0tdjm4sjw.css" data-n-g=""/><link rel="stylesheet" href="/_next/static/immutable/chunks/1jrowddgarsos.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/immutable/chunks/0c0hxoamwjsbw.js"></script><script src="/_next/static/immutable/chunks/0e2l_8l8hvsi9.js" defer=""></script><script src="/_next/static/immutable/chunks/2d8ajbjqt4lnf.js" defer=""></script><script src="/_next/static/immutable/chunks/2b7rt_ij8_u8w.js" defer=""></script><script src="/_next/static/immutable/chunks/2pk2y2y2fhhpc.js" defer=""></script><script src="/_next/static/immutable/chunks/26hjol_2qr7ee.js" defer=""></script><script src="/_next/static/immutable/chunks/turbopack-3wn0-9p7a62_s.js" defer=""></script><script src="/_next/static/immutable/chunks/2pwo1-kc8824v.js" defer=""></script><script src="/_next/static/immutable/chunks/2i-flpu-1alc3.js" defer=""></script><script src="/_next/static/immutable/chunks/0tyf7rka77z0n.js" defer=""></script><script src="/_next/static/immutable/chunks/10jebjzg_96z0.js" defer=""></script><script src="/_next/static/immutable/chunks/3jl4mgz_fpods.js" defer=""></script><script src="/_next/static/immutable/chunks/0kztjy3fho4da.js" defer=""></script><script src="/_next/static/immutable/chunks/13u-0pdqwwfnu.js" defer=""></script><script src="/_next/static/immutable/chunks/0p9y1ks0nksq0.js" defer=""></script><script src="/_next/static/immutable/chunks/3s22r4o5dr_i8.js" defer=""></script><script src="/_next/static/immutable/chunks/turbopack-0_e_-t-r9vuuh.js" defer=""></script><script src="/_next/static/Q3knVdD3C65MAkfAAiAvI/_buildManifest.js" defer=""></script><script src="/_next/static/Q3knVdD3C65MAkfAAiAvI/_ssgManifest.js" defer=""></script><script src="/_next/static/Q3knVdD3C65MAkfAAiAvI/_clientMiddlewareManifest.js" defer=""></script></head><body><link rel="preload" as="image" href="/assets/blog/zero-downtime-postgres-upgrades/snapshot-backfill.png"/><div id="__next"><main class="inter_5d8977a8-module__8niREW__variable opensauce_387b48b9-module__Dt6MAa__variable inconsolata_dc3c1145-module__wcgKRG__variable ppneuemontreal_142289b5-module__bzIvDG__variable font-sans"><script type="application/ld+json">{"@context":"https://schema.org","@type":"Organization","name":"Knock","alternateName":"Knock Labs, Inc.","description":"Product and customer messaging infrastructure for email, SMS, in-app, push, Slack, and MS Teams. The most powerful notification system you'll never build.","url":"https://knock.app","email":"support@knock.app","logo":"https://knock.app/assets/knock-logo.png","sameAs":["https://twitter.com/knocklabs","https://github.com/knocklabs","https://www.linkedin.com/company/knocklabs/","https://www.youtube.com/@knocklabs"],"contactPoint":{"@type":"ContactPoint","email":"support@knock.app"},"address":{"@type":"PostalAddress","streetAddress":"175 Varick St, #413","addressLocality":"New York","addressRegion":"NY","postalCode":"10014","addressCountry":{"@type":"Country","name":"US"}},"numberOfEmployees":{"@type":"QuantitativeValue","minValue":11,"maxValue":50}}</script><script type="application/ld+json">{"@context":"https://schema.org","@type":"SoftwareApplication","name":"Knock","operatingSystem":"All","applicationCategory":"BusinessApplication","url":"https://knock.app","downloadUrl":"https://dashboard.knock.app/signup","description":"Product and customer messaging infrastructure for email, SMS, in-app, push, Slack, and MS Teams. The most powerful notification system you'll never build.","aggregateRating":{"@type":"AggregateRating","ratingValue":"4.6","ratingCount":"107"},"publisher":{"@type":"Organization","name":"Knock"}}</script><header class="sticky top-0 z-50 box-border border-b-[0.5px] border-b-solid-black/0 dark:border-b-solid-white/0 bg-white/80 transition-all duration-200"><div class="px-6 md:px-24"><div class="flex items-center justify-between h-16 max-w-[1216px] w-full mx-auto"><div class="flex items-center gap-4"><div class="flex justify-between flex-1 xl:gap-20"><a class="flex w-fit" href="/"><span class="cursor-pointer"><span class="sr-only">Go to home</span><svg width="74" height="24" viewBox="0 0 76 25" fill="white" xmlns="http://www.w3.org/2000/svg" class="fill-black w-[67px] relative -top-[2px]"><path d="M0 24.1383V3.0076H4.31054V15.1253H4.43111L9.16366 9.18704H13.9866L8.74165 15.3363L14.3785 24.1383H9.58568L6.20959 18.3206L4.31054 20.4608V24.1383H0Z"></path><path d="M15.3085 24.1383V9.18704H18.9257L19.2272 11.0861H19.3477C19.9908 10.4832 20.6942 9.96073 21.4578 9.51862C22.2214 9.05642 23.1157 8.82532 24.1406 8.82532C25.7683 8.82532 26.9439 9.35786 27.6674 10.4229C28.4109 11.488 28.7827 12.965 28.7827 14.854V24.1383H24.3516V15.4268C24.3516 14.3416 24.2009 13.5981 23.8994 13.1961C23.6181 12.7942 23.1559 12.5933 22.5128 12.5933C21.9501 12.5933 21.4678 12.7239 21.0659 12.9851C20.664 13.2263 20.2219 13.578 19.7396 14.0402V24.1383H15.3085Z"></path><path d="M37.7735 24.5C36.829 24.5 35.9146 24.3292 35.0304 23.9876C34.1462 23.6258 33.3625 23.1134 32.6792 22.4502C32.0161 21.7871 31.4835 20.9732 31.0816 20.0086C30.6797 19.0239 30.4787 17.9086 30.4787 16.6627C30.4787 15.4167 30.6797 14.3115 31.0816 13.3469C31.4835 12.3622 32.0161 11.5382 32.6792 10.8751C33.3625 10.2119 34.1462 9.70953 35.0304 9.36791C35.9146 9.00618 36.829 8.82532 37.7735 8.82532C38.718 8.82532 39.6223 9.00618 40.4864 9.36791C41.3706 9.70953 42.1443 10.2119 42.8075 10.8751C43.4907 11.5382 44.0333 12.3622 44.4352 13.3469C44.8372 14.3115 45.0381 15.4167 45.0381 16.6627C45.0381 17.9086 44.8372 19.0239 44.4352 20.0086C44.0333 20.9732 43.4907 21.7871 42.8075 22.4502C42.1443 23.1134 41.3706 23.6258 40.4864 23.9876C39.6223 24.3292 38.718 24.5 37.7735 24.5ZM37.7735 20.9129C38.6778 20.9129 39.3611 20.5311 39.8233 19.7675C40.2855 19.0038 40.5166 17.9689 40.5166 16.6627C40.5166 15.3564 40.2855 14.3215 39.8233 13.5579C39.3611 12.7942 38.6778 12.4124 37.7735 12.4124C36.8491 12.4124 36.1558 12.7942 35.6936 13.5579C35.2515 14.3215 35.0304 15.3564 35.0304 16.6627C35.0304 17.9689 35.2515 19.0038 35.6936 19.7675C36.1558 20.5311 36.8491 20.9129 37.7735 20.9129Z"></path><path d="M53.3785 24.5C52.3335 24.5 51.3589 24.3292 50.4546 23.9876C49.5703 23.6258 48.7967 23.1134 48.1335 22.4502C47.4703 21.7871 46.9478 20.9732 46.566 20.0086C46.1842 19.0239 45.9933 17.9086 45.9933 16.6627C45.9933 15.4167 46.2043 14.3115 46.6263 13.3469C47.0483 12.3622 47.611 11.5382 48.3144 10.8751C49.0378 10.2119 49.8718 9.70953 50.8163 9.36791C51.7608 9.00618 52.7455 8.82532 53.7704 8.82532C54.6948 8.82532 55.5086 8.97604 56.212 9.27747C56.9354 9.57891 57.5785 9.96073 58.1412 10.4229L56.0613 13.2866C55.3579 12.7038 54.6847 12.4124 54.0416 12.4124C52.9565 12.4124 52.1024 12.7942 51.4794 13.5579C50.8565 14.3215 50.545 15.3564 50.545 16.6627C50.545 17.9689 50.8565 19.0038 51.4794 19.7675C52.1225 20.5311 52.9263 20.9129 53.8909 20.9129C54.3732 20.9129 54.8354 20.8124 55.2775 20.6115C55.7397 20.3904 56.1718 20.1292 56.5737 19.8277L58.322 22.7215C57.5785 23.3646 56.7747 23.8268 55.9106 24.1081C55.0464 24.3694 54.2024 24.5 53.3785 24.5Z"></path><path d="M59.7327 24.1383V3.0076H64.0432V15.1253H64.1638L68.8963 9.18704H73.7193L68.4743 15.3363L74.1112 24.1383H69.3183L65.9423 18.3206L64.0432 20.4608V24.1383H59.7327Z"></path><path d="M75.1388 4.04252C75.1388 5.999 73.5527 7.58504 71.5962 7.58504C69.6398 7.58504 68.0537 5.999 68.0537 4.04252C68.0537 2.08604 69.6398 0.5 71.5962 0.5C73.5527 0.5 75.1388 2.08604 75.1388 4.04252Z" fill="#FF6C52"></path></svg></span></a></div><nav class="items-center justify-center hidden lg:flex flex-2"><div class="relative"><a href="/blog/zero-downtime-postgres-upgrades#" class="inline-block hover:!text-solid-red xl:!px-4 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px] !text-sm"><div class="relative flex flex-col"><div class="flex flex-row items-center"><span class="mr-1 font-sans !text-sm">Product</span><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-chevron-down transition-transform duration-200 ease-in-out" aria-hidden="true"><path d="m6 9 6 6 6-6"></path></svg></div></div></span></a></div><div class="relative"><a href="/blog/zero-downtime-postgres-upgrades#" class="inline-block hover:!text-solid-red xl:!px-4 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px] !text-sm"><div class="relative flex flex-col"><div class="flex flex-row items-center"><span class="mr-1 font-sans !text-sm">Solutions</span><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-chevron-down transition-transform duration-200 ease-in-out" aria-hidden="true"><path d="m6 9 6 6 6-6"></path></svg></div></div></span></a></div><div class="relative"><a href="/blog/zero-downtime-postgres-upgrades#" class="inline-block hover:!text-solid-red xl:!px-4 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px] !text-sm"><div class="relative flex flex-col"><div class="flex flex-row items-center"><span class="mr-1 font-sans !text-sm">Developers</span><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-chevron-down transition-transform duration-200 ease-in-out" aria-hidden="true"><path d="m6 9 6 6 6-6"></path></svg></div></div></span></a></div><div class="relative"><a href="/blog/zero-downtime-postgres-upgrades#" class="inline-block hover:!text-solid-red xl:!px-4 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px] !text-sm"><div class="relative flex flex-col"><div class="flex flex-row items-center"><span class="mr-1 font-sans !text-sm">Resources</span><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-chevron-down transition-transform duration-200 ease-in-out" aria-hidden="true"><path d="m6 9 6 6 6-6"></path></svg></div><span class="absolute w-1 h-1 mx-auto -ml-2 rounded-full top-8 left-1/2 bg-solid-red"></span></div></span></a></div><a href="/pricing" class="inline-block hover:!text-solid-red xl:!px-4 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px] !text-sm"><div class="relative flex flex-col font-sans text-sm">Pricing</div></span></a></nav></div><div class="flex items-center justify-end flex-1 gap-3"><a href="/login" class="inline-block !px-3 !py-2 border-[0.5px] border-[#CDCED6] bg-white box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] font-sans !font-medium hover:text-[#E33A1E] focus:text-primary-700"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px] !text-sm">Log in<span aria-hidden="true" class="hidden h-[18px] min-w-[18px] items-center justify-center rounded-[3px] border px-1 font-sans text-[10px] font-semibold leading-none tracking-normal md:inline-flex border-[#CDCED6] bg-[#F9F9FB] text-[#60646C]">L</span></span></a><a href="https://dashboard.knock.app/signup" class="inline-block !px-3 !py-2 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] text-solid-white !font-semibold hover:bg-[#E33A1E] bg-[#FF6C52] focus:outline-1 focus:outline-offset-4 focus:outline-surface-actionFilledHovered"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px] !text-sm">Sign up<span aria-hidden="true" class="hidden h-[18px] min-w-[18px] items-center justify-center rounded-[3px] border px-1 font-sans text-[10px] font-semibold leading-none tracking-normal md:inline-flex border-white/35 bg-white/10 text-white">S</span></span></a><div class="flex items-center lg:hidden"><button aria-label="Menu" class="px-0 align-top dark:text-solid-white box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16" fill="none"><path fill-rule="evenodd" clip-rule="evenodd" d="M2 7.99992C2 7.63173 2.29848 7.33325 2.66667 7.33325H13.3333C13.7015 7.33325 14 7.63173 14 7.99992C14 8.36811 13.7015 8.66659 13.3333 8.66659H2.66667C2.29848 8.66659 2 8.36811 2 7.99992Z" fill="#60646C"></path><path fill-rule="evenodd" clip-rule="evenodd" d="M2 3.99992C2 3.63173 2.29848 3.33325 2.66667 3.33325H13.3333C13.7015 3.33325 14 3.63173 14 3.99992C14 4.36811 13.7015 4.66659 13.3333 4.66659H2.66667C2.29848 4.66659 2 4.36811 2 3.99992Z" fill="#60646C"></path><path fill-rule="evenodd" clip-rule="evenodd" d="M2 11.9999C2 11.6317 2.29848 11.3333 2.66667 11.3333H13.3333C13.7015 11.3333 14 11.6317 14 11.9999C14 12.3681 13.7015 12.6666 13.3333 12.6666H2.66667C2.29848 12.6666 2 12.3681 2 11.9999Z" fill="#60646C"></path></svg></span></button></div></div></div></div></header><main class="-mt-[98px] pt-[88px] dark:bg-solid-black bg-solid-white"><div class="grid w-full min-w-0 grid-cols-12 col-span-12 mx-auto max-w-[1440px]"><div class="relative z-10 hidden col-span-1 !overflow-visible md:flex grid-cols-2"><div class="w-[0.5px] absolute h-full bg-linear15 z-40 hidden md:flex -right-[0.5px] -right-[0px] md:-right-[1px] lg:-right-[0.5px] xl:-right-[0.5px]"></div></div><section class="z-0 col-span-12 md:col-span-10"><div class="flex flex-col items-center gap-2 border-[#CDCED6] bg-linear11 p-12 text-left"><header class="py-8 w-full -mb-12 mr-12"><nav class="text-[13px] text-[#60646C]" aria-label="Breadcrumb"><a class="text-[#1C2024]" href="/blog">Blog</a><span> / <a href="/blog/topics/engineering">Engineering</a></span><span aria-current="page"> / <!-- -->Zero downtime Postgres upgrades</span></nav></header></div></section><div class="relative z-10 hidden col-span-1 !overflow-visible md:flex"><div class="w-[0.5px] absolute h-full bg-linear15 z-40 hidden md:flex -left-[0.5px] -left-[0px] md:-left-[1px] lg:-left-[0.75px] xl:-left-[0.5px]"></div></div></div><div class="w-screen border-t border-[#CDCED6] border-t-[0.5px] border-b-0 border-[0.5px] border-t-[0.5px border-r-0 border-[0.5px] md:border-r-[0.5px]"><div class="max-w-[1440px] mx-auto"><div class="grid grid-cols-12"><div class="hidden md:grid grid-cols-2 col-span-1 border-[#CDCED6] "><div class="grid grid-cols-2 col-span-2"><div class="col-span-1 border-r border-dashed border-[#E0E1E6]"></div></div><div class="grid grid-cols-2 col-span-2 border-t border-dashed border-[#E0E1E6] mask-gradient-fade-left"><div class="col-span-1 border-r border-dashed border-[#E0E1E6]"></div></div></div><div class="w-full h-full col-span-12 md:col-span-10 md:col-start-2 lg:col-start-2 lg:col-span-10 border-l-[0.5px] border-r-[0.5px]"><article class="relative flex flex-col bg-white"><main class="flex flex-col lgCustom:flex-row lgCustom:items-stretch"><section class="flex-1 min-w-0 py-6 md:py-20"><div class="max-w-[680px] mx-auto px-6 md:pr-8 lg:pb-12 lgCustom:mx-0 xlCustom:mx-auto"><h1 class="font-open mb-6 text-[32px] lg:text-[48px] leading-[1.1] font-ppNeueMontreal">Zero downtime Postgres upgrades</h1><div class="flex flex-wrap items-center gap-2 text-[13px] text-[#60646C] mb-6"><div class="flex items-center gap-2"><span class="whitespace-nowrap">Last updated:</span><span class="whitespace-nowrap">12 Dec 2023</span></div><span class="mx-1">|</span><div class="flex items-center gap-2"><span class="whitespace-nowrap">29 min read</span></div><span class="mx-1">|</span><div class="flex items-center gap-2"><span class="whitespace-nowrap">Summarize:</span><div class="flex items-center gap-1"><a href="https://chat.openai.com/?q=Please%20provide%20a%20concise%20summary%20of%20this%20blog%20post%3A%20Zero%20downtime%20Postgres%20upgrades%20-%20https%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades.%20Highlight%20the%20main%20points%20and%20key%20takeaways." target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="Summarize with ChatGPT"><svg width="16" height="16" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg"><text x="-9999" y="-9999">ChatGPT</text><path d="M9.20509 8.76511V6.50545C9.20509 6.31513 9.27649 6.17234 9.44293 6.0773L13.9861 3.46088C14.6046 3.10413 15.342 2.93769 16.103 2.93769C18.9573 2.93769 20.7651 5.14983 20.7651 7.50454C20.7651 7.67098 20.7651 7.86129 20.7412 8.05161L16.0316 5.2924C15.7462 5.12596 15.4607 5.12596 15.1753 5.2924L9.20509 8.76511ZM19.8135 17.5659V12.1664C19.8135 11.8333 19.6708 11.5955 19.3854 11.429L13.4152 7.95633L15.3656 6.83833C15.5321 6.74328 15.6749 6.74328 15.8413 6.83833L20.3845 9.45474C21.6928 10.216 22.5728 11.8333 22.5728 13.4031C22.5728 15.2108 21.5025 16.8758 19.8135 17.5657V17.5659ZM7.80173 12.8088L5.8513 11.6671C5.68486 11.5721 5.61346 11.4293 5.61346 11.239V6.00613C5.61346 3.46111 7.56389 1.53433 10.2042 1.53433C11.2033 1.53433 12.1307 1.86743 12.9159 2.46202L8.2301 5.17371C7.94475 5.34015 7.80195 5.57798 7.80195 5.91109V12.809L7.80173 12.8088ZM12 15.2349L9.20509 13.6651V10.3351L12 8.76534L14.7947 10.3351V13.6651L12 15.2349ZM13.7958 22.4659C12.7967 22.4659 11.8693 22.1328 11.0841 21.5382L15.7699 18.8265C16.0553 18.6601 16.198 18.4222 16.198 18.0891V11.1912L18.1723 12.3329C18.3388 12.4279 18.4102 12.5707 18.4102 12.761V17.9939C18.4102 20.5389 16.4359 22.4657 13.7958 22.4657V22.4659ZM8.15848 17.1617L3.61528 14.5452C2.30696 13.784 1.42701 12.1667 1.42701 10.5969C1.42701 8.76534 2.52115 7.12414 4.20987 6.43428V11.8574C4.20987 12.1905 4.35266 12.4284 4.63802 12.5948L10.5846 16.0436L8.63415 17.1617C8.46771 17.2567 8.32492 17.2567 8.15848 17.1617ZM7.897 21.0625C5.20919 21.0625 3.23488 19.0407 3.23488 16.5432C3.23488 16.3529 3.25875 16.1626 3.2824 15.9723L7.96817 18.6839C8.25352 18.8504 8.53911 18.8504 8.82446 18.6839L14.7947 15.2351V17.4948C14.7947 17.6851 14.7233 17.8279 14.5568 17.9229L10.0136 20.5393C9.39518 20.8961 8.6578 21.0625 7.89677 21.0625H7.897ZM13.7958 23.8929C16.6739 23.8929 19.0762 21.8474 19.6235 19.1357C22.2874 18.4459 24 15.9484 24 13.4034C24 11.7383 23.2865 10.121 22.002 8.95542C22.121 8.45588 22.1924 7.95633 22.1924 7.45702C22.1924 4.0557 19.4331 1.51045 16.2458 1.51045C15.6037 1.51045 14.9852 1.60549 14.3668 1.81968C13.2963 0.773071 11.8215 0.107086 10.2042 0.107086C7.32606 0.107086 4.92383 2.15256 4.37653 4.86425C1.7126 5.55411 0 8.05161 0 10.5966C0 12.2617 0.713506 13.879 1.99795 15.0446C1.87904 15.5441 1.80764 16.0436 1.80764 16.543C1.80764 19.9443 4.56685 22.4895 7.75421 22.4895C8.39632 22.4895 9.01478 22.3945 9.63324 22.1803C10.7035 23.2269 12.1783 23.8929 13.7958 23.8929Z" fill="currentColor"></path></svg></a><a href="https://claude.ai/new?q=Please%20provide%20a%20concise%20summary%20of%20this%20blog%20post%3A%20Zero%20downtime%20Postgres%20upgrades%20-%20https%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades.%20Highlight%20the%20main%20points%20and%20key%20takeaways." target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="Summarize with Claude"><svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg" fill="currentColor" width="16" height="16"><title>Claude</title><path d="m4.7144 15.9555 4.7174-2.6471.079-.2307-.079-.1275h-.2307l-.7893-.0486-2.6956-.0729-2.3375-.0971-2.2646-.1214-.5707-.1215-.5343-.7042.0546-.3522.4797-.3218.686.0608 1.5179.1032 2.2767.1578 1.6514.0972 2.4468.255h.3886l.0546-.1579-.1336-.0971-.1032-.0972L6.973 9.8356l-2.55-1.6879-1.3356-.9714-.7225-.4918-.3643-.4614-.1578-1.0078.6557-.7225.8803.0607.2246.0607.8925.686 1.9064 1.4754 2.4893 1.8336.3643.3035.1457-.1032.0182-.0728-.164-.2733-1.3539-2.4467-1.445-2.4893-.6435-1.032-.17-.6194c-.0607-.255-.1032-.4674-.1032-.7285L6.287.1335 6.6997 0l.9957.1336.419.3642.6192 1.4147 1.0018 2.2282 1.5543 3.0296.4553.8985.2429.8318.091.255h.1579v-.1457l.1275-1.706.2368-2.0947.2307-2.6957.0789-.7589.3764-.9107.7468-.4918.5828.2793.4797.686-.0668.4433-.2853 1.8517-.5586 2.9021-.3643 1.9429h.2125l.2429-.2429.9835-1.3053 1.6514-2.0643.7286-.8196.85-.9046.5464-.4311h1.0321l.759 1.1293-.34 1.1657-1.0625 1.3478-.8804 1.1414-1.2628 1.7-.7893 1.36.0729.1093.1882-.0183 2.8535-.607 1.5421-.2794 1.8396-.3157.8318.3886.091.3946-.3278.8075-1.967.4857-2.3072.4614-3.4364.8136-.0425.0304.0486.0607 1.5482.1457.6618.0364h1.621l3.0175.2247.7892.522.4736.6376-.079.4857-1.2142.6193-1.6393-.3886-3.825-.9107-1.3113-.3279h-.1822v.1093l1.0929 1.0686 2.0035 1.8092 2.5075 2.3314.1275.5768-.3218.4554-.34-.0486-2.2039-1.6575-.85-.7468-1.9246-1.621h-.1275v.17l.4432.6496 2.3436 3.5214.1214 1.0807-.17.3521-.6071.2125-.6679-.1214-1.3721-1.9246L14.38 17.959l-1.1414-1.9428-.1397.079-.674 7.2552-.3156.3703-.7286.2793-.6071-.4614-.3218-.7468.3218-1.4753.3886-1.9246.3157-1.53.2853-1.9004.17-.6314-.0121-.0425-.1397.0182-1.4328 1.9672-2.1796 2.9446-1.7243 1.8456-.4128.164-.7164-.3704.0667-.6618.4008-.5889 2.386-3.0357 1.4389-1.882.929-1.0868-.0062-.1579h-.0546l-6.3385 4.1164-1.1293.1457-.4857-.4554.0608-.7467.2307-.2429 1.9064-1.3114Z"></path></svg></a><a href="https://www.perplexity.ai/search?q=Please%20provide%20a%20concise%20summary%20of%20this%20blog%20post%3A%20Zero%20downtime%20Postgres%20upgrades%20-%20https%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades.%20Highlight%20the%20main%20points%20and%20key%20takeaways." target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="Summarize with Perplexity"><svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg" fill="currentColor" width="16" height="16"><title>Perplexity</title><path d="M22.3977 7.0896h-2.3106V.0676l-7.5094 6.3542V.1577h-1.1554v6.1966L4.4904 0v7.0896H1.6023v10.3976h2.8882V24l6.932-6.3591v6.2005h1.1554v-6.0469l6.9318 6.1807v-6.4879h2.8882V7.0896zm-3.4657-4.531v4.531h-5.355l5.355-4.531zm-13.2862.0676 4.8691 4.4634H5.6458V2.6262zM2.7576 16.332V8.245h7.8476l-6.1149 6.1147v1.9723H2.7576zm2.8882 5.0404v-3.8852h.0001v-2.6488l5.7763-5.7764v7.0111l-5.7764 5.2993zm12.7086.0248-5.7766-5.1509V9.0618l5.7766 5.7766v6.5588zm2.8882-5.0652h-1.733v-1.9723L13.3948 8.245h7.8478v8.087z"></path></svg></a><a href="https://gemini.google.com/?q=Please%20provide%20a%20concise%20summary%20of%20this%20blog%20post%3A%20Zero%20downtime%20Postgres%20upgrades%20-%20https%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades.%20Highlight%20the%20main%20points%20and%20key%20takeaways." target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="Summarize with Gemini"><svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg" fill="currentColor" width="16" height="16"><title>Google Gemini</title><path d="M11.04 19.32Q12 21.51 12 24q0-2.49.93-4.68.96-2.19 2.58-3.81t3.81-2.55Q21.51 12 24 12q-2.49 0-4.68-.93a12.3 12.3 0 0 1-3.81-2.58 12.3 12.3 0 0 1-2.58-3.81Q12 2.49 12 0q0 2.49-.96 4.68-.93 2.19-2.55 3.81a12.3 12.3 0 0 1-3.81 2.58Q2.49 12 0 12q2.49 0 4.68.96 2.19.93 3.81 2.55t2.55 3.81"></path></svg></a><a href="https://www.bing.com/search?showconv=1&amp;sendquery=1&amp;q=Please%20provide%20a%20concise%20summary%20of%20this%20blog%20post%3A%20Zero%20downtime%20Postgres%20upgrades%20-%20https%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades.%20Highlight%20the%20main%20points%20and%20key%20takeaways." target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="Summarize with Copilot"><svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg" fill="currentColor" width="16" height="16"><title>GitHub Copilot</title><path d="M23.922 16.997C23.061 18.492 18.063 22.02 12 22.02 5.937 22.02.939 18.492.078 16.997A.641.641 0 0 1 0 16.741v-2.869a.883.883 0 0 1 .053-.22c.372-.935 1.347-2.292 2.605-2.656.167-.429.414-1.055.644-1.517a10.098 10.098 0 0 1-.052-1.086c0-1.331.282-2.499 1.132-3.368.397-.406.89-.717 1.474-.952C7.255 2.937 9.248 1.98 11.978 1.98c2.731 0 4.767.957 6.166 2.093.584.235 1.077.546 1.474.952.85.869 1.132 2.037 1.132 3.368 0 .368-.014.733-.052 1.086.23.462.477 1.088.644 1.517 1.258.364 2.233 1.721 2.605 2.656a.841.841 0 0 1 .053.22v2.869a.641.641 0 0 1-.078.256Zm-11.75-5.992h-.344a4.359 4.359 0 0 1-.355.508c-.77.947-1.918 1.492-3.508 1.492-1.725 0-2.989-.359-3.782-1.259a2.137 2.137 0 0 1-.085-.104L4 11.746v6.585c1.435.779 4.514 2.179 8 2.179 3.486 0 6.565-1.4 8-2.179v-6.585l-.098-.104s-.033.045-.085.104c-.793.9-2.057 1.259-3.782 1.259-1.59 0-2.738-.545-3.508-1.492a4.359 4.359 0 0 1-.355-.508Zm2.328 3.25c.549 0 1 .451 1 1v2c0 .549-.451 1-1 1-.549 0-1-.451-1-1v-2c0-.549.451-1 1-1Zm-5 0c.549 0 1 .451 1 1v2c0 .549-.451 1-1 1-.549 0-1-.451-1-1v-2c0-.549.451-1 1-1Zm3.313-6.185c.136 1.057.403 1.913.878 2.497.442.544 1.134.938 2.344.938 1.573 0 2.292-.337 2.657-.751.384-.435.558-1.15.558-2.361 0-1.14-.243-1.847-.705-2.319-.477-.488-1.319-.862-2.824-1.025-1.487-.161-2.192.138-2.533.529-.269.307-.437.808-.438 1.578v.021c0 .265.021.562.063.893Zm-1.626 0c.042-.331.063-.628.063-.894v-.02c-.001-.77-.169-1.271-.438-1.578-.341-.391-1.046-.69-2.533-.529-1.505.163-2.347.537-2.824 1.025-.462.472-.705 1.179-.705 2.319 0 1.211.175 1.926.558 2.361.365.414 1.084.751 2.657.751 1.21 0 1.902-.394 2.344-.938.475-.584.742-1.44.878-2.497Z"></path></svg></a><a href="https://grok.com/?q=Please%20provide%20a%20concise%20summary%20of%20this%20blog%20post%3A%20Zero%20downtime%20Postgres%20upgrades%20-%20https%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades.%20Highlight%20the%20main%20points%20and%20key%20takeaways." target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="Summarize with Grok"><svg width="16" height="16" viewBox="0 0 512 492" fill="none" xmlns="http://www.w3.org/2000/svg"><title>Grok</title><path fill-rule="evenodd" clip-rule="evenodd" d="M197.76 315.52l170.197-125.803c8.342-6.186 20.267-3.776 24.256 5.803 20.907 50.539 11.563 111.253-30.08 152.939-41.621 41.685-99.562 50.816-152.512 29.994l-57.834 26.816c82.965 56.768 183.701 42.731 246.656-20.33 49.941-50.006 65.408-118.166 50.944-179.627l.128.149c-20.971-90.282 5.162-126.378 58.666-200.17 1.28-1.75 2.56-3.499 3.819-5.291l-70.421 70.507v-.214l-243.883 245.27m-35.072 30.528c-59.563-56.96-49.28-145.088 1.515-195.926 37.568-37.61 99.136-52.97 152.874-30.4l57.707-26.666a166.554 166.554 0 00-39.019-21.334 191.467 191.467 0 00-208.042 41.942c-54.038 54.101-71.04 137.301-41.856 208.298 21.802 53.056-13.931 90.582-49.92 128.47C23.104 463.915 10.304 477.333 0 491.541l162.56-145.386" fill="currentColor"></path></svg></a></div></div></div><div class="mt-6 post-content text-[#1C2024]"><p><strong>👋 We’re Knock. We provide a set of simple APIs developers use to introduce
notifications into their products, without needing to build and
maintain a notification system in-house.</strong></p>
<div class="callout flex w-full items-center gap-3 rounded-md border-sm border-1 border-solid border-transparent p-4 my-6" style="background:rgba(235, 236, 237, 0.3)"><div class="flex h-8 w-8 shrink-0 items-center justify-center rounded-sm text-2xl leading-none" aria-hidden="true">✅</div><div class="min-w-0 w-full text-lg leading-normal"><p><strong>Tl;dr:</strong> We recently upgraded from Postgres 11.9 to 15.3 with
zero downtime by using logical replication, a suite of support scripts,
and tools in Elixir &amp; Erlang’s BEAM virtual machine.</p><p>This post will go into far too much detail explaining how we did it,
and considerations you might need to make along the way if you try to do the same.</p><p>It is more of a manual than anything, and includes things we learned along the
way that we wish we’d known up front.</p></div></div>
<p>Knock relies on Postgres to power our notification workflow engine. From storing
workflow configurations and message templates, to <a href="https://knock.app/blog/how-we-use-postgres-ltrees">ingesting millions of logs</a>
and <a href="https://getoban.pro/">enqueuing background jobs</a>, Postgres sits at the heart of everything our systems do.
Our Postgres databases running on AWS RDS Aurora have been consistently reliable,
performant, and extensible. This foundation to Knock’s service lets us support
with confidence every customer that joins our platform.</p>
<p>Unlike SaaS software that can be constantly upgraded in the background with little notice,
upgrading relational databases like Postgres generally requires at least a reboot of the database.
In the case of major version upgrades, the database often needs to shut down completely for several minutes
in order to upgrade how data is stored and indexed on disk.</p>
<p><strong>The more data you have, the longer the upgrade will take.</strong></p>
<p>In Knock’s case, we have been running Postgres 11.9 since we started the company.
Although it has reliably served us at every step along the way,
<a href="https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html#Release.Calendar">Postgres 11.9 is being retired by Amazon’s RDS service on February 29, 2024</a>.
Without taking action (i.e. arranging a long-term support contract with RDS),
teams that use Postgres 11.9 on AWS RDS will be forcibly upgraded at that point,
likely resulting in forced downtime.</p>
<p>No amount of downtime - scheduled or otherwise - is acceptable for a service like Knock.
Our customers rely on us to be online 24/7. Although no service can guarantee
perfect uptime, responsible developer teams work to proactively address service
issues before they happen.</p>
<p>We added this upgrade to our roadmap in June of this year, with the following constraints:</p>
<ol>
<li>Upgrade as many versions ahead as possible, skipping to the latest available version (at the time, Postgres 15.3 for Aurora).</li>
<li>Any downtime beyond 60 seconds was completely unacceptable, and ideally we would have zero system downtime.</li>
<li>The upgrade must happen well in advance of Amazon’s February deadline.</li>
<li>Minimize customer impact (e.g. zero API error responses).</li>
<li>Operationalize the process so that next time we need to upgrade the database, it is a well-established runbook.</li>
</ol>
<p>Each of our Postgres databases would need to run through this process, and going
from 11.9 to 15.3 would comprise <strong>four</strong> major version upgrades. If doing an
in-place upgrade for each major version would trigger downtime, doing <strong>four</strong>
in a row was out of the question.</p>
<p>In order to meet our requirements, we knew we’d have to get creative.</p>
<h2 id="preparing-for-any-postgres-upgrade"><a href="#preparing-for-any-postgres-upgrade" aria-hidden="true" tabindex="-1"><span></span></a>Preparing for any Postgres upgrade</h2>
<p>More than anything, teams seeking to upgrade Postgres in any way should focus on
<strong>de-risking</strong> the upgrade process as much as possible:</p>
<ol>
<li>
<p>Make a list of the risks involved in making the migration. For example:</p>
<ul>
<li>Unacceptably long downtime</li>
<li>Data loss</li>
<li>Changes in database performance for your application’s workload</li>
<li>Changes in vacuum frequency or behavior</li>
<li>Are there any replication slots that need to be migrated (this can be tricky - <a href="#a-note-about-moving-replication-slots">see below</a>)</li>
</ul>
</li>
<li>
<p>Figure out which risks are the most critical to the project, and which ones
might be the easiest to explore/rule out/fix in advance.</p>
<p>Sort the list so the risks with the biggest impact yet easiest to address are at the top.</p>
</li>
<li>
<p>As you develop solutions, consider your list of risks:</p>
<ul>
<li>Are there solutions that rule out risks completely?</li>
<li>Which solutions spread out the risk over time? (So we can more gradually
address each step of the migration without taking on too much risk at once.)</li>
</ul>
</li>
<li>
<p>As you work through the project, always revisit your list of risks,
and keep it up to date as you learn new things - including discovering new risks!</p>
<div class="callout flex w-full items-center gap-3 rounded-md border-sm border-1 border-solid border-transparent p-4 my-6" style="background:rgba(235, 236, 237, 0.3)"><div class="flex h-8 w-8 shrink-0 items-center justify-center rounded-sm text-2xl leading-none" aria-hidden="true">✅</div><div class="min-w-0 w-full text-lg leading-normal"><p><strong>Incrementally and continually de-risk projects like this until you are
confident in being able to deliver on your project goals.</strong></p></div></div>
</li>
</ol>
<p>To plan out our upgrade, we started with <a href="https://www.postgresql.org/docs/release/">Postgres’ release notes</a>
to get a sense of what was going to change between database versions.
This helped us identify more risks (e.g. changes in how Postgres’ vacuum works,
requirement to reindex the database when performing certain upgrades) while ruling out others.</p>
<p>As we moved through our planning process, we maintained this list of risks,
adding new concerns and updating old ones as we collected more information.
While working through the upgrade, we systematically addressed each concern
until we were confident we could deliver on our project goals without
risking our reliability.</p>
<h3 id="a-word-about-monitoring--metrics"><a href="#a-word-about-monitoring--metrics" aria-hidden="true" tabindex="-1"><span></span></a>A word about monitoring &amp; metrics</h3>
<p>Having thorough instrumentation (thanks DataDog!) to monitor the health of your
system and database makes it possible to monitor each step of the migration.</p>
<p>A few key metrics to watch:</p>
<ul>
<li>Max TXN ID to avoid <a href="https://www.crunchydata.com/blog/managing-transaction-id-wraparound-in-postgresql">transaction wraparound</a> - if this gets too high, your database can shut down and go into emergency maintenance mode</li>
<li>DB CPU Utilization</li>
<li>Waiting sessions on your writer instance</li>
<li>Query latency</li>
<li>API response latencies for your application</li>
</ul>
<p>At Knock, we monitor all of these metrics as well as some that are unique to our application,
like the time it takes to turn an API request into a notification.</p>
<p><strong>Without timely metrics, you’re flying blind.</strong></p>
<h2 id="options-for-upgrading-postgres"><a href="#options-for-upgrading-postgres" aria-hidden="true" tabindex="-1"><span></span></a>Options for upgrading Postgres</h2>
<p>Part of our research process included looking for <a href="https://retool.com/blog/how-we-upgraded-postgresql-database">prior examples</a>
of database migrations and <a href="https://www.postgresql.org/docs/current/pgupgrade.html">how the Postgres docs</a>
recommend performing an update. Here are a few strategies:</p>
<h3 id="in-place-upgrades-a-non-starter-for-zero-downtime-upgrades"><a href="#in-place-upgrades-a-non-starter-for-zero-downtime-upgrades" aria-hidden="true" tabindex="-1"><span></span></a>In-place upgrades (a non-starter for zero-downtime upgrades)</h3>
<p>The most basic upgrade option for Postgres is an in-place upgrade.
On AWS RDS, this upgrade is executed from the AWS console. When performing an
in-place upgrade, AWS will shut down the database, run upgrade scripts, and then
bring the system back online. Doing this often requires some preparation,
including dropping Postgres replication slots,
like those used to synchronize with a data warehouse or other systems.</p>
<p>This in-place upgrade process can take anywhere from a few minutes to potentially
hours or more - it entirely depends on how much data needs to be updated
between Postgres versions.</p>
<p>Often, the system is still not in a fully usable state when it comes online, and
administrators must run maintenance tasks like Postgres’ <code>VACUUM</code> command,
or <code>REINDEX</code> to update indexes to support the new version’s format.</p>
<p><strong>Because an in-place upgrade would require far more downtime than we wanted to
tolerate, it was out of the question for us.</strong></p>
<p>A similar approach to an in-place upgrade is to use <code>pg_dump</code> and <code>pg_restore</code>
to transfer the contents of a database once it has shut down.
This <a href="https://www.postgresql.org/docs/current/app-pgdump.html">dump &amp; restore</a>
approach would also not work for us due to the required downtime involved,
mostly because you need to disconnect all applications from the old database in
order to get a reliable database backup. Even then, for large databases, it can
take prohibitively long to dump and restore the database.</p>
<h3 id="replication-based-upgrades"><a href="#replication-based-upgrades" aria-hidden="true" tabindex="-1"><span></span></a>Replication-based upgrades</h3>
<p>This approach relies on Postgres’ excellent replication primitives:
the <code>PUBLICATION</code> and the <code>SUBSCRIPTION</code>.</p>
<p>It works something like this:</p>
<ol>
<li>Spin up a new database on your target Postgres version</li>
<li>Copy over settings, extensions, table configurations, users, etc.</li>
<li>Set up a publication on the old database and a subscription to that publication on the new database</li>
<li>Add your tables to the publication (there is a lot of nuance here - <a href="#choosing-tables-to-replicate">more below</a>)</li>
<li>Once it&#x27;s fully replicated, run tests to satisfy any remaining risks</li>
<li>Once you are confident in the new database&#x27;s configuration, point your application at the new database</li>
<li>Tear down the old database</li>
</ol>
<p><strong>In the end, this is the option that we chose at Knock for a few reasons:</strong></p>
<ol>
<li>It gave us gradual steps we could take towards a migration instead of one big upgrade</li>
<li>We could test the new database with real workloads and real data to avoid any regressions</li>
<li>It gave us the most control over when and how to perform the upgrade:
once the new database was fully ready, cutting over to the new database took just a few seconds</li>
</ol>
<p>Although that may sound straightforward, there are several points to consider in
this solution that will depend on your application &amp; circumstances.</p>
<p><strong>Configuring your source and destination databases</strong></p>
<p>Publications and subscriptions depend on a few configuration parameters for
setting up replication slots (how the database keeps track of what needs to be
copied from the primary to the follower database).
<a href="https://www.postgresql.org/docs/16/logical-replication-config.html">The Postgres docs</a>
have plenty of detail on these parameters. These parameters will need to be
tuned for your particular application. For simple applications, the only change
necessary is that <code>wal_level</code> should be set to <code>logical</code>.</p>
<p>If you already use replication slots (e.g. to manage a read replica,
database failover, or to keep a data warehouse in sync), then consider setting
<code>max_replication_slots</code> and the other parameters according to the guidance in the docs.</p>
<p><strong>Setting up basic replication</strong></p>
<ol>
<li>
<p>Start a new Postgres server on your target version of Postgres (in our case v15.3).</p>
</li>
<li>
<p>Set up your desired databases, schemas, tables, partitions, users &amp; passwords, and everything else.</p>
<p><strong>The target database’s tables must have an identical structure to the
source database, but these tables must be empty.</strong></p>
<p>To get a snapshot of the database schema, run <a href="https://www.postgresql.org/docs/current/app-pg-dumpall.html"><code>pg_dumpall</code></a>
on the old DB (pass the <code>--schema-only</code> and <code>--no-role-passwords</code> options to
keep it focused), and then adapt that command for the new DB. You can then
compare the generated SQL files to identify and fix discrepancies between the
old and the new DB.</p>
<p>It may be worth periodically comparing both databases to detect any drift,
especially if you have schema migrations happening in the source database.
Consider running migrations against both databases to keep them in sync.</p>
</li>
<li>
<p>On the primary instance of the <strong>old</strong> database, run <code>CREATE PUBLICATION pg_upgrade_pub;</code>.</p>
<div class="callout flex w-full items-center gap-3 rounded-md border-sm border-1 border-solid border-transparent p-4 my-6" style="background:rgba(235, 236, 237, 0.3)"><div class="flex h-8 w-8 shrink-0 items-center justify-center rounded-sm text-2xl leading-none" aria-hidden="true">🚨</div><div class="min-w-0 w-full text-lg leading-normal"><p>Although you can tack on <code>FOR ALL TABLES</code> and that will set up the publication
for every table, we found that for large databases, this can lead to performance problems.</p><p>Instead, we found it worked much better to incrementally add one table
at a time to the publication via <code>ALTER PUBLICATION pg_upgrade_pub ADD TABLE table_name</code>.
More on this <a href="#choosing-tables-to-replicate">below</a>.</p></div></div>
</li>
<li>
<p>On the primary instance of the <strong>new</strong> database, set up the new subscription pointing to that publication:</p>
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#ffffff;color:#24292eff" tabindex="0" data-language="sql" data-theme="min-light"><code data-language="sql" data-theme="min-light" style="display:grid"><span data-line=""><span style="color:#C2C3C5">-- Note the _sub suffix, you can call this whatever you like</span></span>
<span data-line=""><span style="color:#D32F2F">CREATE</span><span style="color:#24292EFF"> SUBSCRIPTION pg_upgrade_sub</span></span>
<span data-line=""><span style="color:#C2C3C5">-- The connection string can be any standard Postgres connection string.</span></span>
<span data-line=""><span style="color:#C2C3C5">-- More details here:</span></span>
<span data-line=""><span style="color:#C2C3C5">-- https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING</span></span>
<span data-line=""><span style="color:#D32F2F">CONNECTION</span><span style="color:#22863A"> &#x27;host=old-db.cloud.com dbname=your_app user=root password=&lt;password&gt;&#x27;</span></span>
<span data-line=""><span style="color:#C2C3C5">-- The publication name MUST match the publication created on the old database</span></span>
<span data-line=""><span style="color:#24292EFF">PUBLICATION pg_upgrade_pub </span><span style="color:#D32F2F">with</span><span style="color:#24292EFF"> (</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- This subscription will not start syncing until you enable it,</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- which can be helpful when getting started</span></span>
<span data-line=""><span style="color:#D32F2F"> enabled</span><span style="color:#D32F2F"> =</span><span style="color:#24292EFF"> false,</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- Replication slots track the subscription&#x27;s progress.</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- By default, you want Postgres to manage this.</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- If you don&#x27;t create a slot here, you will need to supply one yourself.</span></span>
<span data-line=""><span style="color:#24292EFF"> create_slot </span><span style="color:#D32F2F">=</span><span style="color:#24292EFF"> true,</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- Generally you want Postgres to copy the contents of each table,</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- however for very large tables you may not want this option.</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- More details below.</span></span>
<span data-line=""><span style="color:#24292EFF"> copy_data </span><span style="color:#D32F2F">=</span><span style="color:#24292EFF"> true,</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- This will halt the subscription if something unexpected happens.</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- This is usually because of a unique constraint violation, or</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- a mismatched schema (e.g. a missing or renamed column).</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- We found it helpful to halt the subscription on error so we could</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- fix the problem and then resume replication.</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- Errors are logged to the database&#x27;s logs.</span></span>
<span data-line=""><span style="color:#24292EFF"> disable_on_error </span><span style="color:#D32F2F">=</span><span style="color:#24292EFF"> true</span></span>
<span data-line=""><span style="color:#24292EFF">);</span></span></code></pre></figure>
<p>At this point, you now have a replication pipeline
from the old database to the new one.</p>
<p>To enable the subscription:</p>
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#ffffff;color:#24292eff" tabindex="0" data-language="sql" data-theme="min-light"><code data-language="sql" data-theme="min-light" style="display:grid"><span data-line=""><span style="color:#D32F2F">ALTER</span><span style="color:#24292EFF"> SUBSCRIPTION pg_upgrade_sub </span><span style="color:#D32F2F">ENABLE</span><span style="color:#24292EFF">;</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#C2C3C5">-- To check the status of the subscription...</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#C2C3C5">-- Watch out for subenabled - if it turns false,</span></span>
<span data-line=""><span style="color:#C2C3C5">-- replication is stopped and potentially backing up on the primary!</span></span>
<span data-line=""><span style="color:#D32F2F">SELECT</span><span style="color:#D32F2F"> *</span><span style="color:#D32F2F"> FROM</span><span style="color:#24292EFF"> pg_subscription;</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#C2C3C5">-- More details on monitoring subscriptions using that table here:</span></span>
<span data-line=""><span style="color:#C2C3C5">-- https://www.postgresql.org/docs/16/catalog-pg-subscription.html</span></span></code></pre></figure>
</li>
</ol>
<h3 id="choosing-tables-to-replicate"><a href="#choosing-tables-to-replicate" aria-hidden="true" tabindex="-1"><span></span></a>Choosing tables to replicate</h3>
<p>The next step in the process is to build a list of tables you’d like to replicate.
You will want to add tables one at a time, watching each table until all of them
are fully replicated. Later in this post we will show you <a href="#checking-a-tables-replication-status">how to monitor replication
for all the tables</a>.</p>
<p>Generally, the tables will fall into three based on their disk size and the
number of tuples stored in the database.</p>
<ol>
<li>Small enough to synchronize in a few minutes: These can be replicated by just
adding them to the publication and refreshing the subscription</li>
<li>Large, append-only tables: These can be synchronize by first replicating only
future changes, and then separately backfilling old data from a backup or snapshot</li>
<li>Large, frequently updated tables: These are the hardest to synchronize, and
will require some extra care</li>
</ol>
<p>For us, &quot;small&quot; was any table using less than 50 GB of storage and 10 million tuples.</p>
<p>Anything over those thresholds we considered &quot;large&quot;.</p>
<div class="callout flex w-full items-center gap-3 rounded-md border-sm border-1 border-solid border-transparent p-4 my-6" style="background:rgba(235, 236, 237, 0.3)"><div class="flex h-8 w-8 shrink-0 items-center justify-center rounded-sm text-2xl leading-none" aria-hidden="true">❓</div><div class="min-w-0 w-full text-lg leading-normal"><p><strong>What is a tuple?</strong></p><p>Each insert or update to a Postgres table is stored as a &quot;tuple&quot;. If a table has
3 inserts followed by 2 updates, the table would have 5 tuples. Tuples are used
by Postgres’ concurrency mechanism (<a href="https://www.postgresql.org/docs/16/mvcc-intro.html">more in the docs</a>).
Postgres’ <code>VACUUM</code> procedure cleans up old tuples that are no longer needed.</p><p>When we replicate a table, we replicate all of the tuples that make up the
tables contents - inserts and updates. A table with a few rows but many tuples
that haven’t been cleaned up will take longer to replicate than a similar table
with fewer tuples.</p></div></div>
<p>The following query can help determine the size of a database table in terms of
disk space and tuple counts:</p>
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#ffffff;color:#24292eff" tabindex="0" data-language="sql" data-theme="min-light"><code data-language="sql" data-theme="min-light" style="display:grid"><span data-line=""><span style="color:#D32F2F">SELECT</span></span>
<span data-line=""><span style="color:#24292EFF">relname </span><span style="color:#D32F2F">AS</span><span style="color:#24292EFF"> tablename,</span></span>
<span data-line=""><span style="color:#24292EFF">n_live_tup </span><span style="color:#D32F2F">+</span><span style="color:#24292EFF"> n_dead_tup </span><span style="color:#D32F2F">+</span><span style="color:#24292EFF"> n_mod_since_analyze </span><span style="color:#D32F2F">as</span><span style="color:#24292EFF"> total_tuple_count,</span></span>
<span data-line=""><span style="color:#24292EFF">pg_size_pretty(pg_total_relation_size(quote_ident(relname))) </span><span style="color:#D32F2F">AS</span><span style="color:#24292EFF"> simple_size,</span></span>
<span data-line=""><span style="color:#24292EFF">pg_relation_size(quote_ident(relname)) </span><span style="color:#D32F2F">as</span><span style="color:#24292EFF"> size_in_bytes</span></span>
<span data-line=""><span style="color:#D32F2F">FROM</span><span style="color:#24292EFF"> pg_stat_user_tables;</span></span></code></pre></figure>
<p>One way to prepare your source database for replication is to <code>VACUUM</code> your tables,
which should help the source database reduce the number of tuples it needs to copy
to the target database. This can help reduce the amount of time it takes to replicate a table.</p>
<p>Before using <code>VACUUM</code>, consult the <a href="https://www.postgresql.org/docs/current/sql-vacuum.html">Postgres docs</a>.</p>
<div class="callout flex w-full items-center gap-3 rounded-md border-sm border-1 border-solid border-transparent p-4 my-6" style="background:rgba(235, 236, 237, 0.3)"><div class="flex h-8 w-8 shrink-0 items-center justify-center rounded-sm text-2xl leading-none" aria-hidden="true">🤔</div><div class="min-w-0 w-full text-lg leading-normal"><p><strong>Why does table size matter?</strong></p><p>The time it takes to synchronize a table is directly correlated to its size on disk
and the number of tuples it contains. The larger the table, the longer it takes to replicate.
This is because Postgres needs to copy the entire table over to the new database,
and then apply any changes that happen after the initial copy.</p><p>The problem with long synchronization time is that it can prevent your primary
Postgres instance from performing <code>VACUUM</code> operations, which can lead to degraded
performance over time. Left unchecked, it can even lead to transaction wraparound
and a forced shutdown of the database.</p><p>For these reasons, we added tables one at a time to replication, used different
strategies based on the size &amp; write patterns of each table, and closely monitored
the system’s performance to ensure we didn’t degrade our service.</p><p>If migrating a table becomes problematic, you can remove a table from replication
at any time, and then re-add it later (although you will need to truncate the
target table and start from scratch).</p></div></div>
<h3 id="how-to-replicate-small-tables"><a href="#how-to-replicate-small-tables" aria-hidden="true" tabindex="-1"><span></span></a>How to replicate &quot;small&quot; tables</h3>
<p>To migrate small tables, you just add it to the publication and then refresh the subscription:</p>
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#ffffff;color:#24292eff" tabindex="0" data-language="sql" data-theme="min-light"><code data-language="sql" data-theme="min-light" style="display:grid"><span data-line=""><span style="color:#C2C3C5">-- On the old database</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#D32F2F">ALTER</span><span style="color:#24292EFF"> PUBLICATION pg_upgrade_pub </span><span style="color:#D32F2F">ADD</span><span style="color:#D32F2F"> TABLE</span><span style="color:#24292EFF"> my_table_name;</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#C2C3C5">-- ON the new database</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#D32F2F">ALTER</span><span style="color:#24292EFF"> SUBSCRIPTION pg_upgrade_sub REFRESH PUBLICATION;</span></span></code></pre></figure>
<p>Postgres will handle copying the table over, getting it synchronized, and
applying any further operations to the table. For very small tables,
synchronization can happen in less than a second.</p>
<h3 id="large-append-only-tables"><a href="#large-append-only-tables" aria-hidden="true" tabindex="-1"><span></span></a>Large, append-only tables</h3>
<p>Tables that are too large but generally append-only, with no updates (or, if
updates are <em>always</em> on rows that are recent, like within the past week),
then you can set up a separate <code>PUBLICATION</code> and <code>SUBSCRIPTION</code> following the
same steps as above, but setting the <code>copy_data</code> option on the subscription to
false. Suffix the name of the new publication and new subscription with <code>_nocopy</code>
to make it distinct.</p>
<p>When you are ready to migrate these large, append-only tables, you can add them
to this <code>nocopy</code> publication, and refresh the subscription on the target using
the <code>copy_data = false</code> option:</p>
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#ffffff;color:#24292eff" tabindex="0" data-language="sql" data-theme="min-light"><code data-language="sql" data-theme="min-light" style="display:grid"><span data-line=""><span style="color:#C2C3C5">-- On the old database</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#D32F2F">ALTER</span><span style="color:#24292EFF"> PUBLICATION pg_upgrade_pub_nocopy </span><span style="color:#D32F2F">ADD</span><span style="color:#D32F2F"> TABLE</span><span style="color:#24292EFF"> my_append_only_table_name;</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#C2C3C5">-- On the new database</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#D32F2F">ALTER</span><span style="color:#24292EFF"> SUBSCRIPTION pg_upgrade_sub REFRESH PUBLICATION </span><span style="color:#D32F2F">WITH</span><span style="color:#24292EFF"> ( copy_data </span><span style="color:#D32F2F">=</span><span style="color:#24292EFF"> false );</span></span>
<span data-line=""> </span></code></pre></figure>
<p>We found this approach worked really well for our partitioned tables that stored
various types of logs for our customers.
We did not need to migrate the root of a partitioned table, we only migrated the
underlying tables, and that seemed to work pretty well.</p>
<p>Once the subscription is running, you should start seeing logs appear on the
target databases table:</p>
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#ffffff;color:#24292eff" tabindex="0" data-language="sql" data-theme="min-light"><code data-language="sql" data-theme="min-light" style="display:grid"><span data-line=""><span style="color:#D32F2F">SELECT</span><span style="color:#6F42C1"> COUNT</span><span style="color:#24292EFF">(</span><span style="color:#D32F2F">*</span><span style="color:#24292EFF">) </span><span style="color:#D32F2F">FROM</span><span style="color:#24292EFF"> my_append_only_table_name; </span><span style="color:#C2C3C5">-- Returns more than zero</span></span></code></pre></figure>
<p>From here, you can backfill any records older than those now visible in the
database using whatever means you like (e.g. <code>pg_dump</code>).</p>
<p>Here is how we did it on AWS RDS Aurora:</p>
<ol>
<li>
<p>Take a snapshot of your production database in the AWS Console</p>
</li>
<li>
<p>Restore that snapshot into a new database instance (the snapshot DB)</p>
</li>
<li>
<p>Rename the table(s) on the snapshot DB that you want to replicate by adding
a suffix like <code>_snapshot</code>. This prevents us having two replication pipelines
feeding into the same table on the target database.</p>
</li>
<li>
<p>Create the same table(s) on the target database with the same schema as the
snapshot database. Use the same suffix as above.</p>
</li>
<li>
<p>Create a publication on the snapshot database and a subscription on the target
database to replicate these snapshot table(s) from the snapshot database to the
target database</p>
</li>
<li>
<p>Enable the subscription and monitor its progress</p>
</li>
<li>
<p>Once the subscription is caught up, you can merge the tables together using
<code>INSERT...ON CONFLICT</code>:</p>
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#ffffff;color:#24292eff" tabindex="0" data-language="sql" data-theme="min-light"><code data-language="sql" data-theme="min-light" style="display:grid"><span data-line=""><span style="color:#D32F2F">INSERT INTO</span><span style="color:#24292EFF"> my_append_only_table_name</span></span>
<span data-line=""><span style="color:#D32F2F">SELECT</span><span style="color:#D32F2F"> *</span><span style="color:#D32F2F"> FROM</span><span style="color:#24292EFF"> my_append_only_table_name_snapshot</span></span>
<span data-line=""><span style="color:#D32F2F">ON</span><span style="color:#24292EFF"> CONFLICT (id) DO NOTHING;</span></span></code></pre></figure>
</li>
</ol>
<div class="cursor-pointer relative group overflow-hidden"><img src="/assets/blog/zero-downtime-postgres-upgrades/snapshot-backfill.png" alt="Diagram showing how to backfill data from a snapshot" class="rounded-md mx-auto border border-gray-200"/></div>
<p>For very large tables, this can still take several days, but because it’s all
in the background it shouldn’t affect your production environment.</p>
<p>Once the tables are fully merged, compare them to ensure a consistent row count
(more on that later). Once you are confident the tables are identical,
drop the snapshot table on the target DB, drop the subscription to the snapshot DB,
and terminate the snapshot database instance.</p>
<h3 id="large-tables-with-many-updates-over-most-of-the-rows"><a href="#large-tables-with-many-updates-over-most-of-the-rows" aria-hidden="true" tabindex="-1"><span></span></a>Large tables with many updates over most of the rows</h3>
<p>These are the hard tables. Because they have so much data in them, they can
take a long time to replicate, which can affect system performance on the
source database if it prevents <code>AUTOVUACUUM</code> from running. Because they have
so many updates, we can’t treat it as an append-only table.</p>
<p>A few points to consider:</p>
<ol>
<li>Is there any housekeeping you can do to reduce the table’s size?</li>
<li>Have you vacuumed the table recently?</li>
<li>Can you partition the table into smaller pieces?</li>
<li>Do rows stop receiving updates after a reliable time frame (e.g. 1 week?) -
this could be used to treat the table as an append-only table, and then after
that time frame has elapsed you can backfill old rows from a snapshot.</li>
</ol>
<p>If your source database is not on PG 15 or greater, your options are limited.
Follow the steps in the &quot;small tables&quot; section. Rely on the monitoring you have
in place (you do have monitoring, right?) to ensure replication doesn’t degrade
your service. If needed, you can rollback by removing the table from the
publication, and refreshing the subscription (<a href="#aborting-the-replication-of-one-table">See below</a>).</p>
<p>If the table is still too big, try to start replication during low traffic times to
reduce load and write activity. This will hopefully minimize the impact on your
system.</p>
<h3 id="large-tables-coming-from-pg-15-or-greater"><a href="#large-tables-coming-from-pg-15-or-greater" aria-hidden="true" tabindex="-1"><span></span></a>Large tables coming from PG 15 or greater</h3>
<p>If your source database is on PG 15 or greater, you may be able to split up replication
across multiple publications (similar to partitioning or sharding). You can then
migrate the table in smaller chunks, at the expense of using more replication
slots. <a href="https://www.postgresql.org/docs/16/logical-replication-config.html">The Postgres docs</a>
have more information on setting these parameters.</p>
<div class="callout flex w-full items-center gap-3 rounded-md border-sm border-1 border-solid border-transparent p-4 my-6" style="background:rgba(235, 236, 237, 0.3)"><div class="flex h-8 w-8 shrink-0 items-center justify-center rounded-sm text-2xl leading-none" aria-hidden="true">🤞</div><div class="min-w-0 w-full text-lg leading-normal"><p>Because we migrated from 11.9 to 15.3, we did not have this option available
to us. As such, we have not tested this approach. Even so, as we considered
our options we noticed that this approach might be possible. If you try it
out, let us know, we’d love to hear how it goes!</p></div></div>
<p>The goal is to have enough publications to split your largest table into manageable
pieces (for us, this was about 100 GB of non-index data stored). We’ll assume we
are splitting across three partitions in this example. The trick is adding a <code>WHERE</code>
clause that splits up the rows handled by each subscription:</p>
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#ffffff;color:#24292eff" tabindex="0" data-language="sql" data-theme="min-light"><code data-language="sql" data-theme="min-light" style="display:grid"><span data-line=""><span style="color:#C2C3C5">-- On the source database</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#C2C3C5">-- For three partitions...</span></span>
<span data-line=""><span style="color:#D32F2F">CREATE</span><span style="color:#24292EFF"> PUBLICATION pg_upgrade_pub_0;</span></span>
<span data-line=""><span style="color:#D32F2F">CREATE</span><span style="color:#24292EFF"> PUBLICATION pg_upgrade_pub_1;</span></span>
<span data-line=""><span style="color:#D32F2F">CREATE</span><span style="color:#24292EFF"> PUBLICATION pg_upgrade_pub_2;</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#D32F2F">ALTER</span><span style="color:#24292EFF"> PUBLICATION pg_upgrade_pub_0 </span><span style="color:#D32F2F">ADD</span><span style="color:#D32F2F"> TABLE</span><span style="color:#24292EFF"> big_table</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- id must be the primary key.</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- Use hashint4 for int IDs, hashint8 for bigint IDs</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- Use hashtext(id::text) for UUID or other key types</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- If you have a composite PK, concatenate the columns together before hashing as text</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- Postgres&#x27; hash functions return positive &amp; negative numbers - we abs() the result to make it positive</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- % 3 is used to pick which of three partitions. Adjust the integer for the number of partitions you will create.</span></span>
<span data-line=""><span style="color:#C2C3C5"> -- = 0 assigns rows to the first partition (zero-indexed, so we will finish with partitions 0, 1, and 2)</span></span>
<span data-line=""><span style="color:#D32F2F"> WHERE</span><span style="color:#6F42C1"> abs</span><span style="color:#24292EFF">(hashint4(id)) % </span><span style="color:#1976D2">3</span><span style="color:#D32F2F"> =</span><span style="color:#1976D2"> 0</span><span style="color:#24292EFF">;</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#C2C3C5">-- Repeat the above ALTER statement for each publication, adjust the where clause accordingly.</span></span></code></pre></figure>
<p>On the destination database, create a subscription for each partition.</p>
<p>You only want to migrate one slice of each table at a time. Generally, you will
follow the same instructions as adding a &quot;small&quot; table, but with the extra <code>WHERE</code>
clause added when setting up the table for each publication.</p>
<p>In this way, you can slice up large tables into smaller, more workable pieces.</p>
<p>Consider only using this approach if having too many replication slots is a
problem: you can still add &quot;small&quot; tables using this approach,
just add the table to the <code>_0</code> publication without a <code>WHERE</code> clause.
This can help reduce the number of replication slots required when migrating.</p>
<h3 id="checking-a-tables-replication-status"><a href="#checking-a-tables-replication-status" aria-hidden="true" tabindex="-1"><span></span></a>Checking a table’s replication status</h3>
<p>When a table is added to a subscription, it moves through five distinct states
(visible on the target database under the system table <code>pg_subscription_rel</code>
in the <code>srsubstate</code> column):</p>
<ol>
<li>Initializing the table’s subscription (State code <code>i</code> )</li>
<li>Copying the table’s contents in one efficient operation (State code <code>d</code>)</li>
</ol>
<div class="callout flex w-full items-center gap-3 rounded-md border-sm border-1 border-solid border-transparent p-4 my-6" style="background:rgba(235, 236, 237, 0.3)"><div class="flex h-8 w-8 shrink-0 items-center justify-center rounded-sm text-2xl leading-none" aria-hidden="true">🚨</div><div class="min-w-0 w-full text-lg leading-normal"><p>This step requires keeping old Postgres transaction IDs around, which
prevents vacuum from running effectively and can lead to system performance
issues and (if left running long enough) even Postgres transaction ID
wraparound which can halt the system.</p><p>This is the step that requires replicating only one table at a time.</p></div></div>
<ol start="3">
<li>Copy finished, waiting for final sync (State code <code>f</code>)</li>
<li>Finalizing initial sync (State code <code>s</code>)</li>
<li>Ready and running under normal replication (State code <code>r</code>)</li>
</ol>
<p>In order to prevent the issues found in step 2 above, we found it was necessary
to add one table at a time to replication, and to closely watch the system’s
performance. The worst-case scenario (transaction wraparound) must be avoided.</p>
<p><strong>If you get anywhere close to wraparound, it is better to <a href="#aborting-the-replication-of-one-table">abort the migration</a>
and break it up into smaller pieces.</strong></p>
<p>If we had created our publication using the <code>FOR ALL TABLES</code> option, Postgres
would have started to sync our very large source database all at once,
preventing automatic <code>VACUUM</code> operations from completing necessary maintenance.
We found this to gradually degrade database performance over time,
leading to increased risk to system stability.</p>
<p>Adding one table at a time has the added advantage of allowing teams to
incrementally migrate each table. Replication does come with CPU and other costs
for the source and destination databases. By adding one table at a time,
administrators can control how replication affects the running system.</p>
<h3 id="aborting-the-replication-of-one-table"><a href="#aborting-the-replication-of-one-table" aria-hidden="true" tabindex="-1"><span></span></a>Aborting the replication of one table</h3>
<p>If you need to halt the replication of a table, you reverse the instructions for
adding the table in the first place:</p>
<figure data-rehype-pretty-code-figure=""><pre style="background-color:#ffffff;color:#24292eff" tabindex="0" data-language="sql" data-theme="min-light"><code data-language="sql" data-theme="min-light" style="display:grid"><span data-line=""><span style="color:#C2C3C5">-- On the old database</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#D32F2F">ALTER</span><span style="color:#24292EFF"> PUBLICATION pg_upgrade_pub_nocopy </span><span style="color:#D32F2F">DROP</span><span style="color:#D32F2F"> TABLE</span><span style="color:#6F42C1"> my_append_only_table_name</span><span style="color:#24292EFF">;</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#C2C3C5">-- ON the new database</span></span>
<span data-line=""> </span>
<span data-line=""><span style="color:#D32F2F">ALTER</span><span style="color:#24292EFF"> SUBSCRIPTION pg_upgrade_sub REFRESH PUBLICATION;</span></span></code></pre></figure>
<p>In an emergency, you can also drop the publications and subscriptions entirely,
and start the process over. Postgres will clean up any replication slots that
were created as part of the publication and subscription, which should relieve
any pressure on the source database.</p>
<div class="callout flex w-full items-center gap-3 rounded-md border-sm border-1 border-solid border-transparent p-4 my-6" style="background:rgba(235, 236, 237, 0.3)"><div class="flex h-8 w-8 shrink-0 items-center justify-center rounded-sm text-2xl leading-none" aria-hidden="true">🚨</div><div class="min-w-0 w-full text-lg leading-normal"><p>Be advised that if you just disable the subscription without removing the table
from the publication and refreshing the subscription, the source database
will continue to hold onto old transaction IDs, which can lead to transaction
wraparound and a forced shutdown of the database.</p><p>Just disabling the subscription will not resolve any replication-related
performance problems.</p></div></div>
<h3 id="a-note-about-moving-replication-slots"><a href="#a-note-about-moving-replication-slots" aria-hidden="true" tabindex="-1"><span></span></a>A note about moving replication slots</h3>
<p>Replication slots in Postgres store a log of database activity that can be
consumed on another database or in another application. Postgres tracks slot
progress using a Log Sequence Number (LSN). LSNs are unique to the primary
Postgres database. This means that if you have a replication slot on your
database (e.g. to copy changes to a data warehouse or as part of your own
application), you will not be able to copy the replication slot&#x27;s LSN over from
the old database to the new database.</p>
<p>You will need to consult the documentation of the application consuming the
replication slot to decide how to best migrate (e.g. for data warehousing tools,
they may have a way to merge duplicated information between both databases).
If you’re using replication slots as part of your own application, you already
know that you’re on your own to roll your own solution. Having some idempotence
mechanism to deduplicate transactions from the old and the new database will
definitely be helpful.</p>
<h2 id="finalizing-the-migration"><a href="#finalizing-the-migration" aria-hidden="true" tabindex="-1"><span></span></a>Finalizing the migration</h2>
<p>Once you have added all of your tables to publications, and the subscriptions
have caught up on everything, you need to now verify that the tables match.</p>
<p>Unfortunately, eventual consistency (the lag between a write being applied to
the old database and it showing up on the new database) will prevent both
databases from being perfect matches at the same time, you can still count table
rows to make sure you’re close enough to know it’s working.</p>
<p>At Knock, we wrote a script that iterated through each table and asked both
databases to count the total number of rows in each table on the old and new
database, and compared the results. For tables with an <code>inserted_at</code> column, we
filtered to rows older than 10 seconds. This interval is more than enough to
prove that the tables match, with the assumption that the remaining 10 seconds
will replicate across in short order.</p>
<p>You may need to come up with a strategy that fits your application’s needs. We
felt that as long as row counts were accurate within a few seconds, we could
otherwise assume that Postgres replication was reliable.</p>
<p>In a few instances, we also spot-checked the contents of a few tables to ensure
they matched to confirm this assumption. Collecting a random sample of rows from
tables and comparing them between the old and the new database can help verify
that the tables are identical.</p>
<h3 id="application-level-changes"><a href="#application-level-changes" aria-hidden="true" tabindex="-1"><span></span></a>Application-level changes</h3>
<p>Parallel to all of this database work, you may need to change your application
to connect to both databases. When you are finally ready to cut over,
you need a strategy to shift traffic to your new database.</p>
<p>When the final cutover happens, you could change your application’s configuration
to point to the new database, and then reboot your app. This is simple,
straightforward, and is precisely how we migrated one of our
lower-traffic databases.</p>
<p>For applications with lots of concurrent activity, you may need to get creative.
We wanted to avoid a situation with conflicting writes between the old and new
database. Such conflicts could have caused a service outage for us, requiring
manually reconciling database state.</p>
<p>At Knock, we configured our application to connect to both databases.
When we were ready to execute the cutover, we ran a script that did the following:</p>
<ol>
<li>
<p>Tell all instances of our application to send new queries to the new database</p>
</li>
<li>
<p>All currently running database queries had 500 ms to complete before being forcefully cancelled</p>
</li>
<li>
<p>For the first second after flipping the flag, our application artificially paused
any new database requests for one second. This allowed pending transactions
to replicate to the new database so that new queries wouldn’t have stale reads</p>
<p>500 ms is far higher than most of our db queries, and we saw zero errors due to forced disconnections</p>
</li>
<li>
<p>After that first second, database activity returned to normal behavior, but pointing at the new database.</p>
</li>
<li>
<p>In the middle of the cutover, we had some specialized database workloads that
the script shut down and restarted in order to reconnect to the new database.</p>
</li>
</ol>
<h3 id="one-more-thing-sequences"><a href="#one-more-thing-sequences" aria-hidden="true" tabindex="-1"><span></span></a>One more thing: sequences</h3>
<p>One thing that replication doesn’t synchronize is any Postgres sequence.
Sequences are monotonically increasing integers that are guaranteed to never
duplicate. Unfortunately, they are not incremented on the new database as
sequence values are used up on the old database.</p>
<p>Fortunately, this is pretty easy to control for. Part of our cutover procedure
was to run a script right before flipping our feature flag that did the following:</p>
<ol>
<li>
<p>Connect to both databases</p>
</li>
<li>
<p>Get the next value of all of the sequences in the database using <code>SELECT nextval(&#x27;sequence_name&#x27;)</code></p>
</li>
<li>
<p>Set that value in the new database using <code>SELECT setval(&#x27;sequence_name&#x27;, value::int4 + 100000)</code>
to advance the sequence and offer a little bit of buffer (in this case, 100k
rows can be added between setting this value on the new database and cutting over).
This will introduce a gap in the sequence, but that’s generally not a problem.
For us, our sequences are bigints. 100k values skipped in the sequence is a
rounding error off of 0% used up sequence values in that case.</p>
<p>You will want to tune how big of a gap you introduce so you don’t use too
much of your sequence’s usable space. If you only expect the sequence to
use a few hundred values during your cutover window,
then maybe advance it only by 5000.</p>
</li>
</ol>
<h2 id="final-checklist-before-cutting-over"><a href="#final-checklist-before-cutting-over" aria-hidden="true" tabindex="-1"><span></span></a>Final checklist before cutting over</h2>
<p>Here are some of the things we considered before executing our final cutover:</p>
<ol>
<li>Do the rows on all the tables match as expected?</li>
<li>Are all the subscriptions enabled and running without error?</li>
<li>Do the schemas match? Can you freeze any new schema migrations from being
released to reduce the risk of something changing while you’re migrating?</li>
<li>Is your new database properly sized for your workloads?</li>
<li>Do you have to add any read replicas so the database cluster topology is the
same between the old and the new database?</li>
<li>Have you reindexed and performed basic VACUUM maintenance on the new database
to ensure it’s fresh and ready for production traffic?</li>
<li>Have you double checked Postgres’ release notes for anything that might cause
a regression in your app?</li>
<li>Have you run automated and manual tests against a staging database on the new
version to verify system performance?</li>
<li>Have you run load tests of your most demanding queries using <code>pg_bench</code>
against your new version to verify performance?</li>
<li>If there’s one thing that you can de-risk still, what is it?</li>
<li>Do practice runs in a staging or test environment until you have fully
exercised the cutover process multiple times. Dry runs like this will help
reveal gaps in your plan before you go to production.</li>
<li>Right before cutover, take a database backup - just in case.</li>
</ol>
<h2 id="cutting-over"><a href="#cutting-over" aria-hidden="true" tabindex="-1"><span></span></a>Cutting over</h2>
<p>At Knock, we took a few weeks replicating tables one at a time. We generally did
this after business hours and during our lowest traffic time frames. We practiced
cutover in our staging environment multiple times, ironing out the process until
it just worked without much operator involvement.</p>
<p>Once we had a replica running PG 15 and had the application code in place to
cut over from the old to the new database, we ran one final set of checks and
flipped the flag.</p>
<p>After months of preparation, the actual cutover was uneventful: our
application cut over within a few seconds, we had a brief blip of (intentional)
latency as queries waited to allow for replication, and our application
continued running without skipping a beat. Reading this paragraph took longer
than the cutover itself.</p>
<p>From there, we rolled back the application changes we introduced, permanently
pointed everything at the new database, removed the subscriptions on the new
database, and tore down the old database. We had successfully jumped from
Postgres 11.9 to 15.3 with zero downtime!</p>
<h2 id="conclusion"><a href="#conclusion" aria-hidden="true" tabindex="-1"><span></span></a>Conclusion</h2>
<p>Although jumping four major versions of Postgres in one leap is a painstaking
process, it can be done, and in many ways it’s safer than scheduled downtime:
it can be practiced, tested, and reworked multiple times before performing the
actual cutover. At any point in the process, we could have dropped the
publications from the old database and started over without degrading our service.</p>
<p>Modern customers expect 100% availability. While that is not technically possible,
zero downtime migrations make it easier to keep systems running smoothly without
major service interruptions.</p></div><footer class="mt-8"><div class="flex items-center gap-2 text-[13px] text-[#60646C]"><span>Share:</span><div class="flex items-center gap-1"><button type="button" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="Copy Link"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-link" aria-hidden="true"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"></path><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"></path></svg></button><a href="https://twitter.com/intent/tweet?text=Zero%20downtime%20Postgres%20upgrades&amp;url=https%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades" target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="X (Twitter)"><svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg" fill="currentColor" width="16" height="16"><title>X</title><path d="M14.234 10.162 22.977 0h-2.072l-7.591 8.824L7.251 0H.258l9.168 13.343L.258 24H2.33l8.016-9.318L16.749 24h6.993zm-2.837 3.299-.929-1.329L3.076 1.56h3.182l5.965 8.532.929 1.329 7.754 11.09h-3.182z"></path></svg></a><a href="https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades" target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="LinkedIn"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-linkedin" aria-hidden="true"><path d="M16 8a6 6 0 0 1 6 6v7h-4v-7a2 2 0 0 0-2-2 2 2 0 0 0-2 2v7h-4v-7a6 6 0 0 1 6-6z"></path><rect width="4" height="12" x="2" y="9"></rect><circle cx="4" cy="4" r="2"></circle></svg></a><a href="https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades" target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="Facebook"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-facebook" aria-hidden="true"><path d="M18 2h-3a5 5 0 0 0-5 5v3H7v4h3v8h4v-8h3l1-4h-4V7a1 1 0 0 1 1-1h3z"></path></svg></a><a href="https://www.reddit.com/submit?url=https%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades&amp;title=Zero%20downtime%20Postgres%20upgrades" target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="Reddit"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.75" stroke-linecap="round" stroke-linejoin="round" role="img" aria-label="Reddit Icon"><path stroke="none" d="M0 0h24v24H0z" fill="none"></path><path d="M12 8c2.648 0 5.028 .826 6.675 2.14a2.5 2.5 0 0 1 2.326 4.36c0 3.59 -4.03 6.5 -9 6.5c-4.875 0 -8.845 -2.8 -9 -6.294l-1 -.206a2.5 2.5 0 0 1 2.326 -4.36c1.646 -1.313 4.026 -2.14 6.674 -2.14z"></path><path d="M12 8l1 -5l6 1"></path><path d="M19 4m-1 0a1 1 0 1 0 2 0a1 1 0 1 0 -2 0"></path><circle cx="9" cy="13" r=".5" fill="currentColor"></circle><circle cx="15" cy="13" r=".5" fill="currentColor"></circle><path d="M10 17c.667 .333 1.333 .5 2 .5s1.333 -.167 2 -.5"></path></svg></a><a href="https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades&amp;t=Zero%20downtime%20Postgres%20upgrades" target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="Hacker News"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.75" stroke-linecap="round" stroke-linejoin="round" role="img" aria-label="HackerNews Icon"><path stroke="none" d="M0 0h24v24H0z" fill="none"></path><path d="M4 4m0 2a2 2 0 0 1 2 -2h12a2 2 0 0 1 2 2v12a2 2 0 0 1 -2 2h-12a2 2 0 0 1 -2 -2z"></path><path d="M8 7l4 6l4 -6"></path><path d="M12 17l0 -4"></path></svg></a><a href="https://dev.to/new?prefill=---%0Atitle%3A%20Zero%20downtime%20Postgres%20upgrades%0Apublished%3A%20false%0A---%0A%0AZero%20downtime%20Postgres%20upgrades%0A%0Ahttps%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades" target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="Dev.to"><svg width="16" height="16" viewBox="0 0 512 512" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M80.9021 180.224H55.3265V332.827H80.9168C95.809 332.031 106.372 320.164 106.536 307.395V205.656C105.516 190.517 95.1037 180.426 80.9021 180.224ZM147.965 307.673C147.965 335.149 130.939 376.78 77.0453 376.692H9V135H78.4824C130.455 135 147.936 176.572 147.95 204.063L147.965 307.673ZM295.611 178.165H217.477V234.286H265.24V277.48H217.477V333.586H295.626V376.78H204.439C188.073 377.204 174.464 364.32 174.054 348.018V165.266C173.658 148.979 186.607 135.453 202.958 135.044H295.626L295.611 178.165ZM447.599 346.572C428.241 391.49 393.558 382.55 378.028 346.572L321.524 135.058H369.288L412.857 301.173L456.222 135.058H504L447.599 346.572Z" fill="currentColor"></path></svg></a><a href="https://wa.me/?text=Zero%20downtime%20Postgres%20upgrades%20https%3A%2F%2Fknock.app%2Fblog%2Fzero-downtime-postgres-upgrades" target="_blank" rel="noopener noreferrer" class="text-[#60646C] hover:text-[#1C2024] transition-colors px-1.5 py-1 flex items-center justify-center" aria-label="WhatsApp"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.75" stroke-linecap="round" stroke-linejoin="round" role="img" aria-label="WhatsApp Icon"><path stroke="none" d="M0 0h24v24H0z" fill="none"></path><path d="M3 21l1.65 -3.8a9 9 0 1 1 3.4 2.9l-5.05 .9"></path><path d="M9 10a.5 .5 0 0 0 1 0v-1a.5 .5 0 0 0 -1 0v1a5 5 0 0 0 5 5h1a.5 .5 0 0 0 0 -1h-1a.5 .5 0 0 0 0 1"></path></svg></a></div></div></footer></div></section><aside class="lgCustom:border-l-[0.5px] lgCustom:w-[240px] lg:w-[320px] xlCustom:w-[320px] flex-shrink-0 flex-grow-0 relative flex flex-col"><div><div class="p-6 border-t-[0.5px] lgCustom:border-t-0 border-b-[0.5px] border-[#CDCED6] bg-white"><h4 class="text-[#60646C] text-[13px] leading-5 mb-3">Written by</h4><div class="flex items-start gap-3 "><div class="relative w-12 h-12 flex-shrink-0 overflow-hidden rounded-full"><img alt="Brent Anderson" loading="lazy" decoding="async" data-nimg="fill" style="position:absolute;height:100%;width:100%;left:0;top:0;right:0;bottom:0;object-fit:cover;color:transparent" sizes="100vw" srcSet="/_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fbrent.jpeg&amp;w=640&amp;q=75 640w, /_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fbrent.jpeg&amp;w=750&amp;q=75 750w, /_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fbrent.jpeg&amp;w=828&amp;q=75 828w, /_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fbrent.jpeg&amp;w=1080&amp;q=75 1080w, /_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fbrent.jpeg&amp;w=1200&amp;q=75 1200w, /_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fbrent.jpeg&amp;w=1920&amp;q=75 1920w, /_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fbrent.jpeg&amp;w=2048&amp;q=75 2048w, /_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fbrent.jpeg&amp;w=3840&amp;q=75 3840w" src="/_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fbrent.jpeg&amp;w=3840&amp;q=75"/></div><div class="flex flex-col justify-center min-h-[48px]"><span class="text-[15px] font-medium text-[#1C2024] dark:text-solid-white">Brent Anderson</span><span class="text-[13px] text-[#60646C] dark:text-solid-grey1 mt-0.5">Platform Engineer</span></div></div></div></div><div class="hidden lg:flex flex-col flex-1 min-h-0"><div class="sticky top-16 flex flex-col w-full bg-white z-10"><div class="bg-white flex-shrink-0"><div class="border-t-[0.5px] border-b-[0.5px] border-[#CDCED6]"><div class="p-6"><h3 class="text-lg font-medium mb-2">Keep in touch</h3><p class="text-sm text-gray-600 dark:text-gray-400 mb-4">We write about all things notifications and our experience building Knock</p><div class="flex flex-col justify-end"><form class="flex gap-3 flex-col"><input placeholder="Enter your email" type="email" class="h-10 pl-3 pr-3 border border-[#E0E1E6] bg-[transparent] rounded outline-none focus:outline-none focus:border-black text-[#80838D] text-sm transition-all w-full min-w-[50px]" required="" style="opacity:0;transform:translateY(5px)" name="email"/><div><button type="submit" class="!py-2 !px-3 !w-full box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] text-solid-white !font-semibold hover:bg-[#E33A1E] bg-[#FF6C52] focus:outline-1 focus:outline-offset-4 focus:outline-surface-actionFilledHovered"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px] mx-auto !text-sm font-medium">Subscribe</span></button></div></form></div></div></div></div></div><div class="flex-1"></div></div><div class="border-t-[0.5px] border-[#CDCED6] lgCustom:hidden"><div class="flex flex-col divide-y-[0.5px] divide-[#CDCED6]"><article class="relative overflow-hidden h-full flex flex-col"><a class="flex flex-col flex-1 bg-white" href="/blog/how-we-built-liquid-autocomplete-in-tiptap"><div class="pt-4 px-4"><img alt="" loading="lazy" width="250" height="150" decoding="async" data-nimg="1" class="w-full h-auto object-contain rounded" style="color:transparent;max-width:100%;height:auto" sizes="100vw" srcSet="/_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=640&amp;q=75 640w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=750&amp;q=75 750w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=828&amp;q=75 828w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=1080&amp;q=75 1080w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=1200&amp;q=75 1200w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=1920&amp;q=75 1920w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=2048&amp;q=75 2048w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=3840&amp;q=75 3840w" src="/_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=3840&amp;q=75"/></div><div class="p-4 flex flex-col flex-1"><div class="mb-2 flex items-center"><span class="text-xs text-gray-500">Engineering</span><time class="text-xs text-gray-500 ml-auto">23 Jul 2026</time></div><h2 class="text-lg font-semibold mb-3 leading-6">How we built Liquid autocomplete in Tiptap</h2><p class="text-gray-600 text-sm flex-1">Learn how we rebuilt Liquid autocomplete for Knock&#x27;s new Tiptap-based template editor, from scope-aware suggestions inside loops to keeping it working in documents that won&#x27;t parse.</p><div class="flex items-center text-sm text-gray-500 mt-4 pt-4 justify-between"><div class="flex items-center"><div class="w-8 h-8 rounded-full overflow-hidden mr-2"><img alt="Kyle McDonald" loading="lazy" width="32" height="32" decoding="async" data-nimg="1" class="object-cover" style="color:transparent;max-width:100%;height:auto" srcSet="/_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fkyle.png&amp;w=32&amp;q=75 1x, /_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fkyle.png&amp;w=64&amp;q=75 2x" src="/_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fkyle.png&amp;w=64&amp;q=75"/></div><span class="text-xs">Kyle McDonald</span></div></div></div></a></article><article class="relative overflow-hidden h-full flex flex-col"><a class="flex flex-col flex-1 bg-white" href="/blog/how-we-built-the-knock-agent-virtual-filesystem-and-bash"><div class="pt-4 px-4"><img alt="" loading="lazy" width="250" height="150" decoding="async" data-nimg="1" class="w-full h-auto object-contain rounded" style="color:transparent;max-width:100%;height:auto" sizes="100vw" srcSet="/_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=640&amp;q=75 640w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=750&amp;q=75 750w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=828&amp;q=75 828w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=1080&amp;q=75 1080w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=1200&amp;q=75 1200w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=1920&amp;q=75 1920w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=2048&amp;q=75 2048w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=3840&amp;q=75 3840w" src="/_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=3840&amp;q=75"/></div><div class="p-4 flex flex-col flex-1"><div class="mb-2 flex items-center"><span class="text-xs text-gray-500">Engineering</span><time class="text-xs text-gray-500 ml-auto">09 Jul 2026</time></div><h2 class="text-lg font-semibold mb-3 leading-6">Files over tools: how we built the Knock Agent using a virtual file system and bash</h2><p class="text-gray-600 text-sm flex-1">A look at how we architected the Knock Agent with bash, a virtual file system, and the management API.</p><div class="flex items-center text-sm text-gray-500 mt-4 pt-4 justify-between"><div class="flex items-center"><div class="w-8 h-8 rounded-full overflow-hidden mr-2"><img alt="Chris Bell" loading="lazy" width="32" height="32" decoding="async" data-nimg="1" class="object-cover" style="color:transparent;max-width:100%;height:auto" srcSet="/_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fchris.jpeg&amp;w=32&amp;q=75 1x, /_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fchris.jpeg&amp;w=64&amp;q=75 2x" src="/_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fchris.jpeg&amp;w=64&amp;q=75"/></div><span class="text-xs">Chris Bell</span></div></div></div></a></article></div></div><div class="relative z-0 mt-auto hidden border-t-[0.5px] border-[#CDCED6] bg-white lgCustom:block"><div class="flex flex-col divide-y-[0.5px] divide-[#CDCED6]"><article class="relative overflow-hidden h-full flex flex-col"><a class="flex flex-col flex-1 bg-white" href="/blog/how-we-built-liquid-autocomplete-in-tiptap"><div class="pt-4 px-4"><img alt="" loading="lazy" width="250" height="150" decoding="async" data-nimg="1" class="w-full h-auto object-contain rounded" style="color:transparent;max-width:100%;height:auto" sizes="100vw" srcSet="/_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=640&amp;q=75 640w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=750&amp;q=75 750w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=828&amp;q=75 828w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=1080&amp;q=75 1080w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=1200&amp;q=75 1200w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=1920&amp;q=75 1920w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=2048&amp;q=75 2048w, /_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=3840&amp;q=75 3840w" src="/_next/image?url=%2Fassets%2Fblog%2Fpreview%2Fhow-we-built-liquid-autocomplete-in-tiptap.png&amp;w=3840&amp;q=75"/></div><div class="p-4 flex flex-col flex-1"><div class="mb-2 flex items-center"><span class="text-xs text-gray-500">Engineering</span><time class="text-xs text-gray-500 ml-auto">23 Jul 2026</time></div><h2 class="text-lg font-semibold mb-3 leading-6">How we built Liquid autocomplete in Tiptap</h2><p class="text-gray-600 text-sm flex-1">Learn how we rebuilt Liquid autocomplete for Knock&#x27;s new Tiptap-based template editor, from scope-aware suggestions inside loops to keeping it working in documents that won&#x27;t parse.</p><div class="flex items-center text-sm text-gray-500 mt-4 pt-4 justify-between"><div class="flex items-center"><div class="w-8 h-8 rounded-full overflow-hidden mr-2"><img alt="Kyle McDonald" loading="lazy" width="32" height="32" decoding="async" data-nimg="1" class="object-cover" style="color:transparent;max-width:100%;height:auto" srcSet="/_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fkyle.png&amp;w=32&amp;q=75 1x, /_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fkyle.png&amp;w=64&amp;q=75 2x" src="/_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fkyle.png&amp;w=64&amp;q=75"/></div><span class="text-xs">Kyle McDonald</span></div></div></div></a></article><article class="relative overflow-hidden h-full flex flex-col"><a class="flex flex-col flex-1 bg-white" href="/blog/how-we-built-the-knock-agent-virtual-filesystem-and-bash"><div class="pt-4 px-4"><img alt="" loading="lazy" width="250" height="150" decoding="async" data-nimg="1" class="w-full h-auto object-contain rounded" style="color:transparent;max-width:100%;height:auto" sizes="100vw" srcSet="/_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=640&amp;q=75 640w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=750&amp;q=75 750w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=828&amp;q=75 828w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=1080&amp;q=75 1080w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=1200&amp;q=75 1200w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=1920&amp;q=75 1920w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=2048&amp;q=75 2048w, /_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=3840&amp;q=75 3840w" src="/_next/image?url=%2Fassets%2Fblog%2Fhow-we-built-the-knock-agent%2Fcover.png&amp;w=3840&amp;q=75"/></div><div class="p-4 flex flex-col flex-1"><div class="mb-2 flex items-center"><span class="text-xs text-gray-500">Engineering</span><time class="text-xs text-gray-500 ml-auto">09 Jul 2026</time></div><h2 class="text-lg font-semibold mb-3 leading-6">Files over tools: how we built the Knock Agent using a virtual file system and bash</h2><p class="text-gray-600 text-sm flex-1">A look at how we architected the Knock Agent with bash, a virtual file system, and the management API.</p><div class="flex items-center text-sm text-gray-500 mt-4 pt-4 justify-between"><div class="flex items-center"><div class="w-8 h-8 rounded-full overflow-hidden mr-2"><img alt="Chris Bell" loading="lazy" width="32" height="32" decoding="async" data-nimg="1" class="object-cover" style="color:transparent;max-width:100%;height:auto" srcSet="/_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fchris.jpeg&amp;w=32&amp;q=75 1x, /_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fchris.jpeg&amp;w=64&amp;q=75 2x" src="/_next/image?url=%2Fassets%2Fblog%2Fauthors%2Fchris.jpeg&amp;w=64&amp;q=75"/></div><span class="text-xs">Chris Bell</span></div></div></div></a></article></div></div></aside></main></article></div><div class="hidden md:grid grid-cols-2 col-span-1 border-[#CDCED6] "><div class="grid grid-cols-2 col-span-2"><div class="col-span-1 border-r border-dashed border-[#E0E1E6]"></div></div><div class="grid grid-cols-2 col-span-2 border-t border-dashed border-[#E0E1E6] mask-gradient-fade-right"><div class="col-span-1 border-r border-dashed border-[#E0E1E6]"></div></div></div></div></div></div><div class="w-full min-w-0 border-t border-[#CDCED6] border-t-[0.5px]"></div><div class="bg-[#F9F9FB] w-full"><div class="max-w-[1440px] mx-auto"><div class="hidden md:grid grid-cols-12 md:h-14 lg:h-[120px] bg-[#F9F9FB]"><div class="col-span-10 col-start-2 border-x-[0.5px] mask-gradient-fade-bottom bg-white"></div></div></div></div><footer class="pt-10 pb-10 md:pt-20 bg-linear18 lg:pb-14"><div class="max-w-[1440px] mx-auto"><div class="flex flex-col relative w-full px-[16px] sm:px-10 mx-auto max-w-[1120px]"><div class="flex flex-col justify-between gap-16 lg:items-end lg:flex-row"><div><a class="dark:text-solid-white" href="/"><span class="sr-only">Logo</span><svg width="74" height="24" viewBox="0 0 76 25" fill="white" xmlns="http://www.w3.org/2000/svg" class="fill-current"><path d="M0 24.1383V3.0076H4.31054V15.1253H4.43111L9.16366 9.18704H13.9866L8.74165 15.3363L14.3785 24.1383H9.58568L6.20959 18.3206L4.31054 20.4608V24.1383H0Z"></path><path d="M15.3085 24.1383V9.18704H18.9257L19.2272 11.0861H19.3477C19.9908 10.4832 20.6942 9.96073 21.4578 9.51862C22.2214 9.05642 23.1157 8.82532 24.1406 8.82532C25.7683 8.82532 26.9439 9.35786 27.6674 10.4229C28.4109 11.488 28.7827 12.965 28.7827 14.854V24.1383H24.3516V15.4268C24.3516 14.3416 24.2009 13.5981 23.8994 13.1961C23.6181 12.7942 23.1559 12.5933 22.5128 12.5933C21.9501 12.5933 21.4678 12.7239 21.0659 12.9851C20.664 13.2263 20.2219 13.578 19.7396 14.0402V24.1383H15.3085Z"></path><path d="M37.7735 24.5C36.829 24.5 35.9146 24.3292 35.0304 23.9876C34.1462 23.6258 33.3625 23.1134 32.6792 22.4502C32.0161 21.7871 31.4835 20.9732 31.0816 20.0086C30.6797 19.0239 30.4787 17.9086 30.4787 16.6627C30.4787 15.4167 30.6797 14.3115 31.0816 13.3469C31.4835 12.3622 32.0161 11.5382 32.6792 10.8751C33.3625 10.2119 34.1462 9.70953 35.0304 9.36791C35.9146 9.00618 36.829 8.82532 37.7735 8.82532C38.718 8.82532 39.6223 9.00618 40.4864 9.36791C41.3706 9.70953 42.1443 10.2119 42.8075 10.8751C43.4907 11.5382 44.0333 12.3622 44.4352 13.3469C44.8372 14.3115 45.0381 15.4167 45.0381 16.6627C45.0381 17.9086 44.8372 19.0239 44.4352 20.0086C44.0333 20.9732 43.4907 21.7871 42.8075 22.4502C42.1443 23.1134 41.3706 23.6258 40.4864 23.9876C39.6223 24.3292 38.718 24.5 37.7735 24.5ZM37.7735 20.9129C38.6778 20.9129 39.3611 20.5311 39.8233 19.7675C40.2855 19.0038 40.5166 17.9689 40.5166 16.6627C40.5166 15.3564 40.2855 14.3215 39.8233 13.5579C39.3611 12.7942 38.6778 12.4124 37.7735 12.4124C36.8491 12.4124 36.1558 12.7942 35.6936 13.5579C35.2515 14.3215 35.0304 15.3564 35.0304 16.6627C35.0304 17.9689 35.2515 19.0038 35.6936 19.7675C36.1558 20.5311 36.8491 20.9129 37.7735 20.9129Z"></path><path d="M53.3785 24.5C52.3335 24.5 51.3589 24.3292 50.4546 23.9876C49.5703 23.6258 48.7967 23.1134 48.1335 22.4502C47.4703 21.7871 46.9478 20.9732 46.566 20.0086C46.1842 19.0239 45.9933 17.9086 45.9933 16.6627C45.9933 15.4167 46.2043 14.3115 46.6263 13.3469C47.0483 12.3622 47.611 11.5382 48.3144 10.8751C49.0378 10.2119 49.8718 9.70953 50.8163 9.36791C51.7608 9.00618 52.7455 8.82532 53.7704 8.82532C54.6948 8.82532 55.5086 8.97604 56.212 9.27747C56.9354 9.57891 57.5785 9.96073 58.1412 10.4229L56.0613 13.2866C55.3579 12.7038 54.6847 12.4124 54.0416 12.4124C52.9565 12.4124 52.1024 12.7942 51.4794 13.5579C50.8565 14.3215 50.545 15.3564 50.545 16.6627C50.545 17.9689 50.8565 19.0038 51.4794 19.7675C52.1225 20.5311 52.9263 20.9129 53.8909 20.9129C54.3732 20.9129 54.8354 20.8124 55.2775 20.6115C55.7397 20.3904 56.1718 20.1292 56.5737 19.8277L58.322 22.7215C57.5785 23.3646 56.7747 23.8268 55.9106 24.1081C55.0464 24.3694 54.2024 24.5 53.3785 24.5Z"></path><path d="M59.7327 24.1383V3.0076H64.0432V15.1253H64.1638L68.8963 9.18704H73.7193L68.4743 15.3363L74.1112 24.1383H69.3183L65.9423 18.3206L64.0432 20.4608V24.1383H59.7327Z"></path><path d="M75.1388 4.04252C75.1388 5.999 73.5527 7.58504 71.5962 7.58504C69.6398 7.58504 68.0537 5.999 68.0537 4.04252C68.0537 2.08604 69.6398 0.5 71.5962 0.5C73.5527 0.5 75.1388 2.08604 75.1388 4.04252Z" fill="#FF6C52"></path></svg></a><h2 class="font-ppNeueMontreal max-w-2xl mt-4 text-[28px] leading-[32px]
md:text-[34px] md:!leading-[40px]
lg:text-[40px] lg:!leading-[48px]">Agentic customer engagement<br/>for growth.</h2></div><div class="flex flex-col gap-4 lg:flex-row lg:h-full lg:gap-6 lg:items-end"><a href="https://dashboard.knock.app/signup" class="inline-block !p-0 flex gap-2 lg:hover:!bg-inherit hover:text-[#E33A1E] box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] bg-solid-white lg:hover:bg-[#D1D1D1] focus:outline-1 focus:outline-offset-4 focus:outline-solid-white text-solid-black active:bg-[#D1D1D1]"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px] font-ppNeueMontreal !text-xl lg:!text-2xl font-medium">Get started<svg width="24" height="24" fill="none" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" class="w-[22px] h-[22px]"><path d="m12.563 5.25 6.75 6.75-6.75 6.75M18.375 12H4.687" stroke="currentColor" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round"></path></svg></span></a><a href="/contact-sales" class="inline-block !p-0 flex gap-2 lg:hover:!bg-inherit hover:text-[#E33A1E] box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] bg-solid-white lg:hover:bg-[#D1D1D1] focus:outline-1 focus:outline-offset-4 focus:outline-solid-white text-solid-black active:bg-[#D1D1D1]"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px] font-ppNeueMontreal !text-xl lg:!text-2xl font-medium">Book a demo<svg width="24" height="24" fill="none" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" class="w-[22px] h-[22px]"><path d="m12.563 5.25 6.75 6.75-6.75 6.75M18.375 12H4.687" stroke="currentColor" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round"></path></svg></span></a></div></div><div class="grid grid-cols-1 mt-[72px] sm:grid-cols-2 md:grid-cols-3 lg:grid-cols-5 gap-y-8 sm:gap-y-12 md:gap-y-12 lg:gap-x-16 lg:justify-between"><ul class="w-full list-none pl-0"><li class="mb-3 w-full"><h6 class="font-semibold font-inconsolata text-[15px] uppercase text-[#80838D] tracking-[0.42px]">PRODUCT</h6></li><li class="w-full list-none pl-0"><a href="/features/workflows" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Workflows</span></a></li><li class="w-full list-none pl-0"><a href="/features/guides" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Guides</span></a></li><li class="w-full list-none pl-0"><a href="/features/broadcasts" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Broadcasts</span></a></li><li class="w-full list-none pl-0"><a href="/features/agents" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Agents</span></a></li><li class="w-full list-none pl-0"><a href="/features/template-management" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Template management</span></a></li><li class="w-full list-none pl-0"><a href="/features/preferences" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Preferences</span></a></li><li class="w-full list-none pl-0"><a href="/features/observability" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Observability + analytics</span></a></li><li class="w-full list-none pl-0"><a href="/cli" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Infrastructure as code</span></a></li><li class="w-full list-none pl-0"><a href="/pricing" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Pricing</span></a></li></ul><div><ul class="w-full list-none pl-0"><li class="mb-3 w-full"><h6 class="font-semibold font-inconsolata text-[15px] uppercase text-[#80838D] tracking-[0.42px]">SOLUTIONS</h6></li><li class="w-full list-none pl-0"><a href="/solutions/engineering" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Engineering</span></a></li><li class="w-full list-none pl-0"><a href="/solutions/growth-product" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Growth + product</span></a></li><li class="w-full list-none pl-0"><a href="/solutions/marketing-operations" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Marketing operations</span></a></li></ul><div class="mt-6"><ul class="w-full list-none pl-0"><li class="mb-3 w-full"><h6 class="font-semibold font-inconsolata text-[15px] uppercase text-[#80838D] tracking-[0.42px]">INTEGRATIONS</h6></li><li class="w-full list-none pl-0"><a href="/channels/in-app-notifications" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">In-app</span></a></li><li class="w-full list-none pl-0"><a href="/channels/email" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Email</span></a></li><li class="w-full list-none pl-0"><a href="/channels/push" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Push</span></a></li><li class="w-full list-none pl-0"><a href="/channels/slackkit" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Chat</span></a></li><li class="w-full list-none pl-0"><a href="/integrations" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">More</span></a></li></ul></div></div><ul class="w-full list-none pl-0"><li class="mb-3 w-full"><h6 class="font-semibold font-inconsolata text-[15px] uppercase text-[#80838D] tracking-[0.42px]">DEVELOPERS</h6></li><li class="w-full list-none pl-0"><a href="https://docs.knock.app" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Documentation</span></a></li><li class="w-full list-none pl-0"><a href="https://docs.knock.app/getting-started/quick-start" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Quick start</span></a></li><li class="w-full list-none pl-0"><a href="https://docs.knock.app/guides/alerting" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Tutorials</span></a></li><li class="w-full list-none pl-0"><a href="https://docs.knock.app/getting-started/example-app" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Example apps</span></a></li><li class="w-full list-none pl-0"><a href="/cli" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Knock CLI</span></a></li><li class="w-full list-none pl-0"><a href="https://docs.knock.app/developer-tools/mcp-server" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Knock MCP</span></a></li><li class="w-full list-none pl-0"><a href="https://docs.knock.app/reference" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">API reference</span></a></li><li class="w-full list-none pl-0"><a href="https://github.com/knocklabs" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">GitHub</span></a></li><li class="w-full list-none pl-0"><a href="https://knockstatus.com" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">API status</span></a></li></ul><ul class="w-full list-none pl-0"><li class="mb-3 w-full"><h6 class="font-semibold font-inconsolata text-[15px] uppercase text-[#80838D] tracking-[0.42px]">RESOURCES</h6></li><li class="w-full list-none pl-0"><a href="/changelog" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Changelog</span></a></li><li class="w-full list-none pl-0"><a href="/customers" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Customers</span></a></li><li class="w-full list-none pl-0"><a href="/blog" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Blog</span></a></li><li class="w-full list-none pl-0"><a href="/manuals" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Manuals</span></a></li><li class="w-full list-none pl-0"><a href="/template-library" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Templates</span></a></li><li class="w-full list-none pl-0"><a href="/tools" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Tools</span></a></li><li class="w-full list-none pl-0"><a href="/compare" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Compare</span></a></li></ul><ul class="w-full list-none pl-0"><li class="mb-3 w-full"><h6 class="font-semibold font-inconsolata text-[15px] uppercase text-[#80838D] tracking-[0.42px]">COMPANY</h6></li><li class="w-full list-none pl-0"><a href="/about" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">About us</span></a></li><li class="w-full list-none pl-0"><a href="https://jobs.ashbyhq.com/knock" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Careers</span></a></li><li class="w-full list-none pl-0"><a href="/contact-sales" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Contact us</span></a></li><li class="w-full list-none pl-0"><a href="https://docs.knock.app/security" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Security</span></a></li><li class="w-full list-none pl-0"><a href="/legal/privacy-policy" class="inline-block hover:!text-solid-red font-normal !w-fit !px-0 box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium rounded-[4px] xl:text-base text-sm leading-[22px] xl:px-6 px-4 xl:py-2 py-[10px] dark:hover:text-solid-grey1 hover:text-solid-grey2 dark:text-solid-white text-solid-black focus:outline-1 focus:outline-offset-4 dark:focus:outline-solid-white focus:outline-solid-black text-solid-grey3 hover:text-solid-grey3"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]">Legal</span></a></li></ul></div><div class="flex flex-col items-center gap-4 mt-16 md:flex-row md:justify-between md:items-center xl:flex-row xl:items-center order-last"><div class="flex items-center h-6 gap-6"><a target="_blank" rel="noopener noreferrer" class="dark:text-solid-white hover:!text-solid-red dark:outline-solid-white outline-solid-black" aria-label="X" href="https://x.com/knocklabs"><svg viewBox="-0.1 0 24.1 22" fill="none" xmlns="http://www.w3.org/2000/svg" class="w-6 h-6" style="transform:scale(.8) translateY(1px)"><path d="M18.8799 0H22.5599L14.4799 9.2L23.9199 21.68H16.5119L10.7119 14.096L4.07187 21.68H0.391865L8.95187 11.84L-0.0881348 0H7.50387L12.7439 6.928L18.8799 0ZM17.5919 19.52H19.6319L6.43187 2.08H4.23986L17.5919 19.52Z" fill="currentColor"></path></svg></a><a target="_blank" rel="noopener noreferrer" class="dark:text-solid-white hover:!text-solid-red dark:outline-solid-white outline-solid-black" aria-label="github" href="https://github.com/knocklabs"><svg width="24" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg" class="w-6 h-6"><path fill-rule="evenodd" clip-rule="evenodd" d="M12.227 3A9.228 9.228 0 0 0 3 12.228a9.23 9.23 0 0 0 6.31 8.756c.462.085.63-.2.63-.445 0-.22-.007-.8-.012-1.57-2.567.558-3.109-1.236-3.109-1.236-.42-1.067-1.024-1.35-1.024-1.35-.838-.572.063-.561.063-.561.926.065 1.413.95 1.413.95.824 1.41 2.16 1.004 2.686.767.084-.596.322-1.002.586-1.233-2.049-.233-4.203-1.025-4.203-4.56 0-1.008.36-1.831.95-2.477-.096-.233-.412-1.171.09-2.442 0 0 .775-.248 2.538.946a8.861 8.861 0 0 1 2.31-.31 8.84 8.84 0 0 1 2.31.31c1.762-1.194 2.535-.946 2.535-.946.503 1.27.187 2.209.092 2.442.592.646.948 1.469.948 2.476 0 3.546-2.157 4.325-4.213 4.554.331.285.627.848.627 1.709 0 1.233-.012 2.228-.012 2.53 0 .248.167.535.635.445A9.228 9.228 0 0 0 12.227 3" fill="currentColor"></path></svg></a><a target="_blank" rel="noopener noreferrer" class="dark:text-solid-white hover:!text-solid-red dark:outline-solid-white outline-solid-black" aria-label="linkedin" href="https://www.linkedin.com/company/knocklabs/"><svg width="24" height="24" fill="none" xmlns="http://www.w3.org/2000/svg" class="w-6 h-6"><path fill-rule="evenodd" clip-rule="evenodd" d="M7.128 4.975c0 1.076-.82 1.947-2.101 1.947-1.232 0-2.052-.87-2.026-1.947C2.975 3.845 3.795 3 5.05 3c1.258 0 2.053.846 2.077 1.975ZM3.104 21V8.46H7V21H3.104Zm7.02-8.54c0-1.564-.052-2.897-.103-4h3.385l.18 1.718h.077c.512-.795 1.794-2 3.871-2 2.564 0 4.487 1.693 4.487 5.384v7.437h-3.897V14.05c0-1.616-.564-2.718-1.974-2.718-1.077 0-1.718.744-1.974 1.461-.103.257-.153.615-.153.975V21h-3.897l-.002-8.539Z" fill="currentColor"></path></svg></a><a target="_blank" rel="noopener noreferrer" class="dark:text-solid-white hover:!text-solid-red dark:outline-solid-white outline-solid-black" aria-label="youtube" href="https://www.youtube.com/@knocklabs"><svg width="24" height="24" viewBox="0 0 90 90" fill="none" xmlns="http://www.w3.org/2000/svg" class="w-6 h-6"><path d="M 88.119 23.338 c -1.035 -3.872 -4.085 -6.922 -7.957 -7.957 C 73.144 13.5 45 13.5 45 13.5 s -28.144 0 -35.162 1.881 c -3.872 1.035 -6.922 4.085 -7.957 7.957 C 0 30.356 0 45 0 45 s 0 14.644 1.881 21.662 c 1.035 3.872 4.085 6.922 7.957 7.957 C 16.856 76.5 45 76.5 45 76.5 s 28.144 0 35.162 -1.881 c 3.872 -1.035 6.922 -4.085 7.957 -7.957 C 90 59.644 90 45 90 45 S 90 30.356 88.119 23.338 z M 36 58.5 v -27 L 59.382 45 L 36 58.5 z" fill="currentColor"></path></svg></a></div><div class="flex items-center justify-center gap-2 flex-wrap"><a href="/legal/terms-of-service" class="inline-block font-normal text-solid-black hover:!text-solid-red box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]"><span class="text-[11px] text-gray-600">Terms of use</span></span></a><span class="h-6 w-px mx-2 bg-gray-100"></span><a href="/legal/privacy-policy" class="inline-block font-normal text-solid-black hover:!text-solid-red box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]"><span class="text-[11px] text-gray-600">Privacy policy</span></span></a><span class="h-6 w-px mx-2 bg-gray-100"></span><a href="/blog/zero-downtime-postgres-upgrades#cookie" class="inline-block font-normal text-solid-black hover:!text-solid-red box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]"><span class="text-[11px] text-gray-600">Cookie preferences</span></span></a><span class="h-6 w-px mx-2 bg-gray-100"></span><a href="/llms.txt" class="inline-block font-normal text-solid-black hover:!text-solid-red box-border antialiased cursor-pointer inline-flex items-center transition-colors ease-in duration-200 font-medium"><span class="whitespace-nowrap xl:text-base text-sm leading-[22px] flex items-center gap-[10px]"><span class="text-[11px] text-gray-600">LLMs.txt</span></span></a></div></div></div></div></footer></main></main></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"relatedPosts":[{"title":"How we built Liquid autocomplete in Tiptap","date":"2026-07-23T12:00:00.000Z","slug":"how-we-built-liquid-autocomplete-in-tiptap","author":{"name":"Kyle McDonald","picture":"/assets/blog/authors/kyle.png","title":"Product Engineer"},"excerpt":"Learn how we rebuilt Liquid autocomplete for Knock's new Tiptap-based template editor, from scope-aware suggestions inside loops to keeping it working in documents that won't parse.","categories":["engineering"],"coverImage":"/assets/blog/preview/how-we-built-liquid-autocomplete-in-tiptap.png"},{"title":"Files over tools: how we built the Knock Agent using a virtual file system and bash","date":"2026-07-09T12:00:00.000Z","slug":"how-we-built-the-knock-agent-virtual-filesystem-and-bash","author":{"name":"Chris Bell","picture":"/assets/blog/authors/chris.jpeg","title":"Co-Founder + CTO"},"excerpt":"A look at how we architected the Knock Agent with bash, a virtual file system, and the management API.","categories":["engineering"],"coverImage":"/assets/blog/how-we-built-the-knock-agent/cover.png"}],"post":{"title":"Zero downtime Postgres upgrades","date":"2023-12-12T12:00:00.000Z","slug":"zero-downtime-postgres-upgrades","excerpt":"How we upgraded from Postgres 11.9 to 15.3 with zero downtime","author":{"name":"Brent Anderson","picture":"/assets/blog/authors/brent.jpeg","title":"Platform Engineer"},"content":"\n\n**👋 We’re Knock. We provide a set of simple APIs developers use to introduce\nnotifications into their products, without needing to build and\nmaintain a notification system in-house.**\n\n\u003cCallout emoji=\"✅\"\u003e\n \u003cstrong\u003eTl;dr:\u003c/strong\u003e We recently upgraded from Postgres 11.9 to 15.3 with \n zero downtime by using logical replication, a suite of support scripts, \n and tools in Elixir \u0026 Erlang’s BEAM virtual machine.\n \n This post will go into far too much detail explaining how we did it, \n and considerations you might need to make along the way if you try to do the same.\n\nIt is more of a manual than anything, and includes things we learned along the\nway that we wish we’d known up front.\n\n\u003c/Callout\u003e\n\nKnock relies on Postgres to power our notification workflow engine. From storing\nworkflow configurations and message templates, to [ingesting millions of logs](https://knock.app/blog/how-we-use-postgres-ltrees)\nand [enqueuing background jobs](https://getoban.pro/), Postgres sits at the heart of everything our systems do.\nOur Postgres databases running on AWS RDS Aurora have been consistently reliable,\nperformant, and extensible. This foundation to Knock’s service lets us support\nwith confidence every customer that joins our platform.\n\nUnlike SaaS software that can be constantly upgraded in the background with little notice,\nupgrading relational databases like Postgres generally requires at least a reboot of the database.\nIn the case of major version upgrades, the database often needs to shut down completely for several minutes\nin order to upgrade how data is stored and indexed on disk.\n\n**The more data you have, the longer the upgrade will take.**\n\nIn Knock’s case, we have been running Postgres 11.9 since we started the company.\nAlthough it has reliably served us at every step along the way,\n[Postgres 11.9 is being retired by Amazon’s RDS service on February 29, 2024](https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html#Release.Calendar).\nWithout taking action (i.e. arranging a long-term support contract with RDS),\nteams that use Postgres 11.9 on AWS RDS will be forcibly upgraded at that point,\nlikely resulting in forced downtime.\n\nNo amount of downtime - scheduled or otherwise - is acceptable for a service like Knock.\nOur customers rely on us to be online 24/7. Although no service can guarantee\nperfect uptime, responsible developer teams work to proactively address service\nissues before they happen.\n\nWe added this upgrade to our roadmap in June of this year, with the following constraints:\n\n1. Upgrade as many versions ahead as possible, skipping to the latest available version (at the time, Postgres 15.3 for Aurora).\n2. Any downtime beyond 60 seconds was completely unacceptable, and ideally we would have zero system downtime.\n3. The upgrade must happen well in advance of Amazon’s February deadline.\n4. Minimize customer impact (e.g. zero API error responses).\n5. Operationalize the process so that next time we need to upgrade the database, it is a well-established runbook.\n\nEach of our Postgres databases would need to run through this process, and going\nfrom 11.9 to 15.3 would comprise **four** major version upgrades. If doing an\nin-place upgrade for each major version would trigger downtime, doing **four**\nin a row was out of the question.\n\nIn order to meet our requirements, we knew we’d have to get creative.\n\n## Preparing for any Postgres upgrade\n\nMore than anything, teams seeking to upgrade Postgres in any way should focus on\n**de-risking** the upgrade process as much as possible:\n\n1. Make a list of the risks involved in making the migration. For example:\n\n - Unacceptably long downtime\n - Data loss\n - Changes in database performance for your application’s workload\n - Changes in vacuum frequency or behavior\n - Are there any replication slots that need to be migrated (this can be tricky - [see below](#a-note-about-moving-replication-slots))\n\n2. Figure out which risks are the most critical to the project, and which ones\n might be the easiest to explore/rule out/fix in advance.\n\n Sort the list so the risks with the biggest impact yet easiest to address are at the top.\n\n3. As you develop solutions, consider your list of risks:\n - Are there solutions that rule out risks completely?\n - Which solutions spread out the risk over time? (So we can more gradually\n address each step of the migration without taking on too much risk at once.)\n4. As you work through the project, always revisit your list of risks,\n and keep it up to date as you learn new things - including discovering new risks!\n\n \u003cCallout emoji=\"✅\"\u003e\n **Incrementally and continually de-risk projects like this until you are\n confident in being able to deliver on your project goals.**\n \u003c/Callout\u003e\n\nTo plan out our upgrade, we started with [Postgres’ release notes](https://www.postgresql.org/docs/release/)\nto get a sense of what was going to change between database versions.\nThis helped us identify more risks (e.g. changes in how Postgres’ vacuum works,\nrequirement to reindex the database when performing certain upgrades) while ruling out others.\n\nAs we moved through our planning process, we maintained this list of risks,\nadding new concerns and updating old ones as we collected more information.\nWhile working through the upgrade, we systematically addressed each concern\nuntil we were confident we could deliver on our project goals without\nrisking our reliability.\n\n### A word about monitoring \u0026 metrics\n\nHaving thorough instrumentation (thanks DataDog!) to monitor the health of your\nsystem and database makes it possible to monitor each step of the migration.\n\nA few key metrics to watch:\n\n- Max TXN ID to avoid [transaction wraparound](https://www.crunchydata.com/blog/managing-transaction-id-wraparound-in-postgresql) - if this gets too high, your database can shut down and go into emergency maintenance mode\n- DB CPU Utilization\n- Waiting sessions on your writer instance\n- Query latency\n- API response latencies for your application\n\nAt Knock, we monitor all of these metrics as well as some that are unique to our application,\nlike the time it takes to turn an API request into a notification.\n\n**Without timely metrics, you’re flying blind.**\n\n## Options for upgrading Postgres\n\nPart of our research process included looking for [prior examples](https://retool.com/blog/how-we-upgraded-postgresql-database)\nof database migrations and [how the Postgres docs](https://www.postgresql.org/docs/current/pgupgrade.html)\nrecommend performing an update. Here are a few strategies:\n\n### In-place upgrades (a non-starter for zero-downtime upgrades)\n\nThe most basic upgrade option for Postgres is an in-place upgrade.\nOn AWS RDS, this upgrade is executed from the AWS console. When performing an\nin-place upgrade, AWS will shut down the database, run upgrade scripts, and then\nbring the system back online. Doing this often requires some preparation,\nincluding dropping Postgres replication slots,\nlike those used to synchronize with a data warehouse or other systems.\n\nThis in-place upgrade process can take anywhere from a few minutes to potentially\nhours or more - it entirely depends on how much data needs to be updated\nbetween Postgres versions.\n\nOften, the system is still not in a fully usable state when it comes online, and\nadministrators must run maintenance tasks like Postgres’ `VACUUM` command,\nor `REINDEX` to update indexes to support the new version’s format.\n\n**Because an in-place upgrade would require far more downtime than we wanted to\ntolerate, it was out of the question for us.**\n\nA similar approach to an in-place upgrade is to use `pg_dump` and `pg_restore`\nto transfer the contents of a database once it has shut down.\nThis [dump \u0026 restore](https://www.postgresql.org/docs/current/app-pgdump.html)\napproach would also not work for us due to the required downtime involved,\nmostly because you need to disconnect all applications from the old database in\norder to get a reliable database backup. Even then, for large databases, it can\ntake prohibitively long to dump and restore the database.\n\n### Replication-based upgrades\n\nThis approach relies on Postgres’ excellent replication primitives:\nthe `PUBLICATION` and the `SUBSCRIPTION`.\n\nIt works something like this:\n\n1. Spin up a new database on your target Postgres version\n2. Copy over settings, extensions, table configurations, users, etc.\n3. Set up a publication on the old database and a subscription to that publication on the new database\n4. Add your tables to the publication (there is a lot of nuance here - [more below](#choosing-tables-to-replicate))\n5. Once it's fully replicated, run tests to satisfy any remaining risks\n6. Once you are confident in the new database's configuration, point your application at the new database\n7. Tear down the old database\n\n**In the end, this is the option that we chose at Knock for a few reasons:**\n\n1. It gave us gradual steps we could take towards a migration instead of one big upgrade\n2. We could test the new database with real workloads and real data to avoid any regressions\n3. It gave us the most control over when and how to perform the upgrade:\n once the new database was fully ready, cutting over to the new database took just a few seconds\n\nAlthough that may sound straightforward, there are several points to consider in\nthis solution that will depend on your application \u0026 circumstances.\n\n**Configuring your source and destination databases**\n\nPublications and subscriptions depend on a few configuration parameters for\nsetting up replication slots (how the database keeps track of what needs to be\ncopied from the primary to the follower database).\n[The Postgres docs](https://www.postgresql.org/docs/16/logical-replication-config.html)\nhave plenty of detail on these parameters. These parameters will need to be\ntuned for your particular application. For simple applications, the only change\nnecessary is that `wal_level` should be set to `logical`.\n\nIf you already use replication slots (e.g. to manage a read replica,\ndatabase failover, or to keep a data warehouse in sync), then consider setting\n`max_replication_slots` and the other parameters according to the guidance in the docs.\n\n**Setting up basic replication**\n\n1. Start a new Postgres server on your target version of Postgres (in our case v15.3).\n2. Set up your desired databases, schemas, tables, partitions, users \u0026 passwords, and everything else.\n\n **The target database’s tables must have an identical structure to the\n source database, but these tables must be empty.**\n\n To get a snapshot of the database schema, run [`pg_dumpall`](https://www.postgresql.org/docs/current/app-pg-dumpall.html)\n on the old DB (pass the `--schema-only` and `--no-role-passwords` options to\n keep it focused), and then adapt that command for the new DB. You can then\n compare the generated SQL files to identify and fix discrepancies between the\n old and the new DB.\n\n It may be worth periodically comparing both databases to detect any drift,\n especially if you have schema migrations happening in the source database.\n Consider running migrations against both databases to keep them in sync.\n\n3. On the primary instance of the **old** database, run `CREATE PUBLICATION pg_upgrade_pub;`.\n\n \u003cCallout emoji=\"🚨\"\u003e\n\n Although you can tack on `FOR ALL TABLES` and that will set up the publication\n for every table, we found that for large databases, this can lead to performance problems.\n\n Instead, we found it worked much better to incrementally add one table\n at a time to the publication via `ALTER PUBLICATION pg_upgrade_pub ADD TABLE table_name`.\n More on this [below](#choosing-tables-to-replicate).\n\n \u003c/Callout\u003e\n\n4. On the primary instance of the **new** database, set up the new subscription pointing to that publication:\n\n ```sql\n -- Note the _sub suffix, you can call this whatever you like\n CREATE SUBSCRIPTION pg_upgrade_sub\n -- The connection string can be any standard Postgres connection string.\n -- More details here:\n -- https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING\n CONNECTION 'host=old-db.cloud.com dbname=your_app user=root password=\u003cpassword\u003e'\n -- The publication name MUST match the publication created on the old database\n PUBLICATION pg_upgrade_pub with (\n -- This subscription will not start syncing until you enable it,\n -- which can be helpful when getting started\n enabled = false,\n -- Replication slots track the subscription's progress.\n -- By default, you want Postgres to manage this.\n -- If you don't create a slot here, you will need to supply one yourself.\n create_slot = true,\n -- Generally you want Postgres to copy the contents of each table,\n -- however for very large tables you may not want this option.\n -- More details below.\n copy_data = true,\n -- This will halt the subscription if something unexpected happens.\n -- This is usually because of a unique constraint violation, or\n -- a mismatched schema (e.g. a missing or renamed column).\n -- We found it helpful to halt the subscription on error so we could\n -- fix the problem and then resume replication.\n -- Errors are logged to the database's logs.\n disable_on_error = true\n );\n ```\n\n At this point, you now have a replication pipeline\n from the old database to the new one.\n\n To enable the subscription:\n\n ```sql\n ALTER SUBSCRIPTION pg_upgrade_sub ENABLE;\n\n -- To check the status of the subscription...\n\n -- Watch out for subenabled - if it turns false,\n -- replication is stopped and potentially backing up on the primary!\n SELECT * FROM pg_subscription;\n\n -- More details on monitoring subscriptions using that table here:\n -- https://www.postgresql.org/docs/16/catalog-pg-subscription.html\n ```\n\n### Choosing tables to replicate\n\nThe next step in the process is to build a list of tables you’d like to replicate.\nYou will want to add tables one at a time, watching each table until all of them\nare fully replicated. Later in this post we will show you [how to monitor replication\nfor all the tables](#checking-a-tables-replication-status).\n\nGenerally, the tables will fall into three based on their disk size and the\nnumber of tuples stored in the database.\n\n1. Small enough to synchronize in a few minutes: These can be replicated by just\n adding them to the publication and refreshing the subscription\n2. Large, append-only tables: These can be synchronize by first replicating only\n future changes, and then separately backfilling old data from a backup or snapshot\n3. Large, frequently updated tables: These are the hardest to synchronize, and\n will require some extra care\n\nFor us, \"small\" was any table using less than 50 GB of storage and 10 million tuples.\n\nAnything over those thresholds we considered \"large\".\n\n\u003cCallout emoji=\"❓\"\u003e\n**What is a tuple?**\n\nEach insert or update to a Postgres table is stored as a \"tuple\". If a table has\n3 inserts followed by 2 updates, the table would have 5 tuples. Tuples are used\nby Postgres’ concurrency mechanism ([more in the docs](https://www.postgresql.org/docs/16/mvcc-intro.html)).\nPostgres’ `VACUUM` procedure cleans up old tuples that are no longer needed.\n\nWhen we replicate a table, we replicate all of the tuples that make up the\ntables contents - inserts and updates. A table with a few rows but many tuples\nthat haven’t been cleaned up will take longer to replicate than a similar table\nwith fewer tuples.\n\n\u003c/Callout\u003e\n\nThe following query can help determine the size of a database table in terms of\ndisk space and tuple counts:\n\n```sql\nSELECT\nrelname AS tablename,\nn_live_tup + n_dead_tup + n_mod_since_analyze as total_tuple_count,\npg_size_pretty(pg_total_relation_size(quote_ident(relname))) AS simple_size,\npg_relation_size(quote_ident(relname)) as size_in_bytes\nFROM pg_stat_user_tables;\n```\n\nOne way to prepare your source database for replication is to `VACUUM` your tables,\nwhich should help the source database reduce the number of tuples it needs to copy\nto the target database. This can help reduce the amount of time it takes to replicate a table.\n\nBefore using `VACUUM`, consult the [Postgres docs](https://www.postgresql.org/docs/current/sql-vacuum.html).\n\n\u003cCallout emoji=\"🤔\"\u003e\n **Why does table size matter?**\n\nThe time it takes to synchronize a table is directly correlated to its size on disk\nand the number of tuples it contains. The larger the table, the longer it takes to replicate.\nThis is because Postgres needs to copy the entire table over to the new database,\nand then apply any changes that happen after the initial copy.\n\nThe problem with long synchronization time is that it can prevent your primary\nPostgres instance from performing `VACUUM` operations, which can lead to degraded\nperformance over time. Left unchecked, it can even lead to transaction wraparound\nand a forced shutdown of the database.\n\nFor these reasons, we added tables one at a time to replication, used different\nstrategies based on the size \u0026 write patterns of each table, and closely monitored\nthe system’s performance to ensure we didn’t degrade our service.\n\nIf migrating a table becomes problematic, you can remove a table from replication\nat any time, and then re-add it later (although you will need to truncate the\ntarget table and start from scratch).\n\n\u003c/Callout\u003e\n \n### How to replicate \"small\" tables\n\nTo migrate small tables, you just add it to the publication and then refresh the subscription:\n\n```sql\n-- On the old database\n\nALTER PUBLICATION pg_upgrade_pub ADD TABLE my_table_name;\n\n-- ON the new database\n\nALTER SUBSCRIPTION pg_upgrade_sub REFRESH PUBLICATION;\n```\n\nPostgres will handle copying the table over, getting it synchronized, and\napplying any further operations to the table. For very small tables,\nsynchronization can happen in less than a second.\n\n### Large, append-only tables\n\nTables that are too large but generally append-only, with no updates (or, if\nupdates are _always_ on rows that are recent, like within the past week),\nthen you can set up a separate `PUBLICATION` and `SUBSCRIPTION` following the\nsame steps as above, but setting the `copy_data` option on the subscription to\nfalse. Suffix the name of the new publication and new subscription with `_nocopy`\nto make it distinct.\n\nWhen you are ready to migrate these large, append-only tables, you can add them\nto this `nocopy` publication, and refresh the subscription on the target using\nthe `copy_data = false` option:\n\n```sql\n-- On the old database\n\nALTER PUBLICATION pg_upgrade_pub_nocopy ADD TABLE my_append_only_table_name;\n\n-- On the new database\n\nALTER SUBSCRIPTION pg_upgrade_sub REFRESH PUBLICATION WITH ( copy_data = false );\n\n```\n\nWe found this approach worked really well for our partitioned tables that stored\nvarious types of logs for our customers.\nWe did not need to migrate the root of a partitioned table, we only migrated the\nunderlying tables, and that seemed to work pretty well.\n\nOnce the subscription is running, you should start seeing logs appear on the\ntarget databases table:\n\n```sql\nSELECT COUNT(*) FROM my_append_only_table_name; -- Returns more than zero\n```\n\nFrom here, you can backfill any records older than those now visible in the\ndatabase using whatever means you like (e.g. `pg_dump`).\n\nHere is how we did it on AWS RDS Aurora:\n\n1. Take a snapshot of your production database in the AWS Console\n2. Restore that snapshot into a new database instance (the snapshot DB)\n3. Rename the table(s) on the snapshot DB that you want to replicate by adding\n a suffix like `_snapshot`. This prevents us having two replication pipelines\n feeding into the same table on the target database.\n4. Create the same table(s) on the target database with the same schema as the\n snapshot database. Use the same suffix as above.\n5. Create a publication on the snapshot database and a subscription on the target\n database to replicate these snapshot table(s) from the snapshot database to the\n target database\n6. Enable the subscription and monitor its progress\n7. Once the subscription is caught up, you can merge the tables together using\n `INSERT...ON CONFLICT`:\n\n ```sql\n INSERT INTO my_append_only_table_name\n SELECT * FROM my_append_only_table_name_snapshot\n ON CONFLICT (id) DO NOTHING;\n ```\n\n\u003cImageModal\n src=\"/assets/blog/zero-downtime-postgres-upgrades/snapshot-backfill.png\"\n alt=\"Diagram showing how to backfill data from a snapshot\"\n\u003e\n \u003cimg\n src=\"/assets/blog/zero-downtime-postgres-upgrades/snapshot-backfill.png\"\n alt=\"Diagram showing how to backfill data from a snapshot\"\n className=\"rounded-md mx-auto border border-gray-200\"\n /\u003e\n\u003c/ImageModal\u003e\n\nFor very large tables, this can still take several days, but because it’s all\nin the background it shouldn’t affect your production environment.\n\nOnce the tables are fully merged, compare them to ensure a consistent row count\n(more on that later). Once you are confident the tables are identical,\ndrop the snapshot table on the target DB, drop the subscription to the snapshot DB,\nand terminate the snapshot database instance.\n\n### Large tables with many updates over most of the rows\n\nThese are the hard tables. Because they have so much data in them, they can\ntake a long time to replicate, which can affect system performance on the\nsource database if it prevents `AUTOVUACUUM` from running. Because they have\nso many updates, we can’t treat it as an append-only table.\n\nA few points to consider:\n\n1. Is there any housekeeping you can do to reduce the table’s size?\n2. Have you vacuumed the table recently?\n3. Can you partition the table into smaller pieces?\n4. Do rows stop receiving updates after a reliable time frame (e.g. 1 week?) -\n this could be used to treat the table as an append-only table, and then after\n that time frame has elapsed you can backfill old rows from a snapshot.\n\nIf your source database is not on PG 15 or greater, your options are limited.\nFollow the steps in the \"small tables\" section. Rely on the monitoring you have\nin place (you do have monitoring, right?) to ensure replication doesn’t degrade\nyour service. If needed, you can rollback by removing the table from the\npublication, and refreshing the subscription ([See below](#aborting-the-replication-of-one-table)).\n\nIf the table is still too big, try to start replication during low traffic times to\nreduce load and write activity. This will hopefully minimize the impact on your\nsystem.\n\n### Large tables coming from PG 15 or greater\n\nIf your source database is on PG 15 or greater, you may be able to split up replication\nacross multiple publications (similar to partitioning or sharding). You can then\nmigrate the table in smaller chunks, at the expense of using more replication\nslots. [The Postgres docs](https://www.postgresql.org/docs/16/logical-replication-config.html)\nhave more information on setting these parameters.\n\n\u003cCallout emoji=\"🤞\"\u003e\n Because we migrated from 11.9 to 15.3, we did not have this option available\n to us. As such, we have not tested this approach. Even so, as we considered\n our options we noticed that this approach might be possible. If you try it\n out, let us know, we’d love to hear how it goes!\n\u003c/Callout\u003e\n\nThe goal is to have enough publications to split your largest table into manageable\npieces (for us, this was about 100 GB of non-index data stored). We’ll assume we\nare splitting across three partitions in this example. The trick is adding a `WHERE`\nclause that splits up the rows handled by each subscription:\n\n```sql\n-- On the source database\n\n-- For three partitions...\nCREATE PUBLICATION pg_upgrade_pub_0;\nCREATE PUBLICATION pg_upgrade_pub_1;\nCREATE PUBLICATION pg_upgrade_pub_2;\n\nALTER PUBLICATION pg_upgrade_pub_0 ADD TABLE big_table\n -- id must be the primary key.\n -- Use hashint4 for int IDs, hashint8 for bigint IDs\n -- Use hashtext(id::text) for UUID or other key types\n -- If you have a composite PK, concatenate the columns together before hashing as text\n -- Postgres' hash functions return positive \u0026 negative numbers - we abs() the result to make it positive\n -- % 3 is used to pick which of three partitions. Adjust the integer for the number of partitions you will create.\n -- = 0 assigns rows to the first partition (zero-indexed, so we will finish with partitions 0, 1, and 2)\n WHERE abs(hashint4(id)) % 3 = 0;\n\n-- Repeat the above ALTER statement for each publication, adjust the where clause accordingly.\n```\n\nOn the destination database, create a subscription for each partition.\n\nYou only want to migrate one slice of each table at a time. Generally, you will\nfollow the same instructions as adding a \"small\" table, but with the extra `WHERE`\nclause added when setting up the table for each publication.\n\nIn this way, you can slice up large tables into smaller, more workable pieces.\n\nConsider only using this approach if having too many replication slots is a\nproblem: you can still add \"small\" tables using this approach,\njust add the table to the `_0` publication without a `WHERE` clause.\nThis can help reduce the number of replication slots required when migrating.\n\n### Checking a table’s replication status\n\nWhen a table is added to a subscription, it moves through five distinct states\n(visible on the target database under the system table `pg_subscription_rel`\nin the `srsubstate` column):\n\n1. Initializing the table’s subscription (State code `i` )\n2. Copying the table’s contents in one efficient operation (State code `d`)\n\n \u003cCallout emoji=\"🚨\"\u003e\n\n This step requires keeping old Postgres transaction IDs around, which\n prevents vacuum from running effectively and can lead to system performance\n issues and (if left running long enough) even Postgres transaction ID\n wraparound which can halt the system.\n\n This is the step that requires replicating only one table at a time.\n\n \u003c/Callout\u003e\n\n3. Copy finished, waiting for final sync (State code `f`)\n4. Finalizing initial sync (State code `s`)\n5. Ready and running under normal replication (State code `r`)\n\nIn order to prevent the issues found in step 2 above, we found it was necessary\nto add one table at a time to replication, and to closely watch the system’s\nperformance. The worst-case scenario (transaction wraparound) must be avoided.\n\n**If you get anywhere close to wraparound, it is better to [abort the migration](#aborting-the-replication-of-one-table)\nand break it up into smaller pieces.**\n\nIf we had created our publication using the `FOR ALL TABLES` option, Postgres\nwould have started to sync our very large source database all at once,\npreventing automatic `VACUUM` operations from completing necessary maintenance.\nWe found this to gradually degrade database performance over time,\nleading to increased risk to system stability.\n\nAdding one table at a time has the added advantage of allowing teams to\nincrementally migrate each table. Replication does come with CPU and other costs\nfor the source and destination databases. By adding one table at a time,\nadministrators can control how replication affects the running system.\n\n### Aborting the replication of one table\n\nIf you need to halt the replication of a table, you reverse the instructions for\nadding the table in the first place:\n\n```sql\n-- On the old database\n\nALTER PUBLICATION pg_upgrade_pub_nocopy DROP TABLE my_append_only_table_name;\n\n-- ON the new database\n\nALTER SUBSCRIPTION pg_upgrade_sub REFRESH PUBLICATION;\n```\n\nIn an emergency, you can also drop the publications and subscriptions entirely,\nand start the process over. Postgres will clean up any replication slots that\nwere created as part of the publication and subscription, which should relieve\nany pressure on the source database.\n\n\u003cCallout emoji=\"🚨\"\u003e\n Be advised that if you just disable the subscription without removing the table\n from the publication and refreshing the subscription, the source database\n will continue to hold onto old transaction IDs, which can lead to transaction\n wraparound and a forced shutdown of the database.\n\nJust disabling the subscription will not resolve any replication-related\nperformance problems.\n\n\u003c/Callout\u003e\n\n### A note about moving replication slots\n\nReplication slots in Postgres store a log of database activity that can be\nconsumed on another database or in another application. Postgres tracks slot\nprogress using a Log Sequence Number (LSN). LSNs are unique to the primary\nPostgres database. This means that if you have a replication slot on your\ndatabase (e.g. to copy changes to a data warehouse or as part of your own\napplication), you will not be able to copy the replication slot's LSN over from\nthe old database to the new database.\n\nYou will need to consult the documentation of the application consuming the\nreplication slot to decide how to best migrate (e.g. for data warehousing tools,\nthey may have a way to merge duplicated information between both databases).\nIf you’re using replication slots as part of your own application, you already\nknow that you’re on your own to roll your own solution. Having some idempotence\nmechanism to deduplicate transactions from the old and the new database will\ndefinitely be helpful.\n\n## Finalizing the migration\n\nOnce you have added all of your tables to publications, and the subscriptions\nhave caught up on everything, you need to now verify that the tables match.\n\nUnfortunately, eventual consistency (the lag between a write being applied to\nthe old database and it showing up on the new database) will prevent both\ndatabases from being perfect matches at the same time, you can still count table\nrows to make sure you’re close enough to know it’s working.\n\nAt Knock, we wrote a script that iterated through each table and asked both\ndatabases to count the total number of rows in each table on the old and new\ndatabase, and compared the results. For tables with an `inserted_at` column, we\nfiltered to rows older than 10 seconds. This interval is more than enough to\nprove that the tables match, with the assumption that the remaining 10 seconds\nwill replicate across in short order.\n\nYou may need to come up with a strategy that fits your application’s needs. We\nfelt that as long as row counts were accurate within a few seconds, we could\notherwise assume that Postgres replication was reliable.\n\nIn a few instances, we also spot-checked the contents of a few tables to ensure\nthey matched to confirm this assumption. Collecting a random sample of rows from\ntables and comparing them between the old and the new database can help verify\nthat the tables are identical.\n\n### Application-level changes\n\nParallel to all of this database work, you may need to change your application\nto connect to both databases. When you are finally ready to cut over,\nyou need a strategy to shift traffic to your new database.\n\nWhen the final cutover happens, you could change your application’s configuration\nto point to the new database, and then reboot your app. This is simple,\nstraightforward, and is precisely how we migrated one of our\nlower-traffic databases.\n\nFor applications with lots of concurrent activity, you may need to get creative.\nWe wanted to avoid a situation with conflicting writes between the old and new\ndatabase. Such conflicts could have caused a service outage for us, requiring\nmanually reconciling database state.\n\nAt Knock, we configured our application to connect to both databases.\nWhen we were ready to execute the cutover, we ran a script that did the following:\n\n1. Tell all instances of our application to send new queries to the new database\n2. All currently running database queries had 500 ms to complete before being forcefully cancelled\n3. For the first second after flipping the flag, our application artificially paused\n any new database requests for one second. This allowed pending transactions\n to replicate to the new database so that new queries wouldn’t have stale reads\n\n 500 ms is far higher than most of our db queries, and we saw zero errors due to forced disconnections\n\n4. After that first second, database activity returned to normal behavior, but pointing at the new database.\n5. In the middle of the cutover, we had some specialized database workloads that\n the script shut down and restarted in order to reconnect to the new database.\n\n### One more thing: sequences\n\nOne thing that replication doesn’t synchronize is any Postgres sequence.\nSequences are monotonically increasing integers that are guaranteed to never\nduplicate. Unfortunately, they are not incremented on the new database as\nsequence values are used up on the old database.\n\nFortunately, this is pretty easy to control for. Part of our cutover procedure\nwas to run a script right before flipping our feature flag that did the following:\n\n1. Connect to both databases\n2. Get the next value of all of the sequences in the database using `SELECT nextval('sequence_name')`\n3. Set that value in the new database using `SELECT setval('sequence_name', value::int4 + 100000)`\n to advance the sequence and offer a little bit of buffer (in this case, 100k\n rows can be added between setting this value on the new database and cutting over).\n This will introduce a gap in the sequence, but that’s generally not a problem.\n For us, our sequences are bigints. 100k values skipped in the sequence is a\n rounding error off of 0% used up sequence values in that case.\n\n You will want to tune how big of a gap you introduce so you don’t use too\n much of your sequence’s usable space. If you only expect the sequence to\n use a few hundred values during your cutover window,\n then maybe advance it only by 5000.\n\n## Final checklist before cutting over\n\nHere are some of the things we considered before executing our final cutover:\n\n1. Do the rows on all the tables match as expected?\n2. Are all the subscriptions enabled and running without error?\n3. Do the schemas match? Can you freeze any new schema migrations from being\n released to reduce the risk of something changing while you’re migrating?\n4. Is your new database properly sized for your workloads?\n5. Do you have to add any read replicas so the database cluster topology is the\n same between the old and the new database?\n6. Have you reindexed and performed basic VACUUM maintenance on the new database\n to ensure it’s fresh and ready for production traffic?\n7. Have you double checked Postgres’ release notes for anything that might cause\n a regression in your app?\n8. Have you run automated and manual tests against a staging database on the new\n version to verify system performance?\n9. Have you run load tests of your most demanding queries using `pg_bench`\n against your new version to verify performance?\n10. If there’s one thing that you can de-risk still, what is it?\n11. Do practice runs in a staging or test environment until you have fully\n exercised the cutover process multiple times. Dry runs like this will help\n reveal gaps in your plan before you go to production.\n12. Right before cutover, take a database backup - just in case.\n\n## Cutting over\n\nAt Knock, we took a few weeks replicating tables one at a time. We generally did\nthis after business hours and during our lowest traffic time frames. We practiced\ncutover in our staging environment multiple times, ironing out the process until\nit just worked without much operator involvement.\n\nOnce we had a replica running PG 15 and had the application code in place to\ncut over from the old to the new database, we ran one final set of checks and\nflipped the flag.\n\nAfter months of preparation, the actual cutover was uneventful: our\napplication cut over within a few seconds, we had a brief blip of (intentional)\nlatency as queries waited to allow for replication, and our application\ncontinued running without skipping a beat. Reading this paragraph took longer\nthan the cutover itself.\n\nFrom there, we rolled back the application changes we introduced, permanently\npointed everything at the new database, removed the subscriptions on the new\ndatabase, and tore down the old database. We had successfully jumped from\nPostgres 11.9 to 15.3 with zero downtime!\n\n## Conclusion\n\nAlthough jumping four major versions of Postgres in one leap is a painstaking\nprocess, it can be done, and in many ways it’s safer than scheduled downtime:\nit can be practiced, tested, and reworked multiple times before performing the\nactual cutover. At any point in the process, we could have dropped the\npublications from the old database and started over without degrading our service.\n\nModern customers expect 100% availability. While that is not technically possible,\nzero downtime migrations make it easier to keep systems running smoothly without\nmajor service interruptions.","ogImage":{"url":"/assets/blog/preview/zero-downtime-postgres-upgrades-og.png"},"coverImage":"/assets/blog/preview/zero-downtime-postgres-upgrades-og.png","readingTime":{"text":"29 min read","minutes":28.61,"time":1716600,"words":5722},"categories":["engineering"]},"mdxSource":{"compiledSource":"\"use strict\";\nconst {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n a: \"a\",\n code: \"code\",\n em: \"em\",\n figure: \"figure\",\n h2: \"h2\",\n h3: \"h3\",\n li: \"li\",\n ol: \"ol\",\n p: \"p\",\n pre: \"pre\",\n span: \"span\",\n strong: \"strong\",\n ul: \"ul\",\n ..._provideComponents(),\n ...props.components\n }, {Callout, ImageModal} = _components;\n if (!Callout) _missingMdxReference(\"Callout\", true);\n if (!ImageModal) _missingMdxReference(\"ImageModal\", true);\n return _jsxs(_Fragment, {\n children: [_jsx(_components.p, {\n children: _jsx(_components.strong, {\n children: \"👋 We’re Knock. We provide a set of simple APIs developers use to introduce\\nnotifications into their products, without needing to build and\\nmaintain a notification system in-house.\"\n })\n }), \"\\n\", _jsxs(Callout, {\n emoji: \"✅\",\n children: [_jsxs(_components.p, {\n children: [_jsx(\"strong\", {\n children: \"Tl;dr:\"\n }), \" We recently upgraded from Postgres 11.9 to 15.3 with\\nzero downtime by using logical replication, a suite of support scripts,\\nand tools in Elixir \u0026 Erlang’s BEAM virtual machine.\"]\n }), _jsx(_components.p, {\n children: \"This post will go into far too much detail explaining how we did it,\\nand considerations you might need to make along the way if you try to do the same.\"\n }), _jsx(_components.p, {\n children: \"It is more of a manual than anything, and includes things we learned along the\\nway that we wish we’d known up front.\"\n })]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"Knock relies on Postgres to power our notification workflow engine. From storing\\nworkflow configurations and message templates, to \", _jsx(_components.a, {\n href: \"https://knock.app/blog/how-we-use-postgres-ltrees\",\n children: \"ingesting millions of logs\"\n }), \"\\nand \", _jsx(_components.a, {\n href: \"https://getoban.pro/\",\n children: \"enqueuing background jobs\"\n }), \", Postgres sits at the heart of everything our systems do.\\nOur Postgres databases running on AWS RDS Aurora have been consistently reliable,\\nperformant, and extensible. This foundation to Knock’s service lets us support\\nwith confidence every customer that joins our platform.\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"Unlike SaaS software that can be constantly upgraded in the background with little notice,\\nupgrading relational databases like Postgres generally requires at least a reboot of the database.\\nIn the case of major version upgrades, the database often needs to shut down completely for several minutes\\nin order to upgrade how data is stored and indexed on disk.\"\n }), \"\\n\", _jsx(_components.p, {\n children: _jsx(_components.strong, {\n children: \"The more data you have, the longer the upgrade will take.\"\n })\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"In Knock’s case, we have been running Postgres 11.9 since we started the company.\\nAlthough it has reliably served us at every step along the way,\\n\", _jsx(_components.a, {\n href: \"https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html#Release.Calendar\",\n children: \"Postgres 11.9 is being retired by Amazon’s RDS service on February 29, 2024\"\n }), \".\\nWithout taking action (i.e. arranging a long-term support contract with RDS),\\nteams that use Postgres 11.9 on AWS RDS will be forcibly upgraded at that point,\\nlikely resulting in forced downtime.\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"No amount of downtime - scheduled or otherwise - is acceptable for a service like Knock.\\nOur customers rely on us to be online 24/7. Although no service can guarantee\\nperfect uptime, responsible developer teams work to proactively address service\\nissues before they happen.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"We added this upgrade to our roadmap in June of this year, with the following constraints:\"\n }), \"\\n\", _jsxs(_components.ol, {\n children: [\"\\n\", _jsx(_components.li, {\n children: \"Upgrade as many versions ahead as possible, skipping to the latest available version (at the time, Postgres 15.3 for Aurora).\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Any downtime beyond 60 seconds was completely unacceptable, and ideally we would have zero system downtime.\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"The upgrade must happen well in advance of Amazon’s February deadline.\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Minimize customer impact (e.g. zero API error responses).\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Operationalize the process so that next time we need to upgrade the database, it is a well-established runbook.\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"Each of our Postgres databases would need to run through this process, and going\\nfrom 11.9 to 15.3 would comprise \", _jsx(_components.strong, {\n children: \"four\"\n }), \" major version upgrades. If doing an\\nin-place upgrade for each major version would trigger downtime, doing \", _jsx(_components.strong, {\n children: \"four\"\n }), \"\\nin a row was out of the question.\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"In order to meet our requirements, we knew we’d have to get creative.\"\n }), \"\\n\", _jsxs(_components.h2, {\n id: \"preparing-for-any-postgres-upgrade\",\n children: [_jsx(_components.a, {\n href: \"#preparing-for-any-postgres-upgrade\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Preparing for any Postgres upgrade\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"More than anything, teams seeking to upgrade Postgres in any way should focus on\\n\", _jsx(_components.strong, {\n children: \"de-risking\"\n }), \" the upgrade process as much as possible:\"]\n }), \"\\n\", _jsxs(_components.ol, {\n children: [\"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"Make a list of the risks involved in making the migration. For example:\"\n }), \"\\n\", _jsxs(_components.ul, {\n children: [\"\\n\", _jsx(_components.li, {\n children: \"Unacceptably long downtime\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Data loss\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Changes in database performance for your application’s workload\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Changes in vacuum frequency or behavior\"\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"Are there any replication slots that need to be migrated (this can be tricky - \", _jsx(_components.a, {\n href: \"#a-note-about-moving-replication-slots\",\n children: \"see below\"\n }), \")\"]\n }), \"\\n\"]\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"Figure out which risks are the most critical to the project, and which ones\\nmight be the easiest to explore/rule out/fix in advance.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"Sort the list so the risks with the biggest impact yet easiest to address are at the top.\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"As you develop solutions, consider your list of risks:\"\n }), \"\\n\", _jsxs(_components.ul, {\n children: [\"\\n\", _jsx(_components.li, {\n children: \"Are there solutions that rule out risks completely?\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Which solutions spread out the risk over time? (So we can more gradually\\naddress each step of the migration without taking on too much risk at once.)\"\n }), \"\\n\"]\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"As you work through the project, always revisit your list of risks,\\nand keep it up to date as you learn new things - including discovering new risks!\"\n }), \"\\n\", _jsx(Callout, {\n emoji: \"✅\",\n children: _jsx(_components.p, {\n children: _jsx(_components.strong, {\n children: \"Incrementally and continually de-risk projects like this until you are\\nconfident in being able to deliver on your project goals.\"\n })\n })\n }), \"\\n\"]\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"To plan out our upgrade, we started with \", _jsx(_components.a, {\n href: \"https://www.postgresql.org/docs/release/\",\n children: \"Postgres’ release notes\"\n }), \"\\nto get a sense of what was going to change between database versions.\\nThis helped us identify more risks (e.g. changes in how Postgres’ vacuum works,\\nrequirement to reindex the database when performing certain upgrades) while ruling out others.\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"As we moved through our planning process, we maintained this list of risks,\\nadding new concerns and updating old ones as we collected more information.\\nWhile working through the upgrade, we systematically addressed each concern\\nuntil we were confident we could deliver on our project goals without\\nrisking our reliability.\"\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"a-word-about-monitoring--metrics\",\n children: [_jsx(_components.a, {\n href: \"#a-word-about-monitoring--metrics\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"A word about monitoring \u0026 metrics\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"Having thorough instrumentation (thanks DataDog!) to monitor the health of your\\nsystem and database makes it possible to monitor each step of the migration.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"A few key metrics to watch:\"\n }), \"\\n\", _jsxs(_components.ul, {\n children: [\"\\n\", _jsxs(_components.li, {\n children: [\"Max TXN ID to avoid \", _jsx(_components.a, {\n href: \"https://www.crunchydata.com/blog/managing-transaction-id-wraparound-in-postgresql\",\n children: \"transaction wraparound\"\n }), \" - if this gets too high, your database can shut down and go into emergency maintenance mode\"]\n }), \"\\n\", _jsx(_components.li, {\n children: \"DB CPU Utilization\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Waiting sessions on your writer instance\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Query latency\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"API response latencies for your application\"\n }), \"\\n\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"At Knock, we monitor all of these metrics as well as some that are unique to our application,\\nlike the time it takes to turn an API request into a notification.\"\n }), \"\\n\", _jsx(_components.p, {\n children: _jsx(_components.strong, {\n children: \"Without timely metrics, you’re flying blind.\"\n })\n }), \"\\n\", _jsxs(_components.h2, {\n id: \"options-for-upgrading-postgres\",\n children: [_jsx(_components.a, {\n href: \"#options-for-upgrading-postgres\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Options for upgrading Postgres\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"Part of our research process included looking for \", _jsx(_components.a, {\n href: \"https://retool.com/blog/how-we-upgraded-postgresql-database\",\n children: \"prior examples\"\n }), \"\\nof database migrations and \", _jsx(_components.a, {\n href: \"https://www.postgresql.org/docs/current/pgupgrade.html\",\n children: \"how the Postgres docs\"\n }), \"\\nrecommend performing an update. Here are a few strategies:\"]\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"in-place-upgrades-a-non-starter-for-zero-downtime-upgrades\",\n children: [_jsx(_components.a, {\n href: \"#in-place-upgrades-a-non-starter-for-zero-downtime-upgrades\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"In-place upgrades (a non-starter for zero-downtime upgrades)\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"The most basic upgrade option for Postgres is an in-place upgrade.\\nOn AWS RDS, this upgrade is executed from the AWS console. When performing an\\nin-place upgrade, AWS will shut down the database, run upgrade scripts, and then\\nbring the system back online. Doing this often requires some preparation,\\nincluding dropping Postgres replication slots,\\nlike those used to synchronize with a data warehouse or other systems.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"This in-place upgrade process can take anywhere from a few minutes to potentially\\nhours or more - it entirely depends on how much data needs to be updated\\nbetween Postgres versions.\"\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"Often, the system is still not in a fully usable state when it comes online, and\\nadministrators must run maintenance tasks like Postgres’ \", _jsx(_components.code, {\n children: \"VACUUM\"\n }), \" command,\\nor \", _jsx(_components.code, {\n children: \"REINDEX\"\n }), \" to update indexes to support the new version’s format.\"]\n }), \"\\n\", _jsx(_components.p, {\n children: _jsx(_components.strong, {\n children: \"Because an in-place upgrade would require far more downtime than we wanted to\\ntolerate, it was out of the question for us.\"\n })\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"A similar approach to an in-place upgrade is to use \", _jsx(_components.code, {\n children: \"pg_dump\"\n }), \" and \", _jsx(_components.code, {\n children: \"pg_restore\"\n }), \"\\nto transfer the contents of a database once it has shut down.\\nThis \", _jsx(_components.a, {\n href: \"https://www.postgresql.org/docs/current/app-pgdump.html\",\n children: \"dump \u0026 restore\"\n }), \"\\napproach would also not work for us due to the required downtime involved,\\nmostly because you need to disconnect all applications from the old database in\\norder to get a reliable database backup. Even then, for large databases, it can\\ntake prohibitively long to dump and restore the database.\"]\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"replication-based-upgrades\",\n children: [_jsx(_components.a, {\n href: \"#replication-based-upgrades\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Replication-based upgrades\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"This approach relies on Postgres’ excellent replication primitives:\\nthe \", _jsx(_components.code, {\n children: \"PUBLICATION\"\n }), \" and the \", _jsx(_components.code, {\n children: \"SUBSCRIPTION\"\n }), \".\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"It works something like this:\"\n }), \"\\n\", _jsxs(_components.ol, {\n children: [\"\\n\", _jsx(_components.li, {\n children: \"Spin up a new database on your target Postgres version\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Copy over settings, extensions, table configurations, users, etc.\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Set up a publication on the old database and a subscription to that publication on the new database\"\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"Add your tables to the publication (there is a lot of nuance here - \", _jsx(_components.a, {\n href: \"#choosing-tables-to-replicate\",\n children: \"more below\"\n }), \")\"]\n }), \"\\n\", _jsx(_components.li, {\n children: \"Once it's fully replicated, run tests to satisfy any remaining risks\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Once you are confident in the new database's configuration, point your application at the new database\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Tear down the old database\"\n }), \"\\n\"]\n }), \"\\n\", _jsx(_components.p, {\n children: _jsx(_components.strong, {\n children: \"In the end, this is the option that we chose at Knock for a few reasons:\"\n })\n }), \"\\n\", _jsxs(_components.ol, {\n children: [\"\\n\", _jsx(_components.li, {\n children: \"It gave us gradual steps we could take towards a migration instead of one big upgrade\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"We could test the new database with real workloads and real data to avoid any regressions\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"It gave us the most control over when and how to perform the upgrade:\\nonce the new database was fully ready, cutting over to the new database took just a few seconds\"\n }), \"\\n\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"Although that may sound straightforward, there are several points to consider in\\nthis solution that will depend on your application \u0026 circumstances.\"\n }), \"\\n\", _jsx(_components.p, {\n children: _jsx(_components.strong, {\n children: \"Configuring your source and destination databases\"\n })\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"Publications and subscriptions depend on a few configuration parameters for\\nsetting up replication slots (how the database keeps track of what needs to be\\ncopied from the primary to the follower database).\\n\", _jsx(_components.a, {\n href: \"https://www.postgresql.org/docs/16/logical-replication-config.html\",\n children: \"The Postgres docs\"\n }), \"\\nhave plenty of detail on these parameters. These parameters will need to be\\ntuned for your particular application. For simple applications, the only change\\nnecessary is that \", _jsx(_components.code, {\n children: \"wal_level\"\n }), \" should be set to \", _jsx(_components.code, {\n children: \"logical\"\n }), \".\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"If you already use replication slots (e.g. to manage a read replica,\\ndatabase failover, or to keep a data warehouse in sync), then consider setting\\n\", _jsx(_components.code, {\n children: \"max_replication_slots\"\n }), \" and the other parameters according to the guidance in the docs.\"]\n }), \"\\n\", _jsx(_components.p, {\n children: _jsx(_components.strong, {\n children: \"Setting up basic replication\"\n })\n }), \"\\n\", _jsxs(_components.ol, {\n children: [\"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"Start a new Postgres server on your target version of Postgres (in our case v15.3).\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"Set up your desired databases, schemas, tables, partitions, users \u0026 passwords, and everything else.\"\n }), \"\\n\", _jsx(_components.p, {\n children: _jsx(_components.strong, {\n children: \"The target database’s tables must have an identical structure to the\\nsource database, but these tables must be empty.\"\n })\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"To get a snapshot of the database schema, run \", _jsx(_components.a, {\n href: \"https://www.postgresql.org/docs/current/app-pg-dumpall.html\",\n children: _jsx(_components.code, {\n children: \"pg_dumpall\"\n })\n }), \"\\non the old DB (pass the \", _jsx(_components.code, {\n children: \"--schema-only\"\n }), \" and \", _jsx(_components.code, {\n children: \"--no-role-passwords\"\n }), \" options to\\nkeep it focused), and then adapt that command for the new DB. You can then\\ncompare the generated SQL files to identify and fix discrepancies between the\\nold and the new DB.\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"It may be worth periodically comparing both databases to detect any drift,\\nespecially if you have schema migrations happening in the source database.\\nConsider running migrations against both databases to keep them in sync.\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsxs(_components.p, {\n children: [\"On the primary instance of the \", _jsx(_components.strong, {\n children: \"old\"\n }), \" database, run \", _jsx(_components.code, {\n children: \"CREATE PUBLICATION pg_upgrade_pub;\"\n }), \".\"]\n }), \"\\n\", _jsxs(Callout, {\n emoji: \"🚨\",\n children: [_jsxs(_components.p, {\n children: [\"Although you can tack on \", _jsx(_components.code, {\n children: \"FOR ALL TABLES\"\n }), \" and that will set up the publication\\nfor every table, we found that for large databases, this can lead to performance problems.\"]\n }), _jsxs(_components.p, {\n children: [\"Instead, we found it worked much better to incrementally add one table\\nat a time to the publication via \", _jsx(_components.code, {\n children: \"ALTER PUBLICATION pg_upgrade_pub ADD TABLE table_name\"\n }), \".\\nMore on this \", _jsx(_components.a, {\n href: \"#choosing-tables-to-replicate\",\n children: \"below\"\n }), \".\"]\n })]\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsxs(_components.p, {\n children: [\"On the primary instance of the \", _jsx(_components.strong, {\n children: \"new\"\n }), \" database, set up the new subscription pointing to that publication:\"]\n }), \"\\n\", _jsx(_components.figure, {\n \"data-rehype-pretty-code-figure\": \"\",\n children: _jsx(_components.pre, {\n style: {\n backgroundColor: \"#ffffff\",\n color: \"#24292eff\"\n },\n tabIndex: \"0\",\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n children: _jsxs(_components.code, {\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n style: {\n display: \"grid\"\n },\n children: [_jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- Note the _sub suffix, you can call this whatever you like\"\n })\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"CREATE\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" SUBSCRIPTION pg_upgrade_sub\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- The connection string can be any standard Postgres connection string.\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- More details here:\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING\"\n })\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"CONNECTION\"\n }), _jsx(_components.span, {\n style: {\n color: \"#22863A\"\n },\n children: \" 'host=old-db.cloud.com dbname=your_app user=root password=\u003cpassword\u003e'\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- The publication name MUST match the publication created on the old database\"\n })\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \"PUBLICATION pg_upgrade_pub \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"with\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" (\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- This subscription will not start syncing until you enable it,\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- which can be helpful when getting started\"\n })\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \" enabled\"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \" =\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" false,\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- Replication slots track the subscription's progress.\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- By default, you want Postgres to manage this.\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- If you don't create a slot here, you will need to supply one yourself.\"\n })\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" create_slot \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"=\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" true,\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- Generally you want Postgres to copy the contents of each table,\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- however for very large tables you may not want this option.\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- More details below.\"\n })\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" copy_data \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"=\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" true,\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- This will halt the subscription if something unexpected happens.\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- This is usually because of a unique constraint violation, or\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- a mismatched schema (e.g. a missing or renamed column).\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- We found it helpful to halt the subscription on error so we could\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- fix the problem and then resume replication.\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- Errors are logged to the database's logs.\"\n })\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" disable_on_error \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"=\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" true\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \");\"\n })\n })]\n })\n })\n }), \"\\n\", _jsx(_components.p, {\n children: \"At this point, you now have a replication pipeline\\nfrom the old database to the new one.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"To enable the subscription:\"\n }), \"\\n\", _jsx(_components.figure, {\n \"data-rehype-pretty-code-figure\": \"\",\n children: _jsx(_components.pre, {\n style: {\n backgroundColor: \"#ffffff\",\n color: \"#24292eff\"\n },\n tabIndex: \"0\",\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n children: _jsxs(_components.code, {\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n style: {\n display: \"grid\"\n },\n children: [_jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ALTER\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" SUBSCRIPTION pg_upgrade_sub \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ENABLE\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \";\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- To check the status of the subscription...\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- Watch out for subenabled - if it turns false,\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- replication is stopped and potentially backing up on the primary!\"\n })\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"SELECT\"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \" *\"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \" FROM\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" pg_subscription;\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- More details on monitoring subscriptions using that table here:\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- https://www.postgresql.org/docs/16/catalog-pg-subscription.html\"\n })\n })]\n })\n })\n }), \"\\n\"]\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"choosing-tables-to-replicate\",\n children: [_jsx(_components.a, {\n href: \"#choosing-tables-to-replicate\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Choosing tables to replicate\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"The next step in the process is to build a list of tables you’d like to replicate.\\nYou will want to add tables one at a time, watching each table until all of them\\nare fully replicated. Later in this post we will show you \", _jsx(_components.a, {\n href: \"#checking-a-tables-replication-status\",\n children: \"how to monitor replication\\nfor all the tables\"\n }), \".\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"Generally, the tables will fall into three based on their disk size and the\\nnumber of tuples stored in the database.\"\n }), \"\\n\", _jsxs(_components.ol, {\n children: [\"\\n\", _jsx(_components.li, {\n children: \"Small enough to synchronize in a few minutes: These can be replicated by just\\nadding them to the publication and refreshing the subscription\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Large, append-only tables: These can be synchronize by first replicating only\\nfuture changes, and then separately backfilling old data from a backup or snapshot\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Large, frequently updated tables: These are the hardest to synchronize, and\\nwill require some extra care\"\n }), \"\\n\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"For us, \\\"small\\\" was any table using less than 50 GB of storage and 10 million tuples.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"Anything over those thresholds we considered \\\"large\\\".\"\n }), \"\\n\", _jsxs(Callout, {\n emoji: \"❓\",\n children: [_jsx(_components.p, {\n children: _jsx(_components.strong, {\n children: \"What is a tuple?\"\n })\n }), _jsxs(_components.p, {\n children: [\"Each insert or update to a Postgres table is stored as a \\\"tuple\\\". If a table has\\n3 inserts followed by 2 updates, the table would have 5 tuples. Tuples are used\\nby Postgres’ concurrency mechanism (\", _jsx(_components.a, {\n href: \"https://www.postgresql.org/docs/16/mvcc-intro.html\",\n children: \"more in the docs\"\n }), \").\\nPostgres’ \", _jsx(_components.code, {\n children: \"VACUUM\"\n }), \" procedure cleans up old tuples that are no longer needed.\"]\n }), _jsx(_components.p, {\n children: \"When we replicate a table, we replicate all of the tuples that make up the\\ntables contents - inserts and updates. A table with a few rows but many tuples\\nthat haven’t been cleaned up will take longer to replicate than a similar table\\nwith fewer tuples.\"\n })]\n }), \"\\n\", _jsx(_components.p, {\n children: \"The following query can help determine the size of a database table in terms of\\ndisk space and tuple counts:\"\n }), \"\\n\", _jsx(_components.figure, {\n \"data-rehype-pretty-code-figure\": \"\",\n children: _jsx(_components.pre, {\n style: {\n backgroundColor: \"#ffffff\",\n color: \"#24292eff\"\n },\n tabIndex: \"0\",\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n children: _jsxs(_components.code, {\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n style: {\n display: \"grid\"\n },\n children: [_jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"SELECT\"\n })\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \"relname \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"AS\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" tablename,\"\n })]\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \"n_live_tup \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"+\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" n_dead_tup \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"+\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" n_mod_since_analyze \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"as\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" total_tuple_count,\"\n })]\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \"pg_size_pretty(pg_total_relation_size(quote_ident(relname))) \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"AS\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" simple_size,\"\n })]\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \"pg_relation_size(quote_ident(relname)) \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"as\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" size_in_bytes\"\n })]\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"FROM\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" pg_stat_user_tables;\"\n })]\n })]\n })\n })\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"One way to prepare your source database for replication is to \", _jsx(_components.code, {\n children: \"VACUUM\"\n }), \" your tables,\\nwhich should help the source database reduce the number of tuples it needs to copy\\nto the target database. This can help reduce the amount of time it takes to replicate a table.\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"Before using \", _jsx(_components.code, {\n children: \"VACUUM\"\n }), \", consult the \", _jsx(_components.a, {\n href: \"https://www.postgresql.org/docs/current/sql-vacuum.html\",\n children: \"Postgres docs\"\n }), \".\"]\n }), \"\\n\", _jsxs(Callout, {\n emoji: \"🤔\",\n children: [_jsx(_components.p, {\n children: _jsx(_components.strong, {\n children: \"Why does table size matter?\"\n })\n }), _jsx(_components.p, {\n children: \"The time it takes to synchronize a table is directly correlated to its size on disk\\nand the number of tuples it contains. The larger the table, the longer it takes to replicate.\\nThis is because Postgres needs to copy the entire table over to the new database,\\nand then apply any changes that happen after the initial copy.\"\n }), _jsxs(_components.p, {\n children: [\"The problem with long synchronization time is that it can prevent your primary\\nPostgres instance from performing \", _jsx(_components.code, {\n children: \"VACUUM\"\n }), \" operations, which can lead to degraded\\nperformance over time. Left unchecked, it can even lead to transaction wraparound\\nand a forced shutdown of the database.\"]\n }), _jsx(_components.p, {\n children: \"For these reasons, we added tables one at a time to replication, used different\\nstrategies based on the size \u0026 write patterns of each table, and closely monitored\\nthe system’s performance to ensure we didn’t degrade our service.\"\n }), _jsx(_components.p, {\n children: \"If migrating a table becomes problematic, you can remove a table from replication\\nat any time, and then re-add it later (although you will need to truncate the\\ntarget table and start from scratch).\"\n })]\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"how-to-replicate-small-tables\",\n children: [_jsx(_components.a, {\n href: \"#how-to-replicate-small-tables\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"How to replicate \\\"small\\\" tables\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"To migrate small tables, you just add it to the publication and then refresh the subscription:\"\n }), \"\\n\", _jsx(_components.figure, {\n \"data-rehype-pretty-code-figure\": \"\",\n children: _jsx(_components.pre, {\n style: {\n backgroundColor: \"#ffffff\",\n color: \"#24292eff\"\n },\n tabIndex: \"0\",\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n children: _jsxs(_components.code, {\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n style: {\n display: \"grid\"\n },\n children: [_jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- On the old database\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ALTER\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" PUBLICATION pg_upgrade_pub \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ADD\"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \" TABLE\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" my_table_name;\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- ON the new database\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ALTER\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" SUBSCRIPTION pg_upgrade_sub REFRESH PUBLICATION;\"\n })]\n })]\n })\n })\n }), \"\\n\", _jsx(_components.p, {\n children: \"Postgres will handle copying the table over, getting it synchronized, and\\napplying any further operations to the table. For very small tables,\\nsynchronization can happen in less than a second.\"\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"large-append-only-tables\",\n children: [_jsx(_components.a, {\n href: \"#large-append-only-tables\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Large, append-only tables\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"Tables that are too large but generally append-only, with no updates (or, if\\nupdates are \", _jsx(_components.em, {\n children: \"always\"\n }), \" on rows that are recent, like within the past week),\\nthen you can set up a separate \", _jsx(_components.code, {\n children: \"PUBLICATION\"\n }), \" and \", _jsx(_components.code, {\n children: \"SUBSCRIPTION\"\n }), \" following the\\nsame steps as above, but setting the \", _jsx(_components.code, {\n children: \"copy_data\"\n }), \" option on the subscription to\\nfalse. Suffix the name of the new publication and new subscription with \", _jsx(_components.code, {\n children: \"_nocopy\"\n }), \"\\nto make it distinct.\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"When you are ready to migrate these large, append-only tables, you can add them\\nto this \", _jsx(_components.code, {\n children: \"nocopy\"\n }), \" publication, and refresh the subscription on the target using\\nthe \", _jsx(_components.code, {\n children: \"copy_data = false\"\n }), \" option:\"]\n }), \"\\n\", _jsx(_components.figure, {\n \"data-rehype-pretty-code-figure\": \"\",\n children: _jsx(_components.pre, {\n style: {\n backgroundColor: \"#ffffff\",\n color: \"#24292eff\"\n },\n tabIndex: \"0\",\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n children: _jsxs(_components.code, {\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n style: {\n display: \"grid\"\n },\n children: [_jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- On the old database\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ALTER\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" PUBLICATION pg_upgrade_pub_nocopy \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ADD\"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \" TABLE\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" my_append_only_table_name;\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- On the new database\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ALTER\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" SUBSCRIPTION pg_upgrade_sub REFRESH PUBLICATION \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"WITH\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" ( copy_data \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"=\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" false );\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n })]\n })\n })\n }), \"\\n\", _jsx(_components.p, {\n children: \"We found this approach worked really well for our partitioned tables that stored\\nvarious types of logs for our customers.\\nWe did not need to migrate the root of a partitioned table, we only migrated the\\nunderlying tables, and that seemed to work pretty well.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"Once the subscription is running, you should start seeing logs appear on the\\ntarget databases table:\"\n }), \"\\n\", _jsx(_components.figure, {\n \"data-rehype-pretty-code-figure\": \"\",\n children: _jsx(_components.pre, {\n style: {\n backgroundColor: \"#ffffff\",\n color: \"#24292eff\"\n },\n tabIndex: \"0\",\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n children: _jsx(_components.code, {\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n style: {\n display: \"grid\"\n },\n children: _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"SELECT\"\n }), _jsx(_components.span, {\n style: {\n color: \"#6F42C1\"\n },\n children: \" COUNT\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \"(\"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"*\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \") \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"FROM\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" my_append_only_table_name; \"\n }), _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- Returns more than zero\"\n })]\n })\n })\n })\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"From here, you can backfill any records older than those now visible in the\\ndatabase using whatever means you like (e.g. \", _jsx(_components.code, {\n children: \"pg_dump\"\n }), \").\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"Here is how we did it on AWS RDS Aurora:\"\n }), \"\\n\", _jsxs(_components.ol, {\n children: [\"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"Take a snapshot of your production database in the AWS Console\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"Restore that snapshot into a new database instance (the snapshot DB)\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsxs(_components.p, {\n children: [\"Rename the table(s) on the snapshot DB that you want to replicate by adding\\na suffix like \", _jsx(_components.code, {\n children: \"_snapshot\"\n }), \". This prevents us having two replication pipelines\\nfeeding into the same table on the target database.\"]\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"Create the same table(s) on the target database with the same schema as the\\nsnapshot database. Use the same suffix as above.\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"Create a publication on the snapshot database and a subscription on the target\\ndatabase to replicate these snapshot table(s) from the snapshot database to the\\ntarget database\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"Enable the subscription and monitor its progress\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsxs(_components.p, {\n children: [\"Once the subscription is caught up, you can merge the tables together using\\n\", _jsx(_components.code, {\n children: \"INSERT...ON CONFLICT\"\n }), \":\"]\n }), \"\\n\", _jsx(_components.figure, {\n \"data-rehype-pretty-code-figure\": \"\",\n children: _jsx(_components.pre, {\n style: {\n backgroundColor: \"#ffffff\",\n color: \"#24292eff\"\n },\n tabIndex: \"0\",\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n children: _jsxs(_components.code, {\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n style: {\n display: \"grid\"\n },\n children: [_jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"INSERT INTO\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" my_append_only_table_name\"\n })]\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"SELECT\"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \" *\"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \" FROM\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" my_append_only_table_name_snapshot\"\n })]\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ON\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" CONFLICT (id) DO NOTHING;\"\n })]\n })]\n })\n })\n }), \"\\n\"]\n }), \"\\n\"]\n }), \"\\n\", _jsx(ImageModal, {\n src: \"/assets/blog/zero-downtime-postgres-upgrades/snapshot-backfill.png\",\n alt: \"Diagram showing how to backfill data from a snapshot\",\n children: _jsx(\"img\", {\n src: \"/assets/blog/zero-downtime-postgres-upgrades/snapshot-backfill.png\",\n alt: \"Diagram showing how to backfill data from a snapshot\",\n className: \"rounded-md mx-auto border border-gray-200\"\n })\n }), \"\\n\", _jsx(_components.p, {\n children: \"For very large tables, this can still take several days, but because it’s all\\nin the background it shouldn’t affect your production environment.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"Once the tables are fully merged, compare them to ensure a consistent row count\\n(more on that later). Once you are confident the tables are identical,\\ndrop the snapshot table on the target DB, drop the subscription to the snapshot DB,\\nand terminate the snapshot database instance.\"\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"large-tables-with-many-updates-over-most-of-the-rows\",\n children: [_jsx(_components.a, {\n href: \"#large-tables-with-many-updates-over-most-of-the-rows\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Large tables with many updates over most of the rows\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"These are the hard tables. Because they have so much data in them, they can\\ntake a long time to replicate, which can affect system performance on the\\nsource database if it prevents \", _jsx(_components.code, {\n children: \"AUTOVUACUUM\"\n }), \" from running. Because they have\\nso many updates, we can’t treat it as an append-only table.\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"A few points to consider:\"\n }), \"\\n\", _jsxs(_components.ol, {\n children: [\"\\n\", _jsx(_components.li, {\n children: \"Is there any housekeeping you can do to reduce the table’s size?\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Have you vacuumed the table recently?\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Can you partition the table into smaller pieces?\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Do rows stop receiving updates after a reliable time frame (e.g. 1 week?) -\\nthis could be used to treat the table as an append-only table, and then after\\nthat time frame has elapsed you can backfill old rows from a snapshot.\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"If your source database is not on PG 15 or greater, your options are limited.\\nFollow the steps in the \\\"small tables\\\" section. Rely on the monitoring you have\\nin place (you do have monitoring, right?) to ensure replication doesn’t degrade\\nyour service. If needed, you can rollback by removing the table from the\\npublication, and refreshing the subscription (\", _jsx(_components.a, {\n href: \"#aborting-the-replication-of-one-table\",\n children: \"See below\"\n }), \").\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"If the table is still too big, try to start replication during low traffic times to\\nreduce load and write activity. This will hopefully minimize the impact on your\\nsystem.\"\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"large-tables-coming-from-pg-15-or-greater\",\n children: [_jsx(_components.a, {\n href: \"#large-tables-coming-from-pg-15-or-greater\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Large tables coming from PG 15 or greater\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"If your source database is on PG 15 or greater, you may be able to split up replication\\nacross multiple publications (similar to partitioning or sharding). You can then\\nmigrate the table in smaller chunks, at the expense of using more replication\\nslots. \", _jsx(_components.a, {\n href: \"https://www.postgresql.org/docs/16/logical-replication-config.html\",\n children: \"The Postgres docs\"\n }), \"\\nhave more information on setting these parameters.\"]\n }), \"\\n\", _jsx(Callout, {\n emoji: \"🤞\",\n children: _jsx(_components.p, {\n children: \"Because we migrated from 11.9 to 15.3, we did not have this option available\\nto us. As such, we have not tested this approach. Even so, as we considered\\nour options we noticed that this approach might be possible. If you try it\\nout, let us know, we’d love to hear how it goes!\"\n })\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"The goal is to have enough publications to split your largest table into manageable\\npieces (for us, this was about 100 GB of non-index data stored). We’ll assume we\\nare splitting across three partitions in this example. The trick is adding a \", _jsx(_components.code, {\n children: \"WHERE\"\n }), \"\\nclause that splits up the rows handled by each subscription:\"]\n }), \"\\n\", _jsx(_components.figure, {\n \"data-rehype-pretty-code-figure\": \"\",\n children: _jsx(_components.pre, {\n style: {\n backgroundColor: \"#ffffff\",\n color: \"#24292eff\"\n },\n tabIndex: \"0\",\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n children: _jsxs(_components.code, {\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n style: {\n display: \"grid\"\n },\n children: [_jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- On the source database\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- For three partitions...\"\n })\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"CREATE\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" PUBLICATION pg_upgrade_pub_0;\"\n })]\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"CREATE\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" PUBLICATION pg_upgrade_pub_1;\"\n })]\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"CREATE\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" PUBLICATION pg_upgrade_pub_2;\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ALTER\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" PUBLICATION pg_upgrade_pub_0 \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ADD\"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \" TABLE\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" big_table\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- id must be the primary key.\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- Use hashint4 for int IDs, hashint8 for bigint IDs\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- Use hashtext(id::text) for UUID or other key types\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- If you have a composite PK, concatenate the columns together before hashing as text\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- Postgres' hash functions return positive \u0026 negative numbers - we abs() the result to make it positive\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- % 3 is used to pick which of three partitions. Adjust the integer for the number of partitions you will create.\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \" -- = 0 assigns rows to the first partition (zero-indexed, so we will finish with partitions 0, 1, and 2)\"\n })\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \" WHERE\"\n }), _jsx(_components.span, {\n style: {\n color: \"#6F42C1\"\n },\n children: \" abs\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \"(hashint4(id)) % \"\n }), _jsx(_components.span, {\n style: {\n color: \"#1976D2\"\n },\n children: \"3\"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \" =\"\n }), _jsx(_components.span, {\n style: {\n color: \"#1976D2\"\n },\n children: \" 0\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \";\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- Repeat the above ALTER statement for each publication, adjust the where clause accordingly.\"\n })\n })]\n })\n })\n }), \"\\n\", _jsx(_components.p, {\n children: \"On the destination database, create a subscription for each partition.\"\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"You only want to migrate one slice of each table at a time. Generally, you will\\nfollow the same instructions as adding a \\\"small\\\" table, but with the extra \", _jsx(_components.code, {\n children: \"WHERE\"\n }), \"\\nclause added when setting up the table for each publication.\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"In this way, you can slice up large tables into smaller, more workable pieces.\"\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"Consider only using this approach if having too many replication slots is a\\nproblem: you can still add \\\"small\\\" tables using this approach,\\njust add the table to the \", _jsx(_components.code, {\n children: \"_0\"\n }), \" publication without a \", _jsx(_components.code, {\n children: \"WHERE\"\n }), \" clause.\\nThis can help reduce the number of replication slots required when migrating.\"]\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"checking-a-tables-replication-status\",\n children: [_jsx(_components.a, {\n href: \"#checking-a-tables-replication-status\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Checking a table’s replication status\"]\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"When a table is added to a subscription, it moves through five distinct states\\n(visible on the target database under the system table \", _jsx(_components.code, {\n children: \"pg_subscription_rel\"\n }), \"\\nin the \", _jsx(_components.code, {\n children: \"srsubstate\"\n }), \" column):\"]\n }), \"\\n\", _jsxs(_components.ol, {\n children: [\"\\n\", _jsxs(_components.li, {\n children: [\"Initializing the table’s subscription (State code \", _jsx(_components.code, {\n children: \"i\"\n }), \" )\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"Copying the table’s contents in one efficient operation (State code \", _jsx(_components.code, {\n children: \"d\"\n }), \")\"]\n }), \"\\n\"]\n }), \"\\n\", _jsxs(Callout, {\n emoji: \"🚨\",\n children: [_jsx(_components.p, {\n children: \"This step requires keeping old Postgres transaction IDs around, which\\nprevents vacuum from running effectively and can lead to system performance\\nissues and (if left running long enough) even Postgres transaction ID\\nwraparound which can halt the system.\"\n }), _jsx(_components.p, {\n children: \"This is the step that requires replicating only one table at a time.\"\n })]\n }), \"\\n\", _jsxs(_components.ol, {\n start: \"3\",\n children: [\"\\n\", _jsxs(_components.li, {\n children: [\"Copy finished, waiting for final sync (State code \", _jsx(_components.code, {\n children: \"f\"\n }), \")\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"Finalizing initial sync (State code \", _jsx(_components.code, {\n children: \"s\"\n }), \")\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"Ready and running under normal replication (State code \", _jsx(_components.code, {\n children: \"r\"\n }), \")\"]\n }), \"\\n\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"In order to prevent the issues found in step 2 above, we found it was necessary\\nto add one table at a time to replication, and to closely watch the system’s\\nperformance. The worst-case scenario (transaction wraparound) must be avoided.\"\n }), \"\\n\", _jsx(_components.p, {\n children: _jsxs(_components.strong, {\n children: [\"If you get anywhere close to wraparound, it is better to \", _jsx(_components.a, {\n href: \"#aborting-the-replication-of-one-table\",\n children: \"abort the migration\"\n }), \"\\nand break it up into smaller pieces.\"]\n })\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"If we had created our publication using the \", _jsx(_components.code, {\n children: \"FOR ALL TABLES\"\n }), \" option, Postgres\\nwould have started to sync our very large source database all at once,\\npreventing automatic \", _jsx(_components.code, {\n children: \"VACUUM\"\n }), \" operations from completing necessary maintenance.\\nWe found this to gradually degrade database performance over time,\\nleading to increased risk to system stability.\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"Adding one table at a time has the added advantage of allowing teams to\\nincrementally migrate each table. Replication does come with CPU and other costs\\nfor the source and destination databases. By adding one table at a time,\\nadministrators can control how replication affects the running system.\"\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"aborting-the-replication-of-one-table\",\n children: [_jsx(_components.a, {\n href: \"#aborting-the-replication-of-one-table\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Aborting the replication of one table\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"If you need to halt the replication of a table, you reverse the instructions for\\nadding the table in the first place:\"\n }), \"\\n\", _jsx(_components.figure, {\n \"data-rehype-pretty-code-figure\": \"\",\n children: _jsx(_components.pre, {\n style: {\n backgroundColor: \"#ffffff\",\n color: \"#24292eff\"\n },\n tabIndex: \"0\",\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n children: _jsxs(_components.code, {\n \"data-language\": \"sql\",\n \"data-theme\": \"min-light\",\n style: {\n display: \"grid\"\n },\n children: [_jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- On the old database\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ALTER\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" PUBLICATION pg_upgrade_pub_nocopy \"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"DROP\"\n }), _jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \" TABLE\"\n }), _jsx(_components.span, {\n style: {\n color: \"#6F42C1\"\n },\n children: \" my_append_only_table_name\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \";\"\n })]\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: _jsx(_components.span, {\n style: {\n color: \"#C2C3C5\"\n },\n children: \"-- ON the new database\"\n })\n }), \"\\n\", _jsx(_components.span, {\n \"data-line\": \"\",\n children: \" \"\n }), \"\\n\", _jsxs(_components.span, {\n \"data-line\": \"\",\n children: [_jsx(_components.span, {\n style: {\n color: \"#D32F2F\"\n },\n children: \"ALTER\"\n }), _jsx(_components.span, {\n style: {\n color: \"#24292EFF\"\n },\n children: \" SUBSCRIPTION pg_upgrade_sub REFRESH PUBLICATION;\"\n })]\n })]\n })\n })\n }), \"\\n\", _jsx(_components.p, {\n children: \"In an emergency, you can also drop the publications and subscriptions entirely,\\nand start the process over. Postgres will clean up any replication slots that\\nwere created as part of the publication and subscription, which should relieve\\nany pressure on the source database.\"\n }), \"\\n\", _jsxs(Callout, {\n emoji: \"🚨\",\n children: [_jsx(_components.p, {\n children: \"Be advised that if you just disable the subscription without removing the table\\nfrom the publication and refreshing the subscription, the source database\\nwill continue to hold onto old transaction IDs, which can lead to transaction\\nwraparound and a forced shutdown of the database.\"\n }), _jsx(_components.p, {\n children: \"Just disabling the subscription will not resolve any replication-related\\nperformance problems.\"\n })]\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"a-note-about-moving-replication-slots\",\n children: [_jsx(_components.a, {\n href: \"#a-note-about-moving-replication-slots\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"A note about moving replication slots\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"Replication slots in Postgres store a log of database activity that can be\\nconsumed on another database or in another application. Postgres tracks slot\\nprogress using a Log Sequence Number (LSN). LSNs are unique to the primary\\nPostgres database. This means that if you have a replication slot on your\\ndatabase (e.g. to copy changes to a data warehouse or as part of your own\\napplication), you will not be able to copy the replication slot's LSN over from\\nthe old database to the new database.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"You will need to consult the documentation of the application consuming the\\nreplication slot to decide how to best migrate (e.g. for data warehousing tools,\\nthey may have a way to merge duplicated information between both databases).\\nIf you’re using replication slots as part of your own application, you already\\nknow that you’re on your own to roll your own solution. Having some idempotence\\nmechanism to deduplicate transactions from the old and the new database will\\ndefinitely be helpful.\"\n }), \"\\n\", _jsxs(_components.h2, {\n id: \"finalizing-the-migration\",\n children: [_jsx(_components.a, {\n href: \"#finalizing-the-migration\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Finalizing the migration\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"Once you have added all of your tables to publications, and the subscriptions\\nhave caught up on everything, you need to now verify that the tables match.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"Unfortunately, eventual consistency (the lag between a write being applied to\\nthe old database and it showing up on the new database) will prevent both\\ndatabases from being perfect matches at the same time, you can still count table\\nrows to make sure you’re close enough to know it’s working.\"\n }), \"\\n\", _jsxs(_components.p, {\n children: [\"At Knock, we wrote a script that iterated through each table and asked both\\ndatabases to count the total number of rows in each table on the old and new\\ndatabase, and compared the results. For tables with an \", _jsx(_components.code, {\n children: \"inserted_at\"\n }), \" column, we\\nfiltered to rows older than 10 seconds. This interval is more than enough to\\nprove that the tables match, with the assumption that the remaining 10 seconds\\nwill replicate across in short order.\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"You may need to come up with a strategy that fits your application’s needs. We\\nfelt that as long as row counts were accurate within a few seconds, we could\\notherwise assume that Postgres replication was reliable.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"In a few instances, we also spot-checked the contents of a few tables to ensure\\nthey matched to confirm this assumption. Collecting a random sample of rows from\\ntables and comparing them between the old and the new database can help verify\\nthat the tables are identical.\"\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"application-level-changes\",\n children: [_jsx(_components.a, {\n href: \"#application-level-changes\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Application-level changes\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"Parallel to all of this database work, you may need to change your application\\nto connect to both databases. When you are finally ready to cut over,\\nyou need a strategy to shift traffic to your new database.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"When the final cutover happens, you could change your application’s configuration\\nto point to the new database, and then reboot your app. This is simple,\\nstraightforward, and is precisely how we migrated one of our\\nlower-traffic databases.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"For applications with lots of concurrent activity, you may need to get creative.\\nWe wanted to avoid a situation with conflicting writes between the old and new\\ndatabase. Such conflicts could have caused a service outage for us, requiring\\nmanually reconciling database state.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"At Knock, we configured our application to connect to both databases.\\nWhen we were ready to execute the cutover, we ran a script that did the following:\"\n }), \"\\n\", _jsxs(_components.ol, {\n children: [\"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"Tell all instances of our application to send new queries to the new database\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"All currently running database queries had 500 ms to complete before being forcefully cancelled\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"For the first second after flipping the flag, our application artificially paused\\nany new database requests for one second. This allowed pending transactions\\nto replicate to the new database so that new queries wouldn’t have stale reads\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"500 ms is far higher than most of our db queries, and we saw zero errors due to forced disconnections\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"After that first second, database activity returned to normal behavior, but pointing at the new database.\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"In the middle of the cutover, we had some specialized database workloads that\\nthe script shut down and restarted in order to reconnect to the new database.\"\n }), \"\\n\"]\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.h3, {\n id: \"one-more-thing-sequences\",\n children: [_jsx(_components.a, {\n href: \"#one-more-thing-sequences\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"One more thing: sequences\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"One thing that replication doesn’t synchronize is any Postgres sequence.\\nSequences are monotonically increasing integers that are guaranteed to never\\nduplicate. Unfortunately, they are not incremented on the new database as\\nsequence values are used up on the old database.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"Fortunately, this is pretty easy to control for. Part of our cutover procedure\\nwas to run a script right before flipping our feature flag that did the following:\"\n }), \"\\n\", _jsxs(_components.ol, {\n children: [\"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsx(_components.p, {\n children: \"Connect to both databases\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsxs(_components.p, {\n children: [\"Get the next value of all of the sequences in the database using \", _jsx(_components.code, {\n children: \"SELECT nextval('sequence_name')\"\n })]\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"\\n\", _jsxs(_components.p, {\n children: [\"Set that value in the new database using \", _jsx(_components.code, {\n children: \"SELECT setval('sequence_name', value::int4 + 100000)\"\n }), \"\\nto advance the sequence and offer a little bit of buffer (in this case, 100k\\nrows can be added between setting this value on the new database and cutting over).\\nThis will introduce a gap in the sequence, but that’s generally not a problem.\\nFor us, our sequences are bigints. 100k values skipped in the sequence is a\\nrounding error off of 0% used up sequence values in that case.\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"You will want to tune how big of a gap you introduce so you don’t use too\\nmuch of your sequence’s usable space. If you only expect the sequence to\\nuse a few hundred values during your cutover window,\\nthen maybe advance it only by 5000.\"\n }), \"\\n\"]\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.h2, {\n id: \"final-checklist-before-cutting-over\",\n children: [_jsx(_components.a, {\n href: \"#final-checklist-before-cutting-over\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Final checklist before cutting over\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"Here are some of the things we considered before executing our final cutover:\"\n }), \"\\n\", _jsxs(_components.ol, {\n children: [\"\\n\", _jsx(_components.li, {\n children: \"Do the rows on all the tables match as expected?\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Are all the subscriptions enabled and running without error?\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Do the schemas match? Can you freeze any new schema migrations from being\\nreleased to reduce the risk of something changing while you’re migrating?\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Is your new database properly sized for your workloads?\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Do you have to add any read replicas so the database cluster topology is the\\nsame between the old and the new database?\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Have you reindexed and performed basic VACUUM maintenance on the new database\\nto ensure it’s fresh and ready for production traffic?\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Have you double checked Postgres’ release notes for anything that might cause\\na regression in your app?\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Have you run automated and manual tests against a staging database on the new\\nversion to verify system performance?\"\n }), \"\\n\", _jsxs(_components.li, {\n children: [\"Have you run load tests of your most demanding queries using \", _jsx(_components.code, {\n children: \"pg_bench\"\n }), \"\\nagainst your new version to verify performance?\"]\n }), \"\\n\", _jsx(_components.li, {\n children: \"If there’s one thing that you can de-risk still, what is it?\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Do practice runs in a staging or test environment until you have fully\\nexercised the cutover process multiple times. Dry runs like this will help\\nreveal gaps in your plan before you go to production.\"\n }), \"\\n\", _jsx(_components.li, {\n children: \"Right before cutover, take a database backup - just in case.\"\n }), \"\\n\"]\n }), \"\\n\", _jsxs(_components.h2, {\n id: \"cutting-over\",\n children: [_jsx(_components.a, {\n href: \"#cutting-over\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Cutting over\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"At Knock, we took a few weeks replicating tables one at a time. We generally did\\nthis after business hours and during our lowest traffic time frames. We practiced\\ncutover in our staging environment multiple times, ironing out the process until\\nit just worked without much operator involvement.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"Once we had a replica running PG 15 and had the application code in place to\\ncut over from the old to the new database, we ran one final set of checks and\\nflipped the flag.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"After months of preparation, the actual cutover was uneventful: our\\napplication cut over within a few seconds, we had a brief blip of (intentional)\\nlatency as queries waited to allow for replication, and our application\\ncontinued running without skipping a beat. Reading this paragraph took longer\\nthan the cutover itself.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"From there, we rolled back the application changes we introduced, permanently\\npointed everything at the new database, removed the subscriptions on the new\\ndatabase, and tore down the old database. We had successfully jumped from\\nPostgres 11.9 to 15.3 with zero downtime!\"\n }), \"\\n\", _jsxs(_components.h2, {\n id: \"conclusion\",\n children: [_jsx(_components.a, {\n href: \"#conclusion\",\n \"aria-hidden\": \"true\",\n tabIndex: \"-1\",\n children: _jsx(_components.span, {})\n }), \"Conclusion\"]\n }), \"\\n\", _jsx(_components.p, {\n children: \"Although jumping four major versions of Postgres in one leap is a painstaking\\nprocess, it can be done, and in many ways it’s safer than scheduled downtime:\\nit can be practiced, tested, and reworked multiple times before performing the\\nactual cutover. At any point in the process, we could have dropped the\\npublications from the old database and started over without degrading our service.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"Modern customers expect 100% availability. While that is not technically possible,\\nzero downtime migrations make it easier to keep systems running smoothly without\\nmajor service interruptions.\"\n })]\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\nfunction _missingMdxReference(id, component) {\n throw new Error(\"Expected \" + (component ? \"component\" : \"object\") + \" `\" + id + \"` to be defined: you likely forgot to import, pass, or provide it.\");\n}\n","frontmatter":{},"scope":{"benchmarkData":null,"smsBenchmarkData":null,"pushBenchmarkData":null}},"fullUrl":"https://knock.app/blog/zero-downtime-postgres-upgrades"},"__N_SSG":true},"page":"/blog/[slug]","query":{"slug":"zero-downtime-postgres-upgrades"},"buildId":"Q3knVdD3C65MAkfAAiAvI","isFallback":false,"gsp":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>