Files
nexus/sreweekly/articles/254/05-the-next-gen-database-servers-powering-let-s-encrypt.html
2026-09-12 17:23:01 +08:00

861 lines
33 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en-US" dir="ltr">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<title>
The Next Gen Database Servers Powering Let&#39;s Encrypt - Let&#39;s Encrypt
</title>
<meta name="description" content="Let’s Encrypt helps to protect a huge portion of the Web by providing TLS certificates to more than 235 million websites. A database is at the heart of how Let’s Encrypt manages certificate issuance. If this database isn’t performing well enough, it can cause API errors and timeouts for our subscribers. Database performance is the single most critical factor in our ability to scale while meeting service level objectives. In late 2020, we upgraded our database servers and we’ve been very happy with the results." />
<meta property="og:site_name" content="Let&#39;s Encrypt" />
<meta property="og:type" content="article" />
<meta property="og:title" content="The Next Gen Database Servers Powering Let&#39;s Encrypt" />
<meta property="og:description" content="Let’s Encrypt helps to protect a huge portion of the Web by providing TLS certificates to more than 235 million websites. A database is at the heart of how Let’s Encrypt manages certificate issuance. If this database isn’t performing well enough, it can cause API errors and timeouts for our subscribers. Database performance is the single most critical factor in our ability to scale while meeting service level objectives. In late 2020, we upgraded our database servers and we’ve been very happy with the results." />
<meta property="og:url" content="https://letsencrypt.org/2021/01/21/next-gen-database-servers.html" />
<meta property="og:image" content="https://letsencrypt.org/images/LetsEncrypt-SocialShare.png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="630" />
<link rel="alternate" type="application/rss+xml" title="Let&#39;s Encrypt" href="/feed.xml">
<link rel="stylesheet" href="/css/le-2025-theme-input.3984d9f694d0d451615d24f63240ad420ae9b1fcbdb2a79749fa2503c4541017.css" integrity="sha256-OYTZ9pTQ1FFhXST2MkCtQgrpsfy9sqeXSfolA8RUEBc=" crossorigin="anonymous">
<link rel="stylesheet" href="/fontawesome-free-6.5.2-web/css/all.min.css" />
<script src="/js/le-2025-theme.baca0a37d412391799a4e5f6d58968c12b948b4c91c053a3c025ab758ffb70bc.js" integrity="sha256-usoKN9QSOReZpOX21YlowSuUi0yRwFOjwCWrdY/7cLw=" crossorigin="anonymous"></script>
<script src="/js/autoanchor.44783525c73cfeef565e5b8441cd42a848cb630d9f365def73edce4867e6f4c8.js" integrity="sha256-RHg1Jcc8/u9WXluEQc1CqEjLYw2fNl3vc&#43;3OSGfm9Mg=" crossorigin="anonymous"></script>
<script src="/js/number-animation.137c25102252dc89411a5c2d9eb00a383f12425cb7845c8d8ac4c615c0af1f8e.js" integrity="sha256-E3wlECJS3IlBGlwtnrAKOD8SQly3hFyNisTGFcCvH44=" crossorigin="anonymous"></script>
</head>
<body>
<header class="relative bg-white shadow-sm">
<div class="language-section">
<div class="language-bar bg-gray-100 border-b border-gray-200">
<div class="container mx-auto px-4">
<div class="flex justify-end">
<button
id="language-toggle"
aria-expanded="false"
aria-controls="language-section"
class="flex items-center space-x-2 py-1 text-xs text-gray-600 hover:text-gray-900 focus:outline-none focus:ring-2 focus:ring-blue-500">
<span class="text-xs">Languages</span>
<img src="/images/language-icon128px-black.png" class="w-3 h-3" alt="" aria-hidden="true">
</button>
</div>
</div>
</div>
<div id="language-section"
class="hidden h-0 opacity-0 border-b border-gray-200 bg-white"
role="region"
aria-labelledby="language-toggle">
<div class="container mx-auto px-4 py-4">
<div class="grid grid-cols-2 md:grid-cols-3 lg:grid-cols-4 gap-1">
<a href="/"
lang="en-US"
hreflang="en-US"
aria-current="true"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md bg-blue-50 text-blue-700">
<svg class="w-5 h-5 text-blue-500" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M5 13l4 4L19 7"></path>
</svg>
<span>English</span>
</a>
<a href="/ca/"
lang="ca"
hreflang="ca"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Català</span>
</a>
<a href="/cs/"
lang="cs-CZ"
hreflang="cs-CZ"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Čeština</span>
</a>
<a href="/da/"
lang="da"
hreflang="da"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Dansk</span>
</a>
<a href="/de/"
lang="de-DE"
hreflang="de-DE"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Deutsch</span>
</a>
<a href="/el/"
lang="el"
hreflang="el"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Greek</span>
</a>
<a href="/es/"
lang="es-US"
hreflang="es-US"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Español</span>
</a>
<a href="/fi/"
lang="fi"
hreflang="fi"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Suomi</span>
</a>
<a href="/fr/"
lang="fr-FR"
hreflang="fr-FR"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Français</span>
</a>
<a href="/he/"
lang="he"
hreflang="he"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>עברית</span>
</a>
<a href="/hu/"
lang="hu"
hreflang="hu"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Magyar</span>
</a>
<a href="/id/"
lang="id-ID"
hreflang="id-ID"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Bahasa Indonesia</span>
</a>
<a href="/it/"
lang="it-IT"
hreflang="it-IT"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Italiano</span>
</a>
<a href="/ja/"
lang="ja"
hreflang="ja"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>日本語</span>
</a>
<a href="/ko/"
lang="ko-KR"
hreflang="ko-KR"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>한국어</span>
</a>
<a href="/pl/"
lang="pl"
hreflang="pl"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Polish</span>
</a>
<a href="/pt-br/"
lang="pt-BR"
hreflang="pt-BR"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Português do Brasil</span>
</a>
<a href="/ru/"
lang="ru-RU"
hreflang="ru-RU"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Русский</span>
</a>
<a href="/si/"
lang="si"
hreflang="si"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>සිංහල</span>
</a>
<a href="/sr/"
lang="sr"
hreflang="sr"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Srpski</span>
</a>
<a href="/sv/"
lang="sv"
hreflang="sv"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Svenska</span>
</a>
<a href="/ta/"
lang="ta"
hreflang="ta"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>தமிழ்</span>
</a>
<a href="/th/"
lang="th"
hreflang="th"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Thai</span>
</a>
<a href="/tr/"
lang="tr-TR"
hreflang="tr-TR"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Türkçe</span>
</a>
<a href="/uk/"
lang="uk-UA"
hreflang="uk-UA"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Українська</span>
</a>
<a href="/vi/"
lang="vi"
hreflang="vi"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Tiếng Việt</span>
</a>
<a href="/zh-cn/"
lang="zh-Hans-CN"
hreflang="zh-Hans-CN"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>简体中文</span>
</a>
<a href="/zh-tw/"
lang="zh-Hant-TW"
hreflang="zh-Hant-TW"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>繁體中文</span>
</a>
</div>
</div>
</div>
</div>
<div id="main-header" class="container mx-auto px-4 py-3 md:py-4">
<a id="skiplink"
href="#main-content"
class="sr-only focus:not-sr-only focus:absolute focus:top-4 focus:left-4 bg-white px-4 py-2 rounded-md shadow-md">
Skip navigation links
</a>
<div class="flex items-center justify-between">
<a class="site-logo shrink-0" href="/">
<img src="/images/letsencrypt-logo-horizontal.svg" alt="Let's Encrypt">
</a>
<button
id="mobile-menu-toggle"
class="md:hidden p-2 rounded-md text-gray-600 hover:text-gray-900 hover:bg-gray-100 focus:outline-none focus:ring-2 focus:ring-blue-500"
aria-expanded="false"
aria-controls="main-nav"
aria-label="Toggle menu">
<svg class="w-6 h-6" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 6h16M4 12h16M4 18h16"></path>
</svg>
</button>
<nav id="main-nav"
class="md:flex absolute md:!relative inset-x-0 top-full md:top-auto bg-white md:bg-transparent border-b md:border-b-0 border-gray-200 shadow-lg md:shadow-none md:grow md:ml-8 z-50">
<div class="container mx-auto">
<div class="md:p-0 flex justify-center">
<ul class="site-main-nav flex flex-col md:flex-row md:justify-center md:gap-5 py-2 md:py-0 w-full" role="menubar">
<li class="relative " role="none">
<a role="menuitem" class="nav-item" href="/docs/">
Documentation
</a>
</li>
<li class="relative " role="none">
<a role="menuitem" class="nav-item" href="https://community.letsencrypt.org/">
Get Help
</a>
</li>
<li class="relative " role="none">
<a role="menuitem" class="nav-item" href="/blog/">
Blog
</a>
</li>
<li class="relative group" role="none">
<div class="dropdown-container">
<button
class="nav-item flex items-center justify-between w-full md:w-auto space-x-1"
aria-expanded="false"
aria-controls="dropdown-donate"
role="menuitem">
<span>Donate</span>
<svg class="w-4 h-4 text-gray-500" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M19 9l-7 7-7-7"></path>
</svg>
</button>
<ul id="dropdown-donate"
class="hidden static md:absolute left-0 w-full md:w-auto py-1 bg-white md:rounded-md md:shadow-lg md:border md:border-gray-200 z-60 mt-1"
role="menu">
<li class="relative" role="none">
<a href="https://www.abetterinternet.org/sponsor/"
class="nav-dropdown-item"
role="menuitem">
Become a Sponsor
</a>
</li>
<li class="relative" role="none">
<a href="https://www.abetterinternet.org/sponsors/"
class="nav-dropdown-item"
role="menuitem">
Current Sponsors &amp; Funders
</a>
</li>
<li class="relative" role="none">
<a href="/getinvolved/"
class="nav-dropdown-item"
role="menuitem">
Get Involved
</a>
</li>
<li class="relative" role="none">
<a href="/donate/"
class="nav-dropdown-item"
role="menuitem">
Donate
</a>
</li>
</ul>
</div>
</li>
<li class="relative group" role="none">
<div class="dropdown-container">
<button
class="nav-item flex items-center justify-between w-full md:w-auto space-x-1"
aria-expanded="false"
aria-controls="dropdown-about"
role="menuitem">
<span>About Us</span>
<svg class="w-4 h-4 text-gray-500" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M19 9l-7 7-7-7"></path>
</svg>
</button>
<ul id="dropdown-about"
class="hidden static md:absolute left-0 w-full md:w-auto py-1 bg-white md:rounded-md md:shadow-lg md:border md:border-gray-200 z-60 mt-1"
role="menu">
<li class="relative" role="none">
<a href="/about/"
class="nav-dropdown-item"
role="menuitem">
Let&#39;s Encrypt
</a>
</li>
<li class="relative" role="none">
<a href="/docs/faq/"
class="nav-dropdown-item"
role="menuitem">
Frequently Asked Questions (FAQ)
</a>
</li>
<li class="relative" role="none">
<a href="/repository/"
class="nav-dropdown-item"
role="menuitem">
Policy and Legal Repository
</a>
</li>
<li class="relative" role="none">
<a href="https://letsencrypt.status.io/"
class="nav-dropdown-item"
role="menuitem">
Service Status
</a>
</li>
<li class="relative" role="none">
<a href="/stats/"
class="nav-dropdown-item"
role="menuitem">
Statistics
</a>
</li>
<li class="relative" role="none">
<a href="/contact/"
class="nav-dropdown-item"
role="menuitem">
Contact
</a>
</li>
<li class="relative" role="none">
<a href="https://www.abetterinternet.org/careers/"
class="nav-dropdown-item"
role="menuitem">
Careers
</a>
</li>
<li class="relative" role="none">
<a href="https://www.abetterinternet.org/annual-reports/"
class="nav-dropdown-item"
role="menuitem">
Annual reports
</a>
</li>
<li class="relative" role="none">
<a href="https://www.abetterinternet.org/about/"
class="nav-dropdown-item"
role="menuitem">
Internet Security Research Group (ISRG)
</a>
</li>
</ul>
</div>
</li>
<li class="block md:hidden pt-2.5"><a class="btn btn-yellow btn-sm" href="/donate/">Donate Now</a></li>
</ul>
</div>
</div>
</nav>
<a class="btn btn-yellow hidden md:block shrink-0" href="/donate/">Donate Now</a>
</div>
</div>
</header>
<div id="main-content"></div>
<main>
<section class="post-hero py-8">
<div class="narrow-container mx-auto">
<div class="post-hero__inner border-l border-le-yellow box-border pl-4 flex flex-col gap-2">
<div class="post-hero__breadcrumb">
<a class="" href="/blog/">Blog</a>
</div>
<h1 class="post-hero__title mb-0">The Next Gen Database Servers Powering Let&#39;s Encrypt</h1>
<div class="post-hero__meta">
By Josh Aas and James Renken ·
<time datetime="2021-01-21T00:00:00&#43;00:00">January 21, 2021</time>
</div>
</div>
</div>
</section>
<div class="narrow-container mx-auto py-8 prose-content">
<p>Let’s Encrypt helps to protect a huge portion of the Web by providing TLS certificates to more than <a href="https://letsencrypt.org/stats/">235 million websites</a>. A database is at the heart of how Let’s Encrypt manages certificate issuance. If this database isn’t performing well enough, it can cause API errors and timeouts for our subscribers. Database performance is the single most critical factor in our ability to scale while meeting service level objectives. In late 2020, we upgraded our database servers and we’ve been very happy with the results.</p>
<h2 id="what-exactly-are-we-doing-with-these-servers">What exactly are we doing with these servers?</h2>
<p>Our CA software, <a href="https://github.com/letsencrypt/boulder">Boulder</a>, uses MySQL-style schemas and queries to manage subscriber accounts and the entire certificate issuance process. It&rsquo;s designed to work with a single MySQL, MariaDB, or Percona database. We currently use MariaDB, with the InnoDB database engine.</p>
<p>We run the CA against a single database in order to minimize complexity. Minimizing complexity is good for security, reliability, and reducing maintenance burden. We have a number of replicas of the database active at any given time, and we direct some read operations to replica database servers to reduce load on the primary.</p>
<p>One consequence of this design is that our database machines need to be pretty powerful. Eventually we may need to shard or break the single database into multiple databases, but hardware advancements have allowed us to avoid that so far.</p>
<h2 id="hardware-specifications">Hardware Specifications</h2>
<p>The previous generation of database hardware was powerful but it was regularly being pushed to its limits. For the next generation, we wanted to more than double almost every performance metric in the same 2U form factor. In order to pull that off, we needed AMD EPYC chips and Dell’s <a href="https://www.dell.com/en-us/work/shop/cty/pdp/spd/poweredge-r7525/">PowerEdge R7525</a> was ideal. Here are the specifications:</p>
<div class="flex flex-col items-center">
<table>
<tr>
<td class="p-[10px]"></td>
<td class="p-[10px]"><b>Previous Generation</b></td>
<td class="p-[10px]"><b>Next Generation</b></td>
</tr>
<tr>
<td class="p-[10px] align-top"><b>CPU</b></td>
<td class="p-[10px]">2x Intel Xeon E5-2650<br>Total 24 cores / 48 threads</td>
<td class="p-[10px]">2x <a href="https://www.amd.com/en/products/cpu/amd-epyc-7542">AMD EPYC 7542</a><br>Total 64 cores / 128 threads
</td>
</tr>
<tr>
<td class="p-[10px] align-top"><b>Memory</b></td>
<td class="p-[10px]">1TB 2400MT/s</td>
<td class="p-[10px]">2TB 3200MT/s</td>
</tr>
<tr>
<td class="p-[10px] align-top"><b>Storage</b></td>
<td class="p-[10px]">24x 3.8TB Samsung PM883<br>SATA SSD<br>560/540 MB/s read/write</td>
<td class="p-[10px]">24x 6.4TB Intel P4610<br>NVMe SSD<br>3200/3200 MB/s read/write</td>
</tr>
</table>
</div>
<figure class="flex flex-col items-center text-center">
<img src="/images/2021.01.21-next-gen-db-chassis.jpg" width="600" alt="Dell PowerEdge R7525 Chassis">
<figcaption>Dell PowerEdge R7525 internals. The two silver rectangles in the middle are the CPUs. The RAM sticks, each 64GB, are above and below the CPUs. The 24x NVMe drives are in the front of the server, on the far left.</figcaption>
</figure>
<p>By going with AMD EPYC, we were able to get 64 physical CPU cores while keeping clock speeds high: 2.9GHz base with 3.4GHz boost. More importantly, EPYC provides 128 PCIe v4.0 lanes, which allows us to put 24 NVMe drives in a single machine. NVMe is incredibly fast (~5.7x faster than the SATA SSDs in our previous-gen database servers) because it uses PCIe instead of SATA. However, PCIe lanes are typically very limited: modern consumer chips typically have only 16 lanes, and Intel’s Xeon chips have 48. By providing 128 PCI lanes per chip (v4.0, no less), AMD EPYC has made it possible to pack large numbers of NVMe drives into a single machine. We’ll talk more about NVMe later.</p>
<h2 id="performance-impact">Performance Impact</h2>
<p>We’ll start by looking at our median time to process a request because it best reflects subscribers’ experience. Before the upgrade, we turned around the median API request in ~90 ms. The upgrade decimated that metric to ~9 ms!</p>
<p class="text-center"><img src="/images/2021.01.21-next-gen-db-api-latency.png" alt="API Latency"></p>
<p>We can clearly see how our old CPUs were reaching their limit. In the week before we upgraded our primary database server, its CPU usage (from /proc/stat) averaged over 90%:</p>
<p class="text-center"><img src="/images/2021.01.21-next-gen-db-cpu-before.png" alt="CPU Usage Before Upgrade"></p>
<p>The new AMD EPYC CPUs sit at about 25%. You can see in this graph where we promoted the new database server from replica (read-only) to primary (read/write) on September 15.</p>
<p class="text-center"><img src="/images/2021.01.21-next-gen-db-cpu-after.png" alt="CPU Usage After Upgrade"></p>
<p>The upgrade greatly reduced our overall database latency. The average query response time (from INFORMATION_SCHEMA) used to be ~0.45ms.</p>
<p class="text-center"><img src="/images/2021.01.21-next-gen-db-db-latency-before.png" alt="Database Latency Before Upgrade"></p>
<p>Queries now average <em>three times faster</em>, about 0.15ms.</p>
<p class="text-center"><img src="/images/2021.01.21-next-gen-db-db-latency-after.png" alt="Database Latency After Upgrade"></p>
<h2 id="openzfs-and-nvme">OpenZFS and NVMe</h2>
<p>NVMe drives are becoming increasingly popular because of their incredible performance. Up until recently, though, it was nearly impossible to get many of them in a single machine because NVMe uses PCIe lanes. Those were very limited: Intel’s Xeon processors come with just 48 PCIe v3 lanes, and a number of those are used up by the chipset and add-on cards such as network adapters and GPUs. You can’t fit many NVMe drives in the remaining lanes.</p>
<p>AMD’s latest generation of EPYC processors come with 128 PCIe lanes - more than double what Intel offers - and they’re PCIe v4! This is enough to pack a 2U server full of NVMe drives (24 in our case).</p>
<p>Once you have a server full of NVMe drives, you have to decide how to manage them. Our previous generation of database servers used hardware RAID in a RAID-10 configuration, but there is no effective hardware RAID for NVMe, so we needed another solution. One option was software RAID (Linux mdraid), but we got several recommendations for OpenZFS and decided to give it a shot. We’ve been very happy with it!</p>
<p>There wasn’t a lot of information out there about how best to set up and optimize OpenZFS for a pool of NVMe drives and a database workload, so we want to share what we learned. You can find detailed information about our setup in <a href="https://github.com/letsencrypt/openzfs-nvme-databases">this GitHub repository</a>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>This database upgrade was necessary as more people rely on Let’s Encrypt for the security and privacy that TLS/SSL provides. The equipment is quite expensive and it was a sizable undertaking for our SRE team to plan and execute the transition, but we gained a lot through the process.</p>
<h2 id="support-let-s-encrypt">Support Let&rsquo;s Encrypt</h2>
<p>We depend on contributions from our supporters in order to provide our services. If your company or organization would like to <a href="https://www.abetterinternet.org/sponsor/">sponsor</a> Let’s Encrypt please email us at <a href="mailto:sponsor@letsencrypt.org">sponsor@letsencrypt.org</a>. We ask that you make an <a href="https://letsencrypt.org/donate/">individual contribution</a> if it is within your means.</p>
</div>
</main>
<footer class="bg-white text-gray-800 pt-10 pb-4">
<div class="container">
<div class="grid md:grid-cols-2 gap-12 lg:gap-24">
<div>
<img src="/images/ISRG-Logo-Blue.svg"
alt="Internet Security Research Group (ISRG)"
class="h-12 w-auto mb-6">
<p class="mb-6">Let's Encrypt is a free, automated, and open Certificate Authority brought to you by the nonprofit <a href="https://www.abetterinternet.org/">Internet Security Research Group (ISRG)</a>. Read all about our nonprofit work this year in our <a href="https://www.abetterinternet.org/annual-reports/">2025 Annual Report</a>.
</p>
<div class="grid grid-cols-2 gap-6 mt-10">
<div>
<div class="text-eyebrow text-eyebrow-sm">Legal Address</div>
<div itemscope itemtype="http://schema.org/PostalAddress">
<span itemprop="streetAddress">548 Market St, PMB 77519</span><br />
<span itemprop="addressLocality">San Francisco</span>,
<span itemprop="addressRegion">CA</span>
<span itemprop="postalCode">94104-5401</span><br />
<span itemprop="addressCountry">USA</span>
</div>
</div>
<div>
<div class="text-eyebrow text-eyebrow-sm">Send all mail or inquiries to:</div>
<div itemscope itemtype="http://schema.org/PostalAddress">
<span itemprop="streetAddress">PO Box 18666</span><br />
<span itemprop="addressLocality">Minneapolis</span>,
<span itemprop="addressRegion">MN</span>
<span itemprop="postalCode">55418-0666</span><br />
<span itemprop="addressCountry">USA</span>
</div>
</div>
</div>
</div>
<div class="pt-0 md:pt-[70px]">
<div class="border border-gray-300 rounded-lg pt-3 pb-0 px-2">
<h4 class="text-eyebrow mb-5">Subscribe for email updates about Let&#39;s Encrypt and other ISRG projects</h4>
<iframe id="newsletter-iframe" src="https://outreach.abetterinternet.org/l/1011011/2025-01-14/31v6r" title="newsletter" class="w-full border-0 overflow-hidden"></iframe>
<script>
(function() {
window.addEventListener('message', function(e) {
if (e.origin !== 'https://outreach.abetterinternet.org') return;
if (e.data && typeof e.data === 'object' && e.data.type === 'resize' && e.data.height) {
document.getElementById('newsletter-iframe').setAttribute('height', e.data.height + 20);
}
});
})();
</script>
</div>
</div>
</div>
</div>
</footer>
<div class="footer-copyright-bar bg-le-blue text-white py-6">
<div class="container">
<div class="flex flex-col md:flex-row md:justify-between items-center justify-center">
<div class="mb-4 text-center md:text-left md:mb-0">
&copy; 2026 <a href="https://www.abetterinternet.org" target="_blank" rel="noopener noreferrer">Internet Security Research Group</a>
</div>
<ul class="flex flex-wrap gap-4 justify-center md:justify-end">
<li>
<a href="https://github.com/letsencrypt" target="_blank" rel="noopener noreferrer">
<span>GitHub</span>
</a>
</li>
<li>
<a href="https://www.linkedin.com/company/lets-encrypt/" target="_blank" rel="noopener noreferrer">
<span>LinkedIn</span>
</a>
</li>
<li>
<a href="https://www.abetterinternet.org/terms-of-service" target="_blank" rel="noopener noreferrer">
<span>Terms</span>
</a>
</li>
<li>
<a href="/privacy/">
<span>Privacy Policy</span>
</a>
</li>
<li>
<a href="https://www.abetterinternet.org/trademarks" target="_blank" rel="noopener noreferrer">
<span>Trademark Policy</span>
</a>
</li>
</ul>
</div>
</div>
</div>
<script src="/js/main.03c09aa527a24dbba751011c441d5b78f1fe35f96ccf57317abb09232397e896843267c6489270f0430ccf29681e6a4891bd9af417641f43ffa154d9fbf8a075.js" integrity="sha512-A8CapSeiTbunUQEcRB1bePH&#43;Nflsz1cxersJIyOX6JaEMmfGSJJw8EMMzyloHmpIkb2a9BdkH0P/oVTZ&#43;/igdQ=="></script>
</body>
</html>