Files
nexus/sreweekly/articles/196/02-how-let-s-encrypt-runs-ct-logs.html
2026-09-12 17:23:01 +08:00

857 lines
37 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en-US" dir="ltr">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<title>
How Let&#39;s Encrypt Runs CT Logs - Let&#39;s Encrypt
</title>
<meta name="description" content="Let’s Encrypt launched a Certificate Transparency (CT) log this past spring. We’re excited to share how we built it in hopes that others can learn from what we did. CT has quickly become an important piece of Internet security infrastructure, but unfortunately it’s not trivial to run a good log. The more the CT community can share about what has been done, the better the ecosystem will be. Sectigo and Amazon Web Services have generously provided support to cover a significant portion of the cost of running our CT log. “Sectigo is proud to sponsor the Let’s Encrypt CT Log. We believe this initiative will provide much-needed reinforcement of the CT ecosystem,” said Ed Giaquinto, Sectigo’s CIO." />
<meta property="og:site_name" content="Let&#39;s Encrypt" />
<meta property="og:type" content="article" />
<meta property="og:title" content="How Let&#39;s Encrypt Runs CT Logs" />
<meta property="og:description" content="Let’s Encrypt launched a Certificate Transparency (CT) log this past spring. We’re excited to share how we built it in hopes that others can learn from what we did. CT has quickly become an important piece of Internet security infrastructure, but unfortunately it’s not trivial to run a good log. The more the CT community can share about what has been done, the better the ecosystem will be. Sectigo and Amazon Web Services have generously provided support to cover a significant portion of the cost of running our CT log. “Sectigo is proud to sponsor the Let’s Encrypt CT Log. We believe this initiative will provide much-needed reinforcement of the CT ecosystem,” said Ed Giaquinto, Sectigo’s CIO." />
<meta property="og:url" content="https://letsencrypt.org/2019/11/20/how-le-runs-ct-logs.html" />
<meta property="og:image" content="https://letsencrypt.org/images/LetsEncrypt-SocialShare.png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="630" />
<link rel="alternate" type="application/rss+xml" title="Let&#39;s Encrypt" href="/feed.xml">
<link rel="stylesheet" href="/css/le-2025-theme-input.3984d9f694d0d451615d24f63240ad420ae9b1fcbdb2a79749fa2503c4541017.css" integrity="sha256-OYTZ9pTQ1FFhXST2MkCtQgrpsfy9sqeXSfolA8RUEBc=" crossorigin="anonymous">
<link rel="stylesheet" href="/fontawesome-free-6.5.2-web/css/all.min.css" />
<script src="/js/le-2025-theme.baca0a37d412391799a4e5f6d58968c12b948b4c91c053a3c025ab758ffb70bc.js" integrity="sha256-usoKN9QSOReZpOX21YlowSuUi0yRwFOjwCWrdY/7cLw=" crossorigin="anonymous"></script>
<script src="/js/autoanchor.44783525c73cfeef565e5b8441cd42a848cb630d9f365def73edce4867e6f4c8.js" integrity="sha256-RHg1Jcc8/u9WXluEQc1CqEjLYw2fNl3vc&#43;3OSGfm9Mg=" crossorigin="anonymous"></script>
<script src="/js/number-animation.137c25102252dc89411a5c2d9eb00a383f12425cb7845c8d8ac4c615c0af1f8e.js" integrity="sha256-E3wlECJS3IlBGlwtnrAKOD8SQly3hFyNisTGFcCvH44=" crossorigin="anonymous"></script>
</head>
<body>
<header class="relative bg-white shadow-sm">
<div class="language-section">
<div class="language-bar bg-gray-100 border-b border-gray-200">
<div class="container mx-auto px-4">
<div class="flex justify-end">
<button
id="language-toggle"
aria-expanded="false"
aria-controls="language-section"
class="flex items-center space-x-2 py-1 text-xs text-gray-600 hover:text-gray-900 focus:outline-none focus:ring-2 focus:ring-blue-500">
<span class="text-xs">Languages</span>
<img src="/images/language-icon128px-black.png" class="w-3 h-3" alt="" aria-hidden="true">
</button>
</div>
</div>
</div>
<div id="language-section"
class="hidden h-0 opacity-0 border-b border-gray-200 bg-white"
role="region"
aria-labelledby="language-toggle">
<div class="container mx-auto px-4 py-4">
<div class="grid grid-cols-2 md:grid-cols-3 lg:grid-cols-4 gap-1">
<a href="/"
lang="en-US"
hreflang="en-US"
aria-current="true"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md bg-blue-50 text-blue-700">
<svg class="w-5 h-5 text-blue-500" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M5 13l4 4L19 7"></path>
</svg>
<span>English</span>
</a>
<a href="/ca/"
lang="ca"
hreflang="ca"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Català</span>
</a>
<a href="/cs/"
lang="cs-CZ"
hreflang="cs-CZ"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Čeština</span>
</a>
<a href="/da/"
lang="da"
hreflang="da"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Dansk</span>
</a>
<a href="/de/"
lang="de-DE"
hreflang="de-DE"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Deutsch</span>
</a>
<a href="/el/"
lang="el"
hreflang="el"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Greek</span>
</a>
<a href="/es/"
lang="es-US"
hreflang="es-US"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Español</span>
</a>
<a href="/fi/"
lang="fi"
hreflang="fi"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Suomi</span>
</a>
<a href="/fr/"
lang="fr-FR"
hreflang="fr-FR"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Français</span>
</a>
<a href="/he/"
lang="he"
hreflang="he"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>עברית</span>
</a>
<a href="/hu/"
lang="hu"
hreflang="hu"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Magyar</span>
</a>
<a href="/id/"
lang="id-ID"
hreflang="id-ID"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Bahasa Indonesia</span>
</a>
<a href="/it/"
lang="it-IT"
hreflang="it-IT"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Italiano</span>
</a>
<a href="/ja/"
lang="ja"
hreflang="ja"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>日本語</span>
</a>
<a href="/ko/"
lang="ko-KR"
hreflang="ko-KR"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>한국어</span>
</a>
<a href="/pl/"
lang="pl"
hreflang="pl"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Polish</span>
</a>
<a href="/pt-br/"
lang="pt-BR"
hreflang="pt-BR"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Português do Brasil</span>
</a>
<a href="/ru/"
lang="ru-RU"
hreflang="ru-RU"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Русский</span>
</a>
<a href="/si/"
lang="si"
hreflang="si"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>සිංහල</span>
</a>
<a href="/sr/"
lang="sr"
hreflang="sr"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Srpski</span>
</a>
<a href="/sv/"
lang="sv"
hreflang="sv"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Svenska</span>
</a>
<a href="/ta/"
lang="ta"
hreflang="ta"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>தமிழ்</span>
</a>
<a href="/th/"
lang="th"
hreflang="th"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Thai</span>
</a>
<a href="/tr/"
lang="tr-TR"
hreflang="tr-TR"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Türkçe</span>
</a>
<a href="/uk/"
lang="uk-UA"
hreflang="uk-UA"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Українська</span>
</a>
<a href="/vi/"
lang="vi"
hreflang="vi"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>Tiếng Việt</span>
</a>
<a href="/zh-cn/"
lang="zh-Hans-CN"
hreflang="zh-Hans-CN"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>简体中文</span>
</a>
<a href="/zh-tw/"
lang="zh-Hant-TW"
hreflang="zh-Hant-TW"
class="flex items-start lg:items-center space-x-2 px-4 py-2 rounded-md text-gray-700 hover:bg-gray-50">
<span>繁體中文</span>
</a>
</div>
</div>
</div>
</div>
<div id="main-header" class="container mx-auto px-4 py-3 md:py-4">
<a id="skiplink"
href="#main-content"
class="sr-only focus:not-sr-only focus:absolute focus:top-4 focus:left-4 bg-white px-4 py-2 rounded-md shadow-md">
Skip navigation links
</a>
<div class="flex items-center justify-between">
<a class="site-logo shrink-0" href="/">
<img src="/images/letsencrypt-logo-horizontal.svg" alt="Let's Encrypt">
</a>
<button
id="mobile-menu-toggle"
class="md:hidden p-2 rounded-md text-gray-600 hover:text-gray-900 hover:bg-gray-100 focus:outline-none focus:ring-2 focus:ring-blue-500"
aria-expanded="false"
aria-controls="main-nav"
aria-label="Toggle menu">
<svg class="w-6 h-6" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 6h16M4 12h16M4 18h16"></path>
</svg>
</button>
<nav id="main-nav"
class="md:flex absolute md:!relative inset-x-0 top-full md:top-auto bg-white md:bg-transparent border-b md:border-b-0 border-gray-200 shadow-lg md:shadow-none md:grow md:ml-8 z-50">
<div class="container mx-auto">
<div class="md:p-0 flex justify-center">
<ul class="site-main-nav flex flex-col md:flex-row md:justify-center md:gap-5 py-2 md:py-0 w-full" role="menubar">
<li class="relative " role="none">
<a role="menuitem" class="nav-item" href="/docs/">
Documentation
</a>
</li>
<li class="relative " role="none">
<a role="menuitem" class="nav-item" href="https://community.letsencrypt.org/">
Get Help
</a>
</li>
<li class="relative " role="none">
<a role="menuitem" class="nav-item" href="/blog/">
Blog
</a>
</li>
<li class="relative group" role="none">
<div class="dropdown-container">
<button
class="nav-item flex items-center justify-between w-full md:w-auto space-x-1"
aria-expanded="false"
aria-controls="dropdown-donate"
role="menuitem">
<span>Donate</span>
<svg class="w-4 h-4 text-gray-500" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M19 9l-7 7-7-7"></path>
</svg>
</button>
<ul id="dropdown-donate"
class="hidden static md:absolute left-0 w-full md:w-auto py-1 bg-white md:rounded-md md:shadow-lg md:border md:border-gray-200 z-60 mt-1"
role="menu">
<li class="relative" role="none">
<a href="https://www.abetterinternet.org/sponsor/"
class="nav-dropdown-item"
role="menuitem">
Become a Sponsor
</a>
</li>
<li class="relative" role="none">
<a href="https://www.abetterinternet.org/sponsors/"
class="nav-dropdown-item"
role="menuitem">
Current Sponsors &amp; Funders
</a>
</li>
<li class="relative" role="none">
<a href="/getinvolved/"
class="nav-dropdown-item"
role="menuitem">
Get Involved
</a>
</li>
<li class="relative" role="none">
<a href="/donate/"
class="nav-dropdown-item"
role="menuitem">
Donate
</a>
</li>
</ul>
</div>
</li>
<li class="relative group" role="none">
<div class="dropdown-container">
<button
class="nav-item flex items-center justify-between w-full md:w-auto space-x-1"
aria-expanded="false"
aria-controls="dropdown-about"
role="menuitem">
<span>About Us</span>
<svg class="w-4 h-4 text-gray-500" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M19 9l-7 7-7-7"></path>
</svg>
</button>
<ul id="dropdown-about"
class="hidden static md:absolute left-0 w-full md:w-auto py-1 bg-white md:rounded-md md:shadow-lg md:border md:border-gray-200 z-60 mt-1"
role="menu">
<li class="relative" role="none">
<a href="/about/"
class="nav-dropdown-item"
role="menuitem">
Let&#39;s Encrypt
</a>
</li>
<li class="relative" role="none">
<a href="/docs/faq/"
class="nav-dropdown-item"
role="menuitem">
Frequently Asked Questions (FAQ)
</a>
</li>
<li class="relative" role="none">
<a href="/repository/"
class="nav-dropdown-item"
role="menuitem">
Policy and Legal Repository
</a>
</li>
<li class="relative" role="none">
<a href="https://letsencrypt.status.io/"
class="nav-dropdown-item"
role="menuitem">
Service Status
</a>
</li>
<li class="relative" role="none">
<a href="/stats/"
class="nav-dropdown-item"
role="menuitem">
Statistics
</a>
</li>
<li class="relative" role="none">
<a href="/contact/"
class="nav-dropdown-item"
role="menuitem">
Contact
</a>
</li>
<li class="relative" role="none">
<a href="https://www.abetterinternet.org/careers/"
class="nav-dropdown-item"
role="menuitem">
Careers
</a>
</li>
<li class="relative" role="none">
<a href="https://www.abetterinternet.org/annual-reports/"
class="nav-dropdown-item"
role="menuitem">
Annual reports
</a>
</li>
<li class="relative" role="none">
<a href="https://www.abetterinternet.org/about/"
class="nav-dropdown-item"
role="menuitem">
Internet Security Research Group (ISRG)
</a>
</li>
</ul>
</div>
</li>
<li class="block md:hidden pt-2.5"><a class="btn btn-yellow btn-sm" href="/donate/">Donate Now</a></li>
</ul>
</div>
</div>
</nav>
<a class="btn btn-yellow hidden md:block shrink-0" href="/donate/">Donate Now</a>
</div>
</div>
</header>
<div id="main-content"></div>
<main>
<section class="post-hero py-8">
<div class="narrow-container mx-auto">
<div class="post-hero__inner border-l border-le-yellow box-border pl-4 flex flex-col gap-2">
<div class="post-hero__breadcrumb">
<a class="" href="/blog/">Blog</a>
</div>
<h1 class="post-hero__title mb-0">How Let&#39;s Encrypt Runs CT Logs</h1>
<div class="post-hero__meta">
By Phil Porada, Site Reliability Engineer ·
<time datetime="2019-11-20T00:00:00&#43;00:00">November 20, 2019</time>
</div>
</div>
</div>
</section>
<div class="narrow-container mx-auto py-8 prose-content">
<p>Let’s Encrypt <a href='https://letsencrypt.org/2019/05/15/introducing-oak-ct-log'>launched a Certificate Transparency (CT) log</a> this past spring. We’re excited to share how we built it in hopes that others can learn from what we did. CT has quickly become an important piece of Internet security infrastructure, but unfortunately it’s not trivial to run a good log. The more the CT community can share about what has been done, the better the ecosystem will be.</p>
<p><a href="https://sectigo.com/">Sectigo</a> and <a href="https://aws.amazon.com/">Amazon Web Services</a> have generously provided support to cover a significant portion of the cost of running our CT log. “Sectigo is proud to sponsor the Let’s Encrypt CT Log. We believe this initiative will provide much-needed reinforcement of the CT ecosystem,” said Ed Giaquinto, Sectigo’s CIO.</p>
<p>For more background information about CT and how it works, we recommend reading “<a href="https://www.certificate-transparency.org/how-ct-works">How Certificate Transparency Works</a>.”</p>
<p>If you have questions about any of what we’ve written here, feel free to ask on our <a href="https://community.letsencrypt.org/">community forums</a>.</p>
<h1 id="objectives">Objectives</h1>
<ol>
<li><em>Scale:</em> Let’s Encrypt issues over <a href="https://letsencrypt.org/stats/#daily-issuance">1 million certificates per day</a>, and that number grows each month. We want our log to consume our certificates as well as those from other CAs, so we need to be able to handle as many as 2 million or more certificates per day. To support this ever-increasing number of certificates, CT software and infrastructure need to be architected for scale.</li>
<li><em>Stability and Compliance:</em> We target 99% uptime, with no outage lasting longer than 24 hours, in compliance with the <a href="https://github.com/chromium/ct-policy/blob/master/log_policy.md">Chromium</a> and <a href="https://support.apple.com/en-gb/HT205280">Apple</a> CT policies.</li>
<li><em>Sharding:</em> Best practice for a CT log is to break it into several temporal shards. For more information on temporal sharding, check out these <a href="https://www.digicert.com/blog/scaling-certificate-transparency-logs-temporal-sharding/">blog</a> <a href="https://www.venafi.com/blog/how-temporal-sharding-helps-ease-challenge-growing-log-scale">posts</a>.</li>
<li><em>Low Maintenance:</em> Staff time is expensive, we want to minimize the amount of time spent maintaining infrastructure.</li>
</ol>
<h1 id="system-architecture">System Architecture</h1>
<p><img src="/images/2019-11-20-ct-architecture.png" alt="System Architecture Diagram"></p>
<h1 id="staging-and-production-logs">Staging and Production Logs</h1>
<p>We run two equivalent logs, one for staging and one for production. Any changes we plan to make to the production log are first deployed to the staging log. This is critical for making sure that updates and upgrades don’t cause problems before being deployed to production. You can find access details for these logs in our <a href="https://letsencrypt.org/docs/ct-logs/">documentation</a>.</p>
<p>We keep the staging log continually under production-level load so that any scale-related problems manifest there first. We also use the staging CT log to submit certificates from our staging CA environment, and make it available for use by other CAs’ staging environments.</p>
<p>As a point of clarification, we consider a log to be comprised of several temporal shards. While each shard is technically a separate log, it makes sense to conceptualize the shards as belonging to a single log.</p>
<h1 id="amazon-web-services-aws">Amazon Web Services (AWS)</h1>
<p>We decided to run our CT logs on AWS for two reasons.</p>
<p>One consideration for us was cloud provider diversity. Since there are relatively few trusted logs in the ecosystem, we don’t want multiple logs to go down due to a single cloud provider outage. At the time we made the decision there were logs running on Google and Digital Ocean infrastructure, as well as self-hosted. We were not aware of any on AWS (in hindsight we may have missed the fact that Digicert had started using AWS for logs). If you’re thinking about setting up a trusted log for CAs to use, please consider cloud provider diversity.</p>
<p>Additionally, AWS provides a solid set of features and our team has experience using it for other purposes. We had little doubt that AWS was up to the task.</p>
<h1 id="terraform">Terraform</h1>
<p>Let’s Encrypt uses Hashicorp <a href="https://www.terraform.io/">Terraform</a> for a number of cloud-based projects. We were able to bootstrap our CT log infrastructure by reusing our existing Terraform code. There are roughly 50 components in our CT deployments, including EC2, RDS, EKS, IAM, security groups, and routing. Centrally managing this code allows our small team to reproduce a CT infrastructure in any Amazon region of the globe, prevent configuration drift, and easily test infrastructure changes.</p>
<h1 id="database">Database</h1>
<p>We chose to use MariaDB for our CT log database because we have extensive experience using it to run our certificate authority. MariaDB has scaled well on our journey to becoming the largest publicly trusted certificate authority.</p>
<p>We chose to have our MariaDB instances managed by Amazon RDS because RDS provides synchronous writes to standby cluster members. This allows for automatic database failover and ensures database consistency. Synchronous writes to database replicas are essential for a CT log. One missed write during a database failover can mean a certificate was not included as promised, and could lead to the log being disqualified. Having RDS manage this for us reduces complexity and saves staff time. We are still responsible for managing the database performance, tuning, and monitoring.</p>
<p>It’s important to calculate the necessary amount of storage for a CT log database carefully. Too little storage can result in needing to undertake time-consuming and potentially risky storage migrations. Too much storage can result in unnecessarily high costs.</p>
<p>A back of the napkin storage estimation is 1TB per 100 million entries. We expect to need to store 1 billion certificates and precertificates per annual temporal shard, for which we would need 10TB. We considered having separate database storage per annual temporal shard, with approximately 10TB allocated to each, but that was cost prohibitive. We decided to create a 12TB storage block per log (10TB plus some breathing room), which is duplicated for redundancy by RDS. Each year we plan to freeze the previous year’s shard and move it to a less expensive serving infrastructure, reclaiming its storage for our live shards.</p>
<p>We use 2x db.r5.4xlarge instances for RDS for each CT log. Each of these instances contains 8 CPU cores and 128GB of RAM.</p>
<h1 id="kubernetes">Kubernetes</h1>
<p>After trying a few different strategies for managing application instances, we decided to use Kubernetes. There is a hefty learning curve for Kubernetes and the decision was not made lightly. This was our first project making use of Kubernetes, and part of the reason we went with it was to gain experience and possibly apply that knowledge to other parts of our infrastructure in the future.</p>
<p>Kubernetes provides abstractions for operators such as <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#use-case">deployments</a>, <a href="https://kubernetes.io/docs/tutorials/kubernetes-basics/scale/">scaling</a>, and <a href="https://kubernetes.io/docs/concepts/services-networking/service/#motivation">service discovery</a> that we would not have to build ourselves. We utilized the example Kubernetes deployment manifests in the <a href="https://github.com/google/trillian/">Trillian repository</a> to assist with our deployment.</p>
<p>A Kubernetes cluster is comprised of two main components: the control plane which handles the Kubernetes APIs, and worker nodes where containerized applications run. We chose to have Amazon EKS manage our Kubernetes control plane.</p>
<p>We use 4x c5.2xlarge EC2 instances for the worker node pool for each CT log. Each of these instances contains 8 CPU cores and 16GB of RAM.</p>
<h1 id="application-software">Application Software</h1>
<p>There are three main CT components that we run in a Kubernetes cluster.</p>
<p>The certificate transparency front end, or <a href="https://github.com/google/certificate-transparency-go">CTFE</a>, provides <a href="https://tools.ietf.org/html/rfc6962">RFC 6962</a> endpoints and translates them to gRPC API requests for the Trillian backend.</p>
<p><a href="https://github.com/google/trillian">Trillian</a> describes itself as a “transparent, highly scalable and cryptographically verifiable data store.” Essentially, Trillian implements a generalized verifiable data store via a Merkle tree that can be used as the back-end for a CT log via the CTFE. Trillian consists of two components: the log signer and log server. The <a href="https://github.com/google/trillian/blob/master/docs/images/LogDesign.png">log signer’s function</a> is to periodically process incoming leaf data (certificates in the case of CT) and incorporate them into a Merkle tree. The log server retrieves objects from a Merkle tree in order to fulfill CT API monitoring requests.</p>
<h1 id="load-balancing">Load Balancing</h1>
<p>Traffic enters the CT log through an Amazon ELB which is mapped to a Kubernetes Nginx ingress service. The ingress service balances traffic amongst multiple Nginx pods. The Nginx pods proxy traffic to the CTFE service which balances that traffic to CTFE pods.</p>
<p>We employ IP and user agent based rate limiting at this Nginx layer.</p>
<h1 id="logging-and-monitoring">Logging and Monitoring</h1>
<p>Trillian and the CTFE expose <a href="https://prometheus.io/">Prometheus</a> metrics which we transform into monitoring dashboards and alerts. It is essential to set a <a href="https://en.wikipedia.org/wiki/Service-level_objective">Service Level Objective</a> for the CT log endpoints above the 99% uptime dictated by CT policy to ensure that your log is trusted. A FluentD pod running in a DaemonSet ships logs to centralized storage for further analysis.</p>
<p>We developed a free and open source tool named <a href="https://github.com/letsencrypt/ct-woodpecker">ct-woodpecker</a> that is used to monitor various aspects of log stability and correctness. This tool is an important part of how we ensure we’re meeting our service level objectives. Each ct-woodpecker instance runs externally from Amazon VPCs containing CT logs.</p>
<h1 id="future-efficiency-improvements">Future Efficiency Improvements</h1>
<p>Here are some ways we may be able to improve the efficiency of our system in the future:</p>
<ul>
<li>Trillian stores a copy of each certificate chain, including many duplicate copies of the same intermediate certificates. Being able to de-duplicate these in Trillian would significantly reduce storage costs. We’re planning to look into whether this is possible and reasonable.</li>
<li>See if we can successfully use a cheaper form of storage than IO1 block storage and provisioned IOPS.</li>
<li>See if we can reduce the Kubernetes worker EC2 instance size or use fewer EC2 instances.</li>
</ul>
<h1 id="support-let-s-encrypt">Support Let’s Encrypt</h1>
<p>We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization is interested in learning more about <a href="https://www.abetterinternet.org/sponsor/">sponsorship</a>, please email us at <a href="mailto:sponsor@letsencrypt.org">sponsor@letsencrypt.org</a>. We ask that you make an <a href="https://letsencrypt.org/donate/">individual contribution</a> if it is within your means.</p>
</div>
</main>
<footer class="bg-white text-gray-800 pt-10 pb-4">
<div class="container">
<div class="grid md:grid-cols-2 gap-12 lg:gap-24">
<div>
<img src="/images/ISRG-Logo-Blue.svg"
alt="Internet Security Research Group (ISRG)"
class="h-12 w-auto mb-6">
<p class="mb-6">Let's Encrypt is a free, automated, and open Certificate Authority brought to you by the nonprofit <a href="https://www.abetterinternet.org/">Internet Security Research Group (ISRG)</a>. Read all about our nonprofit work this year in our <a href="https://www.abetterinternet.org/annual-reports/">2025 Annual Report</a>.
</p>
<div class="grid grid-cols-2 gap-6 mt-10">
<div>
<div class="text-eyebrow text-eyebrow-sm">Legal Address</div>
<div itemscope itemtype="http://schema.org/PostalAddress">
<span itemprop="streetAddress">548 Market St, PMB 77519</span><br />
<span itemprop="addressLocality">San Francisco</span>,
<span itemprop="addressRegion">CA</span>
<span itemprop="postalCode">94104-5401</span><br />
<span itemprop="addressCountry">USA</span>
</div>
</div>
<div>
<div class="text-eyebrow text-eyebrow-sm">Send all mail or inquiries to:</div>
<div itemscope itemtype="http://schema.org/PostalAddress">
<span itemprop="streetAddress">PO Box 18666</span><br />
<span itemprop="addressLocality">Minneapolis</span>,
<span itemprop="addressRegion">MN</span>
<span itemprop="postalCode">55418-0666</span><br />
<span itemprop="addressCountry">USA</span>
</div>
</div>
</div>
</div>
<div class="pt-0 md:pt-[70px]">
<div class="border border-gray-300 rounded-lg pt-3 pb-0 px-2">
<h4 class="text-eyebrow mb-5">Subscribe for email updates about Let&#39;s Encrypt and other ISRG projects</h4>
<iframe id="newsletter-iframe" src="https://outreach.abetterinternet.org/l/1011011/2025-01-14/31v6r" title="newsletter" class="w-full border-0 overflow-hidden"></iframe>
<script>
(function() {
window.addEventListener('message', function(e) {
if (e.origin !== 'https://outreach.abetterinternet.org') return;
if (e.data && typeof e.data === 'object' && e.data.type === 'resize' && e.data.height) {
document.getElementById('newsletter-iframe').setAttribute('height', e.data.height + 20);
}
});
})();
</script>
</div>
</div>
</div>
</div>
</footer>
<div class="footer-copyright-bar bg-le-blue text-white py-6">
<div class="container">
<div class="flex flex-col md:flex-row md:justify-between items-center justify-center">
<div class="mb-4 text-center md:text-left md:mb-0">
&copy; 2026 <a href="https://www.abetterinternet.org" target="_blank" rel="noopener noreferrer">Internet Security Research Group</a>
</div>
<ul class="flex flex-wrap gap-4 justify-center md:justify-end">
<li>
<a href="https://github.com/letsencrypt" target="_blank" rel="noopener noreferrer">
<span>GitHub</span>
</a>
</li>
<li>
<a href="https://www.linkedin.com/company/lets-encrypt/" target="_blank" rel="noopener noreferrer">
<span>LinkedIn</span>
</a>
</li>
<li>
<a href="https://www.abetterinternet.org/terms-of-service" target="_blank" rel="noopener noreferrer">
<span>Terms</span>
</a>
</li>
<li>
<a href="/privacy/">
<span>Privacy Policy</span>
</a>
</li>
<li>
<a href="https://www.abetterinternet.org/trademarks" target="_blank" rel="noopener noreferrer">
<span>Trademark Policy</span>
</a>
</li>
</ul>
</div>
</div>
</div>
<script src="/js/main.03c09aa527a24dbba751011c441d5b78f1fe35f96ccf57317abb09232397e896843267c6489270f0430ccf29681e6a4891bd9af417641f43ffa154d9fbf8a075.js" integrity="sha512-A8CapSeiTbunUQEcRB1bePH&#43;Nflsz1cxersJIyOX6JaEMmfGSJJw8EMMzyloHmpIkb2a9BdkH0P/oVTZ&#43;/igdQ=="></script>
</body>
</html>