Files
nexus/sreweekly/articles/379/02-the-story-behind-last-week-s-let-s-encrypt-downtime.html
2026-09-12 17:23:01 +08:00

329 lines
31 KiB
HTML

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xml:lang="en" lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>The Story Behind Last Week's Let's Encrypt Downtime</title><link rel="stylesheet" type="text/css" href="/css/style.css" /><link rel="icon" sizes="16x16" type="image/png" href="/favicon.png" /><link rel="icon" sizes="32x32" type="image/png" href="/art/favicon-32x32.png" /><link rel="icon" sizes="96x96" type="image/png" href="/art/favicon-96x96.png" /><link rel="apple-touch-icon" sizes="152x152" href="/art/appleicon-lightbg-152x152.png" /><link rel="apple-touch-icon" sizes="167x167" href="/art/appleicon-lightbg-167x167.png" /><link rel="apple-touch-icon" sizes="180x180" href="/art/appleicon-lightbg-180x180.png" /><link rel="start" title="Home" href="/" /><link rel="alternate" type="application/atom+xml" title="Blog" href="/blog/feed" /><meta name="generator" content="Terrapin XSLT" /><meta name="author" content="Andrew Ayer" /><meta name="copyright" content="Copyright 2026 Andrew Ayer" /><meta name="robots" content="noarchive" /><meta name="viewport" content="width=device-width, initial-scale=1" /><meta name="twitter:card" content="summary" /><meta name="twitter:site" content="@__agwa" /><meta name="twitter:creator" content="@__agwa" /><meta name="twitter:title" content="The Story Behind Last Week's Let's Encrypt Downtime" /><meta name="twitter:description" content="How I detected that Let's Encrypt issued 645 non-compliant certificates" /><!--[if IE]>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<![endif]--><!--Edition = 'B'--><!--Page ID = '/blog/post/last_weeks_lets_encrypt_downtime'--><!--Date/time = '2026-09-12T05:48:56Z'--></head><body><div id="root"><p id="skiptocontent"><a href="#content" accesskey="c">Skip to Content [alt-c]</a></p><div id="header"><div class="inner_header"><h1><a class="logo" href="/"><svg xmlns="http://www.w3.org/2000/svg" role="img" aria-label="A.G.W.A. logo" width="54" height="30"><use xlink:href="/art/symbols.svg#logo"></use></svg></a><span class="fluff"> </span><a class="title" href="/"><span>Andrew Ayer</span></a></h1><h2>Sections</h2><ul class="tabs"><li><a href="/blog/">Blog</a></li><li><a href="/projects/">Projects</a></li><li><a href="/photos/">Photos</a></li></ul></div><div class="header_background"><svg xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Drawing of an oak tree" version="1.1" viewBox="0 0 4200 114.01"><use class="header_tree" xlink:href="/art/symbols.svg#tree"></use><use class="header_ground" xlink:href="/art/symbols.svg#ground"></use></svg></div></div><div id="content"><div class="blog_page"><div class="blog_sidebar"><div><h2>Blog</h2><ul><li><a href="/blog/">Latest</a></li> <li><a href="/blog/popular">Popular</a></li> <li><a href="/blog/index">Archives</a></li> <li><a href="/blog/feed">RSS</a></li></ul></div></div><div class="blog_content"><div class="blog_post"><p class="date">June 22, 2023</p><h2>The Story Behind Last Week's Let's Encrypt Downtime</h2><div class="content">
<p>
Last Thursday (June 15th, 2023), Let's Encrypt <a href="https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/648b36899c7c1405303ea8c4" rel="external">went down for about
an hour</a>, during which time it was not possible to obtain certificates
from Let's Encrypt. Immediately prior to the outage, Let's Encrypt issued
645 certificates which did not work in Chrome or Safari. In this post,
I'm going to explain what went wrong and how I detected it.
</p>
<h4>The Law of Precertificates</h4>
<p>
Before I can explain the incident, we need to talk about <a href="https://certificate.transparency.dev/" rel="external">Certificate
Transparency</a>. Certificate Transparency (CT) is a system for putting
certificates issued by publicly-trusted CAs, such as Let's Encrypt,
in public, append-only logs. Certificate authorities have a tremendous amount
of power, and if they misuse their power by issuing certificates that they shouldn't, traffic to HTTPS websites
could be intercepted by attackers. Historically, CAs have not used their power well, and Certificate
Transparency is an effort to fix that by letting anyone examine the certificates
that CAs issue.
</p>
<p>
A key concept in Certificate Transparency is the "precertificate".
Before issuing a certificate, the certificate authority creates a
precertificate, which contains all of the information that will
be in the certificate, plus a "poison extension" that prevents the
precertificate from being used like a real certificate. The CA submits
the precertificate to multiple Certificate Transparency logs. Each log
returns a Signed Certificate Timestamp (SCT), which is a signed statement
acknowledging receipt of the precertificate and promising to publish the
precertificate in the log for anyone to download. The CA takes all of
the SCTs and embeds them in the certificate. When a CT-enforcing browser
(like Chrome or Safari) validates the certificate, it makes sure that
the certificate embeds a sufficient number of SCTs from trustworthy logs.
This doesn't prevent the browser from accepting a malicious certificate,
but it does ensure that the precertificate is in public logs, allowing
the attack to be detected and action taken against the CA.
</p>
<p>
The certificate itself may or may not end up in CT logs. Some CAs,
notably Let's Encrypt and Sectigo, automatically submit their
certificates. Certificates from other CAs only end up in logs if
someone else finds and submits them. <strong>Since only the precertificate is
guaranteed to be logged, it is essential that a precertificate be treated
as incontrovertible proof that a certificate containing the same data exists.</strong>
When someone finds a precertificate for a malicious or non-compliant certificate,
the CA can't be allowed to evade responsibility by saying "just kidding,
we never actually issued the real certificate" (and boy, have they tried).
Otherwise, CT would be useless.
</p>
<p>
There are two ways a CA could create a certificate. They could
take the precertificate, remove the poison extension, add the SCTs,
and re-sign it. Or, they could create the certificate from scratch,
making sure to add the same data, in the same order, as used in the
precertificate.
</p>
<p>
The first way is robust because it's guaranteed to produce a certificate
which matches the precertificate. At least one CA, Sectigo, <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1838667#c2" rel="external">uses this
approach</a>. Let's Encrypt uses the second approach. You can probably see
where this is going...
</p>
<h4>The Let's Encrypt incident</h4>
<p>
On June 15, 2023, Let's Encrypt deployed a <a href="https://community.letsencrypt.org/t/small-change-to-end-entity-certificates-cps-url-and-oid-will-not-be-included-from-june-15/198206" rel="external">planned change</a> to their certificate
configuration which altered the contents of the Certificate Policies extension from:
</p>
<code class="block pre" style="white-space: pre;">X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.letsencrypt.org</code>
<p>to:</p>
<code class="block pre" style="white-space: pre;">X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1</code>
<p>
Unfortunately, any certificate which was requested while the change
was being rolled out could have its precertificate and certificate created
with different configurations. For example, when Let's Encrypt issued
the certificate with serial number 03:e2:26:7b:78:6b:7e:33:83:17:dd:d6:2e:76:4f:cb:3c:71,
the <a href="https://understandingwebpki.com/?cert=MIIFGjCCBAKgAwIBAgISA%2BIme3hrfjODF93WLnZPyzxxMA0GCSqGSIb3DQEBCwUAMDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJSMzAeFw0yMzA2MTUxNDM2MTZaFw0yMzA5MTMxNDM2MTVaMB4xHDAaBgNVBAMMEyouN2FjbnIubW9uZ29kYi5uZXQwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCjLiLXI%2FmTBSEkSKVucC3NcnXGu%2FM2qwLIk1uenifnoNMmdJmEyp%2BoWFUSn9rIXtHw27YTlJLRRYLSIzqqujDV5PmXzFrSJ%2F9JrgIbNUowaVF3j9bf1%2BNPENEH81RnNGevtKUN5NoEo3fAmZaMWrGjWioNnpIsegSjvvuHeqMqC7SNrGSvtKLBiPkObL5oScPYj%2FcHzt3RYJ17ru6xWgUDV6aqvEblrxcXvPmd%2F1SxB3Vkdkc%2BbCuSLSNM%2FNmcET0YUhWizanjodJarpYJRuW1SjGmPda0jBAQZQDPmZHCEgwTBcCEIg5J3XzAfFUZPPlTVgE%2B7Mbjd%2FDK7iz46D0uHOigVTZto3lPYRdRiyVFNUMAN0GLAlkaJ7Td0FnAxvhE74lSjI7lFqDNtiyA8ovp%2FJbKfPmnvfH%2BfQa7vEFbR5H9v4UZt0XLeI6WdV4pYoCwuK5mfr0NQLCy%2F015OAU8WF4MLM%2BFyt%2BGG%2BsOk2Maz6ysAShMOvdNH7B3GSn65xBVgBxlPWyYpodW9SS1NSVgrgbKMg0yHzx%2FPdosQehyh9p6OpuTaeEi2iQgyTODKGHX%2BcmjzUx0iCG2ByC9bvMo32eZXiC%2BitZCaHb0FGXh%2BK7UcOCsvsi7NLGRngVKK7u7gZmPu4UkVUBpF3jz%2FOK3OsudHcflZIGd6nf8w4lp0wIDAQABo4IBPDCCATgwDgYDVR0PAQH%2FBAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBREcOX3VXl7%2BuM7aqTQ%2FconiJsAAjAfBgNVHSMEGDAWgBQULrMXt1hWy65QCUDmH6%2BdixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iub3JnLzA4BgNVHREEMTAvghgqLjdhY25yLm1lc2gubW9uZ29kYi5uZXSCEyouN2FjbnIubW9uZ29kYi5uZXQwEwYDVR0gBAwwCjAIBgZngQwBAgEwEwYKKwYBBAHWeQIEAwEB%2FwQCBQAwDQYJKoZIhvcNAQELBQADggEBALIUrHns6TWfT%2FkfJ60D9R1Ek4YGB%2FjVsrh2d3uiIU2hiRBBjgDkCLyKd7oXM761uXX3LL4H4JPegqTrZAPO88tUtzBSb3IF4yA0o1NWhE6ceLnBk9fl5TRCC8QASliApsOigDgRi1VFmyFOHpHnVZdbpPucy6T%2BCdKXKfj4iNw%2BaOZcoQxJ70XECXxQbdqJ7VdYf0B%2Bwtk5HZU8cuVVCj1i%2FiDv1zqITCzaavbz870QugiHO%2F8rj2ctrA07SX3Ovs4JGbCGuMzlpxeIFtQDWVufVbu1ZZltzPlSHFqv6mPKW9stYtt8JCjmPwNW6UdrlBtNgvFgkgDpz%2BQ6%2FVu%2Bu7g%3D" rel="external">precertificate</a> contained the new Certificate Policies extension,
and the <a href="https://understandingwebpki.com/?cert=MIIGRjCCBS6gAwIBAgISA%2BIme3hrfjODF93WLnZPyzxxMA0GCSqGSIb3DQEBCwUAMDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJSMzAeFw0yMzA2MTUxNDM2MTZaFw0yMzA5MTMxNDM2MTVaMB4xHDAaBgNVBAMMEyouN2FjbnIubW9uZ29kYi5uZXQwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCjLiLXI%2FmTBSEkSKVucC3NcnXGu%2FM2qwLIk1uenifnoNMmdJmEyp%2BoWFUSn9rIXtHw27YTlJLRRYLSIzqqujDV5PmXzFrSJ%2F9JrgIbNUowaVF3j9bf1%2BNPENEH81RnNGevtKUN5NoEo3fAmZaMWrGjWioNnpIsegSjvvuHeqMqC7SNrGSvtKLBiPkObL5oScPYj%2FcHzt3RYJ17ru6xWgUDV6aqvEblrxcXvPmd%2F1SxB3Vkdkc%2BbCuSLSNM%2FNmcET0YUhWizanjodJarpYJRuW1SjGmPda0jBAQZQDPmZHCEgwTBcCEIg5J3XzAfFUZPPlTVgE%2B7Mbjd%2FDK7iz46D0uHOigVTZto3lPYRdRiyVFNUMAN0GLAlkaJ7Td0FnAxvhE74lSjI7lFqDNtiyA8ovp%2FJbKfPmnvfH%2BfQa7vEFbR5H9v4UZt0XLeI6WdV4pYoCwuK5mfr0NQLCy%2F015OAU8WF4MLM%2BFyt%2BGG%2BsOk2Maz6ysAShMOvdNH7B3GSn65xBVgBxlPWyYpodW9SS1NSVgrgbKMg0yHzx%2FPdosQehyh9p6OpuTaeEi2iQgyTODKGHX%2BcmjzUx0iCG2ByC9bvMo32eZXiC%2BitZCaHb0FGXh%2BK7UcOCsvsi7NLGRngVKK7u7gZmPu4UkVUBpF3jz%2FOK3OsudHcflZIGd6nf8w4lp0wIDAQABo4ICaDCCAmQwDgYDVR0PAQH%2FBAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBREcOX3VXl7%2BuM7aqTQ%2FconiJsAAjAfBgNVHSMEGDAWgBQULrMXt1hWy65QCUDmH6%2BdixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iub3JnLzA4BgNVHREEMTAvghgqLjdhY25yLm1lc2gubW9uZ29kYi5uZXSCEyouN2FjbnIubW9uZ29kYi5uZXQwTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEEBgorBgEEAdZ5AgQCBIH1BIHyAPAAdgC3Pvsk35xNunXyOcW6WPRsXfxCz3qfNcSeHQmBJe20mQAAAYi%2Fs0QZAAAEAwBHMEUCID4vc7PNWNauTkmkS7CqSwdiyOV%2BLYIT9g8KygWW4atTAiEA6Re4Cz7BsEMi%2B%2FU8G%2Br9LmqbqwGXGS4mXG7RiEfeQEcAdgB6MoxU2LcttiDqOOBSHumEFnAyE4VNO9IrwTpXo1LrUgAAAYi%2Fs0RQAAAEAwBHMEUCIQD95SqDycwXGZ%2BJKBUVBR%2BhBxn4BRIQ7EPIaMTI%2F%2B854gIgDpJm5BFX9vKUf5tKWn9f%2FFagktt5J6hPnrmURSV%2FegAwDQYJKoZIhvcNAQELBQADggEBAKWyDSRmiM9N%2B2AhYgRuzh3JnxtvhmEXUBEgwuFnlQyCm5ZvScvWKmw2sqcj%2BgI2UNUxmWjq3PbIVBrTLDEgXtVN%2BJU6HwC4TdYPIB4LzfrWsGY7cc2aaY76YbWlwEyhN9niQLijZORKhZ6HLM7MI76FM7oJ9eZmvnfypjJ7E0J9ek%2Fy7S1wqg5EM%2BQiAf03YcjSxUCyL3%2F%2BEzlYRz65diLh7Eb6gBd58rWLOa1nbgTOFsToAkBE7qR3HymfWysxApDN8x95jDzubbkqiyuk3dvzjn3oouN1H8NsG%2FxYrYmMMwnJ8xul1AJ31ZMxJ9hr29G122DSEaX9smAyyzWhAwM%3D" rel="external">certificate</a> contained the old Certificate Policies extension.
</p>
<p>
This had two consequences:
</p>
<p>
First, this certificate won't work in Chrome or Safari, because its SCTs
are for a precertificate containing different data from the certificate.
Specifically, the SCTs fail signature validation. When logs sign
SCTs, they compute the signature over the data in the precertificate,
and when browsers verify SCTs, they compute the signature over the data
in the certificate. In this case, that data was not the same.
</p>
<p>
Second, remember how I said that precertificates are treated as
incontrovertible proof that a certificate containing the same data exists?
When Let's Encrypt issued a precertificate with the new Certificate
Policies value, it implied that they also issued a certificate with
the new Certificate Policies value. Thus, according to the Law of
Precertificates, Let's Encrypt issued two certificates with
serial number 03:e2:26:7b:78:6b:7e:33:83:17:dd:d6:2e:76:4f:cb:3c:71:
</p>
<ol class="num-1">
<li>A <a href="https://understandingwebpki.com/?cert=MIIGRjCCBS6gAwIBAgISA%2BIme3hrfjODF93WLnZPyzxxMA0GCSqGSIb3DQEBCwUAMDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJSMzAeFw0yMzA2MTUxNDM2MTZaFw0yMzA5MTMxNDM2MTVaMB4xHDAaBgNVBAMMEyouN2FjbnIubW9uZ29kYi5uZXQwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCjLiLXI%2FmTBSEkSKVucC3NcnXGu%2FM2qwLIk1uenifnoNMmdJmEyp%2BoWFUSn9rIXtHw27YTlJLRRYLSIzqqujDV5PmXzFrSJ%2F9JrgIbNUowaVF3j9bf1%2BNPENEH81RnNGevtKUN5NoEo3fAmZaMWrGjWioNnpIsegSjvvuHeqMqC7SNrGSvtKLBiPkObL5oScPYj%2FcHzt3RYJ17ru6xWgUDV6aqvEblrxcXvPmd%2F1SxB3Vkdkc%2BbCuSLSNM%2FNmcET0YUhWizanjodJarpYJRuW1SjGmPda0jBAQZQDPmZHCEgwTBcCEIg5J3XzAfFUZPPlTVgE%2B7Mbjd%2FDK7iz46D0uHOigVTZto3lPYRdRiyVFNUMAN0GLAlkaJ7Td0FnAxvhE74lSjI7lFqDNtiyA8ovp%2FJbKfPmnvfH%2BfQa7vEFbR5H9v4UZt0XLeI6WdV4pYoCwuK5mfr0NQLCy%2F015OAU8WF4MLM%2BFyt%2BGG%2BsOk2Maz6ysAShMOvdNH7B3GSn65xBVgBxlPWyYpodW9SS1NSVgrgbKMg0yHzx%2FPdosQehyh9p6OpuTaeEi2iQgyTODKGHX%2BcmjzUx0iCG2ByC9bvMo32eZXiC%2BitZCaHb0FGXh%2BK7UcOCsvsi7NLGRngVKK7u7gZmPu4UkVUBpF3jz%2FOK3OsudHcflZIGd6nf8w4lp0wIDAQABo4ICaDCCAmQwDgYDVR0PAQH%2FBAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBREcOX3VXl7%2BuM7aqTQ%2FconiJsAAjAfBgNVHSMEGDAWgBQULrMXt1hWy65QCUDmH6%2BdixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iub3JnLzA4BgNVHREEMTAvghgqLjdhY25yLm1lc2gubW9uZ29kYi5uZXSCEyouN2FjbnIubW9uZ29kYi5uZXQwTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEEBgorBgEEAdZ5AgQCBIH1BIHyAPAAdgC3Pvsk35xNunXyOcW6WPRsXfxCz3qfNcSeHQmBJe20mQAAAYi%2Fs0QZAAAEAwBHMEUCID4vc7PNWNauTkmkS7CqSwdiyOV%2BLYIT9g8KygWW4atTAiEA6Re4Cz7BsEMi%2B%2FU8G%2Br9LmqbqwGXGS4mXG7RiEfeQEcAdgB6MoxU2LcttiDqOOBSHumEFnAyE4VNO9IrwTpXo1LrUgAAAYi%2Fs0RQAAAEAwBHMEUCIQD95SqDycwXGZ%2BJKBUVBR%2BhBxn4BRIQ7EPIaMTI%2F%2B854gIgDpJm5BFX9vKUf5tKWn9f%2FFagktt5J6hPnrmURSV%2FegAwDQYJKoZIhvcNAQELBQADggEBAKWyDSRmiM9N%2B2AhYgRuzh3JnxtvhmEXUBEgwuFnlQyCm5ZvScvWKmw2sqcj%2BgI2UNUxmWjq3PbIVBrTLDEgXtVN%2BJU6HwC4TdYPIB4LzfrWsGY7cc2aaY76YbWlwEyhN9niQLijZORKhZ6HLM7MI76FM7oJ9eZmvnfypjJ7E0J9ek%2Fy7S1wqg5EM%2BQiAf03YcjSxUCyL3%2F%2BEzlYRz65diLh7Eb6gBd58rWLOa1nbgTOFsToAkBE7qR3HymfWysxApDN8x95jDzubbkqiyuk3dvzjn3oouN1H8NsG%2FxYrYmMMwnJ8xul1AJ31ZMxJ9hr29G122DSEaX9smAyyzWhAwM%3D" rel="external">certificate</a> containing the old Certificate Policies extension</li>
<li>A certificate containing the new Certificate Policies extension (implied by the existence of the <a href="https://understandingwebpki.com/?cert=MIIFGjCCBAKgAwIBAgISA%2BIme3hrfjODF93WLnZPyzxxMA0GCSqGSIb3DQEBCwUAMDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJSMzAeFw0yMzA2MTUxNDM2MTZaFw0yMzA5MTMxNDM2MTVaMB4xHDAaBgNVBAMMEyouN2FjbnIubW9uZ29kYi5uZXQwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCjLiLXI%2FmTBSEkSKVucC3NcnXGu%2FM2qwLIk1uenifnoNMmdJmEyp%2BoWFUSn9rIXtHw27YTlJLRRYLSIzqqujDV5PmXzFrSJ%2F9JrgIbNUowaVF3j9bf1%2BNPENEH81RnNGevtKUN5NoEo3fAmZaMWrGjWioNnpIsegSjvvuHeqMqC7SNrGSvtKLBiPkObL5oScPYj%2FcHzt3RYJ17ru6xWgUDV6aqvEblrxcXvPmd%2F1SxB3Vkdkc%2BbCuSLSNM%2FNmcET0YUhWizanjodJarpYJRuW1SjGmPda0jBAQZQDPmZHCEgwTBcCEIg5J3XzAfFUZPPlTVgE%2B7Mbjd%2FDK7iz46D0uHOigVTZto3lPYRdRiyVFNUMAN0GLAlkaJ7Td0FnAxvhE74lSjI7lFqDNtiyA8ovp%2FJbKfPmnvfH%2BfQa7vEFbR5H9v4UZt0XLeI6WdV4pYoCwuK5mfr0NQLCy%2F015OAU8WF4MLM%2BFyt%2BGG%2BsOk2Maz6ysAShMOvdNH7B3GSn65xBVgBxlPWyYpodW9SS1NSVgrgbKMg0yHzx%2FPdosQehyh9p6OpuTaeEi2iQgyTODKGHX%2BcmjzUx0iCG2ByC9bvMo32eZXiC%2BitZCaHb0FGXh%2BK7UcOCsvsi7NLGRngVKK7u7gZmPu4UkVUBpF3jz%2FOK3OsudHcflZIGd6nf8w4lp0wIDAQABo4IBPDCCATgwDgYDVR0PAQH%2FBAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBREcOX3VXl7%2BuM7aqTQ%2FconiJsAAjAfBgNVHSMEGDAWgBQULrMXt1hWy65QCUDmH6%2BdixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iub3JnLzA4BgNVHREEMTAvghgqLjdhY25yLm1lc2gubW9uZ29kYi5uZXSCEyouN2FjbnIubW9uZ29kYi5uZXQwEwYDVR0gBAwwCjAIBgZngQwBAgEwEwYKKwYBBAHWeQIEAwEB%2FwQCBQAwDQYJKoZIhvcNAQELBQADggEBALIUrHns6TWfT%2FkfJ60D9R1Ek4YGB%2FjVsrh2d3uiIU2hiRBBjgDkCLyKd7oXM761uXX3LL4H4JPegqTrZAPO88tUtzBSb3IF4yA0o1NWhE6ceLnBk9fl5TRCC8QASliApsOigDgRi1VFmyFOHpHnVZdbpPucy6T%2BCdKXKfj4iNw%2BaOZcoQxJ70XECXxQbdqJ7VdYf0B%2Bwtk5HZU8cuVVCj1i%2FiDv1zqITCzaavbz870QugiHO%2F8rj2ctrA07SX3Ovs4JGbCGuMzlpxeIFtQDWVufVbu1ZZltzPlSHFqv6mPKW9stYtt8JCjmPwNW6UdrlBtNgvFgkgDpz%2BQ6%2FVu%2Bu7g%3D" rel="external">precertificate</a> with the new Certificate Policies extension)</li>
</ol>
<p>
Issuing two certificates with the same serial number is a violation of
the <a href="https://cabforum.org/baseline-requirements/" rel="external">Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates</a>. Consequentially, Let's Encrypt must
revoke the certificate and post a public incident report,
which must be noted on their next audit statement.
</p>
<p>
You might think that it's harsh to treat this as a compliance incident
if Let's Encrypt didn't really issue two certificates with the same
serial number. Unfortunately, they have no way of proving this, and
the whole reason for Certificate Transparency is so we don't have to take CAs
at their word that they aren't issuing certificates that they shouldn't.
Any exception to the Law of Precertificates creates an opening
for a malicious CA to exploit.
</p>
<h4>How I found this</h4>
<p>
My company, <a href="https://sslmate.com/" rel="external">SSLMate</a>, operates a Certificate Transparency monitor called
<a href="https://sslmate.com/certspotter/" rel="external">Cert Spotter</a>, which continuously downloads and indexes the contents of every
Certificate Transparency log. You can use Cert Spotter to get notifications
when a certificate is issued for one of your domains, or search the database
using a <a href="https://sslmate.com/ct_search_api/" rel="external">JSON API</a>.
</p>
<p>
When Cert Spotter ingests a certificate containing embedded SCTs, it
verifies each SCT's signature and audits that the log really published
the precertificate. (If it detects that a log has broken its promise
to publish a precertificate, I'll publicly disclose the SCT and the log
will be distrusted. Happily, Cert Spotter has never found a bogus SCT,
though it has detected logs violating other requirements.)
</p>
<p>
On Thursday, June 15, 2023 at 15:41 UTC, Cert Spotter began sending me
alerts about certificates containing embedded SCTs with
invalid signatures. Since I was getting hundreds of alerts, I decided
to stop what I was doing and investigate.
</p>
<p>
I had received these alerts several times before, and have gotten pretty
good at zeroing in on the problem. When only one SCT in a certificate
has an invalid signature, it probably means that the CT log screwed up.
When all of the embedded SCTs have an invalid signature, it probably
means the CA screwed up. The most common reason is issuing certificates
that don't match the precertificate. So I took one of the affected
certificates and searched for precertificates containing the same serial
number in Cert Spotter's database of every
(pre)certificate ever logged to Certificate Transparency. Decoding the
certificate and precertificate with the openssl command immediately
revealed the different Certificate Policies extension.
</p>
<p>
Since I was continuing to get alerts from Cert Spotter about invalid
SCT signatures, I quickly fired off an email to Let's Encrypt's problem
reporting address alerting them to the problem.
</p>
<p>
I sent the email at 15:52 UTC. At 16:08, Let's Encrypt replied
that they had paused issuance to investigate. Meanwhile, I filed
a <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1838667" rel="external">CA
Certificate Compliance bug</a> in Bugzilla, which is where Mozilla
and Chrome track compliance incidents by publicly-trusted certificate authorities.
</p>
<p>
At 16:54, Let's Encrypt resumed issuance after confirming that they would
not issue any more certificates with mismatched precertificates.
</p>
<p>
On Friday, June 16, 2023, Let's Encrypt emailed the subscribers of
the affected certificates to inform them of the need to replace
their certificates.
</p>
<p>
On Monday, June 19, 2023 at 18:00 UTC, Let's Encrypt revoked the 645 affected
certificates, as required by the Baseline Requirements. This will cause
the certificates to stop working in any client that checks revocation,
but remember that these certificates were already being rejected by Chrome
and Safari for having invalid SCTs.
</p>
<p>
On Tuesday, June 20, 2023, Let's Encrypt posted their <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1838667#c6" rel="external">public incident report</a>, which explained the root cause of the incident and what they're doing to prevent it
from happening again. Specifically, they plan to add a pre-issuance check
that ensures certificates contain the same data as the precertificate.
</p>
<h4>Hundreds of websites are still serving broken certificates</h4>
<p>
I've been periodically checking port 443 of every DNS name
in the affected certificates, and as of publication time, 261 certificates are still in use,
despite not working in CT-enforcing or revocation-checking
clients.
</p>
<p>
I find it alarming that a week after the incident, 40% of the affected certificates
are still in use, despite being rejected by the most popular browsers and despite affected
subscribers being emailed by Let's Encrypt. I thought
that maybe these certificates were being used by API endpoints which are accessed
by non-browser clients that don't enforce CT or check revocation, but this doesn't
appear to be the case, as most of the DNS names are for bare domains or www subdomains.
It's fortunate that Let's Encrypt issued only a small number of non-compliant
certificates, because otherwise it would have broken a lot of websites.
</p>
<p>
There is a new standard under development
called <a href="https://datatracker.ietf.org/doc/draft-ietf-acme-ari/" rel="external">ACME Renewal Information</a>
which enables certificate authorities to inform ACME clients to renew certificates ahead of
their normal expiration. Let's Encrypt supports ARI, and used it in this incident to trigger
early renewal of the affected certificates. Clearly, more ACME clients need to add support for ARI.
</p>
<h4>This is my 50th CA compliance bug</h4>
<p>
It turns out this is the 50th CA compliance bug that <a href="https://bugzilla.mozilla.org/buglist.cgi?emailtype1=exact&amp;emailreporter1=1&amp;query_format=advanced&amp;resolution=---&amp;resolution=FIXED&amp;product=CA%20Program&amp;component=CA%20Certificate%20Compliance&amp;email1=agwa-bugs%40mm.beanwood.com" rel="external">I've filed
in Bugzilla</a>, and the 5th which was uncovered by Cert Spotter's SCT
signature checks. Additionally, I reported a number of incidents before
2018 which didn't end up in Bugzilla.
</p>
<p>
Some of the problems I uncovered were quite serious (like issuing certificates
without doing domain validation) and snowballed until the CA was ultimately
distrusted. Most are minor in comparison, and ten years ago, no one would
have cared about them: there was no Certificate Transparency to unearth non-compliant
certificates, and even when someone did notice, the revocation requirement was not
enforced, and CAs were not required to file incident reports or document the non-compliance on
their next audit. Thankfully, that's no longer the case,
and even compliance violations that seem minor are treated seriously,
which has led to enormous improvements in the certificate ecosystem:
</p>
<ol class="num-1">
<li>
The improvements which certificate authorities make in response
to seemingly-minor incidents also improve their compliance with
the most security-critical rules.
</li>
<li>TLS clients no longer need to work around non-standards-compliant
certificates, which means they can be simpler. Simpler code is easier to make
secure.</li>
<li>The way that CAs handle minor incidents can uncover much larger problems.
Minor compliance problems are like <a href="https://effectiviology.com/brown-mms/" rel="external">"Brown M&amp;M's"</a>.</li>
</ol>
<p>
Mozilla deserves enormous credit for being the first to require
public incident reports from CAs, as does Google for creating and
fostering Certificate Transparency.
</p>
<h4>You should monitor Certificate Transparency too</h4>
<p>
One limitation of my compliance monitoring is that I am generally only able to
detect certificates that are intrinsically non-compliant, like those which
violate encoding rules or are valid for too many days.
While I do monitor certificates for domains that are likely to be abused,
like <a href="https://groups.google.com/g/mozilla.dev.security.policy/c/fyJ3EK2YOP8" rel="external">example.com</a>
and <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1496088" rel="external">test.com</a>,
I can't tell if a certificate issued for your domain is authorized or not.
Only you know that.
</p>
<p>
Fortunately, it's pretty easy to monitor Certificate Transparency and get
alerts when a certificate is issued for one of your domains.
Cert Spotter has a <a href="https://github.com/SSLMate/certspotter" rel="external">standalone, open source version</a>
that's easy to set up. The <a href="https://sslmate.com/certspotter/" rel="external">paid version</a>
has additional features like expiration monitoring, Slack integration, and ways to filter alerts
so you're not bothered about legitimate certificates. But most importantly, subscribing to the paid
version helps me continue my compliance monitoring of the certificate authority ecosystem.
</p>
</div></div><form class="subscribe" method="post" action="https://buttondown.email/api/emails/embed-subscribe/agwa" rel="noopener" target="_blank"><input type="hidden" name="utm_source" value="blog_footer_form" /><p>Don't miss my next post</p><label for="fd56d400-d297-42ea-a38c-b913fedc67d2">Enter your email address to get my latest posts by email:</label><input id="fd56d400-d297-42ea-a38c-b913fedc67d2" type="email" name="email" required="required" placeholder="Email address" /><button>Subscribe</button><p>
You can also <a href="/blog/feed">subscribe with RSS</a> or follow me on
<a href="https://follow.agwa.name/agwa">the fediverse (Mastodon)</a> or
<a href="https://bsky.app/profile/agwa.name">Bluesky</a>.
</p></form><div class="post_nav"><div class="older"><div><h3>Older (<a href="/blog/index">View Archive</a>)</h3><p><a title="The Difference Between Root Certificate Authorities, Intermediates, and Resellers" href="/blog/post/roots_intermediates_and_resellers">The Difference Between Root Certificate Authorities, Intermediates, and Resellers</a></p></div></div><div class="newer"><div><h3>Newer</h3><p><a title="SQLite's Durability Settings are a Mess" href="/blog/post/sqlite_durability">SQLite's Durability Settings are a Mess</a></p></div></div><div class="clearer"></div></div><div id="comments"><div class="comments"><h3>Comments</h3><div class="level"><div class="comment" id="comment-32424"><h4><span class="poster">Reader mcint</span>
on 2023-06-22 at 21:06:
</h4><p>Excellent post. Thank you for acting as a verifier in the public interest. In</p><blockquote><p> Mozilla deserves enormous credit for being the first to require public incident reports from CAs, as does Google for creating and fostering Certificate Transparency.
</p></blockquote><p>could you link to more information about this Mozilla and Google history, to learn about the evolved consensus you're helping to uphold?</p><p class="actions"><a href="/blog/post/last_weeks_lets_encrypt_downtime/comment/32424">Reply</a></p></div><div class="level"><div class="comment" id="comment-32425"><h4><span class="poster"><a href="https://www.agwa.name" title="https://www.agwa.name" rel="external">Andrew Ayer</a></span>
on 2023-06-22 at 21:33:
</h4><p>Thanks for reading!</p><p>This has some information about Certificate Transparency's history: <a href="https://certificate.transparency.dev/community/" rel="external">https://certificate.transparency.dev/community/</a></p><p>Here's Mozilla's guidelines for CA incident response: <a href="https://wiki.mozilla.org/CA/Responding_To_An_Incident" rel="external">https://wiki.mozilla.org/CA/Responding_To_An_Incident</a></p><p class="actions"><a href="/blog/post/last_weeks_lets_encrypt_downtime/comment/32425">Reply</a></p></div><div class="level"></div></div></div></div></div><div id="reply"><div class="post_comment_form"><h3>Post a Comment</h3><p class="policy">Your comment will be public. To contact me privately, <a href="/">email me</a>. Please keep your comment polite, on-topic, and comprehensible. Your comment may be held for moderation before being published.</p><form action="#reply" method="post"><input type="hidden" name="csrf-token" value="4rzG9IAD4QtcyyVDmTxHReq7XZAviGG9AGT/mwABAAAAAAAAX6kDN0hYpWoAAAAAZiL+WLzdgbOHxzWapYBAsq4chugkRm5j8MvT7kSw4tE=" /><input type="hidden" name="post_comment_internal_data" value="yOekagAAAADB5z22bfRHibKfjmaBNXLvbvW9VsW1+KhyG30PQpMz0h6izcNGjV47Z4eJORMRe0M=" /><p><label for="post_comment_poster">Your Name:</label> <input type="text" name="post_comment_poster" id="post_comment_poster" size="30" maxlength="64" value="" /> <span class="requirement">(Optional; will be published)</span></p><p><label for="post_comment_email">Your Email Address:</label> <input type="text" name="post_comment_email" id="post_comment_email" size="30" maxlength="256" value="" /> <span class="requirement">(Optional; will not be published)</span></p><p><label for="post_comment_website">Your Website:</label> <input type="text" name="post_comment_website" id="post_comment_website" size="30" maxlength="256" value="" /> <span class="requirement">(Optional; will be published)</span></p><p><textarea name="post_comment_text" id="post_comment_text" cols="70" rows="8"></textarea></p><ul class="formatting_help"><li>Blank lines separate paragraphs.</li><li>Lines starting with <code>&gt;</code> are indented as block quotes.</li><li>Lines starting with two spaces are reproduced verbatim (good for code).</li><li>Text surrounded by *asterisks* is <em>italicized</em>.</li><li>Text surrounded by `back ticks` is <code>monospaced</code>.</li><li>URLs are turned into links.</li><li>Use the Preview button to check your formatting.</li></ul><p><input type="submit" name="post_comment_post" value="Post" /> <input type="submit" name="post_comment_preview" value="Preview" /></p></form></div></div></div></div></div><div id="footer"><p class="copyright">© 2026 Andrew Ayer</p><p class="trees"><svg xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Drawing of six trees without leaves" version="1.1" viewBox="0 0 1474 364.7"><use xlink:href="/art/symbols.svg#footer_trees"></use></svg></p></div></div></body></html>