329 lines
31 KiB
HTML
329 lines
31 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xml:lang="en" lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>The Story Behind Last Week's Let's Encrypt Downtime</title><link rel="stylesheet" type="text/css" href="/css/style.css" /><link rel="icon" sizes="16x16" type="image/png" href="/favicon.png" /><link rel="icon" sizes="32x32" type="image/png" href="/art/favicon-32x32.png" /><link rel="icon" sizes="96x96" type="image/png" href="/art/favicon-96x96.png" /><link rel="apple-touch-icon" sizes="152x152" href="/art/appleicon-lightbg-152x152.png" /><link rel="apple-touch-icon" sizes="167x167" href="/art/appleicon-lightbg-167x167.png" /><link rel="apple-touch-icon" sizes="180x180" href="/art/appleicon-lightbg-180x180.png" /><link rel="start" title="Home" href="/" /><link rel="alternate" type="application/atom+xml" title="Blog" href="/blog/feed" /><meta name="generator" content="Terrapin XSLT" /><meta name="author" content="Andrew Ayer" /><meta name="copyright" content="Copyright 2026 Andrew Ayer" /><meta name="robots" content="noarchive" /><meta name="viewport" content="width=device-width, initial-scale=1" /><meta name="twitter:card" content="summary" /><meta name="twitter:site" content="@__agwa" /><meta name="twitter:creator" content="@__agwa" /><meta name="twitter:title" content="The Story Behind Last Week's Let's Encrypt Downtime" /><meta name="twitter:description" content="How I detected that Let's Encrypt issued 645 non-compliant certificates" /><!--[if IE]>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
|
<![endif]--><!--Edition = 'B'--><!--Page ID = '/blog/post/last_weeks_lets_encrypt_downtime'--><!--Date/time = '2026-09-12T05:48:56Z'--></head><body><div id="root"><p id="skiptocontent"><a href="#content" accesskey="c">Skip to Content [alt-c]</a></p><div id="header"><div class="inner_header"><h1><a class="logo" href="/"><svg xmlns="http://www.w3.org/2000/svg" role="img" aria-label="A.G.W.A. logo" width="54" height="30"><use xlink:href="/art/symbols.svg#logo"></use></svg></a><span class="fluff"> </span><a class="title" href="/"><span>Andrew Ayer</span></a></h1><h2>Sections</h2><ul class="tabs"><li><a href="/blog/">Blog</a></li><li><a href="/projects/">Projects</a></li><li><a href="/photos/">Photos</a></li></ul></div><div class="header_background"><svg xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Drawing of an oak tree" version="1.1" viewBox="0 0 4200 114.01"><use class="header_tree" xlink:href="/art/symbols.svg#tree"></use><use class="header_ground" xlink:href="/art/symbols.svg#ground"></use></svg></div></div><div id="content"><div class="blog_page"><div class="blog_sidebar"><div><h2>Blog</h2><ul><li><a href="/blog/">Latest</a></li> <li><a href="/blog/popular">Popular</a></li> <li><a href="/blog/index">Archives</a></li> <li><a href="/blog/feed">RSS</a></li></ul></div></div><div class="blog_content"><div class="blog_post"><p class="date">June 22, 2023</p><h2>The Story Behind Last Week's Let's Encrypt Downtime</h2><div class="content">
|
|
|
|
|
|
|
|
<p>
|
|
Last Thursday (June 15th, 2023), Let's Encrypt <a href="https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/648b36899c7c1405303ea8c4" rel="external">went down for about
|
|
an hour</a>, during which time it was not possible to obtain certificates
|
|
from Let's Encrypt. Immediately prior to the outage, Let's Encrypt issued
|
|
645 certificates which did not work in Chrome or Safari. In this post,
|
|
I'm going to explain what went wrong and how I detected it.
|
|
</p>
|
|
|
|
|
|
<h4>The Law of Precertificates</h4>
|
|
|
|
<p>
|
|
Before I can explain the incident, we need to talk about <a href="https://certificate.transparency.dev/" rel="external">Certificate
|
|
Transparency</a>. Certificate Transparency (CT) is a system for putting
|
|
certificates issued by publicly-trusted CAs, such as Let's Encrypt,
|
|
in public, append-only logs. Certificate authorities have a tremendous amount
|
|
of power, and if they misuse their power by issuing certificates that they shouldn't, traffic to HTTPS websites
|
|
could be intercepted by attackers. Historically, CAs have not used their power well, and Certificate
|
|
Transparency is an effort to fix that by letting anyone examine the certificates
|
|
that CAs issue.
|
|
</p>
|
|
|
|
<p>
|
|
A key concept in Certificate Transparency is the "precertificate".
|
|
Before issuing a certificate, the certificate authority creates a
|
|
precertificate, which contains all of the information that will
|
|
be in the certificate, plus a "poison extension" that prevents the
|
|
precertificate from being used like a real certificate. The CA submits
|
|
the precertificate to multiple Certificate Transparency logs. Each log
|
|
returns a Signed Certificate Timestamp (SCT), which is a signed statement
|
|
acknowledging receipt of the precertificate and promising to publish the
|
|
precertificate in the log for anyone to download. The CA takes all of
|
|
the SCTs and embeds them in the certificate. When a CT-enforcing browser
|
|
(like Chrome or Safari) validates the certificate, it makes sure that
|
|
the certificate embeds a sufficient number of SCTs from trustworthy logs.
|
|
This doesn't prevent the browser from accepting a malicious certificate,
|
|
but it does ensure that the precertificate is in public logs, allowing
|
|
the attack to be detected and action taken against the CA.
|
|
</p>
|
|
|
|
<p>
|
|
The certificate itself may or may not end up in CT logs. Some CAs,
|
|
notably Let's Encrypt and Sectigo, automatically submit their
|
|
certificates. Certificates from other CAs only end up in logs if
|
|
someone else finds and submits them. <strong>Since only the precertificate is
|
|
guaranteed to be logged, it is essential that a precertificate be treated
|
|
as incontrovertible proof that a certificate containing the same data exists.</strong>
|
|
When someone finds a precertificate for a malicious or non-compliant certificate,
|
|
the CA can't be allowed to evade responsibility by saying "just kidding,
|
|
we never actually issued the real certificate" (and boy, have they tried).
|
|
Otherwise, CT would be useless.
|
|
</p>
|
|
|
|
<p>
|
|
There are two ways a CA could create a certificate. They could
|
|
take the precertificate, remove the poison extension, add the SCTs,
|
|
and re-sign it. Or, they could create the certificate from scratch,
|
|
making sure to add the same data, in the same order, as used in the
|
|
precertificate.
|
|
</p>
|
|
|
|
<p>
|
|
The first way is robust because it's guaranteed to produce a certificate
|
|
which matches the precertificate. At least one CA, Sectigo, <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1838667#c2" rel="external">uses this
|
|
approach</a>. Let's Encrypt uses the second approach. You can probably see
|
|
where this is going...
|
|
</p>
|
|
|
|
|
|
<h4>The Let's Encrypt incident</h4>
|
|
|
|
<p>
|
|
On June 15, 2023, Let's Encrypt deployed a <a href="https://community.letsencrypt.org/t/small-change-to-end-entity-certificates-cps-url-and-oid-will-not-be-included-from-june-15/198206" rel="external">planned change</a> to their certificate
|
|
configuration which altered the contents of the Certificate Policies extension from:
|
|
</p>
|
|
|
|
<code class="block pre" style="white-space: pre;">X509v3 Certificate Policies:
|
|
Policy: 2.23.140.1.2.1
|
|
Policy: 1.3.6.1.4.1.44947.1.1.1
|
|
CPS: http://cps.letsencrypt.org</code>
|
|
|
|
<p>to:</p>
|
|
|
|
<code class="block pre" style="white-space: pre;">X509v3 Certificate Policies:
|
|
Policy: 2.23.140.1.2.1</code>
|
|
|
|
<p>
|
|
Unfortunately, any certificate which was requested while the change
|
|
was being rolled out could have its precertificate and certificate created
|
|
with different configurations. For example, when Let's Encrypt issued
|
|
the certificate with serial number 03:e2:26:7b:78:6b:7e:33:83:17:dd:d6:2e:76:4f:cb:3c:71,
|
|
the <a href="https://understandingwebpki.com/?cert=MIIFGjCCBAKgAwIBAgISA%2BIme3hrfjODF93WLnZPyzxxMA0GCSqGSIb3DQEBCwUAMDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJSMzAeFw0yMzA2MTUxNDM2MTZaFw0yMzA5MTMxNDM2MTVaMB4xHDAaBgNVBAMMEyouN2FjbnIubW9uZ29kYi5uZXQwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCjLiLXI%2FmTBSEkSKVucC3NcnXGu%2FM2qwLIk1uenifnoNMmdJmEyp%2BoWFUSn9rIXtHw27YTlJLRRYLSIzqqujDV5PmXzFrSJ%2F9JrgIbNUowaVF3j9bf1%2BNPENEH81RnNGevtKUN5NoEo3fAmZaMWrGjWioNnpIsegSjvvuHeqMqC7SNrGSvtKLBiPkObL5oScPYj%2FcHzt3RYJ17ru6xWgUDV6aqvEblrxcXvPmd%2F1SxB3Vkdkc%2BbCuSLSNM%2FNmcET0YUhWizanjodJarpYJRuW1SjGmPda0jBAQZQDPmZHCEgwTBcCEIg5J3XzAfFUZPPlTVgE%2B7Mbjd%2FDK7iz46D0uHOigVTZto3lPYRdRiyVFNUMAN0GLAlkaJ7Td0FnAxvhE74lSjI7lFqDNtiyA8ovp%2FJbKfPmnvfH%2BfQa7vEFbR5H9v4UZt0XLeI6WdV4pYoCwuK5mfr0NQLCy%2F015OAU8WF4MLM%2BFyt%2BGG%2BsOk2Maz6ysAShMOvdNH7B3GSn65xBVgBxlPWyYpodW9SS1NSVgrgbKMg0yHzx%2FPdosQehyh9p6OpuTaeEi2iQgyTODKGHX%2BcmjzUx0iCG2ByC9bvMo32eZXiC%2BitZCaHb0FGXh%2BK7UcOCsvsi7NLGRngVKK7u7gZmPu4UkVUBpF3jz%2FOK3OsudHcflZIGd6nf8w4lp0wIDAQABo4IBPDCCATgwDgYDVR0PAQH%2FBAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBREcOX3VXl7%2BuM7aqTQ%2FconiJsAAjAfBgNVHSMEGDAWgBQULrMXt1hWy65QCUDmH6%2BdixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iub3JnLzA4BgNVHREEMTAvghgqLjdhY25yLm1lc2gubW9uZ29kYi5uZXSCEyouN2FjbnIubW9uZ29kYi5uZXQwEwYDVR0gBAwwCjAIBgZngQwBAgEwEwYKKwYBBAHWeQIEAwEB%2FwQCBQAwDQYJKoZIhvcNAQELBQADggEBALIUrHns6TWfT%2FkfJ60D9R1Ek4YGB%2FjVsrh2d3uiIU2hiRBBjgDkCLyKd7oXM761uXX3LL4H4JPegqTrZAPO88tUtzBSb3IF4yA0o1NWhE6ceLnBk9fl5TRCC8QASliApsOigDgRi1VFmyFOHpHnVZdbpPucy6T%2BCdKXKfj4iNw%2BaOZcoQxJ70XECXxQbdqJ7VdYf0B%2Bwtk5HZU8cuVVCj1i%2FiDv1zqITCzaavbz870QugiHO%2F8rj2ctrA07SX3Ovs4JGbCGuMzlpxeIFtQDWVufVbu1ZZltzPlSHFqv6mPKW9stYtt8JCjmPwNW6UdrlBtNgvFgkgDpz%2BQ6%2FVu%2Bu7g%3D" rel="external">precertificate</a> contained the new Certificate Policies extension,
|
|
and the <a href="https://understandingwebpki.com/?cert=MIIGRjCCBS6gAwIBAgISA%2BIme3hrfjODF93WLnZPyzxxMA0GCSqGSIb3DQEBCwUAMDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJSMzAeFw0yMzA2MTUxNDM2MTZaFw0yMzA5MTMxNDM2MTVaMB4xHDAaBgNVBAMMEyouN2FjbnIubW9uZ29kYi5uZXQwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCjLiLXI%2FmTBSEkSKVucC3NcnXGu%2FM2qwLIk1uenifnoNMmdJmEyp%2BoWFUSn9rIXtHw27YTlJLRRYLSIzqqujDV5PmXzFrSJ%2F9JrgIbNUowaVF3j9bf1%2BNPENEH81RnNGevtKUN5NoEo3fAmZaMWrGjWioNnpIsegSjvvuHeqMqC7SNrGSvtKLBiPkObL5oScPYj%2FcHzt3RYJ17ru6xWgUDV6aqvEblrxcXvPmd%2F1SxB3Vkdkc%2BbCuSLSNM%2FNmcET0YUhWizanjodJarpYJRuW1SjGmPda0jBAQZQDPmZHCEgwTBcCEIg5J3XzAfFUZPPlTVgE%2B7Mbjd%2FDK7iz46D0uHOigVTZto3lPYRdRiyVFNUMAN0GLAlkaJ7Td0FnAxvhE74lSjI7lFqDNtiyA8ovp%2FJbKfPmnvfH%2BfQa7vEFbR5H9v4UZt0XLeI6WdV4pYoCwuK5mfr0NQLCy%2F015OAU8WF4MLM%2BFyt%2BGG%2BsOk2Maz6ysAShMOvdNH7B3GSn65xBVgBxlPWyYpodW9SS1NSVgrgbKMg0yHzx%2FPdosQehyh9p6OpuTaeEi2iQgyTODKGHX%2BcmjzUx0iCG2ByC9bvMo32eZXiC%2BitZCaHb0FGXh%2BK7UcOCsvsi7NLGRngVKK7u7gZmPu4UkVUBpF3jz%2FOK3OsudHcflZIGd6nf8w4lp0wIDAQABo4ICaDCCAmQwDgYDVR0PAQH%2FBAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBREcOX3VXl7%2BuM7aqTQ%2FconiJsAAjAfBgNVHSMEGDAWgBQULrMXt1hWy65QCUDmH6%2BdixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iub3JnLzA4BgNVHREEMTAvghgqLjdhY25yLm1lc2gubW9uZ29kYi5uZXSCEyouN2FjbnIubW9uZ29kYi5uZXQwTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEEBgorBgEEAdZ5AgQCBIH1BIHyAPAAdgC3Pvsk35xNunXyOcW6WPRsXfxCz3qfNcSeHQmBJe20mQAAAYi%2Fs0QZAAAEAwBHMEUCID4vc7PNWNauTkmkS7CqSwdiyOV%2BLYIT9g8KygWW4atTAiEA6Re4Cz7BsEMi%2B%2FU8G%2Br9LmqbqwGXGS4mXG7RiEfeQEcAdgB6MoxU2LcttiDqOOBSHumEFnAyE4VNO9IrwTpXo1LrUgAAAYi%2Fs0RQAAAEAwBHMEUCIQD95SqDycwXGZ%2BJKBUVBR%2BhBxn4BRIQ7EPIaMTI%2F%2B854gIgDpJm5BFX9vKUf5tKWn9f%2FFagktt5J6hPnrmURSV%2FegAwDQYJKoZIhvcNAQELBQADggEBAKWyDSRmiM9N%2B2AhYgRuzh3JnxtvhmEXUBEgwuFnlQyCm5ZvScvWKmw2sqcj%2BgI2UNUxmWjq3PbIVBrTLDEgXtVN%2BJU6HwC4TdYPIB4LzfrWsGY7cc2aaY76YbWlwEyhN9niQLijZORKhZ6HLM7MI76FM7oJ9eZmvnfypjJ7E0J9ek%2Fy7S1wqg5EM%2BQiAf03YcjSxUCyL3%2F%2BEzlYRz65diLh7Eb6gBd58rWLOa1nbgTOFsToAkBE7qR3HymfWysxApDN8x95jDzubbkqiyuk3dvzjn3oouN1H8NsG%2FxYrYmMMwnJ8xul1AJ31ZMxJ9hr29G122DSEaX9smAyyzWhAwM%3D" rel="external">certificate</a> contained the old Certificate Policies extension.
|
|
</p>
|
|
|
|
<p>
|
|
This had two consequences:
|
|
</p>
|
|
|
|
<p>
|
|
First, this certificate won't work in Chrome or Safari, because its SCTs
|
|
are for a precertificate containing different data from the certificate.
|
|
Specifically, the SCTs fail signature validation. When logs sign
|
|
SCTs, they compute the signature over the data in the precertificate,
|
|
and when browsers verify SCTs, they compute the signature over the data
|
|
in the certificate. In this case, that data was not the same.
|
|
</p>
|
|
|
|
<p>
|
|
Second, remember how I said that precertificates are treated as
|
|
incontrovertible proof that a certificate containing the same data exists?
|
|
When Let's Encrypt issued a precertificate with the new Certificate
|
|
Policies value, it implied that they also issued a certificate with
|
|
the new Certificate Policies value. Thus, according to the Law of
|
|
Precertificates, Let's Encrypt issued two certificates with
|
|
serial number 03:e2:26:7b:78:6b:7e:33:83:17:dd:d6:2e:76:4f:cb:3c:71:
|
|
</p>
|
|
|
|
<ol class="num-1">
|
|
<li>A <a href="https://understandingwebpki.com/?cert=MIIGRjCCBS6gAwIBAgISA%2BIme3hrfjODF93WLnZPyzxxMA0GCSqGSIb3DQEBCwUAMDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJSMzAeFw0yMzA2MTUxNDM2MTZaFw0yMzA5MTMxNDM2MTVaMB4xHDAaBgNVBAMMEyouN2FjbnIubW9uZ29kYi5uZXQwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCjLiLXI%2FmTBSEkSKVucC3NcnXGu%2FM2qwLIk1uenifnoNMmdJmEyp%2BoWFUSn9rIXtHw27YTlJLRRYLSIzqqujDV5PmXzFrSJ%2F9JrgIbNUowaVF3j9bf1%2BNPENEH81RnNGevtKUN5NoEo3fAmZaMWrGjWioNnpIsegSjvvuHeqMqC7SNrGSvtKLBiPkObL5oScPYj%2FcHzt3RYJ17ru6xWgUDV6aqvEblrxcXvPmd%2F1SxB3Vkdkc%2BbCuSLSNM%2FNmcET0YUhWizanjodJarpYJRuW1SjGmPda0jBAQZQDPmZHCEgwTBcCEIg5J3XzAfFUZPPlTVgE%2B7Mbjd%2FDK7iz46D0uHOigVTZto3lPYRdRiyVFNUMAN0GLAlkaJ7Td0FnAxvhE74lSjI7lFqDNtiyA8ovp%2FJbKfPmnvfH%2BfQa7vEFbR5H9v4UZt0XLeI6WdV4pYoCwuK5mfr0NQLCy%2F015OAU8WF4MLM%2BFyt%2BGG%2BsOk2Maz6ysAShMOvdNH7B3GSn65xBVgBxlPWyYpodW9SS1NSVgrgbKMg0yHzx%2FPdosQehyh9p6OpuTaeEi2iQgyTODKGHX%2BcmjzUx0iCG2ByC9bvMo32eZXiC%2BitZCaHb0FGXh%2BK7UcOCsvsi7NLGRngVKK7u7gZmPu4UkVUBpF3jz%2FOK3OsudHcflZIGd6nf8w4lp0wIDAQABo4ICaDCCAmQwDgYDVR0PAQH%2FBAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBREcOX3VXl7%2BuM7aqTQ%2FconiJsAAjAfBgNVHSMEGDAWgBQULrMXt1hWy65QCUDmH6%2BdixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iub3JnLzA4BgNVHREEMTAvghgqLjdhY25yLm1lc2gubW9uZ29kYi5uZXSCEyouN2FjbnIubW9uZ29kYi5uZXQwTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEEBgorBgEEAdZ5AgQCBIH1BIHyAPAAdgC3Pvsk35xNunXyOcW6WPRsXfxCz3qfNcSeHQmBJe20mQAAAYi%2Fs0QZAAAEAwBHMEUCID4vc7PNWNauTkmkS7CqSwdiyOV%2BLYIT9g8KygWW4atTAiEA6Re4Cz7BsEMi%2B%2FU8G%2Br9LmqbqwGXGS4mXG7RiEfeQEcAdgB6MoxU2LcttiDqOOBSHumEFnAyE4VNO9IrwTpXo1LrUgAAAYi%2Fs0RQAAAEAwBHMEUCIQD95SqDycwXGZ%2BJKBUVBR%2BhBxn4BRIQ7EPIaMTI%2F%2B854gIgDpJm5BFX9vKUf5tKWn9f%2FFagktt5J6hPnrmURSV%2FegAwDQYJKoZIhvcNAQELBQADggEBAKWyDSRmiM9N%2B2AhYgRuzh3JnxtvhmEXUBEgwuFnlQyCm5ZvScvWKmw2sqcj%2BgI2UNUxmWjq3PbIVBrTLDEgXtVN%2BJU6HwC4TdYPIB4LzfrWsGY7cc2aaY76YbWlwEyhN9niQLijZORKhZ6HLM7MI76FM7oJ9eZmvnfypjJ7E0J9ek%2Fy7S1wqg5EM%2BQiAf03YcjSxUCyL3%2F%2BEzlYRz65diLh7Eb6gBd58rWLOa1nbgTOFsToAkBE7qR3HymfWysxApDN8x95jDzubbkqiyuk3dvzjn3oouN1H8NsG%2FxYrYmMMwnJ8xul1AJ31ZMxJ9hr29G122DSEaX9smAyyzWhAwM%3D" rel="external">certificate</a> containing the old Certificate Policies extension</li>
|
|
<li>A certificate containing the new Certificate Policies extension (implied by the existence of the <a href="https://understandingwebpki.com/?cert=MIIFGjCCBAKgAwIBAgISA%2BIme3hrfjODF93WLnZPyzxxMA0GCSqGSIb3DQEBCwUAMDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJSMzAeFw0yMzA2MTUxNDM2MTZaFw0yMzA5MTMxNDM2MTVaMB4xHDAaBgNVBAMMEyouN2FjbnIubW9uZ29kYi5uZXQwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCjLiLXI%2FmTBSEkSKVucC3NcnXGu%2FM2qwLIk1uenifnoNMmdJmEyp%2BoWFUSn9rIXtHw27YTlJLRRYLSIzqqujDV5PmXzFrSJ%2F9JrgIbNUowaVF3j9bf1%2BNPENEH81RnNGevtKUN5NoEo3fAmZaMWrGjWioNnpIsegSjvvuHeqMqC7SNrGSvtKLBiPkObL5oScPYj%2FcHzt3RYJ17ru6xWgUDV6aqvEblrxcXvPmd%2F1SxB3Vkdkc%2BbCuSLSNM%2FNmcET0YUhWizanjodJarpYJRuW1SjGmPda0jBAQZQDPmZHCEgwTBcCEIg5J3XzAfFUZPPlTVgE%2B7Mbjd%2FDK7iz46D0uHOigVTZto3lPYRdRiyVFNUMAN0GLAlkaJ7Td0FnAxvhE74lSjI7lFqDNtiyA8ovp%2FJbKfPmnvfH%2BfQa7vEFbR5H9v4UZt0XLeI6WdV4pYoCwuK5mfr0NQLCy%2F015OAU8WF4MLM%2BFyt%2BGG%2BsOk2Maz6ysAShMOvdNH7B3GSn65xBVgBxlPWyYpodW9SS1NSVgrgbKMg0yHzx%2FPdosQehyh9p6OpuTaeEi2iQgyTODKGHX%2BcmjzUx0iCG2ByC9bvMo32eZXiC%2BitZCaHb0FGXh%2BK7UcOCsvsi7NLGRngVKK7u7gZmPu4UkVUBpF3jz%2FOK3OsudHcflZIGd6nf8w4lp0wIDAQABo4IBPDCCATgwDgYDVR0PAQH%2FBAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBREcOX3VXl7%2BuM7aqTQ%2FconiJsAAjAfBgNVHSMEGDAWgBQULrMXt1hWy65QCUDmH6%2BdixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iub3JnLzA4BgNVHREEMTAvghgqLjdhY25yLm1lc2gubW9uZ29kYi5uZXSCEyouN2FjbnIubW9uZ29kYi5uZXQwEwYDVR0gBAwwCjAIBgZngQwBAgEwEwYKKwYBBAHWeQIEAwEB%2FwQCBQAwDQYJKoZIhvcNAQELBQADggEBALIUrHns6TWfT%2FkfJ60D9R1Ek4YGB%2FjVsrh2d3uiIU2hiRBBjgDkCLyKd7oXM761uXX3LL4H4JPegqTrZAPO88tUtzBSb3IF4yA0o1NWhE6ceLnBk9fl5TRCC8QASliApsOigDgRi1VFmyFOHpHnVZdbpPucy6T%2BCdKXKfj4iNw%2BaOZcoQxJ70XECXxQbdqJ7VdYf0B%2Bwtk5HZU8cuVVCj1i%2FiDv1zqITCzaavbz870QugiHO%2F8rj2ctrA07SX3Ovs4JGbCGuMzlpxeIFtQDWVufVbu1ZZltzPlSHFqv6mPKW9stYtt8JCjmPwNW6UdrlBtNgvFgkgDpz%2BQ6%2FVu%2Bu7g%3D" rel="external">precertificate</a> with the new Certificate Policies extension)</li>
|
|
</ol>
|
|
|
|
<p>
|
|
Issuing two certificates with the same serial number is a violation of
|
|
the <a href="https://cabforum.org/baseline-requirements/" rel="external">Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates</a>. Consequentially, Let's Encrypt must
|
|
revoke the certificate and post a public incident report,
|
|
which must be noted on their next audit statement.
|
|
</p>
|
|
|
|
<p>
|
|
You might think that it's harsh to treat this as a compliance incident
|
|
if Let's Encrypt didn't really issue two certificates with the same
|
|
serial number. Unfortunately, they have no way of proving this, and
|
|
the whole reason for Certificate Transparency is so we don't have to take CAs
|
|
at their word that they aren't issuing certificates that they shouldn't.
|
|
Any exception to the Law of Precertificates creates an opening
|
|
for a malicious CA to exploit.
|
|
</p>
|
|
|
|
<h4>How I found this</h4>
|
|
|
|
<p>
|
|
My company, <a href="https://sslmate.com/" rel="external">SSLMate</a>, operates a Certificate Transparency monitor called
|
|
<a href="https://sslmate.com/certspotter/" rel="external">Cert Spotter</a>, which continuously downloads and indexes the contents of every
|
|
Certificate Transparency log. You can use Cert Spotter to get notifications
|
|
when a certificate is issued for one of your domains, or search the database
|
|
using a <a href="https://sslmate.com/ct_search_api/" rel="external">JSON API</a>.
|
|
</p>
|
|
|
|
<p>
|
|
When Cert Spotter ingests a certificate containing embedded SCTs, it
|
|
verifies each SCT's signature and audits that the log really published
|
|
the precertificate. (If it detects that a log has broken its promise
|
|
to publish a precertificate, I'll publicly disclose the SCT and the log
|
|
will be distrusted. Happily, Cert Spotter has never found a bogus SCT,
|
|
though it has detected logs violating other requirements.)
|
|
</p>
|
|
|
|
<p>
|
|
On Thursday, June 15, 2023 at 15:41 UTC, Cert Spotter began sending me
|
|
alerts about certificates containing embedded SCTs with
|
|
invalid signatures. Since I was getting hundreds of alerts, I decided
|
|
to stop what I was doing and investigate.
|
|
</p>
|
|
|
|
<p>
|
|
I had received these alerts several times before, and have gotten pretty
|
|
good at zeroing in on the problem. When only one SCT in a certificate
|
|
has an invalid signature, it probably means that the CT log screwed up.
|
|
When all of the embedded SCTs have an invalid signature, it probably
|
|
means the CA screwed up. The most common reason is issuing certificates
|
|
that don't match the precertificate. So I took one of the affected
|
|
certificates and searched for precertificates containing the same serial
|
|
number in Cert Spotter's database of every
|
|
(pre)certificate ever logged to Certificate Transparency. Decoding the
|
|
certificate and precertificate with the openssl command immediately
|
|
revealed the different Certificate Policies extension.
|
|
</p>
|
|
|
|
<p>
|
|
Since I was continuing to get alerts from Cert Spotter about invalid
|
|
SCT signatures, I quickly fired off an email to Let's Encrypt's problem
|
|
reporting address alerting them to the problem.
|
|
</p>
|
|
|
|
<p>
|
|
I sent the email at 15:52 UTC. At 16:08, Let's Encrypt replied
|
|
that they had paused issuance to investigate. Meanwhile, I filed
|
|
a <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1838667" rel="external">CA
|
|
Certificate Compliance bug</a> in Bugzilla, which is where Mozilla
|
|
and Chrome track compliance incidents by publicly-trusted certificate authorities.
|
|
</p>
|
|
|
|
<p>
|
|
At 16:54, Let's Encrypt resumed issuance after confirming that they would
|
|
not issue any more certificates with mismatched precertificates.
|
|
</p>
|
|
|
|
<p>
|
|
On Friday, June 16, 2023, Let's Encrypt emailed the subscribers of
|
|
the affected certificates to inform them of the need to replace
|
|
their certificates.
|
|
</p>
|
|
|
|
<p>
|
|
On Monday, June 19, 2023 at 18:00 UTC, Let's Encrypt revoked the 645 affected
|
|
certificates, as required by the Baseline Requirements. This will cause
|
|
the certificates to stop working in any client that checks revocation,
|
|
but remember that these certificates were already being rejected by Chrome
|
|
and Safari for having invalid SCTs.
|
|
</p>
|
|
|
|
<p>
|
|
On Tuesday, June 20, 2023, Let's Encrypt posted their <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1838667#c6" rel="external">public incident report</a>, which explained the root cause of the incident and what they're doing to prevent it
|
|
from happening again. Specifically, they plan to add a pre-issuance check
|
|
that ensures certificates contain the same data as the precertificate.
|
|
</p>
|
|
|
|
<h4>Hundreds of websites are still serving broken certificates</h4>
|
|
|
|
<p>
|
|
I've been periodically checking port 443 of every DNS name
|
|
in the affected certificates, and as of publication time, 261 certificates are still in use,
|
|
despite not working in CT-enforcing or revocation-checking
|
|
clients.
|
|
</p>
|
|
|
|
<p>
|
|
I find it alarming that a week after the incident, 40% of the affected certificates
|
|
are still in use, despite being rejected by the most popular browsers and despite affected
|
|
subscribers being emailed by Let's Encrypt. I thought
|
|
that maybe these certificates were being used by API endpoints which are accessed
|
|
by non-browser clients that don't enforce CT or check revocation, but this doesn't
|
|
appear to be the case, as most of the DNS names are for bare domains or www subdomains.
|
|
It's fortunate that Let's Encrypt issued only a small number of non-compliant
|
|
certificates, because otherwise it would have broken a lot of websites.
|
|
</p>
|
|
|
|
<p>
|
|
There is a new standard under development
|
|
called <a href="https://datatracker.ietf.org/doc/draft-ietf-acme-ari/" rel="external">ACME Renewal Information</a>
|
|
which enables certificate authorities to inform ACME clients to renew certificates ahead of
|
|
their normal expiration. Let's Encrypt supports ARI, and used it in this incident to trigger
|
|
early renewal of the affected certificates. Clearly, more ACME clients need to add support for ARI.
|
|
</p>
|
|
|
|
<h4>This is my 50th CA compliance bug</h4>
|
|
|
|
<p>
|
|
It turns out this is the 50th CA compliance bug that <a href="https://bugzilla.mozilla.org/buglist.cgi?emailtype1=exact&emailreporter1=1&query_format=advanced&resolution=---&resolution=FIXED&product=CA%20Program&component=CA%20Certificate%20Compliance&email1=agwa-bugs%40mm.beanwood.com" rel="external">I've filed
|
|
in Bugzilla</a>, and the 5th which was uncovered by Cert Spotter's SCT
|
|
signature checks. Additionally, I reported a number of incidents before
|
|
2018 which didn't end up in Bugzilla.
|
|
</p>
|
|
|
|
<p>
|
|
Some of the problems I uncovered were quite serious (like issuing certificates
|
|
without doing domain validation) and snowballed until the CA was ultimately
|
|
distrusted. Most are minor in comparison, and ten years ago, no one would
|
|
have cared about them: there was no Certificate Transparency to unearth non-compliant
|
|
certificates, and even when someone did notice, the revocation requirement was not
|
|
enforced, and CAs were not required to file incident reports or document the non-compliance on
|
|
their next audit. Thankfully, that's no longer the case,
|
|
and even compliance violations that seem minor are treated seriously,
|
|
which has led to enormous improvements in the certificate ecosystem:
|
|
</p>
|
|
|
|
<ol class="num-1">
|
|
<li>
|
|
The improvements which certificate authorities make in response
|
|
to seemingly-minor incidents also improve their compliance with
|
|
the most security-critical rules.
|
|
</li>
|
|
|
|
<li>TLS clients no longer need to work around non-standards-compliant
|
|
certificates, which means they can be simpler. Simpler code is easier to make
|
|
secure.</li>
|
|
|
|
<li>The way that CAs handle minor incidents can uncover much larger problems.
|
|
Minor compliance problems are like <a href="https://effectiviology.com/brown-mms/" rel="external">"Brown M&M's"</a>.</li>
|
|
</ol>
|
|
|
|
<p>
|
|
Mozilla deserves enormous credit for being the first to require
|
|
public incident reports from CAs, as does Google for creating and
|
|
fostering Certificate Transparency.
|
|
</p>
|
|
|
|
<h4>You should monitor Certificate Transparency too</h4>
|
|
|
|
<p>
|
|
One limitation of my compliance monitoring is that I am generally only able to
|
|
detect certificates that are intrinsically non-compliant, like those which
|
|
violate encoding rules or are valid for too many days.
|
|
While I do monitor certificates for domains that are likely to be abused,
|
|
like <a href="https://groups.google.com/g/mozilla.dev.security.policy/c/fyJ3EK2YOP8" rel="external">example.com</a>
|
|
and <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1496088" rel="external">test.com</a>,
|
|
I can't tell if a certificate issued for your domain is authorized or not.
|
|
Only you know that.
|
|
</p>
|
|
|
|
<p>
|
|
Fortunately, it's pretty easy to monitor Certificate Transparency and get
|
|
alerts when a certificate is issued for one of your domains.
|
|
Cert Spotter has a <a href="https://github.com/SSLMate/certspotter" rel="external">standalone, open source version</a>
|
|
that's easy to set up. The <a href="https://sslmate.com/certspotter/" rel="external">paid version</a>
|
|
has additional features like expiration monitoring, Slack integration, and ways to filter alerts
|
|
so you're not bothered about legitimate certificates. But most importantly, subscribing to the paid
|
|
version helps me continue my compliance monitoring of the certificate authority ecosystem.
|
|
</p>
|
|
|
|
|
|
</div></div><form class="subscribe" method="post" action="https://buttondown.email/api/emails/embed-subscribe/agwa" rel="noopener" target="_blank"><input type="hidden" name="utm_source" value="blog_footer_form" /><p>Don't miss my next post</p><label for="fd56d400-d297-42ea-a38c-b913fedc67d2">Enter your email address to get my latest posts by email:</label><input id="fd56d400-d297-42ea-a38c-b913fedc67d2" type="email" name="email" required="required" placeholder="Email address" /><button>Subscribe</button><p>
|
|
You can also <a href="/blog/feed">subscribe with RSS</a> or follow me on
|
|
<a href="https://follow.agwa.name/agwa">the fediverse (Mastodon)</a> or
|
|
<a href="https://bsky.app/profile/agwa.name">Bluesky</a>.
|
|
</p></form><div class="post_nav"><div class="older"><div><h3>Older (<a href="/blog/index">View Archive</a>)</h3><p><a title="The Difference Between Root Certificate Authorities, Intermediates, and Resellers" href="/blog/post/roots_intermediates_and_resellers">The Difference Between Root Certificate Authorities, Intermediates, and Resellers</a></p></div></div><div class="newer"><div><h3>Newer</h3><p><a title="SQLite's Durability Settings are a Mess" href="/blog/post/sqlite_durability">SQLite's Durability Settings are a Mess</a></p></div></div><div class="clearer"></div></div><div id="comments"><div class="comments"><h3>Comments</h3><div class="level"><div class="comment" id="comment-32424"><h4><span class="poster">Reader mcint</span>
|
|
on 2023-06-22 at 21:06:
|
|
</h4><p>Excellent post. Thank you for acting as a verifier in the public interest. In</p><blockquote><p> Mozilla deserves enormous credit for being the first to require public incident reports from CAs, as does Google for creating and fostering Certificate Transparency.
|
|
</p></blockquote><p>could you link to more information about this Mozilla and Google history, to learn about the evolved consensus you're helping to uphold?</p><p class="actions"><a href="/blog/post/last_weeks_lets_encrypt_downtime/comment/32424">Reply</a></p></div><div class="level"><div class="comment" id="comment-32425"><h4><span class="poster"><a href="https://www.agwa.name" title="https://www.agwa.name" rel="external">Andrew Ayer</a></span>
|
|
on 2023-06-22 at 21:33:
|
|
</h4><p>Thanks for reading!</p><p>This has some information about Certificate Transparency's history: <a href="https://certificate.transparency.dev/community/" rel="external">https://certificate.transparency.dev/community/</a></p><p>Here's Mozilla's guidelines for CA incident response: <a href="https://wiki.mozilla.org/CA/Responding_To_An_Incident" rel="external">https://wiki.mozilla.org/CA/Responding_To_An_Incident</a></p><p class="actions"><a href="/blog/post/last_weeks_lets_encrypt_downtime/comment/32425">Reply</a></p></div><div class="level"></div></div></div></div></div><div id="reply"><div class="post_comment_form"><h3>Post a Comment</h3><p class="policy">Your comment will be public. To contact me privately, <a href="/">email me</a>. Please keep your comment polite, on-topic, and comprehensible. Your comment may be held for moderation before being published.</p><form action="#reply" method="post"><input type="hidden" name="csrf-token" value="4rzG9IAD4QtcyyVDmTxHReq7XZAviGG9AGT/mwABAAAAAAAAX6kDN0hYpWoAAAAAZiL+WLzdgbOHxzWapYBAsq4chugkRm5j8MvT7kSw4tE=" /><input type="hidden" name="post_comment_internal_data" value="yOekagAAAADB5z22bfRHibKfjmaBNXLvbvW9VsW1+KhyG30PQpMz0h6izcNGjV47Z4eJORMRe0M=" /><p><label for="post_comment_poster">Your Name:</label> <input type="text" name="post_comment_poster" id="post_comment_poster" size="30" maxlength="64" value="" /> <span class="requirement">(Optional; will be published)</span></p><p><label for="post_comment_email">Your Email Address:</label> <input type="text" name="post_comment_email" id="post_comment_email" size="30" maxlength="256" value="" /> <span class="requirement">(Optional; will not be published)</span></p><p><label for="post_comment_website">Your Website:</label> <input type="text" name="post_comment_website" id="post_comment_website" size="30" maxlength="256" value="" /> <span class="requirement">(Optional; will be published)</span></p><p><textarea name="post_comment_text" id="post_comment_text" cols="70" rows="8"></textarea></p><ul class="formatting_help"><li>Blank lines separate paragraphs.</li><li>Lines starting with <code>></code> are indented as block quotes.</li><li>Lines starting with two spaces are reproduced verbatim (good for code).</li><li>Text surrounded by *asterisks* is <em>italicized</em>.</li><li>Text surrounded by `back ticks` is <code>monospaced</code>.</li><li>URLs are turned into links.</li><li>Use the Preview button to check your formatting.</li></ul><p><input type="submit" name="post_comment_post" value="Post" /> <input type="submit" name="post_comment_preview" value="Preview" /></p></form></div></div></div></div></div><div id="footer"><p class="copyright">© 2026 Andrew Ayer</p><p class="trees"><svg xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Drawing of six trees without leaves" version="1.1" viewBox="0 0 1474 364.7"><use xlink:href="/art/symbols.svg#footer_trees"></use></svg></p></div></div></body></html> |