Figure out why acme/autocert has not renewed bridge TLS certificates since October
I got emails from Let's Encrypt on 2023-12-28 and 2024-01-05 saying that the certificates for 02.snowflake.torproject.net and snowflake.torproject.net respectively would soon expire.
Subject: Let's Encrypt certificate expiration notice for domain "02.snowflake.torproject.net"
Date: Thu, 28 Dec 2023 09:46:04 +0000
From: Let's Encrypt Expiry Bot <expiry@letsencrypt.org>
To: dcf@torproject.org
Subject: Let's Encrypt certificate expiration notice for domain "02.snowflake.torproject.net"
Hello,
Your certificate (or certificates) for the names listed below will expire in 19 days (on 2024-01-17). Please make sure to renew your certificate before then, or visitors to your web site will encounter errors.
We recommend renewing certificates automatically when they have a third of their total lifetime left. For Let's Encrypt's current 90-day certificates, that means renewing 30 days before expiration. See https://letsencrypt.org/docs/integration-guide/ for details.
02.snowflake.torproject.net
For details about when we send these emails, please visit: https://letsencrypt.org/docs/expiration-emails/ In particular, note that this reminder email is still sent if you've obtained a slightly different certificate by adding or removing names. If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message.
Subject: Let's Encrypt certificate expiration notice for domain "snowflake.torproject.net"
Date: Fri, 05 Jan 2024 17:19:27 +0000
From: Let's Encrypt Expiry Bot <expiry@letsencrypt.org>
To: dcf@torproject.org
Subject: Let's Encrypt certificate expiration notice for domain "snowflake.torproject.net"
Hello,
Your certificate (or certificates) for the names listed below will expire in 19 days (on 2024-01-25). Please make sure to renew your certificate before then, or visitors to your web site will encounter errors.
We recommend renewing certificates automatically when they have a third of their total lifetime left. For Let's Encrypt's current 90-day certificates, that means renewing 30 days before expiration. See https://letsencrypt.org/docs/integration-guide/ for details.
snowflake.torproject.net
For details about when we send these emails, please visit: https://letsencrypt.org/docs/expiration-emails/ In particular, note that this reminder email is still sent if you've obtained a slightly different certificate by adding or removing names. If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message.
Certificate renewal is supposed to happen automatically using the acme/autocert package. But the current certificates are both from last October:
Validity
Not Before: Oct 15 14:03:54 2023 GMT
Not After : Jan 13 14:03:53 2024 GMT
Subject: CN = 02.snowflake.torproject.net
-----BEGIN CERTIFICATE-----
MIIEODCCAyCgAwIBAgISBILJIV5T+hp2SFg3D2z5RGrDMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMzEwMTUxNDAzNTRaFw0yNDAxMTMxNDAzNTNaMCYxJDAiBgNVBAMT
GzAyLnNub3dmbGFrZS50b3Jwcm9qZWN0Lm5ldDBZMBMGByqGSM49AgEGCCqGSM49
AwEHA0IABH5DrT87QcJxDEhNFZ15YHUR16BQAwNQpQ52qWKV1Y4oA8elZvE/XKHY
op2p6JEO3Sz/kjbcNH6dWXWIkRXz0CmjggIdMIICGTAOBgNVHQ8BAf8EBAMCB4Aw
HQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYD
VR0OBBYEFIetn19sSDfKBLDyYT7PfpQELn08MB8GA1UdIwQYMBaAFBQusxe3WFbL
rlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0cDov
L3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5sZW5jci5v
cmcvMCYGA1UdEQQfMB2CGzAyLnNub3dmbGFrZS50b3Jwcm9qZWN0Lm5ldDATBgNV
HSAEDDAKMAgGBmeBDAECATCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB3ADtTd3U+
LbmAToswWwb+QDtn2E/D9Me9AA0tcm/h+tQXAAABizPdegsAAAQDAEgwRgIhAPgR
OZAl/LXlhK8XIqa6MWEct6xODHoT4/pVBMRqsV0mAiEA7fO6V1+Gdc6rzViQBRTk
xJjsHzd5qHhf/ahbpZ/rJUMAdQDatr9rP7W2Ip+bwrtca+hwkXFsu1GEhTS9pD0w
SNf7qwAAAYsz3XogAAAEAwBGMEQCIHV/7jyPSrJe9bjMa2HMvwdu+FWChjvaPnY7
XYM55exqAiA8cdIDAZJymwt6PX/eWtyLhswbBD7iLbhlIRO6kfpcUTANBgkqhkiG
9w0BAQsFAAOCAQEASXh2o1Gh5nhhwKYEeU+wZWpqhIkvf+zbO7PEd/X1IMGRheRJ
UnobWUwzIn8Rrks6y3ktjSRtY2wY5QQgfXClYCMeleZLlp7IY1RDgG4oiqiXQ1Xr
ZJMQ+2cGnDGbdW+Jy2ISo3Mlc6H/TlfC7w6Ef+4NeTgVGbyuGKhHD0szASWfWsdO
jPtKVdxOYAyENAr0Xk/slNfgHubnKz5m1qHl0Lm8IEDgW56PjpLahQNkJM+7XUgz
52ANjQaToIzuafxGTCM2Ik0xry1/P7skI6KenuLfvxevS90qZ6GRTI6aAw/8PMR3
dWAv4LRds9DRL/NBRWTCNRNHwki9fNHciOiGzw==
-----END CERTIFICATE-----
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
04:82:c9:21:5e:53:fa:1a:76:48:58:37:0f:6c:f9:44:6a:c3
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = R3
Validity
Not Before: Oct 15 14:03:54 2023 GMT
Not After : Jan 13 14:03:53 2024 GMT
Subject: CN = 02.snowflake.torproject.net
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:7e:43:ad:3f:3b:41:c2:71:0c:48:4d:15:9d:79:
60:75:11:d7:a0:50:03:03:50:a5:0e:76:a9:62:95:
d5:8e:28:03:c7:a5:66:f1:3f:5c:a1:d8:a2:9d:a9:
e8:91:0e:dd:2c:ff:92:36:dc:34:7e:9d:59:75:88:
91:15:f3:d0:29
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
87:AD:9F:5F:6C:48:37:CA:04:B0:F2:61:3E:CF:7E:94:04:2E:7D:3C
X509v3 Authority Key Identifier:
14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6
Authority Information Access:
OCSP - URI:http://r3.o.lencr.org
CA Issuers - URI:http://r3.i.lencr.org/
X509v3 Subject Alternative Name:
DNS:02.snowflake.torproject.net
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 3B:53:77:75:3E:2D:B9:80:4E:8B:30:5B:06:FE:40:3B:
67:D8:4F:C3:F4:C7:BD:00:0D:2D:72:6F:E1:FA:D4:17
Timestamp : Oct 15 15:03:54.635 2023 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:F8:11:39:90:25:FC:B5:E5:84:AF:17:
22:A6:BA:31:61:1C:B7:AC:4E:0C:7A:13:E3:FA:55:04:
C4:6A:B1:5D:26:02:21:00:ED:F3:BA:57:5F:86:75:CE:
AB:CD:58:90:05:14:E4:C4:98:EC:1F:37:79:A8:78:5F:
FD:A8:5B:A5:9F:EB:25:43
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : DA:B6:BF:6B:3F:B5:B6:22:9F:9B:C2:BB:5C:6B:E8:70:
91:71:6C:BB:51:84:85:34:BD:A4:3D:30:48:D7:FB:AB
Timestamp : Oct 15 15:03:54.656 2023 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:44:02:20:75:7F:EE:3C:8F:4A:B2:5E:F5:B8:CC:6B:
61:CC:BF:07:6E:F8:55:82:86:3B:DA:3E:76:3B:5D:83:
39:E5:EC:6A:02:20:3C:71:D2:03:01:92:72:9B:0B:7A:
3D:7F:DE:5A:DC:8B:86:CC:1B:04:3E:E2:2D:B8:65:21:
13:BA:91:FA:5C:51
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
49:78:76:a3:51:a1:e6:78:61:c0:a6:04:79:4f:b0:65:6a:6a:
84:89:2f:7f:ec:db:3b:b3:c4:77:f5:f5:20:c1:91:85:e4:49:
52:7a:1b:59:4c:33:22:7f:11:ae:4b:3a:cb:79:2d:8d:24:6d:
63:6c:18:e5:04:20:7d:70:a5:60:23:1e:95:e6:4b:96:9e:c8:
63:54:43:80:6e:28:8a:a8:97:43:55:eb:64:93:10:fb:67:06:
9c:31:9b:75:6f:89:cb:62:12:a3:73:25:73:a1:ff:4e:57:c2:
ef:0e:84:7f:ee:0d:79:38:15:19:bc:ae:18:a8:47:0f:4b:33:
01:25:9f:5a:c7:4e:8c:fb:4a:55:dc:4e:60:0c:84:34:0a:f4:
5e:4f:ec:94:d7:e0:1e:e6:e7:2b:3e:66:d6:a1:e5:d0:b9:bc:
20:40:e0:5b:9e:8f:8e:92:da:85:03:64:24:cf:bb:5d:48:33:
e7:60:0d:8d:06:93:a0:8c:ee:69:fc:46:4c:23:36:22:4d:31:
af:2d:7f:3f:bb:24:23:a2:9e:9e:e2:df:bf:17:af:4b:dd:2a:
67:a1:91:4c:8e:9a:03:0f:fc:3c:c4:77:75:60:2f:e0:b4:5d:
b3:d0:d1:2f:f3:41:45:64:c2:35:13:47:c2:48:bd:7c:d1:dc:
88:e8:86:cf
Validity
Not Before: Oct 22 11:23:44 2023 GMT
Not After : Jan 20 11:23:43 2024 GMT
Subject: CN = snowflake.torproject.net
-----BEGIN CERTIFICATE-----
MIIEMjCCAxqgAwIBAgISBMnl0UqZyd2Ds/DcWpQdF13aMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMzEwMjIxMTIzNDRaFw0yNDAxMjAxMTIzNDNaMCMxITAfBgNVBAMT
GHNub3dmbGFrZS50b3Jwcm9qZWN0Lm5ldDBZMBMGByqGSM49AgEGCCqGSM49AwEH
A0IABDGdN4mjaNp0o+146WrLD5iuzDBpd4DrR9FmYbISj9KNQOVbty07VIWtdhel
KO95Phbj8Gmo8OlU2GCsRFuVczOjggIaMIICFjAOBgNVHQ8BAf8EBAMCB4AwHQYD
VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0O
BBYEFMGfwy1PcMQaSDTB+69POTFrNxHRMB8GA1UdIwQYMBaAFBQusxe3WFbLrlAJ
QOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0cDovL3Iz
Lm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5sZW5jci5vcmcv
MCMGA1UdEQQcMBqCGHNub3dmbGFrZS50b3Jwcm9qZWN0Lm5ldDATBgNVHSAEDDAK
MAgGBmeBDAECATCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB2ANq2v2s/tbYin5vC
u1xr6HCRcWy7UYSFNL2kPTBI1/urAAABi1dXW9IAAAQDAEcwRQIgVx42UBDJsH1f
R48qhQrRUqGHF5Xn0W16ONOiTDf3m0gCIQCoUlDJjucW8suM+ZZll1IJrT2by9eM
ik+dLfJfeEIfAwB2ADtTd3U+LbmAToswWwb+QDtn2E/D9Me9AA0tcm/h+tQXAAAB
i1dXW8IAAAQDAEcwRQIhAJgErCNbhsx8+wrpmWJ3NO04EpRrw1qTMyecCM/viUL4
AiAMGXmerZwFANih+eRqPNfPpGYollCcCRsRYVgf9XHcbzANBgkqhkiG9w0BAQsF
AAOCAQEAccSUiBqi7/D7HZSLw6Ji1t3Dvy50wB13QAAJJY1juv8Izxgltcqd2Hvb
NHVI2GQ0gQ0DxHmbRX77t9CmjCYnhFQrVvzPguYECqDfBpxjjezr8T9axfGm2CIU
n2PnUP65p/E49D9PAidUkmz2NW5IkWhvm3tfdbkdbJNpHfnM1rPmcIA3Q2FGE+jj
bGTmBsngUL/XV38crKTssOny5Lcp8W7iYlWTpWKw1r3Ja/yW2aG89oF5RvgPLf1z
Q9bqG7Y7uYtSz1hGNiy4wGypiGD6Oj14cAKOKiPL2MH9j1NYosvi2LbFMN2cIByW
cV8VCezuZbnVpvubv5HTxDyW0gLtuw==
-----END CERTIFICATE-----
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
04:c9:e5:d1:4a:99:c9:dd:83:b3:f0:dc:5a:94:1d:17:5d:da
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = R3
Validity
Not Before: Oct 22 11:23:44 2023 GMT
Not After : Jan 20 11:23:43 2024 GMT
Subject: CN = snowflake.torproject.net
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:31:9d:37:89:a3:68:da:74:a3:ed:78:e9:6a:cb:
0f:98:ae:cc:30:69:77:80:eb:47:d1:66:61:b2:12:
8f:d2:8d:40:e5:5b:b7:2d:3b:54:85:ad:76:17:a5:
28:ef:79:3e:16:e3:f0:69:a8:f0:e9:54:d8:60:ac:
44:5b:95:73:33
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
C1:9F:C3:2D:4F:70:C4:1A:48:34:C1:FB:AF:4F:39:31:6B:37:11:D1
X509v3 Authority Key Identifier:
14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6
Authority Information Access:
OCSP - URI:http://r3.o.lencr.org
CA Issuers - URI:http://r3.i.lencr.org/
X509v3 Subject Alternative Name:
DNS:snowflake.torproject.net
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : DA:B6:BF:6B:3F:B5:B6:22:9F:9B:C2:BB:5C:6B:E8:70:
91:71:6C:BB:51:84:85:34:BD:A4:3D:30:48:D7:FB:AB
Timestamp : Oct 22 12:23:44.850 2023 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:20:57:1E:36:50:10:C9:B0:7D:5F:47:8F:2A:
85:0A:D1:52:A1:87:17:95:E7:D1:6D:7A:38:D3:A2:4C:
37:F7:9B:48:02:21:00:A8:52:50:C9:8E:E7:16:F2:CB:
8C:F9:96:65:97:52:09:AD:3D:9B:CB:D7:8C:8A:4F:9D:
2D:F2:5F:78:42:1F:03
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 3B:53:77:75:3E:2D:B9:80:4E:8B:30:5B:06:FE:40:3B:
67:D8:4F:C3:F4:C7:BD:00:0D:2D:72:6F:E1:FA:D4:17
Timestamp : Oct 22 12:23:44.834 2023 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:21:00:98:04:AC:23:5B:86:CC:7C:FB:0A:E9:
99:62:77:34:ED:38:12:94:6B:C3:5A:93:33:27:9C:08:
CF:EF:89:42:F8:02:20:0C:19:79:9E:AD:9C:05:00:D8:
A1:F9:E4:6A:3C:D7:CF:A4:66:28:96:50:9C:09:1B:11:
61:58:1F:F5:71:DC:6F
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
71:c4:94:88:1a:a2:ef:f0:fb:1d:94:8b:c3:a2:62:d6:dd:c3:
bf:2e:74:c0:1d:77:40:00:09:25:8d:63:ba:ff:08:cf:18:25:
b5:ca:9d:d8:7b:db:34:75:48:d8:64:34:81:0d:03:c4:79:9b:
45:7e:fb:b7:d0:a6:8c:26:27:84:54:2b:56:fc:cf:82:e6:04:
0a:a0:df:06:9c:63:8d:ec:eb:f1:3f:5a:c5:f1:a6:d8:22:14:
9f:63:e7:50:fe:b9:a7:f1:38:f4:3f:4f:02:27:54:92:6c:f6:
35:6e:48:91:68:6f:9b:7b:5f:75:b9:1d:6c:93:69:1d:f9:cc:
d6:b3:e6:70:80:37:43:61:46:13:e8:e3:6c:64:e6:06:c9:e0:
50:bf:d7:57:7f:1c:ac:a4:ec:b0:e9:f2:e4:b7:29:f1:6e:e2:
62:55:93:a5:62:b0:d6:bd:c9:6b:fc:96:d9:a1:bc:f6:81:79:
46:f8:0f:2d:fd:73:43:d6:ea:1b:b6:3b:b9:8b:52:cf:58:46:
36:2c:b8:c0:6c:a9:88:60:fa:3a:3d:78:70:02:8e:2a:23:cb:
d8:c1:fd:8f:53:58:a2:cb:e2:d8:b6:c5:30:dd:9c:20:1c:96:
71:5f:15:09:ec:ee:65:b9:d5:a6:fb:9b:bf:91:d3:c4:3c:96:
d2:02:ed:bbNote also that the "Not After" dates in the certificates are 4–5 days earlier than the dates in the emails.
I started thinking about what might have changed at the most recent deployment, which was on 2023-11-21. This is the diff with the next most recent deployment on 2023-07-01. What's weird is that certificates for the snowflake-01 bridge's alternative domain names snowflake.freehaven.net and snowflake.bamsoftware.com have been renewed as recently as a week ago:
2023-10-03 05:45 snowflake.bamsoftware.com+rsa
2023-10-22 12:23 snowflake.torproject.net
2023-10-27 18:12 snowflake.torproject.net+rsa
2023-12-08 22:50 snowflake.freehaven.net+rsa
2023-12-10 04:39 snowflake.bamsoftware.com
2023-12-30 05:16 snowflake.freehaven.netAbout every 20–30 minutes I can see a file being created in the pt_state directory like:
-rw------- 1 snowflake-server nogroup 903 Jan 7 21:04 snowflake.torproject.net+tokenSo it seems like the HTTP-01 proof renewal machinery is getting invoked, but it's not getting completed for some reason.