For about 2 hours our Let's Encrypt certificates all appeared to be expired in at least some browsers. We tried with Firefox on Linux, Firefox and Chrome on Windows and both were fine. The trouble was real though. I am guessing that those browsers and possibly others had foreseen this coming and made a special case for it.
The problem was that the certificate used in the chain had expired. The Let's Encrypt people did warn that this was coming here:
https://letsencrypt.org/docs/dst-roo...eptember-2021/ I do remember reading that a few months ago and checking and I must have read something backwards because I thought it wasn't going to be an issue for us and then I forgot about it. Luckily it didn't take me long to remember the notice and discover that it really was the problem even if things still worked fine in my browser.
Unfortunately, finding the correct chain file didn't solve the problem. Much of the downtime was due to a problem with our certificate management system. Simply put a chain certificate had never been modified before and the code to do that just didn't work. In the past chain certificates have only ever been replaced by completely new ones leaving the old ones for older certs and the new only applying to new certs. This is why their names tend to end with a number that is incremented.
So, in the end, the tool wouldn't do the job. The data (and other things the tool was supposed to calculate) had to be changed within a MySQL table manually. Anyone who has ever written a MySQL UPDATE query where one of the values is a pasting of a text file will know that the only thing good about that sentence is that it doesn't say binary file.
Our apologies for the outage. Thanks for sending in enough queries quickly enough to make us realize that the problem was real even if we couldn't see it.
Linear Mode
