LetsEncrypt
This wiki page is about the Certificate Authority "Let's Encrypt" and the technical details for obtaining and automatically renewing SSL certificates on a Debian server. In addition, the page focuses on CloudFlare because that is the DNS provider I currently use.
In older versions of Debian (specifically Debian jessie) it was necessary to run Certbot in a Docker image. Consult the history of this page to see how that worked.
Contents
Introduction
From the "Getting Started" page:
In order to get a certificate for your website's domain from Let's Encrypt, you have to demonstrate control over the domain. With Let's Encrypt, you do this using software that uses the ACME protocol, which typically runs on your web host. [...[ We recommend that most people with shell access use the Certbot ACME client. [...] If Certbot does not meet your needs, or you’d like to try something else, there are many more ACME clients to choose from. Once you’ve chosen ACME client software, see the documentation for that client to proceed.
I have been successfully using Certbot, so that's what I am going to write about in the sections below.
Furthermore, because I want to have wildcard certificates, the following is also important (quote from the FAQ):
[...] Wildcard issuance must be done via ACMEv2 using the DNS-01 challenge.
What is a "DNS-01" challenge? Without going into details, the challenge requires you to add a custom TXT record to the domain's DNS configuration so that Let's Encrypt can verify that you are in control of the domain.
References
- Let's Encrypt website
- Let's Encrypt information about wildcard certificates
- List of ACME clients. ACME is the communication protocol employed by Let's Encrypt to automatically issue certificates.
- Certbot website
- Certbot User Guide
- Running Certbot in Docker
- certbot-dns-cloudflare reference. This is the Certbot plugin that automates DNS-01 challenges.
- certbot-dns-cloudflare HOWTO from howtoforge.com
- certbot-dns-cloudflare HOWTO from eigenmagic.com
Preparing to use Certbot
Debian packages
This is the main Debian package to install:
certbot
I want wildcard certificates, so I can't just use plain Certbot because that has no support for automating DNS-01 challenges. Certbot has a plugin concept, and some plugins that handle DNS-01 challenges (called "DNS authenticator plugins") exist for various popular DNS providers. Fortunately a Certbot plugin exists for my DNS provider, which is Cloudflare. The plugin uses the Cloudflare web API to achieve its task. The plugin name is cerbot-dns-cloudflare, the Debian package that contains the plugin is this:
python3-certbot-dns-cloudflare
Filesystem folders
The certbot package sets up the main folder where everything related to Let's Encrypt (configuration, certificates, etc.) will be stored:
/etc/letsencrypt
Configuration files
The Certbot main configuration file is this
/etc/letsencrypt/cli.ini
The file content should look like this (everything except the max-log-backups option added by me):
root@pelargir:~# cat /etc/letsencrypt/cli.ini # Because we are using logrotate for greater flexibility, disable the # internal certbot logrotation. max-log-backups = 0 # Let's Encrypt site-wide configuration dns-cloudflare-credentials = /etc/letsencrypt/herzbube.ch.ini # Use the ACME v2 staging URI for testing things #server = https://acme-staging-v02.api.letsencrypt.org/directory # Production ACME v2 API endpoint server = https://acme-v02.api.letsencrypt.org/directory
Note the reference to /etc/letsencrypt/herzbube.ch.ini: This is the configuration file that will be used by Certbot's cerbot-dns-cloudflare plugin. It contains the Cloudflare API key which authorizes the Certbot plugin to automatically make DNS changes. This makes the configuration file very sensitive, and you need to make absolutely sure that the file is protected against unauthorized access. Note that we could protect against abuse with 2-factor authentication, but this would become a problem later on when we automate certificate renewal - you don't want to wake up in the middle of the night to confirm Certbot's renewal attempt. So this is how the config file looks like:
root@pelargir:~# ls -l /etc/letsencrypt/herzbube.ch.ini -r-------- 1 root root 140 Jun 7 22:28 /etc/letsencrypt/herzbube.ch.ini root@pelargir:~# cat /etc/letsencrypt/herzbube.ch.ini # CloudFlare API key information dns_cloudflare_api_key = 1234567890abcdef1234567890abcdef12345 dns_cloudflare_email = herzbube@herzbube.ch
Disable systemd timer for renewals
The certbot Debian package installs a systemd timer that runs twice per day to perform automated certificate renewals. Because I already have a cron job in place for doing that I am disabling the timer:
systemctl disable certbot.timer
Disabling the timer prevents it from being started when the system reboots the next time. Installing the Debian package also started the timer, so it must be stopped now:
systemctl stop certbot.timer
Obtaining certificates
At the time of writing I had four domains for which I wanted a wildcard certificate, so these were the four commands that I issued:
certbot certonly -d herzbube.ch -d *.herzbube.ch certbot certonly -d moser-naef.ch -d *.moser-naef.ch certbot certonly -d francescamoser.ch -d *.francescamoser.ch certbot certonly -d grunzwanzling.ch -d *.grunzwanzling.ch
Discussion:
certonlytells Certbot to obtain or renew a certificate, but not to install it.- The
-doption specifies the domain name, or "subject", to be used in the certificate. Multiple-doptions create a single certificate that contains multiple subjects. The first-dis used for the main subject. The "Certificate Subject Alt Name" (Firefox term) or "X509v3 Subject Alternative Name" (OpenSSL term) lists all subjects, including the main subject. Certbot stores the certificate material in a folder named after the main subject (i.e. the first-d).
When you run one of the Certbot commands you will be prompted as follows. Obviously you want to select option 1.
1: Obtain certificates using a DNS TXT record (if you are using Cloudflare for DNS). (dns-cloudflare) 2: Spin up a temporary webserver (standalone) 3: Place files in webroot directory (webroot)
We're almost ready to go now. A number of manual steps still have to be performed, though, before we can use the certificates that we just obtained.
Certbot files and folders
Account files
Certbot automatically creates a Let's Encrypt account when the first certificate is obtained. Certbot manages this automatically, so the details are not interesting here.
root@pelargir:~# ls -l /etc/letsencrypt/accounts/ total 12 drwx------ 3 root root 4096 Jun 7 22:14 . drwxr-xr-x 9 root root 4096 Jun 8 01:18 .. drwx------ 3 root root 4096 Jun 7 22:14 acme-v02.api.letsencrypt.org
Private keys
Certbot creates and stores private keys here. These files are duplicates of the files stored in the archive folder, which is discussed further down.
root@pelargir:~# ls -l /etc/letsencrypt/csr/ total 52 drwx------ 2 root root 4096 Jun 8 01:18 . drwxr-xr-x 9 root root 4096 Jun 8 01:18 .. -rw------- 1 root root 1704 Jun 7 22:14 0000_key-certbot.pem -rw------- 1 root root 1704 Jun 7 22:17 0001_key-certbot.pem -rw------- 1 root root 1708 Jun 7 22:23 0002_key-certbot.pem [...]
CSRs
Certbot creates and stores CSRs here:
root@pelargir:~# ls -l /etc/letsencrypt/csr/ total 52 drwxr-xr-x 2 root root 4096 Jun 8 01:18 . drwxr-xr-x 9 root root 4096 Jun 8 01:18 .. -rw-r--r-- 1 root root 924 Jun 7 22:14 0000_csr-certbot.pem -rw-r--r-- 1 root root 924 Jun 7 22:17 0001_csr-certbot.pem -rw-r--r-- 1 root root 924 Jun 7 22:23 0002_csr-certbot.pem [...]
Certificate material
As mentioned in the previous section, Certbot stores certificate material under the main subject's name, i.e. the first -d option specified on the Certbot command line. Note the special permissions of the /etc/letsencrypt/live folder - these are non-standard and will be explained in the next section.
root@pelargir:~# ls -l /etc/letsencrypt/live total 28 drwx--x--- 7 root ssl-cert 4096 Jun 8 01:18 . drwxr-xr-x 9 root root 4096 Jun 8 01:18 .. drwxr-xr-x 2 root root 4096 Jun 8 00:10 francescamoser.ch drwxr-xr-x 2 root root 4096 Jun 8 01:18 francescamoser.ch-0001 drwxr-xr-x 2 root root 4096 Jun 8 01:18 grunzwanzling.ch drwxr-xr-x 2 root root 4096 Jun 8 01:10 herzbube.ch drwxr-xr-x 2 root root 4096 Jun 8 01:17 moser-naef.ch root@pelargir:~# ls -l /etc/letsencrypt/live/herzbube.ch/ total 20 drwxr-xr-x 2 root root 4096 Jun 8 01:10 . drwx--x--- 7 root ssl-cert 4096 Jun 8 01:18 .. lrwxrwxrwx 1 root root 35 Jun 8 01:10 cert.pem -> ../../archive/herzbube.ch/cert2.pem lrwxrwxrwx 1 root root 36 Jun 8 01:10 chain.pem -> ../../archive/herzbube.ch/chain2.pem lrwxrwxrwx 1 root root 40 Jun 8 01:10 fullchain.pem -> ../../archive/herzbube.ch/fullchain2.pem lrwxrwxrwx 1 root root 38 Jun 8 01:10 privkey.pem -> ../../archive/herzbube.ch/privkey2.pem -rw-r--r-- 1 root root 682 Jun 7 22:32 README
As can be seen, the live folder consists of symlinks that together form the currently valid set of certificate material and that point back to an archive folder. Note the special permissions of the /etc/letsencrypt/live and /etc/letsencrypt/archive folders - these are non-standard and will be explained in the next section (Using Let's Encrypt certificates).
root@pelargir:~# ls -l /etc/letsencrypt/archive total 32 drwx--x--- 8 root ssl-cert 4096 Jun 8 01:18 . drwxr-xr-x 9 root root 4096 Jun 8 01:18 .. drwxr-xr-x 2 root root 4096 Jun 8 00:10 francescamoser.ch drwxr-xr-x 2 root root 4096 Jun 8 01:18 francescamoser.ch-0001 drwxr-xr-x 2 root root 4096 Jun 8 01:18 grunzwanzling.ch drwxr-xr-x 2 root root 4096 Jun 8 01:10 herzbube.ch drwxr-xr-x 2 root root 4096 Jun 8 01:08 herzbube.ch.org drwxr-xr-x 2 root root 4096 Jun 8 01:17 moser-naef.ch root@pelargir:~# ls -l /etc/letsencrypt/archive/herzbube.ch total 40 drwxr-xr-x 2 root root 4096 Jun 8 01:10 . drwx--x--- 8 root ssl-cert 4096 Jun 8 01:18 .. -rw-r--r-- 1 root root 2147 Jun 7 22:32 cert1.pem -rw-r--r-- 1 root root 2163 Jun 8 01:10 cert2.pem -rw-r--r-- 1 root root 1647 Jun 7 22:32 chain1.pem -rw-r--r-- 1 root root 1647 Jun 8 01:10 chain2.pem -rw-r--r-- 1 root root 3794 Jun 7 22:32 fullchain1.pem -rw-r--r-- 1 root root 3810 Jun 8 01:10 fullchain2.pem -rw-r--r-- 1 root root 1708 Jun 7 22:32 privkey1.pem -rw-r--r-- 1 root root 1704 Jun 8 01:10 privkey2.pem
Using Let's Encrypt certificates
The Let's Encrypt certificates will be used by the following services:
- Dovecot IMAP: See the Dovecot wiki page for details.
- Exim MTA: See the Exim wiki page for details. Most important is that Exim needs special permissions on the
/etc/letsencrypt/liveand/etc/letsencrypt/archivefolders (groupssl-certand permissions 710). - Apache: See the Apache wiki page for details.
Listing certificates
To list all certificates that Certbot knows about:
certbot certificates
Renewing certificates
Introduction
Because of the high degree of automation involved, certificates issued by Let's Encrypt are valid only for a short time - currently 90 days. It is therefore important to be able to automate the renewal process.
Let's Encrypt recommends to renew certificates when they have a third of their total lifetime left. With the typical 90 days lifetime of a Let's Encrypt certificate, this means renewal when 30 days (or less) are left.
Manual renewal
Manual renewal of certificates is very simple: Just run Certbot with the renew action:
certbot renew
General notes:
- The
renewaction attempts to renew any previously-obtained certificates that expire in less than 30 days. This means that, unlikecertonly, therenewaction operates on multiple certificates. - The same plugin and options that were used at the time the certificate was originally issued will be used for the renewal attempt (unless you specify other plugins or options). This information is taken from one of the domain-specific configuration files located in the folder
/etc/letsencrypt/renewal. Certbot generates these config files automatically when the certificate is obtained for the first time. - The
renewaction performs no action if a certificate is not yet due for renewal, i.e. if it's not yet within the 30 days expiry period. Because of this, therenewaction can be run as frequently as you want and is therefore ideally suited for automation. If you want to ignore the expiry period for some reason, run the command like this:certbot renew --force-renewal
Hooks
Certbot supports running so-called "pre hooks", "post hooks" and "deploy hooks", which are scripts or other executables that run before or after a renewal attempt. A "deploy hook" is run only if the renewal process was successful, whereas the "post hook" runs even if the renewal process failed.
Typical reasons why this capability can be important:
- A daemon must be restarted after a successful renewal so that it picks up the renewed certificate
- A daemon does not read its certificates as the
rootuser, so a post hook is necessary to copy the renewed certificate to another filesystem location and/or apply appropriate file permissions.
General notes about hooks:
- Hooks are run only if renewal is actually taking place. If Certbot finds that a certificate does not need renewal, it won't run any hooks.
- Hooks are run for every renewed certificate
- A successful hook must terminate with exit code 0, a failed hook must terminate with a non-zero exit code. A failed hook does not cause the renewal itself to fail, but it will cause a message printed to
stderr. - Certbot passes a number of environment variables to hooks. I have not found an official documentation that lists them, but a few can be found in the deploy hook example script in the Certbot user guide:
- RENEWED_DOMAINS contains the list of domain names in the renewed certificate
- RENEWED_LINEAGE refers to the
livefolder where the certificate material is stored. Example:/etc/letsencrypt/live/herzbube.ch
On the command line, a hook can be specified like this:
certbot renew --pre-hook "/path/to/hook-script.sh" certbot renew --post-hook "/path/to/hook-script.sh" certbot renew --deploy-hook "/path/to/hook-script.sh"
Certbot also runs any executable files that it finds in one of the following folders. Executables are run in alphabetical order of their file names. Hooks specified on the command line run after hooks found in the filesystem.
/etc/letsencrypt/renewal-hooks/pre /etc/letsencrypt/renewal-hooks/post /etc/letsencrypt/renewal-hooks/deploy
Hooks on pelargir.herzbube.ch
- Note: The solution presented in this section is overcomplicated in that the deploy hook script does not directly restart services, but instead delegates the task to a secondary script. This splitting of responsibilities is a left-over from the time when I used Certbot from within a Docker image: A hook script that runs within a Docker container and not on the host has no access to host system services, so it cannot restart those services on its own but has to delegate the task to an agent that runs outside the Docker container.
- Without a Docker container the deploy hook script could directly restart services. I'm not changing things, though, simply because the two-script solution is working.
On my server, I have only one deploy hook:
root@pelargir:~# ls -l /etc/letsencrypt/renewal-hooks/deploy total 8 -rwxr-xr-x 1 root root 377 Aug 29 16:32 20-notify-cron.sh
While I was using Courier IMAP I had another hook that re-generated a special file which contained the certificate chain and the private key in a single file. This was required by Courier IMAP. Since I have switched from Courier IMAP to Dovecot I no longer need this hook, I just mention it here to illustrate a possible use case for hook scripts.
Anyway, the hook script I have now generates a "notification file" whose name depends on the certificate that was renewed. Currently the script only distinguishes between the certificate for the domain herzbube.ch and all other certificates. The purpose of the "notification file" is explained below.
root@pelargir:~# cat /etc/letsencrypt/renewal-hooks/deploy/20-notify-cron.sh
#!/bin/sh
# It's important to not use Bash because it is not available in the
# Docker container that runs certbot.
# Exit immediately if a command exits with non-zero status
set -e
# The following path must be in /etc/letsencrypt so that this script,
# which runs in a Docker container, has access to it
NOTIFICATIONS_PATH="/etc/letsencrypt/renewal-notifications"
if test ! -d "$NOTIFICATIONS_PATH"; then
mkdir -p "$NOTIFICATIONS_PATH"
fi
case $RENEWED_DOMAINS in
*herzbube.ch*)
NOTIFICATION_FILE="$NOTIFICATIONS_PATH/herzbube.ch"
;;
*)
NOTIFICATION_FILE="$NOTIFICATIONS_PATH/other"
;;
esac
touch "$NOTIFICATION_FILE"
What actually needs to be done after a certificate is renewed is that one or more services that use the certificate are restarted. The hook script stores the information which certificate was renewed for later processing. It does so by creating a "notification file" whose name is significant. Another post-processing script, which must be run separately after the Certbot command has completed its work, then checks for the presence of one or more "notification files". If it finds one, the post-processing script performs the actual restart of the appropriate services. Here is the post-processing script:
root@pelargir:~# cat /usr/local/bin/restart-services-after-letsencrypt-cert-renewal.sh
#!/bin/sh
# It's important to not use Bash because it is not available in the
# Docker container that runs certbot.
# Exit immediately if a command exits with non-zero status
set -e
MYNAME="$(basename $0)"
LOGGER_PATH="/usr/bin/logger"
NOTIFICATIONS_PATH="/etc/letsencrypt/renewal-notifications"
if test ! -d "$NOTIFICATIONS_PATH"; then
logger "$MYNAME: No certificates were renewed"
exit 0
fi
EXIT_STATUS=0
logger "$MYNAME: One or more certificates were renewed"
RESTART_DOVECOT=0
RESTART_EXIM=0
RESTART_APACHE=0
for NOTIFY_FILE in $NOTIFICATIONS_PATH/*; do
NOTIFY_FILE_NAME="$(basename "$NOTIFY_FILE")"
logger "$MYNAME: Certificate for '$NOTIFY_FILE_NAME' was renewed (notification file $NOTIFY_FILE)"
case "$NOTIFY_FILE_NAME" in
# Dovecot and Exim run only under a herzbube.ch subdomain
herzbube.ch)
RESTART_DOVECOT=1
RESTART_EXIM=1
RESTART_APACHE=1
;;
# Apache has vhosts for all domains
other)
RESTART_APACHE=1
;;
*)
logger "$MYNAME: Unrecognized notification file $NOTIFY_FILE"
EXIT_STATUS=1
;;
esac
rm -f "$NOTIFY_FILE"
done
if test "$RESTART_DOVECOT" = "1"; then
logger "$MYNAME: Restarting service Dovecot IMAP"
systemctl restart dovecot.service
fi
if test "$RESTART_EXIM" = "1"; then
logger "$MYNAME: Restarting service Exim4 MTA"
systemctl restart exim4.service
fi
if test "$RESTART_APACHE" = "1"; then
logger "$MYNAME: Restarting service Apache2 HTTPD"
systemctl restart apache2.service
fi
rmdir "$NOTIFICATIONS_PATH"
exit $EXIT_STATUS
Automated renewal
Renewal can be automated by running the Certbot renew command from cron. The following runs the renewal process every Sunday at 04:00 in the morning:
root@pelargir:~# cat /etc/cron.d/pelargir-renew-certificates 0 4 * * 0 root certbot renew --quiet && /usr/local/bin/restart-services-after-letsencrypt-cert-renewal.sh
Notes:
- The
--quietoption suppresses all Certbot output except errors - The post-processing script must also be run to restart system services in case a renewal actually took place. See the previous section for details about the interaction between deploy hooks and the post-processing script.
- The renewal process must run at least twice a month, because Certbot only renews certificates 30 days before expiry but there are months that have 31 days. The renewal process should run at least every ten days, though, to avoid the expiry warning emails that Let's Encrypt is automatically sending 20 days before a certificate expires. On the web you can even find hints that it might be a good idea to run renewal daily or twice per day, to counteract a possible revocation initiated by Let's Encrypt. In my case, I decided to run renewal every week.
Troubleshooting
Unexplained renewal failure
On my first attempt to renew certificates, certbot renew managed to renew 4 out of 5 certificates but failed to renew the 5th certificate. This was the output:
------------------------------------------------------------------------------- Processing /etc/letsencrypt/renewal/grunzwanzling.ch.conf ------------------------------------------------------------------------------- Cert is due for renewal, auto-renewing... Plugins selected: Authenticator dns-cloudflare, Installer None Renewing an existing certificate Performing the following challenges: dns-01 challenge for grunzwanzling.ch dns-01 challenge for grunzwanzling.ch Waiting 10 seconds for DNS changes to propagate Waiting for verification... Cleaning up challenges Attempting to renew cert (grunzwanzling.ch) from /etc/letsencrypt/renewal/grunzwanzling.ch.conf produced an unexpected error: Failed authorization procedure. grunzwanzling.ch (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect TXT record "wr_SDxjaXE0LVjcF3RM-82Dhjh-ua9eSEDx-BWqXszk" found at _acme-challenge.grunzwanzling.ch. Skipping. The following certs could not be renewed: /etc/letsencrypt/live/grunzwanzling.ch/fullchain.pem (failure)
I am not sure why renewal failed. I logged in to my CloudFlare account via web browser to check the domain, but had to enter a challenge passcode first. After successfully logging in I didn't find anything unusual about the domain, so I decided to simply try the renewal again. Lo and behold! - It worked.
Is it possible that CloudFlare is limiting API access?
Revoking certificates
To revoke a certificate:
certbot revoke --cert-path /etc/letsencrypt/live/CERTNAME/cert.pem
After successfully revoking the certificate, Certbot should interactively ask whether you want to remove all certificate material. If it does not ask you can manually delete all traces of the certificate like this:
certbot delete --cert-name CERTNAME
