This wiki page is about the Certificate Authority "Let's Encrypt" and the technical details for obtaining and automatically renewing SSL certificates on a Debian server. In addition, the page focuses on CloudFlare because that is the DNS provider I currently use.
- 1 Introduction
- 2 References
- 3 Preparing to use Certbot
- 4 Obtaining certificates
- 5 Certbot files and folders
- 6 Using Let's Encrypt certificates
- 7 Listing certificates
- 8 Renewing certificates
- 9 Revoking certificates
From the "Getting Started" page:
In order to get a certificate for your website's domain from Let's Encrypt, you have to demonstrate control over the domain. With Let's Encrypt, you do this using software that uses the ACME protocol, which typically runs on your web host. [...[ We recommend that most people with shell access use the Certbot ACME client. [...] If Certbot does not meet your needs, or you’d like to try something else, there are many more ACME clients to choose from. Once you’ve chosen ACME client software, see the documentation for that client to proceed.
I have been successfully using Certbot, so that's what I am going to write about in the sections below.
Furthermore, because I want to have wildcard certificates, the following is also important (quote from the FAQ):
[...] Wildcard issuance must be done via ACMEv2 using the DNS-01 challenge.
What is a "DNS-01" challenge? Without going into details, the challenge requires you to add a custom TXT record to the domain's DNS configuration so that Let's Encrypt can verify that you are in control of the domain.
- Let's Encrypt website
- Let's Encrypt information about wildcard certificates
- List of ACME clients. ACME is the communication protocol employed by Let's Encrypt to automatically issue certificates.
- Certbot website
- Certbot User Guide
- Running Certbot in Docker
- certbot-dns-cloudflare reference. This is the Certbot plugin that automates DNS-01 challenges.
- certbot-dns-cloudflare HOWTO from howtoforge.com
- certbot-dns-cloudflare HOWTO from eigenmagic.com
Preparing to use Certbot
Certbot is available as a Debian package. For Debian jessie, the recommendation is to install Certbot like this:
apt-get install python-certbot-apache -t stretch-backports
Unfortunately this Certbot package does not support issuing wildcard certificates. The workaround is to run Certbot in Docker. As a prerequisite therefore the first thing to do is to install the
apt-get install docker.io
I want wildcard certificates, so I can't use the plain Docker image
certbot/certbot because that has no support for automating DNS-01 challenges. Certbot has a plugin concept, and some plugins that handle DNS-01 challenges (called "DNS authenticator plugins") exist for various popular DNS providers. Fortunately a Certbot plugin exists for my DNS provider, which is Cloudflare. The plugin name is
cerbot-dns-cloudflare, and the corresponding Docker image which contains both Certbot and the plugin and that we are going to use is
Before we can proceed with requesting certificates, we first have to set up a bit of infrastructure on the local system. Later on when we run the Docker image we will instruct it to use that infrastructure. Step 1 is to create some folders:
mkdir /etc/letsencrypt mkdir /var/lib/letsencrypt
Step 2 is to create the Certbot main configuration file
/etc/letsencrypt/cli.ini that will be used by Certbot when it runs within the Docker image. The file content looks like this:
root@pelargir:~# cat /etc/letsencrypt/cli.ini # Let's Encrypt site-wide configuration dns-cloudflare-credentials = /etc/letsencrypt/herzbube.ch.ini # Use the ACME v2 staging URI for testing things #server = https://acme-staging-v02.api.letsencrypt.org/directory # Production ACME v2 API endpoint server = https://acme-v02.api.letsencrypt.org/directory
Note the reference to
/etc/letsencrypt/herzbube.ch.ini: This is the configuration file that will be used by Certbot's
cerbot-dns-cloudflare plugin. It contains the Cloudflare API key which authorizes the Certbot plugin to automatically make DNS changes. This makes the configuration file very sensitive, and you need to make absolutely sure that the file is protected against unauthorized access. Note that we could protect against abuse with 2-factor authentication, but this would become a problem later on when we automate certificate renewal - you don't want to wake up in the middle of the night to confirm Certbot's renewal attempt. So this is how the config file looks like:
root@pelargir:~# ls -l /etc/letsencrypt/herzbube.ch.ini -r-------- 1 root root 140 Jun 7 22:28 /etc/letsencrypt/herzbube.ch.ini root@pelargir:~# cat /etc/letsencrypt/herzbube.ch.ini # CloudFlare API key information dns_cloudflare_api_key = 1234567890abcdef1234567890abcdef12345 dns_cloudflare_email = firstname.lastname@example.org
At the time of writing I had four domains for which I wanted a wildcard certificate, so these were the four commands that I issued:
docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare certonly -d herzbube.ch -d *.herzbube.ch docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare certonly -d moser-naef.ch -d *.moser-naef.ch docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare certonly -d francescamoser.ch -d *.francescamoser.ch docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare certonly -d grunzwanzling.ch -d *.grunzwanzling.ch
- The docker image to run is named "certbot/dns-cloudflare"
--rmoption causes the docker image to be removed after it has run. Note that
--rmis a boolean option which normally wants either "true" or "false" as an argument. The general practice for Docker's command line is that not specifying a boolean option's argument is the same as specifying "true" - so here we are effectively saying
-voption maps a folder from the host into the docker image (aka "mounting a volume" in Docker lingo). The two paths are specified with a colon (":") separator. Multiple
-voptions can be specified.
-itoption is actually two options:
-ioption is a boolean option that makes sure that the Docker process runs interactively (it opens STDIN)
-toption is a boolean option that makes sure that Docker allocates a pseudo TTY and attaches it to the container's STDIN
--nameoption assigns a name to the Docker container. Not important in this scenario
- "certonly" is the command that is passed to the Docker image's entry point (set in the Dockerfile)
-doption is an argument passed to Certbot. Multiple
-doptions create a single certificate that contains multiple subjects. The first
-dis used for the main subject. The "Certificate Subject Alt Name" (Firefox term) or "X509v3 Subject Alternative Name" (OpenSSL term) lists all subjects, including the main subject. Certbot stores the certificate material in a folder named after the main subject (i.e. the first
When you run one of the Docker commands you will be prompted as follows. Obviously you want to select option 1.
1: Obtain certificates using a DNS TXT record (if you are using Cloudflare for DNS). (dns-cloudflare) 2: Spin up a temporary webserver (standalone) 3: Place files in webroot directory (webroot)
We're almost ready to go now. A number of manual steps still have to be performed, though, before we can use the certificates that we just obtained.
Certbot files and folders
Certbot automatically creates a Let's Encrypt account when the first certificate is obtained. Certbot manages this automatically, so the details are not interesting here.
root@pelargir:~# ls -l /etc/letsencrypt/accounts/ total 12 drwx------ 3 root root 4096 Jun 7 22:14 . drwxr-xr-x 9 root root 4096 Jun 8 01:18 .. drwx------ 3 root root 4096 Jun 7 22:14 acme-v02.api.letsencrypt.org
Certbot creates and stores private keys here. These files are duplicates of the files stored in the
archive folder, which is discussed further down.
root@pelargir:~# ls -l /etc/letsencrypt/csr/ total 52 drwx------ 2 root root 4096 Jun 8 01:18 . drwxr-xr-x 9 root root 4096 Jun 8 01:18 .. -rw------- 1 root root 1704 Jun 7 22:14 0000_key-certbot.pem -rw------- 1 root root 1704 Jun 7 22:17 0001_key-certbot.pem -rw------- 1 root root 1708 Jun 7 22:23 0002_key-certbot.pem [...]
Certbot creates and stores CSRs here:
root@pelargir:~# ls -l /etc/letsencrypt/csr/ total 52 drwxr-xr-x 2 root root 4096 Jun 8 01:18 . drwxr-xr-x 9 root root 4096 Jun 8 01:18 .. -rw-r--r-- 1 root root 924 Jun 7 22:14 0000_csr-certbot.pem -rw-r--r-- 1 root root 924 Jun 7 22:17 0001_csr-certbot.pem -rw-r--r-- 1 root root 924 Jun 7 22:23 0002_csr-certbot.pem [...]
As mentioned in the previous section, Certbot stores certificate material under the main subject's name, i.e. the first
-d option specified on the Docker command line. Note the special permissions of the
/etc/letsencrypt/live folder - these are non-standard and will be explained in the next section.
root@pelargir:~# ls -l /etc/letsencrypt/live total 28 drwx--x--- 7 root ssl-cert 4096 Jun 8 01:18 . drwxr-xr-x 9 root root 4096 Jun 8 01:18 .. drwxr-xr-x 2 root root 4096 Jun 8 00:10 francescamoser.ch drwxr-xr-x 2 root root 4096 Jun 8 01:18 francescamoser.ch-0001 drwxr-xr-x 2 root root 4096 Jun 8 01:18 grunzwanzling.ch drwxr-xr-x 2 root root 4096 Jun 8 01:10 herzbube.ch drwxr-xr-x 2 root root 4096 Jun 8 01:17 moser-naef.ch root@pelargir:~# ls -l /etc/letsencrypt/live/herzbube.ch/ total 20 drwxr-xr-x 2 root root 4096 Jun 8 01:10 . drwx--x--- 7 root ssl-cert 4096 Jun 8 01:18 .. lrwxrwxrwx 1 root root 35 Jun 8 01:10 cert.pem -> ../../archive/herzbube.ch/cert2.pem lrwxrwxrwx 1 root root 36 Jun 8 01:10 chain.pem -> ../../archive/herzbube.ch/chain2.pem lrwxrwxrwx 1 root root 40 Jun 8 01:10 fullchain.pem -> ../../archive/herzbube.ch/fullchain2.pem lrwxrwxrwx 1 root root 38 Jun 8 01:10 privkey.pem -> ../../archive/herzbube.ch/privkey2.pem -rw-r--r-- 1 root root 682 Jun 7 22:32 README
As can be seen, the
live folder consists of symlinks that together form the currently valid set of certificate material and that point back to an
archive folder. Note the special permissions of the
/etc/letsencrypt/archive folders - these are non-standard and will be explained in the next section (Using Let's Encrypt certificates).
root@pelargir:~# ls -l /etc/letsencrypt/archive total 32 drwx--x--- 8 root ssl-cert 4096 Jun 8 01:18 . drwxr-xr-x 9 root root 4096 Jun 8 01:18 .. drwxr-xr-x 2 root root 4096 Jun 8 00:10 francescamoser.ch drwxr-xr-x 2 root root 4096 Jun 8 01:18 francescamoser.ch-0001 drwxr-xr-x 2 root root 4096 Jun 8 01:18 grunzwanzling.ch drwxr-xr-x 2 root root 4096 Jun 8 01:10 herzbube.ch drwxr-xr-x 2 root root 4096 Jun 8 01:08 herzbube.ch.org drwxr-xr-x 2 root root 4096 Jun 8 01:17 moser-naef.ch root@pelargir:~# ls -l /etc/letsencrypt/archive/herzbube.ch total 40 drwxr-xr-x 2 root root 4096 Jun 8 01:10 . drwx--x--- 8 root ssl-cert 4096 Jun 8 01:18 .. -rw-r--r-- 1 root root 2147 Jun 7 22:32 cert1.pem -rw-r--r-- 1 root root 2163 Jun 8 01:10 cert2.pem -rw-r--r-- 1 root root 1647 Jun 7 22:32 chain1.pem -rw-r--r-- 1 root root 1647 Jun 8 01:10 chain2.pem -rw-r--r-- 1 root root 3794 Jun 7 22:32 fullchain1.pem -rw-r--r-- 1 root root 3810 Jun 8 01:10 fullchain2.pem -rw-r--r-- 1 root root 1708 Jun 7 22:32 privkey1.pem -rw-r--r-- 1 root root 1704 Jun 8 01:10 privkey2.pem
Using Let's Encrypt certificates
The Let's Encrypt certificates will be used by the following services:
- Dovecot IMAP: See the Dovecot wiki page for details.
- Exim MTA: See the Exim wiki page for details. Most important is that Exim needs special permissions on the
ssl-certand permissions 710).
- Apache: See the Apache wiki page for details.
To list all certificates that Certbot knows about:
The corresponding Docker command line:
docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare certificates
Because of the high degree of automation involved, certificates issued by Let's Encrypt are valid only for a short time - currently 90 days. It is therefore important to be able to automate the renewal process.
Let's Encrypt recommends to renew certificates when they have a third of their total lifetime left. With the typical 90 days lifetime of a Let's Encrypt certificate, this means renewal when 30 days (or less) are left.
Manual renewal of certificates is very simple: Just run Certbot with the
Because I am working with Docker (due to wildcard certificates), the actual command is more complicated, but not more so than the normal command for obtaining a new command.
docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare renew
renewaction attempts to renew any previously-obtained certificates that expire in less than 30 days. This means that, unlike
renewaction operates on multiple certificates.
- The same plugin and options that were used at the time the certificate was originally issued will be used for the renewal attempt (unless you specify other plugins or options). This information is taken from one of the domain-specific configuration files located in the folder
/etc/letsencrypt/renewal. Certbot generates these config files automatically when the certificate is obtained for the first time.
renewaction performs no action if a certificate is not yet due for renewal, i.e. if it's not yet within the 30 days expiry period. Because of this, the
renewaction can be run as frequently as you want and is therefore ideally suited for automation. If you want to ignore the expiry period for some reason, run the command like this:
certbot renew --force-renewal
Certbot supports running so-called "pre hooks", "post hooks" and "deploy hooks", which are scripts or other executables that run before or after a renewal attempt. A "deploy hook" is run only if the renewal process was successful, whereas the "post hook" runs even if the renewal process failed.
Typical reasons why this capability can be important:
- A daemon must be restarted after a successful renewal so that it picks up the renewed certificate
- A daemon does not read its certificates as the
rootuser, so a post hook is necessary to copy the renewed certificate to another filesystem location and/or apply appropriate file permissions.
General notes about hooks:
- Hooks are run only if renewal is actually taking place. If Certbot finds that a certificate does not need renewal, it won't run any hooks.
- Hooks are run for every renewed certificate
- A successful hook must terminate with exit code 0, a failed hook must terminate with a non-zero exit code. A failed hook does not cause the renewal itself to fail, but it will cause a message printed to
- Certbot passes a number of environment variables to hooks. I have not found an official documentation that lists them, but a few can be found in the deploy hook example script in the Certbot user guide:
- RENEWED_DOMAINS contains the list of domain names in the renewed certificate
- RENEWED_LINEAGE refers to the
livefolder where the certificate material is stored. Example:
On the command line, a hook can be specified like this:
certbot renew --pre-hook "/path/to/hook-script.sh" certbot renew --post-hook "/path/to/hook-script.sh" certbot renew --deploy-hook "/path/to/hook-script.sh"
Certbot also runs any executable files that it finds in one of the following folders. Executables are run in alphabetical order of their file names. Hooks specified on the command line run after hooks found in the filesystem.
/etc/letsencrypt/renewal-hooks/pre /etc/letsencrypt/renewal-hooks/post /etc/letsencrypt/renewal-hooks/deploy
On my server, I have only one deploy hook:
root@pelargir:~# ls -l /etc/letsencrypt/renewal-hooks/deploy total 8 -rwxr-xr-x 1 root root 377 Aug 29 16:32 20-notify-cron.sh
While I was using Courier IMAP I had another hook that re-generated a special file which contained the certificate chain and the private key in a single file. This was required by Courier IMAP. Since I have switched from Courier IMAP to Dovecot I no longer need this hook, I just mention it here to illustrate a possible use case for hook scripts.
Anyway, the hook script I have now generates a "notification file" whose name depends on the certificate that was renewed. Currently the script only distinguishes between the certificate for the domain
herzbube.ch and all other certificates. The purpose of the "notification file" is explained below.
root@pelargir:~# cat /etc/letsencrypt/renewal-hooks/deploy/20-notify-cron.sh #!/bin/sh # It's important to not use Bash because it is not available in the # Docker container that runs certbot. # Exit immediately if a command exits with non-zero status set -e # The following path must be in /etc/letsencrypt so that this script, # which runs in a Docker container, has access to it NOTIFICATIONS_PATH="/etc/letsencrypt/renewal-notifications" if test ! -d "$NOTIFICATIONS_PATH"; then mkdir -p "$NOTIFICATIONS_PATH" fi case $RENEWED_DOMAINS in *herzbube.ch*) NOTIFICATION_FILE="$NOTIFICATIONS_PATH/herzbube.ch" ;; *) NOTIFICATION_FILE="$NOTIFICATIONS_PATH/other" ;; esac touch "$NOTIFICATION_FILE"
What actually needs to be done after a certificate is renewed is that one or more services that use the certificate are restarted. Unfortunately the hook script itself cannot restart those services because it runs within a Docker container and not on the host, so it has no access to host system services. The solution is that the hook script stores the information which certificate was renewed for later processing. It does so by creating a "notification file" whose name is significant. Another post-processing script, which must be run separately after the Docker container has completed its work, then checks for the presence of one or more "notification files". If it finds one, the post-processing script performs the actual restart of the appropriate services. Here is the post-processing script:
root@pelargir:~# cat /usr/local/bin/restart-services-after-letsencrypt-cert-renewal.sh #!/bin/sh # It's important to not use Bash because it is not available in the # Docker container that runs certbot. # Exit immediately if a command exits with non-zero status set -e MYNAME="$(basename $0)" LOGGER_PATH="/usr/bin/logger" NOTIFICATIONS_PATH="/etc/letsencrypt/renewal-notifications" if test ! -d "$NOTIFICATIONS_PATH"; then logger "$MYNAME: No certificates were renewed" exit 0 fi EXIT_STATUS=0 logger "$MYNAME: One or more certificates were renewed" RESTART_DOVECOT=0 RESTART_EXIM=0 RESTART_APACHE=0 for NOTIFY_FILE in $NOTIFICATIONS_PATH/*; do NOTIFY_FILE_NAME="$(basename "$NOTIFY_FILE")" logger "$MYNAME: Certificate for '$NOTIFY_FILE_NAME' was renewed (notification file $NOTIFY_FILE)" case "$NOTIFY_FILE_NAME" in # Dovecot and Exim run only under a herzbube.ch subdomain herzbube.ch) RESTART_DOVECOT=1 RESTART_EXIM=1 RESTART_APACHE=1 ;; # Apache has vhosts for all domains other) RESTART_APACHE=1 ;; *) logger "$MYNAME: Unrecognized notification file $NOTIFY_FILE" EXIT_STATUS=1 ;; esac rm -f "$NOTIFY_FILE" done if test "$RESTART_DOVECOT" = "1"; then logger "$MYNAME: Restarting service Dovecot IMAP" systemctl restart dovecot.service fi if test "$RESTART_EXIM" = "1"; then logger "$MYNAME: Restarting service Exim4 MTA" systemctl restart exim4.service fi if test "$RESTART_APACHE" = "1"; then logger "$MYNAME: Restarting service Apache2 HTTPD" systemctl restart apache2.service fi rmdir "$NOTIFICATIONS_PATH" exit $EXIT_STATUS
Renewal can be automated by running the Certbot
renew command from
cron. The following runs the renewal process every Sunday at 04:00 in the morning:
root@pelargir:~# cat /etc/cron.d/pelargir-renew-certificates 0 4 * * 0 root docker run --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare renew --quiet && /usr/local/bin/restart-services-after-letsencrypt-cert-renewal.sh
- Because this runs in
cron, the Docker command must not use the options
-it- these are for interactive use and require a TTY
--quietoption suppresses all Certbot output except errors
- The post-processing script must also be run to restart system services in case a renewal actually took place. See the previous section for details about the interaction between deploy hooks inside the Docker container and the post-processing script.
- The renewal process must run at least twice a month, because Certbot only renews certificates 30 days before expiry but there are months that have 31 days. The renewal process should run at least every ten days, though, to avoid the expiry warning emails that Let's Encrypt is automatically sending 20 days before a certificate expires. On the web you can even find hints that it might be a good idea to run renewal daily or twice per day, to counteract a possible revocation initiated by Let's Encrypt. In my case, I decided to run renewal every week.
Unexplained renewal failure
On my first attempt to renew certificates,
certbot renew managed to renew 4 out of 5 certificates but failed to renew the 5th certificate. This was the output:
------------------------------------------------------------------------------- Processing /etc/letsencrypt/renewal/grunzwanzling.ch.conf ------------------------------------------------------------------------------- Cert is due for renewal, auto-renewing... Plugins selected: Authenticator dns-cloudflare, Installer None Renewing an existing certificate Performing the following challenges: dns-01 challenge for grunzwanzling.ch dns-01 challenge for grunzwanzling.ch Waiting 10 seconds for DNS changes to propagate Waiting for verification... Cleaning up challenges Attempting to renew cert (grunzwanzling.ch) from /etc/letsencrypt/renewal/grunzwanzling.ch.conf produced an unexpected error: Failed authorization procedure. grunzwanzling.ch (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect TXT record "wr_SDxjaXE0LVjcF3RM-82Dhjh-ua9eSEDx-BWqXszk" found at _acme-challenge.grunzwanzling.ch. Skipping. The following certs could not be renewed: /etc/letsencrypt/live/grunzwanzling.ch/fullchain.pem (failure)
I am not sure why renewal failed. I logged in to my CloudFlare account via web browser to check the domain, but had to enter a challenge passcode first. After successfully logging in I didn't find anything unusual about the domain, so I decided to simply try the renewal again. Lo and behold! - It worked.
Is it possible that CloudFlare is limiting API access?
To revoke a certificate:
certbot revoke --cert-path /etc/letsencrypt/live/CERTNAME/cert.pem
The corresponding Docker command line:
docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare revoke --cert-path /etc/letsencrypt/live/CERTNAME/cert.pem
After successfully revoking the certificate, Certbot should interactively ask whether you want to remove all certificate material. If it does not ask you can manually delete all traces of the certificate like this:
certbot delete --cert-name CERTNAME docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare delete --cert-name CERTNAME