LetsEncrypt

From HerzbubeWiki
Jump to: navigation, search

This wiki page is about the Certificate Authority "Let's Encrypt" and the technical details for obtaining and automatically renewing SSL certificates on a Debian server. In addition, the page focuses on CloudFlare because that is the DNS provider I currently use.


Introduction

From the "Getting Started" page:
In order to get a certificate for your website's domain from Let's Encrypt, you have to demonstrate control over the domain. With Let's Encrypt, you do this using software that uses the ACME protocol, which typically runs on your web host. [...[ We recommend that most people with shell access use the Certbot ACME client. [...] If Certbot does not meet your needs, or you’d like to try something else, there are many more ACME clients to choose from. Once you’ve chosen ACME client software, see the documentation for that client to proceed.

I have been successfully using Certbot, so that's what I am going to write about in the sections below.

Furthermore, because I want to have wildcard certificates, the following is also important (quote from the FAQ):
[...] Wildcard issuance must be done via ACMEv2 using the DNS-01 challenge.
What is a "DNS-01" challenge? Without going into details, the challenge requires you to add a custom TXT record to the domain's DNS configuration so that Let's Encrypt can verify that you are in control of the domain.


References


Preparing to use Certbot

Docker

Certbot is available as a Debian package. For Debian jessie, the recommendation is to install Certbot like this:

apt-get install python-certbot-apache -t stretch-backports


Unfortunately this Certbot package does not support issuing wildcard certificates. The workaround is to run Certbot in Docker. As a prerequisite therefore the first thing to do is to install the docker.io package:

apt-get install docker.io


I want wildcard certificates, so I can't use the plain Docker image certbot/certbot because that has no support for automating DNS-01 challenges. Certbot has a plugin concept, and some plugins that handle DNS-01 challenges (called "DNS authenticator plugins") exist for various popular DNS providers. Fortunately a Certbot plugin exists for my DNS provider, which is Cloudflare. The plugin name is cerbot-dns-cloudflare, and the corresponding Docker image which contains both Certbot and the plugin and that we are going to use is

certbot/dns-cloudflare


Filesystem folders

Before we can proceed with requesting certificates, we first have to set up a bit of infrastructure on the local system. Later on when we run the Docker image we will instruct it to use that infrastructure. Step 1 is to create some folders:

mkdir /etc/letsencrypt
mkdir /var/lib/letsencrypt


Configuration files

Step 2 is to create the Certbot main configuration file /etc/letsencrypt/cli.ini that will be used by Certbot when it runs within the Docker image. The file content looks like this:

root@pelargir:~# cat /etc/letsencrypt/cli.ini
# Let's Encrypt site-wide configuration
dns-cloudflare-credentials = /etc/letsencrypt/herzbube.ch.ini
# Use the ACME v2 staging URI for testing things
#server = https://acme-staging-v02.api.letsencrypt.org/directory
# Production ACME v2 API endpoint
server = https://acme-v02.api.letsencrypt.org/directory


Note the reference to /etc/letsencrypt/herzbube.ch.ini: This is the configuration file that will be used by Certbot's cerbot-dns-cloudflare plugin. It contains the Cloudflare API key which authorizes the Certbot plugin to automatically make DNS changes. This makes the configuration file very sensitive, and you need to make absolutely sure that the file is protected against unauthorized access. Note that we could protect against abuse with 2-factor authentication, but this would become a problem later on when we automate certificate renewal - you don't want to wake up in the middle of the night to confirm Certbot's renewal attempt. So this is how the config file looks like:

root@pelargir:~# ls -l /etc/letsencrypt/herzbube.ch.ini 
-r-------- 1 root root 140 Jun  7 22:28 /etc/letsencrypt/herzbube.ch.ini

root@pelargir:~# cat /etc/letsencrypt/herzbube.ch.ini 
# CloudFlare API key information
dns_cloudflare_api_key = 1234567890abcdef1234567890abcdef12345
dns_cloudflare_email = herzbube@herzbube.ch


Obtaining certificates

At the time of writing I had four domains for which I wanted a wildcard certificate, so these were the four commands that I issued:

docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare certonly -d herzbube.ch -d *.herzbube.ch
docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare certonly -d moser-naef.ch -d *.moser-naef.ch
docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare certonly -d francescamoser.ch -d *.francescamoser.ch
docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare certonly -d grunzwanzling.ch -d *.grunzwanzling.ch

Discussion:

  • The docker image to run is named "certbot/dns-cloudflare"
  • The --rm option causes the docker image to be removed after it has run. Note that --rm is a boolean option which normally wants either "true" or "false" as an argument. The general practice for Docker's command line is that not specifying a boolean option's argument is the same as specifying "true" - so here we are effectively saying --rm=true.
  • The -v option maps a folder from the host into the docker image (aka "mounting a volume" in Docker lingo). The two paths are specified with a colon (":") separator. Multiple -v options can be specified.
  • The -it option is actually two options:
    • The -i option is a boolean option that makes sure that the Docker process runs interactively (it opens STDIN)
    • The -t option is a boolean option that makes sure that Docker allocates a pseudo TTY and attaches it to the container's STDIN
  • The --name option assigns a name to the Docker container. Not important in this scenario
  • "certonly" is the command that is passed to the Docker image's entry point (set in the Dockerfile)
  • The -d option is an argument passed to Certbot. Multiple -d options create a single certificate that contains multiple subjects. The first -d is used for the main subject. The "Certificate Subject Alt Name" (Firefox term) or "X509v3 Subject Alternative Name" (OpenSSL term) lists all subjects, including the main subject. Certbot stores the certificate material in a folder named after the main subject (i.e. the first -d).


When you run one of the Docker commands you will be prompted as follows. Obviously you want to select option 1.

1: Obtain certificates using a DNS TXT record (if you are using Cloudflare for DNS). (dns-cloudflare)
2: Spin up a temporary webserver (standalone)
3: Place files in webroot directory (webroot)


We're almost ready to go now. A number of manual steps still have to be performed, though, before we can use the certificates that we just obtained.


Certbot files and folders

Account files

Certbot automatically creates a Let's Encrypt account when the first certificate is obtained. Certbot manages this automatically, so the details are not interesting here.

root@pelargir:~# ls -l /etc/letsencrypt/accounts/
total 12
drwx------ 3 root root 4096 Jun  7 22:14 .
drwxr-xr-x 9 root root 4096 Jun  8 01:18 ..
drwx------ 3 root root 4096 Jun  7 22:14 acme-v02.api.letsencrypt.org


Private keys

Certbot creates and stores private keys here. These files are duplicates of the files stored in the archive folder, which is discussed further down.

root@pelargir:~# ls -l /etc/letsencrypt/csr/
total 52
drwx------ 2 root root 4096 Jun  8 01:18 .
drwxr-xr-x 9 root root 4096 Jun  8 01:18 ..
-rw------- 1 root root 1704 Jun  7 22:14 0000_key-certbot.pem
-rw------- 1 root root 1704 Jun  7 22:17 0001_key-certbot.pem
-rw------- 1 root root 1708 Jun  7 22:23 0002_key-certbot.pem
[...]


CSRs

Certbot creates and stores CSRs here:

root@pelargir:~# ls -l /etc/letsencrypt/csr/
total 52
drwxr-xr-x 2 root root 4096 Jun  8 01:18 .
drwxr-xr-x 9 root root 4096 Jun  8 01:18 ..
-rw-r--r-- 1 root root  924 Jun  7 22:14 0000_csr-certbot.pem
-rw-r--r-- 1 root root  924 Jun  7 22:17 0001_csr-certbot.pem
-rw-r--r-- 1 root root  924 Jun  7 22:23 0002_csr-certbot.pem
[...]


Certificate material

As mentioned in the previous section, Certbot stores certificate material under the main subject's name, i.e. the first -d option specified on the Docker command line. Note the special permissions of the /etc/letsencrypt/live folder - these are non-standard and will be explained in the next section.

root@pelargir:~# ls -l /etc/letsencrypt/live
total 28
drwx--x--- 7 root ssl-cert 4096 Jun  8 01:18 .
drwxr-xr-x 9 root root     4096 Jun  8 01:18 ..
drwxr-xr-x 2 root root     4096 Jun  8 00:10 francescamoser.ch
drwxr-xr-x 2 root root     4096 Jun  8 01:18 francescamoser.ch-0001
drwxr-xr-x 2 root root     4096 Jun  8 01:18 grunzwanzling.ch
drwxr-xr-x 2 root root     4096 Jun  8 01:10 herzbube.ch
drwxr-xr-x 2 root root     4096 Jun  8 01:17 moser-naef.ch
root@pelargir:~# ls -l /etc/letsencrypt/live/herzbube.ch/
total 20
drwxr-xr-x 2 root root     4096 Jun  8 01:10 .
drwx--x--- 7 root ssl-cert 4096 Jun  8 01:18 ..
lrwxrwxrwx 1 root root       35 Jun  8 01:10 cert.pem -> ../../archive/herzbube.ch/cert2.pem
lrwxrwxrwx 1 root root       36 Jun  8 01:10 chain.pem -> ../../archive/herzbube.ch/chain2.pem
-rw-r--r-- 1 root root     5514 Jun  8 01:13 courier.cert-and-key.unsecure
lrwxrwxrwx 1 root root       40 Jun  8 01:10 fullchain.pem -> ../../archive/herzbube.ch/fullchain2.pem
lrwxrwxrwx 1 root root       38 Jun  8 01:10 privkey.pem -> ../../archive/herzbube.ch/privkey2.pem
-rw-r--r-- 1 root root      682 Jun  7 22:32 README


As can be seen, the live folder consists of symlinks that together form the currently valid set of certificate material and that point back to an archive folder. Note the special permissions of the /etc/letsencrypt/live and /etc/letsencrypt/archive folders and the special file courier.cert-and-key.unsecure - these are non-standard and will be explained in the next section (Using Let's Encrypt certificates).

root@pelargir:~# ls -l /etc/letsencrypt/archive
total 32
drwx--x--- 8 root ssl-cert 4096 Jun  8 01:18 .
drwxr-xr-x 9 root root     4096 Jun  8 01:18 ..
drwxr-xr-x 2 root root     4096 Jun  8 00:10 francescamoser.ch
drwxr-xr-x 2 root root     4096 Jun  8 01:18 francescamoser.ch-0001
drwxr-xr-x 2 root root     4096 Jun  8 01:18 grunzwanzling.ch
drwxr-xr-x 2 root root     4096 Jun  8 01:10 herzbube.ch
drwxr-xr-x 2 root root     4096 Jun  8 01:08 herzbube.ch.org
drwxr-xr-x 2 root root     4096 Jun  8 01:17 moser-naef.ch
root@pelargir:~# ls -l /etc/letsencrypt/archive/herzbube.ch
total 40
drwxr-xr-x 2 root root     4096 Jun  8 01:10 .
drwx--x--- 8 root ssl-cert 4096 Jun  8 01:18 ..
-rw-r--r-- 1 root root     2147 Jun  7 22:32 cert1.pem
-rw-r--r-- 1 root root     2163 Jun  8 01:10 cert2.pem
-rw-r--r-- 1 root root     1647 Jun  7 22:32 chain1.pem
-rw-r--r-- 1 root root     1647 Jun  8 01:10 chain2.pem
-rw-r--r-- 1 root root     3794 Jun  7 22:32 fullchain1.pem
-rw-r--r-- 1 root root     3810 Jun  8 01:10 fullchain2.pem
-rw-r--r-- 1 root root     1708 Jun  7 22:32 privkey1.pem
-rw-r--r-- 1 root root     1704 Jun  8 01:10 privkey2.pem


Using Let's Encrypt certificates

The Let's Encrypt certificates will be used by the following services:

  • Courier IMAP: See the Courier wiki page for details. Most important is that Courier needs the certificate chain and the private key to be present in a single file. This is why we must manually create courier.cert-and-key.unsecure.
  • Exim MTA: See the Exim wiki page for details. Most important is that Exim needs special permissions on the /etc/letsencrypt/live and /etc/letsencrypt/archive folders (group ssl-cert and permissions 710).
  • Apache: See the Apache wiki page for details.


Listing certificates

To list all certificates that Certbot knows about:

certbot certificates

The corresponding Docker command line:

docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare certificates


Renewing certificates

Introduction

Because of the high degree of automation involved, certificates issued by Let's Encrypt are valid only for a short time - currently 90 days. It is therefore important to be able to automate the renewal process.

Let's Encrypt recommends to renew certificates when they have a third of their total lifetime left. With the typical 90 days lifetime of a Let's Encrypt certificate, this means renewal when 30 days (or less) are left.


Manual renewal

Manual renewal of certificates is very simple: Just run Certbot with the renew action:

certbot renew

Because I am working with Docker (due to wildcard certificates), the actual command is more complicated, but not more so than the normal command for obtaining a new command.

docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare renew

General notes:

  • The renew action attempts to renew any previously-obtained certificates that expire in less than 30 days. This means that, unlike certonly, the renew action operates on multiple certificates.
  • The same plugin and options that were used at the time the certificate was originally issued will be used for the renewal attempt (unless you specify other plugins or options). This information is taken from one of the domain-specific configuration files located in the folder /etc/letsencrypt/renewal. Certbot generates these config files automatically when the certificate is obtained for the first time.
  • The renew action performs no action if a certificate is not yet due for renewal, i.e. if it's not yet within the 30 days expiry period. Because of this, the renew action can be run as frequently as you want and is therefore ideally suited for automation. If you want to ignore the expiry period for some reason, run the command like this: certbot renew --force-renewal


Hooks

Certbot supports running so-called "pre hooks", "post hooks" and "deploy hooks", which are scripts or other executables that run before or after a renewal attempt. A "deploy hook" is run only if the renewal process was successful, whereas the "post hook" runs even if the renewal process failed.


Typical reasons why this capability can be important:

  • A daemon must be restarted after a successful renewal so that it picks up the renewed certificate
  • A daemon does not read its certificates as the root user, so a post hook is necessary to copy the renewed certificate to another filesystem location and/or apply appropriate file permissions.


General notes about hooks:

  • Hooks are run only if renewal is actually taking place. If Certbot finds that a certificate does not need renewal, it won't run any hooks.
  • Hooks are run for every renewed certificate
  • A successful hook must terminate with exit code 0, a failed hook must terminate with a non-zero exit code. A failed hook does not cause the renewal itself to fail, but it will cause a message printed to stderr.
  • Certbot passes a number of environment variables to hooks. I have not found an official documentation that lists them, but a few can be found in the deploy hook example script in the Certbot user guide:
    • RENEWED_DOMAINS contains the list of domain names in the renewed certificate
    • RENEWED_LINEAGE refers to the live folder where the certificate material is stored. Example: /etc/letsencrypt/live/herzbube.ch


On the command line, a hook can be specified like this:

certbot renew --pre-hook "/path/to/hook-script.sh"
certbot renew --post-hook "/path/to/hook-script.sh"
certbot renew --deploy-hook "/path/to/hook-script.sh"


Certbot also runs any executable files that it finds in one of the following folders. Executables are run in alphabetical order of their file names. Hooks specified on the command line run after hooks found in the filesystem.

/etc/letsencrypt/renewal-hooks/pre
/etc/letsencrypt/renewal-hooks/post
/etc/letsencrypt/renewal-hooks/deploy


On my server, I have these deploy hooks:

root@pelargir:~# ls -l /etc/letsencrypt/renewal-hooks/deploy
total 8
-rwxr-xr-x 1 root root 697 Aug 29 16:35 10-setup-courier.sh
-rwxr-xr-x 1 root root 377 Aug 29 16:32 20-notify-cron.sh


The first hook re-generates the file required by Courier IMAP:

root@pelargir:~# cat /etc/letsencrypt/renewal-hooks/deploy/10-setup-courier.sh 
#!/bin/sh

# It's important to not use Bash because it is not available in the
# Docker container that runs certbot.

# Exit immediately if a command exits with non-zero status
set -e

case $RENEWED_DOMAINS in
  # Courier runs only under a herzbube.ch subdomain
  *herzbube.ch*)

    # We don't care about file permissions because we know that the
    # filesystem folder where we generate the file is not generally
    # accessible
    cat "$RENEWED_LINEAGE/fullchain.pem" "$RENEWED_LINEAGE/privkey.pem" >"$RENEWED_LINEAGE/courier.cert-and-key.unsecure"
    ;;
esac


The second hook script generates a "notification file" whose name depends on the certificate that was renewed. Currently the script only distinguishes between the certificate for the domain herzbube.ch and all other certificates. The purpose of the "notification file" is explained below.

root@pelargir:~# cat /etc/letsencrypt/renewal-hooks/deploy/20-notify-cron.sh
#!/bin/sh

# It's important to not use Bash because it is not available in the
# Docker container that runs certbot.

# Exit immediately if a command exits with non-zero status
set -e

# The following path must be in /etc/letsencrypt so that this script,
# which runs in a Docker container, has access to it
NOTIFICATIONS_PATH="/etc/letsencrypt/renewal-notifications"
if test ! -d "$NOTIFICATIONS_PATH"; then
  mkdir -p "$NOTIFICATIONS_PATH"
fi
  
case $RENEWED_DOMAINS in
  *herzbube.ch*)
    NOTIFICATION_FILE="$NOTIFICATIONS_PATH/herzbube.ch"
    ;;
      
  *)  
    NOTIFICATION_FILE="$NOTIFICATIONS_PATH/other"
    ;;
esac

touch "$NOTIFICATION_FILE"


What actually needs to be done after a certificate is renewed is that one or more services that use the certificate are restarted. Unfortunately the hook script itself cannot restart those services because it runs within a Docker container and not on the host, so it has no access to host system services. The solution is that the hook script stores the information which certificate was renewed for later processing. It does so by creating a "notification file" whose name is significant. Another post-processing script, which must be run separately after the Docker container has completed its work, then checks for the presence of one or more "notification files". If it finds one, the post-processing script performs the actual restart of the appropriate services. Here is the post-processing script:

root@pelargir:~# cat /usr/local/bin/restart-services-after-letsencrypt-cert-renewal.sh
#!/bin/sh

# It's important to not use Bash because it is not available in the
# Docker container that runs certbot.

# Exit immediately if a command exits with non-zero status
set -e

MYNAME="$(basename $0)"
LOGGER_PATH="/usr/bin/logger"
NOTIFICATIONS_PATH="/etc/letsencrypt/renewal-notifications"

if test ! -d "$NOTIFICATIONS_PATH"; then
  logger "$MYNAME: No certificates were renewed"
  exit 0
fi

EXIT_STATUS=0

logger "$MYNAME: One or more certificates were renewed"

RESTART_COURIER=0
RESTART_EXIM=0
RESTART_APACHE=0

for NOTIFY_FILE in $NOTIFICATIONS_PATH/*; do
  NOTIFY_FILE_NAME="$(basename "$NOTIFY_FILE")"

  logger "$MYNAME: Certificate for '$NOTIFY_FILE_NAME' was renewed (notification file $NOTIFY_FILE)"

  case "$NOTIFY_FILE_NAME" in
    # Courier and Exim run only under a herzbube.ch subdomain
    herzbube.ch)
      RESTART_COURIER=1
      RESTART_EXIM=1
      RESTART_APACHE=1
      ;;
    # Apache has vhosts for all domains
    other)
      RESTART_APACHE=1
      ;;
    *)
      logger "$MYNAME: Unrecognized notification file $NOTIFY_FILE"
      EXIT_STATUS=1
      ;;
  esac

  rm -f "$NOTIFY_FILE"
done

if test "$RESTART_COURIER" = "1"; then
  logger "$MYNAME: Restarting service Courier IMAP"
  systemctl restart courier-imap.service
fi

if test "$RESTART_EXIM" = "1"; then
  logger "$MYNAME: Restarting service Exim4 MTA"
  systemctl restart exim4.service
fi

if test "$RESTART_APACHE" = "1"; then
  logger "$MYNAME: Restarting service Apache2 HTTPD"
  systemctl restart apache2.service
fi

rmdir "$NOTIFICATIONS_PATH"

exit $EXIT_STATUS


Automated renewal

Renewal can be automated by running the Certbot renew command from cron. The following runs the renewal process every Sunday at 04:00 in the morning:

root@pelargir:~# cat /etc/cron.d/pelargir-renew-certificates 
0 4 * * 0   root     docker run --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare renew --quiet && /usr/local/bin/restart-services-after-letsencrypt-cert-renewal.sh

Notes:

  • Because this runs in cron, the Docker command must not use the options -it - these are for interactive use and require a TTY
  • The --quiet option suppresses all Certbot output except errors
  • The post-processing script must also be run to restart system services in case a renewal actually took place. See the previous section for details about the interaction between deploy hooks inside the Docker container and the post-processing script.
  • The renewal process must run at least twice a month, because Certbot only renews certificates 30 days before expiry but there are months that have 31 days. The renewal process should run at least every ten days, though, to avoid the expiry warning emails that Let's Encrypt is automatically sending 20 days before a certificate expires. On the web you can even find hints that it might be a good idea to run renewal daily or twice per day, to counteract a possible revocation initiated by Let's Encrypt. In my case, I decided to run renewal every week.


Troubleshooting

Unexplained renewal failure

On my first attempt to renew certificates, certbot renew managed to renew 4 out of 5 certificates but failed to renew the 5th certificate. This was the output:

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/grunzwanzling.ch.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator dns-cloudflare, Installer None
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for grunzwanzling.ch
dns-01 challenge for grunzwanzling.ch
Waiting 10 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (grunzwanzling.ch) from /etc/letsencrypt/renewal/grunzwanzling.ch.conf produced an unexpected error: Failed authorization procedure. grunzwanzling.ch (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect TXT record "wr_SDxjaXE0LVjcF3RM-82Dhjh-ua9eSEDx-BWqXszk" found at _acme-challenge.grunzwanzling.ch. Skipping.
The following certs could not be renewed:
  /etc/letsencrypt/live/grunzwanzling.ch/fullchain.pem (failure)

I am not sure why renewal failed. I logged in to my CloudFlare account via web browser to check the domain, but had to enter a challenge passcode first. After successfully logging in I didn't find anything unusual about the domain, so I decided to simply try the renewal again. Lo and behold! - It worked.

Is it possible that CloudFlare is limiting API access?


Revoking certificates

To revoke a certificate:

certbot revoke --cert-path /etc/letsencrypt/live/CERTNAME/cert.pem

The corresponding Docker command line:

docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare revoke --cert-path /etc/letsencrypt/live/CERTNAME/cert.pem

After successfully revoking the certificate, Certbot should interactively ask whether you want to remove all certificate material. If it does not ask you can manually delete all traces of the certificate like this:

certbot delete --cert-name CERTNAME
docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/dns-cloudflare delete --cert-name CERTNAME