LetsEncrypt

From HerzbubeWiki
Jump to navigation Jump to search

This wiki page is about the Certificate Authority "Let's Encrypt" and the technical details for obtaining and automatically renewing SSL certificates on a Debian server. In addition, the page focuses on CloudFlare because that is the DNS provider I currently use.

In older versions of Debian (specifically Debian jessie) it was necessary to run Certbot in a Docker image. Consult the history of this page to see how that worked.


Introduction

From the "Getting Started" page:

In order to get a certificate for your website's domain from Let's Encrypt, you have to demonstrate control over the domain. With Let's Encrypt, you do this using software that uses the ACME protocol, which typically runs on your web host. [...[ We recommend that most people with shell access use the Certbot ACME client. [...] If Certbot does not meet your needs, or you’d like to try something else, there are many more ACME clients to choose from. Once you’ve chosen ACME client software, see the documentation for that client to proceed.

I have been successfully using Certbot, so that's what I am going to write about in the sections below.

Furthermore, because I want to have wildcard certificates, the following is also important (quote from the FAQ):

[...] Wildcard issuance must be done via ACMEv2 using the DNS-01 challenge.

What is a "DNS-01" challenge? Without going into details, the challenge requires you to add a custom TXT record to the domain's DNS configuration so that Let's Encrypt can verify that you are in control of the domain.


References


Preparing to use Certbot

Debian packages

This is the main Debian package to install: 

certbot

I want wildcard certificates, so I can't just use plain Certbot because that has no support for automating DNS-01 challenges. Certbot has a plugin concept, and some plugins that handle DNS-01 challenges (called "DNS authenticator plugins") exist for various popular DNS providers. Fortunately a Certbot plugin exists for my DNS provider, which is Cloudflare. The plugin uses the Cloudflare web API to achieve its task. The plugin name is cerbot-dns-cloudflare, the Debian package that contains the plugin is this: 

python3-certbot-dns-cloudflare


Filesystem folders

The certbot package sets up the main folder where everything related to Let's Encrypt (configuration, certificates, etc.) will be stored: 

/etc/letsencrypt


Configuration files

The Certbot main configuration file is this 

/etc/letsencrypt/cli.ini


The file content should look like this (everything except the max-log-backups option added by me): 

root@pelargir:~# cat /etc/letsencrypt/cli.ini
# Because we are using logrotate for greater flexibility, disable the
# internal certbot logrotation.
max-log-backups = 0

# Let's Encrypt site-wide configuration
dns-cloudflare-credentials = /etc/letsencrypt/herzbube.ch.ini
# Use the ACME v2 staging URI for testing things
#server = https://acme-staging-v02.api.letsencrypt.org/directory
# Production ACME v2 API endpoint
server = https://acme-v02.api.letsencrypt.org/directory


Note the reference to /etc/letsencrypt/herzbube.ch.ini: This is the configuration file that will be used by Certbot's cerbot-dns-cloudflare plugin. It contains the Cloudflare API key which authorizes the Certbot plugin to automatically make DNS changes. This makes the configuration file very sensitive, and you need to make absolutely sure that the file is protected against unauthorized access. Note that we could protect against abuse with 2-factor authentication, but this would become a problem later on when we automate certificate renewal - you don't want to wake up in the middle of the night to confirm Certbot's renewal attempt. So this is how the config file looks like: 

root@pelargir:~# ls -l /etc/letsencrypt/herzbube.ch.ini 
-r-------- 1 root root 140 Jun  7 22:28 /etc/letsencrypt/herzbube.ch.ini

root@pelargir:~# cat /etc/letsencrypt/herzbube.ch.ini 
# CloudFlare API key information
dns_cloudflare_api_key = 1234567890abcdef1234567890abcdef12345
dns_cloudflare_email = herzbube@herzbube.ch


Disable systemd timer for renewals

The certbot Debian package installs a systemd timer that runs twice per day to perform automated certificate renewals. Because I already have a cron job in place for doing that I am disabling the timer: 

systemctl disable certbot.timer

Disabling the timer prevents it from being started when the system reboots the next time. Installing the Debian package also started the timer, so it must be stopped now: 

systemctl stop certbot.timer


Obtaining certificates

At the time of writing I had four domains for which I wanted a wildcard certificate, so these were the four commands that I issued: 

certbot certonly -d herzbube.ch -d *.herzbube.ch
certbot certonly -d moser-naef.ch -d *.moser-naef.ch
certbot certonly -d francescamoser.ch -d *.francescamoser.ch
certbot certonly -d grunzwanzling.ch -d *.grunzwanzling.ch

Discussion:

  • certonly tells Certbot to obtain or renew a certificate, but not to install it.
  • The -d option specifies the domain name, or "subject", to be used in the certificate. Multiple -d options create a single certificate that contains multiple subjects. The first -d is used for the main subject. The "Certificate Subject Alt Name" (Firefox term) or "X509v3 Subject Alternative Name" (OpenSSL term) lists all subjects, including the main subject. Certbot stores the certificate material in a folder named after the main subject (i.e. the first -d).


When you run one of the Certbot commands you will be prompted as follows. Obviously you want to select option 1. 

1: Obtain certificates using a DNS TXT record (if you are using Cloudflare for DNS). (dns-cloudflare)
2: Spin up a temporary webserver (standalone)
3: Place files in webroot directory (webroot)


We're almost ready to go now. A number of manual steps still have to be performed, though, before we can use the certificates that we just obtained.


Certbot files and folders

Account files

Certbot automatically creates a Let's Encrypt account when the first certificate is obtained. Certbot manages this automatically, so the details are not interesting here. 

root@pelargir:~# ls -l /etc/letsencrypt/accounts/
total 12
drwx------ 3 root root 4096 Jun  7 22:14 .
drwxr-xr-x 9 root root 4096 Jun  8 01:18 ..
drwx------ 3 root root 4096 Jun  7 22:14 acme-v02.api.letsencrypt.org


Private keys

Certbot creates and stores private keys here. These files are duplicates of the files stored in the archive folder, which is discussed further down. 

root@pelargir:~# ls -l /etc/letsencrypt/csr/
total 52
drwx------ 2 root root 4096 Jun  8 01:18 .
drwxr-xr-x 9 root root 4096 Jun  8 01:18 ..
-rw------- 1 root root 1704 Jun  7 22:14 0000_key-certbot.pem
-rw------- 1 root root 1704 Jun  7 22:17 0001_key-certbot.pem
-rw------- 1 root root 1708 Jun  7 22:23 0002_key-certbot.pem
[...]


CSRs

Certbot creates and stores CSRs here: 

root@pelargir:~# ls -l /etc/letsencrypt/csr/
total 52
drwxr-xr-x 2 root root 4096 Jun  8 01:18 .
drwxr-xr-x 9 root root 4096 Jun  8 01:18 ..
-rw-r--r-- 1 root root  924 Jun  7 22:14 0000_csr-certbot.pem
-rw-r--r-- 1 root root  924 Jun  7 22:17 0001_csr-certbot.pem
-rw-r--r-- 1 root root  924 Jun  7 22:23 0002_csr-certbot.pem
[...]


Certificate material

As mentioned in the previous section, Certbot stores certificate material under the main subject's name, i.e. the first -d option specified on the Certbot command line. Note the special permissions of the /etc/letsencrypt/live folder - these are non-standard and will be explained in the next section. 

root@pelargir:~# ls -l /etc/letsencrypt/live
total 28
drwx--x--- 7 root ssl-cert 4096 Jun  8 01:18 .
drwxr-xr-x 9 root root     4096 Jun  8 01:18 ..
drwxr-xr-x 2 root root     4096 Jun  8 00:10 francescamoser.ch
drwxr-xr-x 2 root root     4096 Jun  8 01:18 francescamoser.ch-0001
drwxr-xr-x 2 root root     4096 Jun  8 01:18 grunzwanzling.ch
drwxr-xr-x 2 root root     4096 Jun  8 01:10 herzbube.ch
drwxr-xr-x 2 root root     4096 Jun  8 01:17 moser-naef.ch
root@pelargir:~# ls -l /etc/letsencrypt/live/herzbube.ch/
total 20
drwxr-xr-x 2 root root     4096 Jun  8 01:10 .
drwx--x--- 7 root ssl-cert 4096 Jun  8 01:18 ..
lrwxrwxrwx 1 root root       35 Jun  8 01:10 cert.pem -> ../../archive/herzbube.ch/cert2.pem
lrwxrwxrwx 1 root root       36 Jun  8 01:10 chain.pem -> ../../archive/herzbube.ch/chain2.pem
lrwxrwxrwx 1 root root       40 Jun  8 01:10 fullchain.pem -> ../../archive/herzbube.ch/fullchain2.pem
lrwxrwxrwx 1 root root       38 Jun  8 01:10 privkey.pem -> ../../archive/herzbube.ch/privkey2.pem
-rw-r--r-- 1 root root      682 Jun  7 22:32 README


As can be seen, the live folder consists of symlinks that together form the currently valid set of certificate material and that point back to an archive folder. Note the special permissions of the /etc/letsencrypt/live and /etc/letsencrypt/archive folders - these are non-standard and will be explained in the next section (Using Let's Encrypt certificates). 

root@pelargir:~# ls -l /etc/letsencrypt/archive
total 32
drwx--x--- 8 root ssl-cert 4096 Jun  8 01:18 .
drwxr-xr-x 9 root root     4096 Jun  8 01:18 ..
drwxr-xr-x 2 root root     4096 Jun  8 00:10 francescamoser.ch
drwxr-xr-x 2 root root     4096 Jun  8 01:18 francescamoser.ch-0001
drwxr-xr-x 2 root root     4096 Jun  8 01:18 grunzwanzling.ch
drwxr-xr-x 2 root root     4096 Jun  8 01:10 herzbube.ch
drwxr-xr-x 2 root root     4096 Jun  8 01:08 herzbube.ch.org
drwxr-xr-x 2 root root     4096 Jun  8 01:17 moser-naef.ch
root@pelargir:~# ls -l /etc/letsencrypt/archive/herzbube.ch
total 40
drwxr-xr-x 2 root root     4096 Jun  8 01:10 .
drwx--x--- 8 root ssl-cert 4096 Jun  8 01:18 ..
-rw-r--r-- 1 root root     2147 Jun  7 22:32 cert1.pem
-rw-r--r-- 1 root root     2163 Jun  8 01:10 cert2.pem
-rw-r--r-- 1 root root     1647 Jun  7 22:32 chain1.pem
-rw-r--r-- 1 root root     1647 Jun  8 01:10 chain2.pem
-rw-r--r-- 1 root root     3794 Jun  7 22:32 fullchain1.pem
-rw-r--r-- 1 root root     3810 Jun  8 01:10 fullchain2.pem
-rw-r--r-- 1 root root     1708 Jun  7 22:32 privkey1.pem
-rw-r--r-- 1 root root     1704 Jun  8 01:10 privkey2.pem


Using Let's Encrypt certificates

The Let's Encrypt certificates will be used by the following services:

  • Dovecot IMAP: See the Dovecot wiki page for details.
  • Exim MTA: See the Exim wiki page for details. Most important is that Exim needs special permissions on the /etc/letsencrypt/live and /etc/letsencrypt/archive folders (group ssl-cert and permissions 710).
  • Apache: See the Apache wiki page for details.


Listing certificates

To list all certificates that Certbot knows about: 

certbot certificates


Renewing certificates

Introduction

Because of the high degree of automation involved, certificates issued by Let's Encrypt are valid only for a short time - currently 90 days. It is therefore important to be able to automate the renewal process.

Let's Encrypt recommends to renew certificates when they have a third of their total lifetime left. With the typical 90 days lifetime of a Let's Encrypt certificate, this means renewal when 30 days (or less) are left.


Manual renewal

Manual renewal of certificates is very simple: Just run Certbot with the renew action: 

certbot renew

General notes:

  • The renew action attempts to renew any previously-obtained certificates that expire in less than 30 days. This means that, unlike certonly, the renew action operates on multiple certificates.
  • The same plugin and options that were used at the time the certificate was originally issued will be used for the renewal attempt (unless you specify other plugins or options). This information is taken from one of the domain-specific configuration files located in the folder /etc/letsencrypt/renewal. Certbot generates these config files automatically when the certificate is obtained for the first time.
  • The renew action performs no action if a certificate is not yet due for renewal, i.e. if it's not yet within the 30 days expiry period. Because of this, the renew action can be run as frequently as you want and is therefore ideally suited for automation. If you want to ignore the expiry period for some reason, run the command like this: certbot renew --force-renewal


Hooks

Certbot supports running so-called "pre hooks", "post hooks" and "deploy hooks", which are scripts or other executables that run before or after a renewal attempt. A "deploy hook" is run only if the renewal process was successful, whereas the "post hook" runs even if the renewal process failed.


Typical reasons why this capability can be important:

  • A daemon must be restarted after a successful renewal so that it picks up the renewed certificate
  • A daemon does not read its certificates as the root user, so a post hook is necessary to copy the renewed certificate to another filesystem location and/or apply appropriate file permissions.


General notes about hooks:

  • Hooks are run only if renewal is actually taking place. If Certbot finds that a certificate does not need renewal, it won't run any hooks.
  • Hooks are run for every renewed certificate
  • A successful hook must terminate with exit code 0, a failed hook must terminate with a non-zero exit code. A failed hook does not cause the renewal itself to fail, but it will cause a message printed to stderr.
  • Certbot passes a number of environment variables to hooks. I have not found an official documentation that lists them, but a few can be found in the deploy hook example script in the Certbot user guide:
    • RENEWED_DOMAINS contains the list of domain names in the renewed certificate
    • RENEWED_LINEAGE refers to the live folder where the certificate material is stored. Example: /etc/letsencrypt/live/herzbube.ch


On the command line, a hook can be specified like this: 

certbot renew --pre-hook "/path/to/hook-script.sh"
certbot renew --post-hook "/path/to/hook-script.sh"
certbot renew --deploy-hook "/path/to/hook-script.sh"


Certbot also runs any executable files that it finds in one of the following folders. Executables are run in alphabetical order of their file names. Hooks specified on the command line run after hooks found in the filesystem. 

/etc/letsencrypt/renewal-hooks/pre
/etc/letsencrypt/renewal-hooks/post
/etc/letsencrypt/renewal-hooks/deploy


Hooks on pelargir.herzbube.ch

Note: The solution presented in this section is overcomplicated in that the deploy hook script does not directly restart services, but instead delegates the task to a secondary script. This splitting of responsibilities is a left-over from the time when I used Certbot from within a Docker image: A hook script that runs within a Docker container and not on the host has no access to host system services, so it cannot restart those services on its own but has to delegate the task to an agent that runs outside the Docker container.
Without a Docker container the deploy hook script could directly restart services. I'm not changing things, though, simply because the two-script solution is working.


On my server, I have only one deploy hook: 

root@pelargir:~# ls -l /etc/letsencrypt/renewal-hooks/deploy
total 8
-rwxr-xr-x 1 root root 377 Aug 29 16:32 20-notify-cron.sh

While I was using Courier IMAP I had another hook that re-generated a special file which contained the certificate chain and the private key in a single file. This was required by Courier IMAP. Since I have switched from Courier IMAP to Dovecot I no longer need this hook, I just mention it here to illustrate a possible use case for hook scripts.


Anyway, the hook script I have now generates a "notification file" whose name depends on the certificate that was renewed. Currently the script only distinguishes between the certificate for the domain herzbube.ch and all other certificates. The purpose of the "notification file" is explained below. 

root@pelargir:~# cat /etc/letsencrypt/renewal-hooks/deploy/20-notify-cron.sh
#!/bin/sh

# It's important to not use Bash because it is not available in the
# Docker container that runs certbot.

# Exit immediately if a command exits with non-zero status
set -e

# The following path must be in /etc/letsencrypt so that this script,
# which runs in a Docker container, has access to it
NOTIFICATIONS_PATH="/etc/letsencrypt/renewal-notifications"
if test ! -d "$NOTIFICATIONS_PATH"; then
  mkdir -p "$NOTIFICATIONS_PATH"
fi
  
case $RENEWED_DOMAINS in
  *herzbube.ch*)
    NOTIFICATION_FILE="$NOTIFICATIONS_PATH/herzbube.ch"
    ;;
      
  *)  
    NOTIFICATION_FILE="$NOTIFICATIONS_PATH/other"
    ;;
esac

touch "$NOTIFICATION_FILE"


What actually needs to be done after a certificate is renewed is that one or more services that use the certificate are restarted. The hook script stores the information which certificate was renewed for later processing. It does so by creating a "notification file" whose name is significant. Another post-processing script, which must be run separately after the Certbot command has completed its work, then checks for the presence of one or more "notification files". If it finds one, the post-processing script performs the actual restart of the appropriate services. Here is the post-processing script: 

root@pelargir:~# cat /usr/local/bin/restart-services-after-letsencrypt-cert-renewal.sh
#!/bin/sh

# It's important to not use Bash because it is not available in the
# Docker container that runs certbot.

# Exit immediately if a command exits with non-zero status
set -e

MYNAME="$(basename $0)"
LOGGER_PATH="/usr/bin/logger"
NOTIFICATIONS_PATH="/etc/letsencrypt/renewal-notifications"

if test ! -d "$NOTIFICATIONS_PATH"; then
  logger "$MYNAME: No certificates were renewed"
  exit 0
fi

EXIT_STATUS=0

logger "$MYNAME: One or more certificates were renewed"

RESTART_DOVECOT=0
RESTART_EXIM=0
RESTART_APACHE=0

for NOTIFY_FILE in $NOTIFICATIONS_PATH/*; do
  NOTIFY_FILE_NAME="$(basename "$NOTIFY_FILE")"

  logger "$MYNAME: Certificate for '$NOTIFY_FILE_NAME' was renewed (notification file $NOTIFY_FILE)"

  case "$NOTIFY_FILE_NAME" in
    # Dovecot and Exim run only under a herzbube.ch subdomain
    herzbube.ch)
      RESTART_DOVECOT=1
      RESTART_EXIM=1
      RESTART_APACHE=1
      ;;
    # Apache has vhosts for all domains
    other)
      RESTART_APACHE=1
      ;;
    *)
      logger "$MYNAME: Unrecognized notification file $NOTIFY_FILE"
      EXIT_STATUS=1
      ;;
  esac

  rm -f "$NOTIFY_FILE"
done

if test "$RESTART_DOVECOT" = "1"; then
  logger "$MYNAME: Restarting service Dovecot IMAP"
  systemctl restart dovecot.service
fi

if test "$RESTART_EXIM" = "1"; then
  logger "$MYNAME: Restarting service Exim4 MTA"
  systemctl restart exim4.service
fi

if test "$RESTART_APACHE" = "1"; then
  logger "$MYNAME: Restarting service Apache2 HTTPD"
  systemctl restart apache2.service
fi

rmdir "$NOTIFICATIONS_PATH"

exit $EXIT_STATUS


Automated renewal

Renewal can be automated by running the Certbot renew command from cron. The following runs the renewal process every Sunday at 04:00 in the morning: 

root@pelargir:~# cat /etc/cron.d/pelargir-renew-certificates 
0 4 * * 0   root     certbot renew --quiet && /usr/local/bin/restart-services-after-letsencrypt-cert-renewal.sh

Notes:

  • The --quiet option suppresses all Certbot output except errors
  • The post-processing script must also be run to restart system services in case a renewal actually took place. See the previous section for details about the interaction between deploy hooks and the post-processing script.
  • The renewal process must run at least twice a month, because Certbot only renews certificates 30 days before expiry but there are months that have 31 days. The renewal process should run at least every ten days, though, to avoid the expiry warning emails that Let's Encrypt is automatically sending 20 days before a certificate expires. On the web you can even find hints that it might be a good idea to run renewal daily or twice per day, to counteract a possible revocation initiated by Let's Encrypt. In my case, I decided to run renewal every week.


Troubleshooting

Unexplained renewal failure

On my first attempt to renew certificates, certbot renew managed to renew 4 out of 5 certificates but failed to renew the 5th certificate. This was the output: 

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/grunzwanzling.ch.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator dns-cloudflare, Installer None
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for grunzwanzling.ch
dns-01 challenge for grunzwanzling.ch
Waiting 10 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (grunzwanzling.ch) from /etc/letsencrypt/renewal/grunzwanzling.ch.conf produced an unexpected error: Failed authorization procedure. grunzwanzling.ch (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect TXT record "wr_SDxjaXE0LVjcF3RM-82Dhjh-ua9eSEDx-BWqXszk" found at _acme-challenge.grunzwanzling.ch. Skipping.
The following certs could not be renewed:
  /etc/letsencrypt/live/grunzwanzling.ch/fullchain.pem (failure)

I am not sure why renewal failed. I logged in to my CloudFlare account via web browser to check the domain, but had to enter a challenge passcode first. After successfully logging in I didn't find anything unusual about the domain, so I decided to simply try the renewal again. Lo and behold! - It worked.

Is it possible that CloudFlare is limiting API access?


Revoking certificates

To revoke a certificate: 

certbot revoke --cert-path /etc/letsencrypt/live/CERTNAME/cert.pem

After successfully revoking the certificate, Certbot should interactively ask whether you want to remove all certificate material. If it does not ask you can manually delete all traces of the certificate like this: 

certbot delete --cert-name CERTNAME
Retrieved from "https://wiki.herzbube.ch/index.php?title=LetsEncrypt&oldid=6842"