Nordisch by nature

Advanced dehydration

Posted on Sep 15, 2021

Back when i started using Let’s Encrypt I wanted a lightweight client. Danimo recommended dehydrated. Well it still had a different name back then. But details. So for years I had been a very happy camper with it. Then I learned that many daemons now support that you can run ECDSA and RSA certs at the same time and the server picks the right cert depending on the client. So we will solve this first.

Some basics

Dehydrated supports that you do not have to touch any example files that shipped with the distro package. This makes it easier to update the package because you do not have to merge any files during the upgrade.

Instead of changing /etc/dehydrated/config we will create /etc/dehydrated/config.d/99-salt.sh. Yes the .sh extension is important, but it does not have to be executable. So every setting we want to change we put there. My file looks like this:

CONTACT_EMAIL="me@example.com"
HOOK="${BASEDIR}/hooks.sh"
PRIVATE_KEY_RENEW="yes"
DOMAINS_TXT="${BASEDIR}/salt-domains.txt"
DOMAINS_D="${BASEDIR}/domains.d/"
OCSP_FETCH="yes"

Why we use ${BASEDIR} for everything we will cover in the 2nd part when we learn about how to talk to multiple certificate authorities.

Multiple certificate types

The format of the domains.txt file is well explain in the official documentation. We need to use alias aliases for the certificates as we need multiple certs for the same domain lists. The aliases can be any string but I can recommend to use a format that you can parse easily in your hooks.sh script later.

Our /etc/dehydrated/salt-domains.txt looks like

somehost.example.com example.com > example.com_rsa
somehost.example.com example.com > example.com_ecdsa

This gives us a simple a certificate with 2 hostnames in the subject alternative field. But the suffix at the end of the alias does not do any magic yet. For this we need the directory configured via DOMAINS_D. This directory allows us to specify settings per certificate. In theory we could also use /etc/dehydrated/certs/<certname>/config, but i have the tendenc to just do rm /etc/dehydrated/certs/<certname>/ when i want to reset a certificate. So let us store the settings in a safe place.

The default default key algorithm is nowadays using ECDSA. So we only need to change the type for the RSA certs.

We create a /etc/dehydrated/domains.d/example.com_rsa with the following content:

KEY_ALGO=rsa

You can find the list of all settings you can override here. The last bit we have to do is adapt our hooks.sh.

#!/usr/bin/bash

# install -o dehydrated -g dehydrated -m 0751 -d /etc/ssl/services
export DEPLOY_TARGET_DIRECTORY="/etc/ssl/services"

do_cert() {
    local DOMAIN="${1}" KEYFILE="${2}" CERTFILE="${3}" FULLCHAINFILE="${4}" CHAINFILE="${5}" TIMESTAMP="${6}"
    CERTDIR="$(dirname $FULLCHAINFILE)"
    CERTTYPE="${CERTDIR##*_}"
    DHFILE="${CERTDIR}/dhparams"
    ECFILE="${CERTDIR}/ecparams"
    if [ ! -e $DHFILE ] ; then
      openssl dhparam -outform PEM -out "$DHFILE" "2048"
    fi
    cat "$FULLCHAINFILE" "$KEYFILE" "$DHFILE" > "${DEPLOY_TARGET_DIRECTORY}/${DOMAIN}.with.chain.pem.${CERTTYPE}"
}

do_ocsp() {
    local DOMAIN="${1}" OCSPFILE="${2}" TIMESTAMP="${3}"
    CERTDIR="$(dirname $OCSPFILE)"
    CERTTYPE="${CERTDIR##*_}"
    DEPLOYED_FILE="${DEPLOY_TARGET_DIRECTORY}/${DOMAIN}.with.chain.pem.${CERTTYPE}.ocsp"
    echo "Deploying OCSP response for '${DOMAIN}' at '${TIMESTAMP}' '${DEPLOYED_FILE}'"
    cp "${OCSPFILE}" "${DEPLOYED_FILE}"
}

deploy_cert() {
    local DOMAIN="${1}" KEYFILE="${2}" CERTFILE="${3}" FULLCHAINFILE="${4}" CHAINFILE="${5}" TIMESTAMP="${6}"
    do_cert "${DOMAIN}" "${KEYFILE}" "${CERTFILE}" "${FULLCHAINFILE}" "${CHAINFILE}"
}

deploy_ocsp() {
    local DOMAIN="${1}" OCSPFILE="${2}" TIMESTAMP="${3}"
    do_ocsp "${DOMAIN}" "${OCSPFILE}" "${TIMESTAMP}"
}

unchanged_cert() {
    local DOMAIN="${1}" KEYFILE="${2}" CERTFILE="${3}" FULLCHAINFILE="${4}" CHAINFILE="${5}"
    do_cert "${DOMAIN}" "${KEYFILE}" "${CERTFILE}" "${FULLCHAINFILE}" "${CHAINFILE}"
}

<snip all the other functions, consult the hook.sh example that ship with dehydrated>

HANDLER="$1"; shift
if [[ "${HANDLER}" =~ ^(deploy_challenge|clean_challenge|sync_cert|deploy_cert|deploy_ocsp|unchanged_cert|invalid_challenge|request_failure|generate_csr|startup_hook|exit_hook)$ ]]; then
  "$HANDLER" "$@"
fi

You do not have to follow the same filename format. I am a huge fan of haproxy and it wants the certificate type as suffix in the filename. The full file can be found here.

Voila we can trigger our dehydrated service an it should hopefully all work:

systemctl start dehydrated.service

And check with …

journalctl -u dehydrated.service

Well the latter part actually only works if you use our system improvements. Let us review those next

Systemd Improvements

The past

For a long time our service file in openSUSE looked like this:

[Unit]
Description=Certificate Update Runner for Dehydrated
ConditionPathExists=/etc/dehydrated/config
After=network-online.target
Wants=acmeresponder.socket



[Service]
Type=oneshot
ExecStartPost=-/usr/bin/find -L /etc/dehydrated/postrun-hooks.d -maxdepth 1 -executable -type f -exec {} \;
ExecStart=/usr/bin/dehydrated --cron

# dehydrated --cron will drop permissions and run critical code as dehydrated user.
User=root
Group=root

This gave us 2 features in one go

  1. We do not need sudo calls in hooks.sh to restart services as we put this now into scripts in /etc/dehydrated/postrun-hooks.d and then call those in ExecStartPost
  2. As the comment said … dehydrated will drop the permissions which then will use su/sudo.

But we were never quite happy with this mixture of scopes. One of the major annoyances was that the journalctl commandline above did not contain any output from dehydrated after dehydrated internally dropped the root permissions, as those are not part of the systemd scope of the service anymore.

Split all the services

As you might have guessed we just split the 2 parts into distinct service files:

/usr/lib/systemd/system/dehydrated.service

[Unit]
Description=Certificate Update Runner for Dehydrated
ConditionPathExists=/etc/dehydrated/config
After=network-online.target
Wants=acmeresponder.socket
PartOf=dehydrated.target

[Service]
Type=oneshot
ExecStart=/usr/bin/dehydrated --cron
User=dehydrated
Group=dehydrated

/usr/lib/systemd/system/dehydrated-postrun-hooks.service

[Unit]
Description=Postrun Hooks Runner for Dehydrated
ConditionPathExists=/etc/dehydrated/postrun-hooks.d
After=dehydrated.service
PartOf=dehydrated.target

[Service]
Type=oneshot
ExecStart=-/usr/bin/find -L /etc/dehydrated/postrun-hooks.d -maxdepth 1 -executable -type f -exec {} \;
User=root
Group=root

[Install]
RequiredBy=dehydrated.service

Now if we just use

systemctl enable dehydrated.timer

or

systemctl start dehydrated.service

It will only run the cron part but not the postrun hooks. For this we also need to enable the other unit:

systemctl enable dehydrated-postrun-hooks.service

So when ever the dehydrated.service is finished, it will automatically trigger the postrun hook service.

Multiple Certificate Authorities

Using Let’s Encrypt for your internet facing services is all great but what about services inside your home network or in a DMZ? Yes there are ways to get certificates for those via Let’s Encrypt, but I actually prefer to run an internal certificate authority for that. For a long time it meant running custom tools for special certificate handling tools. Many implementations were either really complex setups like boulder or not meant for production.

Now we have smallstep as a small certificate to run internally, which not only supports their custom protocol for certificate provisioning, but also the ACME protocol. Which leads us to a problem. What do we do when we need multiple CAs? Let’s Encrypt for the external services and smallstep for the internal services. It should be noted that you need at least version 0.7.0 to get it working with smallstep.

Lukas said that implementing multi CA support would be nice but also not just a quick change. So we started to look elsewhere.

If you call /usr/bin/dehydrated --cron --config /path/to/config it will set the ${BASEDIR} variable to /path/to. Because we used ${BASEDIR} in all our settings in 99-salt.sh it will find all the files relative to our new home. So with this we can do

cp -av /etc/dehydrated /etc/dehydrated-internal
cd /etc/dehydrated-internal
rm -rv accounts/* certs/* domains.d/*

After the cleanup is done, we can point our new config in /etc/dehydrated-internal/config.d/99-salt.sh to the smallstep instance by adding this line:

CA="https://smallstep.dmz.example.com/acme/acme/directory"

Once this is done we can register with the new CA and grab our initial certificates

su -s /bin/bash - dehydrated
dehydrated --register --accept-terms
dehydrated --cron

Last but not least … hooking this all up with systemd. For this we will use systemd’s nifty Instantiated Services. First we need some unit files:

Unit files

/usr/lib/systemd/system/dehydrated@.service

[Unit]
Description=Certificate Update Runner for Dehydrated
ConditionPathExists=/etc/dehydrated-%i/config
After=network-online.target
Wants=acmeresponder.socket
PartOf=dehydrated.target

[Service]
Type=oneshot
ExecStart=/usr/bin/dehydrated --cron --config /etc/dehydrated-%i/config
User=dehydrated
Group=dehydrated

/usr/lib/systemd/system/dehydrated-postrun-hooks@.service

[Unit]
Description=Postrun Hooks Runner for Dehydrated: %i
ConditionPathExists=/etc/dehydrated-%i/postrun-hooks.d/
After=dehydrated@%i.service
PartOf=dehydrated.target

[Service]
Type=oneshot
ExecStart=-/usr/bin/find -L /etc/dehydrated-%i/postrun-hooks.d/ -maxdepth 1 -executable -type f -exec {} \;
User=root
Group=root

[Install]
RequiredBy=dehydrated@%i.service

/usr/lib/systemd/system/dehydrated@.timer

[Unit]
Description=Timer for Certificate Update Runner for Dehydrated
PartOf=dehydrated.target

[Timer]
OnCalendar=daily
# Two hour window
RandomizedDelaySec=7200

[Install]
WantedBy=timers.target

Finalize setup

So all left to go is:

systemctl enable dehydrated@internal.timer
systectl enable dehydrated-postrun-hooks@internal.service

If you are an openSUSE Tumbleweed user, all those new unit files are in your package already. Users of older openSUSE versions or other distributions can find packages built with build.opensuse.org.

All this work was done as part of the Hackweek 2021 at SUSE.