Securing an Internet accessible server – Part 3

This post is part of a series. Part 1, Part 2.

In the last part I briefly mentioned load balancers and proxies. After thinking about it for a while, I realized I see no reason not to run one, since it simplifies things a bit when setting up secure web services. In this part, we will be setting up a HAProxy server which won’t actually load balance anything, but which will act as a kind of extensible gatekeeper for our web services. In addition, the HAProxy instance will act as the TLS termination point for secure traffic between clients on the Internet and services hosted on our server(s).

This article is written from the perspective of running HAProxy on a separate virtual machine. That’s just for my own convenience, though. If you’re running pfSense for a firewall, you already have HAProxy as a module. It is also possible to run HAProxy directly on your web server, just logically putting it in front of whatever web server software you’re running.

Let’s get started. This post will be a rather long one.

At this point, we’re close enough to having something to publish that it may be a good idea to get a domain name. This will give us three important benefits:

1) Our name will be easier to remember than our IP address.
2) Our public IP address will likely change over time. Using dymanic DNS, we can keep our name pointing at whatever address our server has at the moment.
3) Using a domain name, it’s easy – and nowadays free – to encrypt web traffic to and from our website. This means that it becomes less trivial to eavesdrop on your logon credentials if you’re logging on to your site from a public network.

I personally went with NameCheap, since they’re, well, cheap, and since they support dynamic DNS out of the box. The yearly cost for a domain is pretty trivial anyway, and there are a bunch of people out there who will help you out for a little fee.

With that sorted out, let’s continue with the server.

Dynamic DNS

The point of dynamic DNS is that we tell our DNS provider what our external IP address is when it changes. With NameCheap, a name record intended for use with dynamic DNS is called an A+ record. The only real difference to a regular A (or host) record is that the time to live (TTL) of the record is held pretty short, so that we’re more likely to get to the correct server soon after an IP address change.

I used a nice little software called ddclient for updating my DNS records. It’s in the Ubuntu repository, and for NameCheap, the config looks like this:

# Configuration file for ddclient generated by debconf
#
# /etc/ddclient.conf

protocol=namecheap
use=web, web=dynamicdns.park-your-domain.com/getip
server=dynamicdns.park-your-domain.com
ssl=yes
login=mydomain.com
password='theverylongapikeypresentedbynamecheap'
www,otherserver,whatever

Everything down to and including line 8 should be common for all namecheap clients. The login string is the domain you bought, the password is an API key that’s visible on the DNS management page, and the last line is a comma separated list of DNS records we want to update using dynamic DNS. Note that they are not suffixed by the domain name, and also note that we need to register the A+ records for any names we want ddclient to update for us.

Installing HAProxy

The first thing we do, as always is to install the software:

$ sudo apt update && sudo apt install haproxy

This installs HAProxy along with a skeleton configuration in /etc/haproxy/haproxy.cfg. Take a backup of the original in case you mess up:

$ sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.original

Getting a TLS certificate

Now we’ll be setting up our first proper web service. We’re still building the infrastructure, but here’s something that will actually listen for http traffic and do something useful with it.

First, we want to install letsencrypt, and then we’ll also get a Lua-based ACME plugin for HAProxy, and finally we’ll configure everything to play nicely together. The HAProxy plugin can be downloaded from GitHub. We only really need the lua file, and it should be placed in /etc/haproxy/.

As usual in the Ubuntu world, getting a program is easy:

$ sudo apt install letsencrypt

The next step is preparing the HAProxy configuration file in /etc/haproxy/haproxy.conf:

global
[...]
        tune.ssl.default-dh-param 2048
        lua-load /etc/haproxy/acme-http01-webroot.lua

frontend web-http
        bind 0.0.0.0:80
        acl url_acme_http01 path_beg /.well-known/acme-challenge/
         http-request use-service lua.acme-http01 if METH_GET url_acme_http01
         redirect scheme https code 301 if !{ ssl_fc }

So what did we do?

Line 3 tells HAProxy to use a stronger encryption method than it does by default. Without this line, the program will throw a warning when started.

The frontend section defines a listener interface for HAProxy. We’re telling it to listen on all its IP addresses (which may or may not be what you want depending on how your server looks), and to do it specifically on port 80 which corresponds to unencrypted http traffic.

Then we tell it to specifically use the ACME plugin we just downloaded if anyone asks to see the path .well-known/acme-challenge. This is how we prove to letsencrypt that we own the domain we just bought, and that we’re valid recipients for certificates for the domain.

The last line tells HAProxy to redirect all other http traffic to https, which will become meaningful in the next part of this tutorial.

Now it’s time to get that certificate!

We’ll start by writing a script that uses the letsencrypt program and the HAProxy listener we just set up to achieve exactly this. Create the file /opt/letsencrypt-haproxy and fill it with this content – just remember to fix the mail address in it.

#!/bin/bash

# Path to the letsencrypt-auto tool
LE_TOOL=/usr/bin/letsencrypt

# Directory where the acme client puts the generated certs
LE_OUTPUT=/etc/letsencrypt/live

# Concat the requested domains
DOMAINS=""
for DOM in "$@"
do
 DOMAINS+=" -d $DOM"
done

# Create or renew certificate for the domain(s) supplied for this tool
$LE_TOOL --agree-tos --renew-by-default certonly $DOMAINS --text --webroot --webroot-path /var/lib/haproxy --email your@mailaddress.com

# Cat the certificate chain and the private key together for haproxy
cat $LE_OUTPUT/$1/{fullchain.pem,privkey.pem} > /etc/haproxy/ssl/${1}.pem

# Reload the haproxy daemon to activate the cert
systemctl reload haproxy

Let’s just fix the permissions on the script, and make sure there’s a place to store the finished certificates where HAProxy can reach them and where they can’t be read by the wrong people:

$ sudo chmod 755 /opt/letsencrypt-haproxy
$ sudo mkdir /etc/haproxy/ssl
$ sudo chmod 700 /etc/haproxy/ssl

Before we do anything else now, remember what we said about securing ssh and enabling firewalls earlier in this series? We should follow that advice now at the latest. Also enable incoming http traffic. That will be all for now. After this, it should be reasonably safe to open our external firewall for HTTP traffic to port 80 on the address corresponding to our HAProxy server. How to achieve this will differ slightly between firewall models, but a good bet in consumer-grade routers is to look for the term “port forwarding”.

OK, all done?

Assuming you’ve set up a DNS A record for www.mydomain.com, you should be able to do this:

$ sudo /opt/letsencrypt-haproxy www.mydomain.com

After a little while, you should receive the following confirmation:

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
 /etc/letsencrypt/live/www.yourdomain.com-0001/fullchain.pem. Your cert
 will expire on 2017-07-30. To obtain a new version of the
 certificate in the future, simply run Let's Encrypt again.
 - If you like Let's Encrypt, please consider supporting our work by:

 Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
 Donating to EFF: https://eff.org/donate-le

Let’s just make sure the entire script worked:

$ sudo find /etc/haproxy/ssl/

If this command results in output similar to the following, you’re done.

/etc/haproxy/ssl/
/etc/haproxy/ssl/www.mydomain.com.pem

Now we just have to make sure that we update our certificates more often than they expire. We edit our /etc/crontab to run the certificate update script on an arbitrary time, the first day of every month:

[...]
# m h dom mon dow user command
[...]
31 1 1 * * root /opt/letsencrypt-haproxy www.mydomain.com

Summary

After this part, we have a server with the ability to update a DNS with our current address, we have a TLS certificate, and we have a HAProxy instance with the ability to renew it without downtime. Now we’ve got the infrastructure in place to serve actual web content in a pretty secure way. We’ll get to that in the next part.

Acknowledgements:
The script for updating our SSL certificate was stolen almost verbatim from Martijn Braam’s post here. I adapted the arguments for it to work with the HAProxy Lua ACME plugin as described in this post, rather than having it present its own web service.

Leave a Reply