Environment variables for services in a systemd-based system

My current config deployment automation project has required me to set up a dev/staging environment for my load balancers, since I don’t want to break stuff by deploying untested configurations.
This environment is functionally identical to a single load balancer and can be used along with a hosts file on a client to not only develop configurations and make sure they are syntactically correct, but also to fully test the functionality of the load balancer rules.

As part of this, I naturally need to change the listener addresses in the dev/staging HAProxy environment compared to the production environment that I keep version controlled in my git repository.
My first instinct was to use a script to pull the latest versions, modify the necessary lines using sed, and copy the config to the correct location. I didn’t really like this concept since it would by definition mean that the configs weren’t fully identical between the production environment+git repo and the dev/staging environment.
If I used environment variables, the version controlled configuration could be kept fully identical across all instances.

The first mistake I made took me a while to grasp. HAProxy parsed an obviously fully valid configuration, but intermittently presented a certificate I didn’t expect, and web services intermittently failed to reply to requests.
It turns out Linux daemons don’t inherit even system-wide environment variables.
So how do we check what environment variables a service does see?
First get the PID(s) of the service:

$ pgrep haproxy
1517
1521
1523
$ 

In the case of my HAProxy installation, I got a list of three processes, so I chose the last one and checked out its environment:

# cat /proc/1523/environ

This results in a list of its current environment variables, and naturally the ones I thought I’d added were nowhere to be seen.

So why did HAProxy start without complaining? Naturally since the environment variables weren’t defined in this context, their implicit value was NULL, and so HAProxy figured I wanted to listen on the assigned ports on all interfaces.

How do we assign environment variables to a service in a modern, systemd-based Linux, then?
On a command-prompt, run systemctl edit servicename. This starts your default editor. A valid config looks like this:

[Service]
Environment=ENVVAR1=value
Environment=ENVVAR2=value

On Ubuntu, this file is stored in /etc/systemd/system/servicename.service.d/override.conf, but naturally this file can be renamed to something more descriptive. The systemctl edit command doesn’t do anything magical, it’s just a shortcut.
After the file is in place, run systemctl daemon-reload to make the configuration active, and then the HAProxy service needs to be restarted, not only reloaded, for the changes to apply.

Of course, I want this config too to be deployable through Ansible.
The relevant lines from my playbook:

---
    -   name: Update environment variables for HAProxy service
        copy:
            src: "{{ config_root }}/etc/systemd/system/haproxy.service.d/10-listeners.conf"
            dest: "/etc/systemd/system/haproxy.service.d/"
        register: ha_envvar_status
    
    -   name: Reload systemd service configuration
        systemd:
            daemon_reload: yes
        when: ha_envvar_status|changed
...
    -   name: Reload HAProxy configuration
        service:
            name: haproxy
            state: reloaded
        when: ha_envvar_status|skipped and haproxy_cfg_status|changed
        register: reloaded_haproxy

    -   name: Restart HAProxy daemon
        service:
            name: haproxy
            state: restarted
        when: ha_envvar_status|changed or (haproxy_cfg_status|changed and not reloaded_haproxy|skipped)

Key lines:
The register line stores the result of the command to a variable. Thanks to that, I can use the when keyword to only reload daemons when anything has actually changed.

Summary

Linux daemons don’t automatically inherent environment variables.
In systemd-based distros (which today means pretty much anyone with corporate backing), environment variables can be added using the Environment keyword in the Service section of a file in /etc/systemd/system/servicename.service.d/.

Continuous Deployment of Load Balancer Configurations

I thought I’d describe some optimizations I’ve made to my load balancers at work, both for the good of the older me, and in case someone would benefit from some of my ideas.

Background

The load balancers are based on four software packages that integrate to create a powerful whole:
Keepalive Daemon provides a common set of virtual IP addresses and ensures that failover happens to a Backup server if the Master would cease responding.
HAProxy does most of the actual load balancing and mangles network traffic when required.
SNMPD throws SNMP trap events from keepalived whenever a failover occurs.
The Zabbix Agent enumerates current configuration and system state for detailed system monitoring.

Now, all of these components get the occasional configuration change, except for HAProxy, which pretty much sees changes on at least a weekly basis.
The procedure for updating the configuration must cover the following steps:

  1. Run a pre-check to confirm that both load balancers in the pair work; we don’t want to initiate an automated update that could kill off service availability completely.
    On the Backup load balancer node:
  2. Backup the current configuration.
  3. Deploy the new configuration.
  4. Reload services.
  5. Run a post-op check on the secondary node to confirm that the new config hasn’t broken anything important.
  6. Fail over operations from the Master load balancer node to the Backup node and repeat steps 2-5 on the Master node.
  7. Perform a final check on the load balanced services to confirm functionality hasn’t been lost.

From experience, this procedure is tedious to say the least. In addition there’s always the risk of introducing a change to an active load balancer and forgetting to deploy the same change to the backup one; something that may not become obvious until after the next major configuration update when the last change disappears and functionality breaks.

These are just the most obvious arguments for an automated and version controlled deployment procedure. So how do we go about that?

Version control

In my case, I use Git connected to a GitLab server for version control, and Ansible for automation.

Configuration changes are prepared in a development environment, from which the relevant files are committed to a git repository.

Other components in the load balancer config – Lua scripts or tools made by our developers are stored in other repositories, and can be pulled by git before a new deployment.

Ansible structure

For each load balancer pair, I’ve built a directory structure containing a playbook directory for the Ansible YAML scripts, and a filesystem directory that mirrors the movable parts of the load balancer, where the relevant parts exist in the etc directory tree.

Automation

Deployment is initialized by a shell script that git-pulls the latest versions of dependencies we have and then ensures that the Ansible playbooks can work on remote computers by wrapping them in an ssh-agent environment.
The execution of Ansible playbooks happens from within a session script called by the ssh-agent.

Ansible-specific tips

The key to ensuring that the production environment doesn’t break lies in the header of the playbook:

---

-   name: Update PRODUCTION load balancer configuration
    hosts: lb_hadmzprod
    serial: true
    any_errors_fatal: true

The serial keyword makes the script work on one server at a time rather than executing in parallel.
The any_errors_fatal parameter is combined with relevant service checks interspersed among the deployment tasks to ensure that the script fails fast and loudly if a backend web service stops responding while deployment is underway, so that we don’t break both servers in a pair. Note that this requires some thought on the part of the person running the scripts, so they fix the problem before re-attempting to run the script, or fecal matter will hit the fan quickly enough.

The most basic of tests just ensures I can reach the statistics page of my load balancer:

    -   name: Fail task if lb1 is unavailable
        uri: 
            url: https://lb1.domain.com:1936

A typical file copying task:

    -   name: Update Keepalived configuration
        copy:
            src: "{{ config_root }}/etc/keepalived/{{ item }}"
            dest: "/etc/keepalived/"
            mode: 0600
        with_items:
        -   keepalived-master.conf
        -   keepalived-slave.conf

As a side note: Since I don’t want the script to have to care about which server is which, I’ve created one config file for the keepalived master instance and one for the slave. On the actual servers, a symlink points to the correct configuration for the instance.

By reloading the HAProxy service, existing sessions are not lost even though the configuration gets updated. As a bonus, in the Ansible service module, the reloaded state request also starts the service if it wasn’t started before.

    -   name: Reload HAProxy configuration
        service:
            name: haproxy
            state: reloaded

With way less than a day’s worth of work, a workflow has been introduced for the deployment process that is repeatable and that mitigates some of the risks involved in letting humans tamper with production systems.

Load balancing a Citrix StoreFront

This is well-documented by Citrix too; I just thought I’d write a few lines on a gotcha I happened upon:
Load balancing the Citrix StoreFront has two prerequisites:

  1. The StoreFront servers must see the load balancer VIP when looking up the service FQDN.
  2. The load balancers must be configured with session affinity, since it doesn’t seem as though StoreFront keeps track of user sessions between StoreFront servers.

In addition, Citrix recommends running a load balancing scheme based on least connections.

In HAProxy, a valid backend config looks something like this:


backend bk_citrixsf
    balance leastconn
    stick-table type ip size 200k expire 30m
    stick on src
    acl h_xff_exists req.hdr(X-Forwarded-For) -m found
    http-request add-header X-Forwarded-For %[src] unless h_xff_exists

    server ctxsf01 ....

Load Balancing Exchange 2016 behind HAProxy

I recently started the upgrade to Exchange 2016 at work. A huge benefit over Exchange 2010, is that REST based client connections are truly stateless. In effect this means that if a server goes down, clients shouldn’t really notice any issues as long as something redirects them to a working server. In my system, this something is HAProxy.

The guys at HAProxy have their own excellent walkthroughs for setting up their load balancer for Exchange 2013, which can pretty much be lifted verbatim to Exchange 2016, but I want to add a few key points to think about:

Service health checks

Each web service has a virtual file to tell its state, called HealthCheck.htm. Let HAProxy use the contents of this file for the server health check. That way it’ll know to redirect clients if one of the services is down, even though the Exchange server in question may still be listening on port 443.

Example config:

    option httpchk GET /owa/HealthCheck.htm
    http-check expect string 200\ OK
    server Exchange1 192.168.1.13:443 maxconn 10000 ssl cafile cacert.pem weight 20 check 

This example shows a test of the Outlook Web Access service state. Naturally the config can be set to test each of the REST services each Exchange server presents.

Exchange server default firewall rules

Our design puts the load balancer in a DMZ outside of our server networks. Clients connecting through the load balancer will be dropped by Windows firewall rules generated by Exchange; specifically the edge traversal rules for the POP3 and IMAP protocols. Make sure you allow edge traversal for these protocols, letting the network firewall take care of limiting external client connections to them. Also take note there are multiple firewall rules for IMAP and POP3 traffic. Only the ones concerned with client traffic are relevant for this change. There’s no point in kicking open holes in your firewall for no good reason.

Exchange and Outlook suck at IMAP

We use IMAP for an internal order management system. Outlook and Exchange aren’t the best tools for this protocol, but unfortunately we have to live with those due to sins committed long ago. I spent quite some time troubleshooting our IMAP connections:
No matter how I configured Outlook I couldn’t get it to open an IMAP connection to the Exchange servers. Error messages varied depending on the client settings, but essentially I couldn’t log on, couldn’t establish a secure connection, or couldn’t synchronize my folders.

I would get the regular banner when telnetting from the client machine, so I knew traffic was getting through all the way from Exchange via the load balancer.
Mozilla Thunderbird could connect perfectly well and sync accounts, both using STARTTLS on port 143 and over a TLS encrypted connection on port 993. After mulling it over, I turned on debug logging in Outlook and quickly saw that the client was trying and failing to perform an NTLM logon to Exchange. Using the error messages as search terms, I found others who had experienced the same issue. Their solution had been to turn off NTLM authentication for the IMAP protocol on the Exchange server. This seems to be a regression in Exchange Server 2016 from an earlier bug in Exchange 2013.
The command in the Exchange Management Shell:

Set-IMAPSettings -EnableGSSAPIAndNTLMAuth $false

After this, Outlook still is incapable of logging on using TLS over port 993, but at least it consistently manages to run STARTTLS over port 143, which is good enough for my use case.

All in all, the most complicated part here wasn’t to make HAProxy do its magic, but to get Exchange and Outlook do what they should.

Securing an Internet accessible server – Part 3

This post is part of a series. Part 1, Part 2.

In the last part I briefly mentioned load balancers and proxies. After thinking about it for a while, I realized I see no reason not to run one, since it simplifies things a bit when setting up secure web services. In this part, we will be setting up a HAProxy server which won’t actually load balance anything, but which will act as a kind of extensible gatekeeper for our web services. In addition, the HAProxy instance will act as the TLS termination point for secure traffic between clients on the Internet and services hosted on our server(s).

This article is written from the perspective of running HAProxy on a separate virtual machine. That’s just for my own convenience, though. If you’re running pfSense for a firewall, you already have HAProxy as a module. It is also possible to run HAProxy directly on your web server, just logically putting it in front of whatever web server software you’re running.

Let’s get started. This post will be a rather long one.

Continue reading “Securing an Internet accessible server – Part 3”

WordPress behind HAProxy with TLS termination

My current project has been to set up a publicly accessible web server with a decent level of security. It has been an interesting exercise in applying “old” knowledge and gathering some new.

This weekend I finished this project for now. The current setup is as follows:
Behind my firewall, where I NAT port 80 and 443 for http and https traffic, I have set up a HAProxy instance. This allows me to do some interesting magic with incoming traffic.

In addition to the traffic manipulation, I also use the HAProxy server for contacting Let’s Encrypt to renew my TLS certificates, and for terminating TLS traffic. The latter has two reasons: a) I’m frankly too lazy to automate installing updated certificates on the web server, and b) I’m running the entire solution on so limited hardware that I’m a little bit worried about putting too much of a strain on it should there ever be a bit more traffic on the machine.

The web server is an Nginx running this very WordPress instance.

Let’s Encrypt configuration

I took the best parts from two different solutions to automate the relatively frequent certificate renewals that Let’s Encrypt enforces. I began by installing the HAProxy ACME Domain Validation Lua Plugin into HAProxy, which ensures that there’s a valid listener to show that I own my domain when I trigger the letsencrypt client program. The beauty of running this through HAProxy is that the process requires no downtime.

For the configuration of the letsencrypt client, I basically stole the scripts from Martijn Braam’s blog BrixIT, just adapting them to the fact that there was a listener provided through the Lua script. The benefit from doing it this way is that the BrixIT method is considerably more flexible than the Lua script when one expects HAProxy to use more than one certificate.

Example config:

/etc/haproxy/haproxy.conf

Global
[...]
    lua-load /etc/haproxy/acme-http01-webroot.lua
[...]

frontend web-http
    bind 0.0.0.0:80
    acl url_acme_http01 path_beg /.well-known/acme-challenge/
    http-request use-service lua.acme-http01 if METH_GET url_acme_http01
    redirect scheme https code 301 if !{ ssl_fc }

The last line also shows how to redirect regular http traffic to a https listener.

/opt/letsencrypt-haproxy:

#!/bin/bash

# Path to the letsencrypt-auto tool
LE_TOOL=/usr/bin/letsencrypt

# Directory where the acme client puts the generated certs
LE_OUTPUT=/etc/letsencrypt/live

# Concat the requested domains
DOMAINS=""
for DOM in "$@"
do
 DOMAINS+=" -d $DOM"
done

# Create or renew certificate for the domain(s) supplied for this tool
$LE_TOOL --agree-tos --renew-by-default certonly $DOMAINS --text --webroot --webroot-path /var/lib/haproxy --email name@somedomain.com

# Cat the certificate chain and the private key together for haproxy
cat $LE_OUTPUT/$1/{fullchain.pem,privkey.pem} > /etc/haproxy/ssl/${1}.pem

# Reload the haproxy daemon to activate the cert
systemctl reload haproxy

TLS termination configuration

The problem with terminating TLS traffic before the web server, is that any good web application should be able to recognize that the client is coming from an insecure connection. Luckily, we can use HAProxy to tell WordPress that the connection was good up until the load balancer and to trust it the rest of the way. Be aware that this is an extremely bad idea if there is any way to reach the web server other than via your HAProxy:

/usr/share/wordpress/wp-config.php:

[...]
/** Make sure WordPress understands it's behind an SSL terminator */
define('FORCE_SSL_ADMIN', true);
define('FORCE_SSL_LOGIN', true);
if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')
$_SERVER['HTTPS']='on';
[...]

/etc/haproxy/haproxy.cfg:

[...]
frontend web-https
    option http-server-close
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
[...]

As a final touch, I copied the brute force sandboxing scheme straight from this blog post by Baptiste Assmann over at haproxy.com.