Fixing “No DKIM keys saved for this domain” in EOP and Office365

Sometimes a newly added domain in Microsoft EOP will not let you enable DKIM from the web user interface. The only workaround I know of is to prepare the domain using PowerShell.

To connect a PS session to O365, I use the following script, ripped straight from Microsoft’s documentation:

$UserCredential = Get-Credential
$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $UserCredential -Authentication Basic -AllowRedirection
Import-PSSession $Session -DisableNameChecking

After waiting for an eternity for the necessary stuff to load, run the following command – and wait another eternity for it to finish:

New-DkimSigningConfig -DomainName "mydomain.tld" -Enabled $true

Note: Unless you’ve already added the necessary _domainkey CNAME records to your DNS zonefile, this command will succeed in generating the DKIM keys, but will fail to enable DKIM signing for the domain. Without looking into it I suspect that the Set-DkimSigningConfig cmdlet could be used to enable signing.

Finally disconnect from your O365 PS session:

Remove-PSSession $Session

Your domain now signs mail sent through O365 or via Exchange Online Protection.

Bonus knowledge: With a recent version of PowerShell Core installed, you can manage situations like this from a regular Mac or Linux box.

IKEv2 IPsec VPN with pfSense and Apple devices

Part 2: Apple VPN clients

(Part 1)

In the first part, we configured the pfSense firewall to allow clients to establish secure VPN connections to it. Now we’ll look at what needs to be done to get the clients to actually connect.

Specifically, we’ll create an Apple configuration profile that we can deliver to devices that we want to use as VPN clients.

We’ll start by getting the necessary certificates.

CA and Server certificates

As usual with a PKI-based solution, we need to trust the Root certificate to trust any certificates signed by the Root. Then we need a copy of the Server certificate’s public key to be able to establish an encrypted connection to it from the client. The VPN host in this case already has the client’s public key since we generated the client key-pair locally on the host.

In System – Cert. manager, select the “CAs” tab. Next to the “mydomain VPN-root-CA [year-month]” certificate we created earlier, there’s a row of blue icons. We’re interested in the middle one that represents a round seal. Press it, and your browser will download a .crt file; named something akin to “mydomain+VPN-root-CA+[year-month].crt

Then select the “Certificates” tab and do the same for the server certificate we created earlier. You will now have an additional file called “mydomain+VPN-server+[year-month].crt” in your Downloads directory.

Now for the only bit of shell magic we’ll need to do:

Client certificate

In System – Cert. manager, select the “Certificates” tab. This time download both the certificate (represented by the round seal icon” and the private key (represented by a key icon). This will store “mydomain+VPN-client+[year-month.crt]” and “mydomain+VPN-client+[year-month].key” in your Downloads directory.

Open a Terminal and run the following two commands:

$ cd ~/Downloads
$ openssl pkcs12 -export \
-in mydomain+VPN-client+[year-month].crt \ 
-inkey mydomain+VPN-client+[year-month].key \
-out mydomain+VPN-client+[year-month].p12

You will be asked for an export passphrase. Generate a secure one and store it in your password manager along with the certificate files.

Create an Apple Configuration Profile

This step requires a Mac with Apple Configurator 2 installed.

Start the program and create a new profile. Store it as “[year-month]-mydomain.tld-VPN.mobileconfig

General

Name: mydomain.tld VPN
Identifier: [Reverse FQDN of the VPN gateway, e.g. “tld.mydomain.vpn”
[The rest of the fields are optional]

Certificates

Using the “+” button, add the Root CA certificate (“mydomain+VPN-root-CA+[year-month].crt“), the Server certificate (“”mydomain+VPN-server+[year-month].crt“), and the client certificate bundle we generated earlier (“mydomain+VPN-client+[year-month].p12“). When adding the latter, we also need to enter the export pass phrase.

VPN

Connection Name: mydomain.tld VPN
Connection Type: IKEv2
Always-on VPN: Unchecked
Server: [The Common Name from the Server certificate]
Remote Identifier: [The Common Name from the Server certificate]
Local Identifier: [The Common Name from the Client certificate]
Machine Authentication: Certificate
Certificate Type: RSA
Server Certificate Issuer Common Name: [The Common Name from the Root CA]
Server Certificate Common Name: [The Common Name from the Server certificate]
Enable EAP: Checked
Disconnect on Idle: Optional – I have it set to Never
EAP Authentication: Certificate
Identity Certificate: Select your Client certificate
Dead Peer Detection Rate: Medium
Disable redirects: Unchecked
Disable Mobility and Multihoming: Unchecked
Use IPv4 / IPv6 Internal Subnet Attributes: Unchecked
Enable perfect forward secrecy: Unchecked
Enable certificate revocation check: Unchecked
[Note: The following checkboxes may be changed depending on requirements, but that is outside the scope for this article]
Disable redirects: Unchecked
Disable Mobility and Multihoming: Unchecked
Use IPv4 / IPv6 Internal Subnet Attributes: Unchecked
Enable perfect forward secrecy: Unchecked
Enable certificate revocation check: Unchecked

Select the “IKE SA Params” tab and fill in the following:
First set the Integrity Algorithm to SHA2-384
Then set the Encryption Algorithm to AES-256-GCM
Diffie-Hellman Group: 20
Lifetime In Minutes: 720
Proxy Setup: [Optional]

Select the “Child SA Params” and fill in the following:
First set the Integrity Algorithm to SHA2-256
Then set the Encryption Algorithm to AES-256-GCM
Diffie-Hellman Group: 20
Lifetime In Minutes: 60
Proxy Setup: [Optional]

Save the .mobileconfig.

Using the profile

macOS

The profile can be installed on a Mac by double-clicking the file and entering administrative credentials to allow it to install. When installed, System Preferences – Network will contain a new “network device” called mydomain.tld VPN, with a padlock as an icon. It’s possible to start the VPN connection from here. It’s also possible to check the “Show VPN status in menu bar” checkbox, and manage the VPN by clicking the resulting icon.

iOS

The simplest way to install the profile on an iOS device is by mailing it and tapping the file from within Mail. After providing the device password to allow system changes, there will be a new “mydomain.tld VPN” profile in Settings – VPN. Select it and change Status to Connected.

Conclusion

We have enabled a simple and secure way to reach our home network and to reach the Internet via a known and trusted gateway from our Apple devices even when on the move.
With the proper client configuration, the same principles should be applicable to a client running any modern operating system.

IKEv2 IPsec VPN with pfSense and Apple devices

Part 1: pfSense configuration

For a long time I’ve been content running a simple SSH gateway into my network, since I was severely bandwidth-limited.

The connection was secured in a number of ways I consider a sort of best practice: no remote login for the root account, key based (as opposed to password based) logon, and a custom port which doesn’t add any security per se, but which let me avoid the most common hammering from Asian botnets looking for a way in.

However now that I have a good connection, I have some use for accessing more bandwidth-hungry services from home – or, for that matter, to redirect my Internet traffic via my home when surfing the web over insecure Internet connections.

Here’s the first part of a howto that works with pfSense 2.4, macOS High Sierra (10.13), and iOS 11:

Certificates

The first thing we need is a set of certificates to for mutual identification and encryption between the clients and the VPN endpoint. We’ll start the process on the pfSense box:

CA Certificate

In SystemCert. manager, choose the “CAs” tab and Add a CA certificate.

Descriptive name: mydomain VPN-root-CA [year-month]
Method: Create an internal Certificate Authority
Key length: 2048
Digest algorithm: SHA256
Lifetime (Days): 3650
[Fill in everything down to but not including Common Name]
Common Name: mydomain.tld-vpnrootca

Save the certificate.

Server certificate

In System – Cert. manager, choose the “Certificates” tab and Add/Sign a Server certificate.

Method: Create an internal Certificate
Descriptive name: mydomain VPN-server [year-month]
Certificate Authority: mydomain.tld-vpnrootca
Key length: 2048
Digest algorithm: SHA256
Lifetime (Days): 3650
[….]
Common name: [FQDN of VPN gateway]
Certificate type: Server certificate
Alternative names: Type: FQDN or Hostname Value: [FQDN of VPN gateway]

NB! Do not forget to add an Alternative name even if it’s identical to the Common name!

Save the certificate.

Client certificate

In System – Cert. manager, choose the “Certificates” tab and Add/Sign a User certificate.

Method: Create an internal Certificate
Descriptive name: mydomain VPN-client [year-month]
Certificate Authority: mydomain.tld-vpnrootca
Key length: 2048
Digest algorithm: SHA256
Lifetime (Days): 3650
[….]
Common name: vpnclient.mydomain.tld
Certificate type: User certificate
Alternative names: Type: FQDN or Hostname Value: vpnclient.mydomain.tld

NB! Do not forget to add an Alternative name even if it’s identical to the Common name!

Save the certificate.

VPN configuration

Mobile Client settings

In VPN – IPsec, choose the “Mobile clients” tab and fill in the following values:

IKE Extensions: Enable IPsec mobile client support – checked
User Authentication: Source: Local Database
Group Authentication: Source: system
Virtual Address Pool: Provide a virtual IP address to clients – checked
Network configuration for Virtual Address Pool: 10.200.250.0/24
Provide a virtual IPv6 address to clients: Unchecked
Provide a list of accessible networks to clients:
Unchecked
Allow clients to save Xauth passwords (Cisco VPN client only).: Unchecked
Provide a default domain name to clients: Checked
Specify domain as DNS Default Domain: mydomain.tld
Provide a list of split DNS domain names to clients.: Unchecked
Provide a DNS server list to clients: Checked
[Fill in your DNS servers]
Provide a WINS server list to clients: Unchecked
Provide the Phase2 PFS group to clients: Unchecked
Provide a login banner to clients: Unchecked

Save the settings.

Phase 1 settings

In VPN – IPsec, choose the “Tunnels” tab and Add P1.

Disabled: Unchecked
Key Exchange version: IKEv2
Internet Protocol: IPv4
Interface: WAN
Description: IKEv2 Phase 1
Authentication Method: EAP-TLS
My identifier: Distinguished Name; [Common Name of your Server certificate]
Peer identifier: Any
My Certificate: [Descriptive Name of your Server certificate]
Peer Certificate Authority: [Descriptive Name of your CA certificate]
Encryption Algorithm: AES256-GCM
Key length: 128 bits
Hash: SHA384
DH Group: 20 (nist ecp384)
Lifetime (Seconds)28800
Disable rekey: Unchecked
Margintime (Seconds): 20
Disable Reauth: Unchecked
Responder Only: Checked
MOBIKE: Enable
Split connections: Unchecked
Dead Peer Detection: Checked
Delay: 10
Max failures: 5

Save the settings.

Phase 2 settings

In VPN – IPsec, choose the “Tunnels” tab, Show Phase 2 Entries, and Add P2.

Disabled: Unchecked
Mode: Tunnel IPv4
Local Network: Type: Network
Address: 0.0.0.0/0
NAT/BINAT translation: None
Description: IKEv2 Phase 2
Protocol: ESP
Encryption Algorithms: Check AES256-GCM/128 bits only
Hash Algorithms: Check SHA256 only
PFS key group: 20 (nist ecp384)
Lifetime: 3600
Automatically ping host: [empty]

Save the settings.

Firewall settings

In Firewall – Rules, choose the “IPsec” tab and Add a rule. In this case we’re not interested in limiting traffic, so it will be an “allow all” type rule:

Action: Pass
Disabled: Unchecked
Interface: IPsec
Address Family: IPv4
Protocol: Any
Source: Any
Destination: Any
Log: Unchecked
Description: Allow all VPN traffic to anywhere.

Save the firewall rule.

This is it for the firewall configuration. In the next part (Part 2) we’ll export the certificates and set up an Apple Configurator config for iOS and macOS devices.

SIP telephony behind a pfSense firewall

Background:
When we got the fibre connection, I decided to use Bahnhof as our service provider. They enable a SIP phone connection at no extra cost, but they don’t support using third-party SIP boxes; you have to use their combined router/wifi AP/SIP converter (a box by Tilgin), which they manage for you.
Naturally, since I’m tinkering a bit, using a third-party router I can’t manage in front of my network would be unacceptable. The next best thing, then, is to put the Tilgin router behind the pfSense box and use it only for SIP.

Setup:
Bahnhof demands opening the following ports for SIP telephony to work:
69 – UDP
5060 – 5080 TCP + UDP
9000 – 14000 UDP
50000 – 60000 UDP

I set up a DHCP reservation for the Tilgin box, gave it an alias in pfSense, and NATed the ports specified above to it.
Second, I connected the WAN port of the Tilgin box to my network, and saw that it started up fine, and I could both call out and receive calls using a phone connected to the router. All fine, right?
Not quite. After a few hours, incoming calls stopped working. A couple of minutes with my search engine provided the following page: https://www.netgate.com/docs/pfsense/nat/configuring-nat-for-voip-phones.html.
The required fix was the first one suggested; to enable hybrid outbound NAT and static ports for UDP traffic from the Tilgin box.

Two steps forward and one step back

I’m happy to report that oxcrag.net has been upgraded from a crappy ADSL line to a fibre based connection, which has improved uplink speed for the server tremendously.

Unfortunately, it looks as though newer technology doesn’t always imply that everything gets better: Unlike what the representative for the fibre project stated, there’s no sign of IPv6 on this network with my chosen provider, Bahnhof. Indeed a mail to their support was answered with the short statement that they do not currently have an agreement with Telia – the network owner – for IPv6 over the current service solution “Öppen Fiber”. If I would want to pay Telias exorbitant fees, I could probably keep using their IPv6rd tunnel, but I don’t see the point in haggling or ISP-hopping. On the other hand, IPv6 tunnel services from other service providers break geographically limited content like Netflix. What I think of that practice is probably subject to a rant by itself, but suffice to say Netflix thinks I’m a pirate when I use the Swedish gateway of Hurricane Electric’s IPv6 tunneling service.

Long story short, what’s called “Telia Öppen Fiber” in Sweden is only “open” in a very Orwellian sense, and so I’ve lost the convenience of IPv6 addressing for my machines – at least for the foreseeable future.

Closed is open. Worse is better. Old is new.

IPv6 guests in KVM

I’ve been experimenting with IPv6 at home, and spent some time trying to get it working in my virtual machines.

The first symptom I got was that VMs got a “Network unreachable” error when trying to ping6 anything but their own address. The cause was a complete brainfart on my side: We need a loopback interface network definition for IPv6 in /etc/network/interfaces:

auto lo
iface lo inet loopback
iface lo inet6 loopback

The second problem took a bit more digging to understand: I would get an IPv6 address, and I could ping stuff both on my own network and on the Internet from the VM, but no other computers could reach the virtual machine over IPv6.

According to this discussion, QEMU/KVM has support for multicast (required for proper IPv6 functioning), but it’s turned off by default. Remedy this by running virsh edit [vm-name] and adding trustGuestRxFilters='yes' to the appropriate network interface definition:

    
      
      
      
      

As usual, when you understand the problem the solution is simple.

Frustrations in Ubuntu 18.04

My first frustration with Ubuntu 18.04 came yesterday. I created a template VM with my basic toolkit that any machine in my network should have. I then deployed the VM and asked vSphere to set the hostname to the value of the VM name. Strangely, this didn’t happen: The new machine booted up alright, but its name remained that of the template.

Remember the old way to manually change the name of a machine in Linux? It went something like this:

  1. Add the new hostname to your /etc/hosts so sudo doesn’t get confused.
  2. Replace the old hostname in /etc/hostname with the new one.
  3. Reboot the computer or restart all affected services.

The new way goes like this:

  1. Add the new hostname to your /etc/hosts so sudo doesn’t get confused.
  2. Replace the old hostname in /etc/hostname with the new one.
  3. Reboot the computer.
  4. Notice that the hostname is the same as it was before you attempted to change it.
  5. Web search “change hostname ubuntu 18.04”.
  6. Discover that there’s a new utility, hostnamectl, which has a command, change-hostname, that takes the new hostname as an argument.
  7. Run hostnamectl change-hostname [newname]
  8. Run hostnamectl without any arguments to confirm that “Static hostname” has the correct value.
  9. Log off and back on again and be happy that everything seems to be working.
  10. Reboot the computer after doing some changes.
  11. Notice that the hostname is back to what it was.
  12. Run hostnamectl change-hostname [newname] again, and check in /etc/hostname just to see that it actually did change the file to contain the new hostname.
  13. Check in /etc/hosts and see that the new name appears there too.
  14. Scour the web some more for additional information.
  15. Find some mention of cloud-init.
  16. Read up on it and see the point of it – but also that it doesn’t apply to my current environment.
  17. Run sudo apt remove cloud-init
  18. Reboot the server and see that it works as expected again.
  19. (In the future: Learn more about cloud-init and re-evaluate whether it should be implemented in my environment as a complement to Ansible).

DNS/DHCP issues in modern Windows versions

Static IP addresses are a solid way to configure machines if you have few enough of them to manage them manually. But the more ability you want to have to change things on the fly, the more limiting such a configuration scheme becomes.

Unfortunately I’ve had severe problems with getting servers with DHCP leases (or even DHCP reservations) to have their names stick in DNS over time. Suddenly, after a reboot a machine would seemingly drop off the network even though it had the same IP address as earlier. Rebooting or manually re-registering its DNS record would solve the problem, but it wasn’t an acceptable solution to the underlying issue.

I found a discussion that gave a few pointers on how to get these things working in Windows, and I’ve shamelessly ripped the relevant points to present them here:

Step one: Define a user in whose context the DHCP server will run

Simply add a domain user with no special rights, and give it a properly strong password. Then open the DHCP management console, right-click the protocol you want to change (IPv4 or IPv6), and select Properties and the Advanced tab. Click Credentials and enter the relevant information for the account.

Step two: Tell DHCP to always attempt to update DNS records

In the same properties window, select the DNS tab. Ensure the following choices are ticked:

Enable DNS Dynamic Updates(…) -> Always dynamically update DNS records
Dynamically update DNS records for DHCP clients that do not request updates

Step three: Ensure DHCP server AD group membership

The DHCP server(s) should exist in the group DNSUpdateProxy. No other user or computer accounts may exist in this group.

Other tips

Make sure DHCP leases are longer than 24 hours, or bad things are likely to happen. A concrete example given is that Microsoft KMS servers have a 24 hour update cycle.

 

SMS notifications from Zabbix

To increase visibility of critical alerts, I’ve set up Zabbix to send messages to the phones of certain technicians via a GSM modem. This post documents what I had to do to get things running.

  • Zabbix server
  • MOXA ethernet-to-serial gateway
  • Siemens GSM modem

The first thing to do is to get the driver for the MOXA gateway running. The documentation is alrightish and covers some of the most used Linux distributions. Dissappointingly, after running through the checklist in the README file, the MOXA gateway will only work until after the next reboot.

The problem here, is that the init script for the MOXA driver removes and then recreates the device nodes on startup, and the scripts being called (mxmknod and mxrmnod) contain references to current dir (./) rather than their absolute path (/usr/lib/npreal2/driver). Edit those scripts, and you’ll have a working driver even after the Zabbix server reboots.

It should now be possible to test the modem by running, for example, socat - /dev/ttyr00 and entering some AT commands like in the olden days.

The next step is to install the gsm-utils package, which will create the possibility to easily send messages. Without any other options set, the gsmsendsms command expects there to be a device called /dev/mobilephone. We can create that one by making a symlink with that name to /dev/ttyr00, or we can tell the command which device to use with the -d argument.

Try sending a message: gsmsmssend -d /dev/ttyr00 <phone_number> "Hello World!"

This does not immediately solve how to send messages from Zabbix, though. Recent versions of Zabbix do have an SMS media type that can be called on, but it doesn’t seem to work when not speaking directly to one of the two modems on the compatibility list. Fortunately it’s very easy to create a new media type:

To prepare for this, create the folder /etc/zabbix/alertscripts, then modify the AlertScriptsPath definition in /etc/zabbix/zabbix-sever.conf to point at this directory, and restart the Zabbix Server service.

Now create the script /etc/zabbix/alertscripts/sendsms with the following contents:

#!/bin/bash
serial=`date +%s | cut -c 6-`
/usr/bin/gsmsendsms -d /dev/ttyr00 -b 115200 -c $serial $1 "$2"

The -c $serial argument is used to create a unique ID when serializing messages longer than 160 characters, as per the SMS standard. $1 will be populated with the contact phone number, and $2 will be populated by the actual message from Zabbix.

Finally let’s create our Media Type definition in Zabbix:

Under Administration -> Media Types -> Create Media Type define the following:

Name: SMS via MOXA
Type: Script
Script Name: sendsms
Script Parameters:
{ALERT.SENDTO}
{ALERT.MESSAGE}

To be able to alert users, Zabbix needs to know how to contact them using this new media type:

Administration -> Users -> <username> -> Media -> Add

Type: SMS via Moxa
Send to: <phonenumber>
When active: schedule during which to receive alerts
Use if severity: Severity levels to send via SMS
Enabled:

And finally you need to create Actions that define when to send messages via this solution, under Configuration -> Actions.

Voilà: Zabbix just got a bit noisier, in a good way.

Transport security with Postfix

I had a “Face: Meet Palm” moment today, and as usual when that happens, I learned something new:

What happened was that I noticed that mail from a Postfix server I use for sending mail from a couple of domains was marked with the red “no encryption” label rather than the expected grey “standard encryption” icon when I looked at the message details in Gmail. I was sure that I had set the server to use what they call “opportunistic TLS”; that is: Attempt to use TLS but fall back to no encryption if that’s unavailable.

Reading the Postfix documentation, however, I saw the problem: there are two sets of TLS rules in the main.cf configuration file: those starting with “smtpd_“, which deal with how the server responds to its clients, and those who start with “smtp_“, which deal with how Postfix acts when working in client mode towards other servers.

So now I have the following two lines in my /etc/postfix/main.cf:

smtp_tls_security_level = may
smtpd_tls_security_level = may