IPsec

IPsec is deployed with strongswan on multiple servers throughout the architecture. It interconnects many of the KVM hosts but also the monitoring server because it can be used as a NAT bypass mechanism for some machines.

Hooking up a new node to the IPsec network

TODO: This is the old way of configuring Puppet nodes. There's now an ipsec module which does that more easily.

This is managed through Puppet, so it's basically a matter of adding the hostname to the ipsec role in modules/torproject_org/misc/local.yaml and adding the network configuration block to modules/ipsec/misc/config.yaml. For example, this was the diff for the new monitoring server:

diff --git c/modules/ipsec/misc/config.yaml w/modules/ipsec/misc/config.yaml
index e4367c38..3b724e77 100644
--- c/modules/ipsec/misc/config.yaml
+++ w/modules/ipsec/misc/config.yaml
@@ -50,3 +49,9 @@ hetzner-hel1-01.torproject.org:
   subnet:
     - 95.216.141.241/32
     - 2a01:4f9:c010:5f1::1/128
+
+hetzner-nbg1-01.torproject.org:
+  address: 195.201.139.202
+  subnet:
+    - 195.201.139.202/32
+    - 2a01:4f8:c2c:1e17::1/128
diff --git c/modules/torproject_org/misc/local.yaml w/modules/torproject_org/misc/local.yaml
index 703254f4..e2dd9ea3 100644
--- c/modules/torproject_org/misc/local.yaml
+++ w/modules/torproject_org/misc/local.yaml
@@ -163,6 +163,7 @@ services:
     - scw-arm-par-01.torproject.org
   ipsec:
     - hetzner-hel1-01.torproject.org
+    - hetzner-nbg1-01.torproject.org
     - kvm4.torproject.org
     - kvm5.torproject.org
     - macrum.torproject.org

Then Puppet needs to run on the various peers and the new peer should be rebooted, otherwise it will not be able to load the new IPsec kernel modules.

Special case: Mikrotik server

Update: we don't have a microtik server anymore. This documentation is kept for historical reference, in case such a manual configuration is required elsewhere.

The Mikrotik server is a special case that is not configured in Puppet, because Puppet can't run on its custom OS. To configure such a pairing, you first need to configure it on the normal server end, using something like this:

conn hetzner-nbg1-01.torproject.org-mikrotik.sbg.torproject.org
  ike = aes128-sha256-modp3072

  left       = 195.201.139.202
  leftsubnet = 195.201.139.202/32

  right = 141.201.12.27
  rightallowany = yes
  rightid     = mikrotik.sbg.torproject.org
  rightsubnet = 172.30.115.0/24

  auto = route

  forceencaps = yes
  dpdaction = hold

The left part is the public IP of the "normal server". The right part has the public and private IPs of the Mikrotik server. Then a secret should be generated:

printf '195.201.139.202 mikrotik.sbg.torproject.org : PSK "%s"' $(base64 < /dev/urandom | head -c 32) > /etc/ipsec.secrets.d/20-local-peers.secrets

In the above, the first field is the IP of the "left" side, the second field is the hostname of the "right" side, and then it's followed by a secret, the "pre-shared key" (PSK) that will be reused below.

That's for the "left" side. The "right" side, the Mikrotik one, is a little more involved. The first step is to gain access to the Mikrotik SSH terminal, details of which are stored in tor-passwords, in hosts-extra-info. A good trick is to look at the output of /export for an existing peer and copy-paste the good stuff. Here is how the nbg1 peer was configured on the "right" side:

[admin@mtsbg] /ip ipsec> peer add address=195.201.139.202 exchange-mode=ike2 name=hetzner-nbg1-01 port=500 profile=profile_1
[admin@mtsbg] /ip ipsec> identity add my-id=fqdn:mikrotik.sbg.torproject.org peer=hetzner-nbg1-01 secret=[REDACTED]
[admin@mtsbg] /ip ipsec> policy add dst-address=195.201.139.202/32 proposal=my-ipsec-proposal sa-dst-address=195.201.139.202 sa-src-address=0.0.0.0 src-address=172.30.115.0/24 tunnel=yes
[admin@mtsbg] /ip firewall filter> add action=accept chain=from-tor-hosts comment=hetzner-hel1-01 src-address=195.201.139.202
[admin@mtsbg] /system script> print
Flags: I - invalid 
 0   name="ping_ipsect_tunnel_peers" owner="admin" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon 
,,
[admin@mtsbg] /system script> remove 0
[admin@mtsbg] /system script> add dont-require-permissions=no name=ping_ipsect_tunnel_peers owner=admin policy=\
\...     ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="/ping count=1 src-address=172.30.115.1 172.30.134.1 ; \
"\...     \n/ping count=1 src-address=172.30.115.1 94.130.28.193 ; \
"\...     \n/ping count=1 src-address=172.30.115.1 94.130.38.33 ; \ 
"\...     \n/ping count=1 src-address=172.30.115.1 95.216.141.241 ; \
"\...     \n/ping count=1 src-address=172.30.115.1 195.201.139.202 ; \
"\...     \n"
[admin@mtsbg] /ip firewall nat> add action=accept chain=srcnat dst-address=195.201.139.202 src-address=172.30.115.0/24

The [REDACTED] part should be the PSK field defined on the left side (what is between quotes).

More information about how to configure IPsec on Mikrotik routers is available in the upstream documentation.

Debugging

To diagnose problems, you can check the state of a given connexion with, for example:

ipsec status hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org

This will show summary information of the current connexion. This shows, for example, an established and working connexion:

root@hetzner-nbg1-01:/home/anarcat# ipsec status hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org
Routed Connections:
hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org{6}:  ROUTED, TUNNEL, reqid 6
hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org{6}:   195.201.139.202/32 2a01:4f8:c2c:1e17::1/128 === 95.216.141.241/32 2a01:4f9:c010:5f1::1/128
Security Associations (3 up, 2 connecting):
hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org[4]: ESTABLISHED 9 minutes ago, 195.201.139.202[195.201.139.202]...95.216.141.241[95.216.141.241]
hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org{7}:  INSTALLED, TUNNEL, reqid 6, ESP SPIs: [redacted]_i [redacted]_o
hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org{7}:   195.201.139.202/32 2a01:4f8:c2c:1e17::1/128 === 95.216.141.241/32 2a01:4f9:c010:5f1::1/128

As a comparison, here is a connexion that is failing to complete:

root@hetzner-hel1-01:/etc/ipsec.secrets.d# ipsec status hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org
Routed Connections:
hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org{6}:  ROUTED, TUNNEL, reqid 6
hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org{6}:   95.216.141.241/32 2a01:4f9:c010:5f1::1/128 === 195.201.139.202/32 2a01:4f8:c2c:1e17::1/128
Security Associations (7 up, 1 connecting):
hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org[18]: CONNECTING, 95.216.141.241[%any]...195.201.139.202[%any]

The following messages are then visible in /var/log/daemon.log on that side of the connexion:

Apr  4 21:32:58 hetzner-hel1-01/hetzner-hel1-01 charon[14592]: 12[IKE] initiating IKE_SA hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org[17] to 195.201.139.202
Apr  4 21:35:44 hetzner-hel1-01/hetzner-hel1-01 charon[14592]: 05[IKE] initiating IKE_SA hetzner-hel1-01.torproject.org-hetzner-nbg1-01.torproject.org[18] to 195.201.139.202

In this case, the other side wasn't able to start the charon daemon properly because of missing kernel modules:

Apr  4 21:38:07 hetzner-nbg1-01/hetzner-nbg1-01 ipsec[25243]: charon has quit: initialization failed
Apr  4 21:38:07 hetzner-nbg1-01/hetzner-nbg1-01 ipsec[25243]: charon refused to be started
Apr  4 21:38:07 hetzner-nbg1-01/hetzner-nbg1-01 ipsec[25243]: ipsec starter stopped

Note that the ipsec statusall can also be used for more detailed status information.

The ipsec up <connexion> command can also be used to start a connexion manually, ipsec down <connexion> for stopping a connexion, naturally. Connexions are defined in /etc/ipsec.conf.d.

The traceroute command can be used to verify a host is well connected over IPsec. For example, this host is directly connected:

root@hetzner-nbg1-01:/home/anarcat# traceroute hetzner-hel1-01.torproject.org 
traceroute to hetzner-hel1-01.torproject.org (95.216.141.241), 30 hops max, 60 byte packets
 1  hetzner-hel1-01.torproject.org (95.216.141.241)  23.780 ms  23.781 ms  23.851 ms

Another example, this host is configured through IPsec, but somehow unreachable:

root@hetzner-nbg1-01:/home/anarcat# traceroute kvm4.torproject.org 
traceroute to kvm4.torproject.org (94.130.38.33), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *

That was because Puppet hadn't run on that other end. This Cumin recipe fixed that:

cumin 'C:ipsec' 'puppet agent -t'

The first run "failed" (as in, Puppet returned a non-zero status because it performed changes) but another run "succeeded").

Finally, never forget to "try to turn it off and on again". Simply rebooting the box can sometimes do wonders:

reboot

In my case, it seems the configuration wasn't being re-read by strongswan and rebooting the machine fixed it.

Traffic inspection

You may need to legitimately inspect the cleartext of an IPsec connexion, for example to diagnose what's taking up all that bandwidth between two nodes. It seems the state of the art on this is doing this by decrypting the ESP packets with Wireshark.

Comments

Please register or sign in to add a comment.