new-machine-hetzner-robot.md

• How to install a new bare metal server at Hetzner
  • Order
  • Automated install procedure
  • Manual install procedure

How to install a new bare metal server at Hetzner

This is for setting up physical metal at Hetzner.

Order

1. get approval for the server, picking the specs from the main website

2. head to the order page and pick the right server. pay close attention to the location, you might want to put it alongside other TPO servers (or not!) depending on redundancy or traffic requirements. Click Add to shopping cart, leaving all other fields as default.

3. in the Server login details page, you should leave Type set to Public key. If you do not recognize your public SSH key in there, head to the server list and click on key management to add your public keys

4. when you're certain of everything, click Checkout in the cart, review the order again and click Order in obligation.

A confirmation email will be sent by Hetzner at the TPA alias when the order is filed. Then you wait for the order to complete before being able to proceed with the install.

Ordering physical servers from Hetzner can be very fast: we've seen 2 minutes turn around times, but it can also take a lot more time in some situations, see their status page for estimates.

Automated install procedure

At this point you should have received an email from Hetzner with a subject like:

Subject: Your ordered SX62 server

It should contain the SSH fingerprint, and IP address of the new host which we'll use below. The machine can be bootstrapped with a basic Debian installer with the howto/Fabric code in the tsa-misc git repository. Here's an example of a commandline:

./install -H root@88.99.194.57 \
          --fingerprint 0d:4a:c0:85:c4:e1:fe:03:15:e0:99:fe:7d:cc:34:f7 \
          --verbose \
          hetzner-robot \
          --fqdn=HOSTNAME.torproject.org \
          --fai-disk-config=installer/disk-config/gnt-fsn-NVMe \
          --package-list=installer/packages \
          --post-scripts-dir=installer/post-scripts/ \
          --mirror=https://mirror.hetzner.de/debian/packages/

Taking that apart:

• -H root@88.99.194.57: the IP address provided by Hetzner in the confirmation email
• --fingerprint: the ed25519 MD5 fingerprint from the same email
• --verbose: important for now because otherwise the job is silent except for errors, which could be confusing
• hetzner-robot: the install job type (only robot supported for now)
• --fqdn=HOSTNAME.torproject.org: the Fully Qualified Domain Name to set on the machine, it is used in a few places, but the hostname is correctly set to the HOSTNAME part only
• --fai-disk-config=installer/disk-config/gnt-fsn-NVMe: the disk configuration, in fai-setup-storage(8) format
• --package-list=installer/packages: the base packages to isntall
• --post-scripts-dir=installer/post-scripts/: post-install scripts, magic glue that does everything

The last two are passed to grml-debootstrap and should rarely be changed (although they could be coverted in to Fabric tasks themselves).

Note that the script will show you lines like:

STEP 1: SSH into server with fingerprint ...

Those correspond to the manual install procedure, below. If the procedure stops before the last step (currently STEP 12), there was a problem in the procedure, but the remaining steps can still be performed by hand.

If a problem occurs in the install, you can login to the rescue shell with:

ssh -o FingerprintHash=md5 -o UserKnownHostsFile=~/.ssh/authorized_keys.hetzner-rescue root@88.99.194.57

... and check the fingerprint against the email provided by Hetzner.

See howto/new-machine for post-install configuration steps, then follow howto/new-machine-mandos for setting up the mandos client on this host.

Manual install procedure

WARNING: this procedure is kept for historical reference, and in case the automatic procedure above fails for some reason. It should not be used.

At this point you should have received an email from Hetzner with a subject like:

Subject: Your ordered SX62 server

It should contain the SSH fingerprint, and IP address of the new host which we'll use below.

1. login to the server using the IP address and host key hash provided above:

   ssh -o FingerprintHash=md5 -o UserKnownHostsFile=~/.ssh/authorized_keys.hetzner-rescue root@159.69.63.226

   Note: the FingerprintHash parameter above is to make sure we match the hashing algorithm used by Hetzner in their email, which is, at the time of writing, MD5 (!). Newer versions of SSH will also encode the hash as base64 instead of hexadecimal, so you might want to decode the base64 into the latter using this: The UserKnownHostsFile is to make sure we don't store the (temporary) SSH host key.

   perl -MMIME::Base64 -e '$h = unpack("H*", decode_base64(<>)); $h =~ s/(..)(?=.)/\1:/g; print $h, "\n"'

2. Set a hostname (short version, not the FQDN):

   echo -n 'New hostname: ' && read hn && hostname "$hn" && exec bash

   TODO: merge this with wrapper script below.

3. Partition disks. This might vary wildly between hosts, but in general, we want:

   • GPT partitionning, with space for a 8MB grub partition and cleartext /boot
   • software RAID (RAID-1 for two drives, RAID-5 for 3, RAID-10 for 4)
   • crypto (LUKS)
   • LVM, with separate volume groups for different medium (SSD vs HDD)

   We are experimenting with FAI's setup-storage to partition disks instead of rolling our own scripts. You first need to checkout the installer's configuration:

       apt install git
       git clone https://git.torproject.org/admin/tsa-misc
       cd tsa-misc/installer
       git show-ref master

   Check that the above hashes match a trusted copy.

   Use the following to setup a Ganeti node, for example:

       apt install fai-setup-storage
   
       setup-storage -f "disk-config/gnt-fsn-NVMe" -X

   TODO: merge this with wrapper script below.

   TODO: convert the other existing tor-install-format-disks-4HDDs script into a setup-storage configuration.

   And finally mount the filesystems:

       . /tmp/fai/disk_var.sh &&
       mkdir /target &&
       mount "$ROOT_PARTITION" /target &&
       mkdir /target/boot &&
       mount "$BOOT_DEVICE" /target/boot

   TODO: test if we can skip that test by passing $ROOT_PARTITION as a --target to grml-debootstrap. Probably not.

   TODO: in any case, this could be all wrapper up in a single wrapper shell script in tsa-misc instead of this long copy-paste. Possibly merge with tor-install-hetzner from howto/new-machine-hetzner-cloud.

4. Install the system. This can be done with grml-debootstrap which will also configure grub, a root password and so on. This should get you started, assuming the formatted root disk is mounted on /target and that the boot device is defined by $BOOT_DEVICE (populated above by FAI). Note that BOOT_DISK is the disk as opposed to the partition which is $BOOT_DEVICE.

   BOOT_DISK=/dev/nvme0n1 &&
   mkdir -p /target/run && mount -t tmpfs tgt-run /target/run &&
   mkdir /target/run/udev && mount -o bind /run/udev /target/run/udev &&
   apt-get install -y grml-debootstrap && \
   grml-debootstrap \
       --grub "$BOOT_DISK" \
       --target /target \
       --hostname `hostname` \
       --release bullseye \
       --mirror https://mirror.hetzner.de/debian/packages/ \
       --packages /root/tsa-misc/installer/packages \
       --post-scripts /root/tsa-misc/installer/post-scripts/ \
       --nopassword \
       --remove-configs \
       --defaultinterfaces &&
   umount /target/run/udev /target/run

5. setup dropbear-initramfs to unlock the filesystem on boot. this should already have been done by the 50-tor-install-luks-setup hook deployed in the grml-debootstrap stage.

   TODO: in an install following the above procedure, a keyfile was left unprotected in /etc. Make sure we have strong mechanisms to avoid that ever happening again. For example:

   chmod 0 /etc/luks/

   TODO: the keyfiles deployed there can be used to bootstrap mandos. Document how to do this better.

6. Review the crypto configuration:

   cat /target/etc/crypttab

   If the backing device is NOT an SSD, remove the ,discard option.

   TODO: remove this step, it is probably unnecessary.

7. Review the network configuration, since it will end up in the installed instance:

   cat /target/etc/network/interfaces

   An example safe configuration is:

   auto lo
   iface lo inet loopback
   
   allow-hotplug eth0
   iface eth0 inet dhcp

   The latter two lines usually need to be added as they are missing from Hetzner rescue shells:

   cat >> /etc/network/interfaces <<EOF
   
   allow-hotplug eth0
   iface eth0 inet dhcp
   EOF

   TODO: fix this in a post-install debootstrap hook, or in grml-debootstrap already, see also upstream issue 105 and issue 136.

   Add the hostname, IP address and domain to /etc/hosts and /etc/resolv.conf:

   grep torproject.org /etc/resolv.conf || ( echo 'domain torproject.org'; echo 'nameserver 8.8.8.8' ) >> /etc/resolv.conf
   if ! hostname -f 2>/dev/null || [ "$(hostname)" = "$(hostname -f)" ]; then
       IPADDRESS=$(ip -br -color=never route get to 8.8.8.8 | head -1 | grep -v linkdown | sed 's/.*  *src  *\([^ ]*\)  *.*/\1/')
       echo "$IPADDRESS $(hostname).torproject.org $(hostname)" >> /etc/hosts
   fi

   TODO: add the above as a post-hook. possibly merge with tor-puppet/3rdparty/modules/ganeti/files/instance-debootstrap/hooks/gnt-debian-interfaces

   TODO: add IPv6 address configuration. look at how tor-install-generate-ldap guesses as well.

8. If any of those latter things changed, you need to regenerate the initramfs:

   chroot /target update-initramfs -u
   chroot /target update-grub

   TODO: remove this step, if the above extra steps are removed.

9. umount things:

   umount /target/run/udev || true &&
   for fs in dev proc run sys  ; do
       umount /target/$fs || true
   done &&
   umount /target/boot &&
   cd / && umount /target

   TODO: merge this with wrapper script.

10. close things

    vgchange -a n cryptsetup luksClose crypt_dev_md1 cryptsetup luksClose crypt_dev_md2 mdadm --stop /dev/md*

    TODO: merge this with wrapper script.

11. Document the LUKS passphrase and root password in tor-passwords

12. Cross fingers and reboot:

    reboot

See howto/new-machine for post-install configuration steps, then follow howto/new-machine-mandos for setting up the mandos client on this host.