How to install a new bare metal server at Hetzner
=================================================
This is for setting up physical metal at Hetzner.
Order
-----
1. get approval for the server, picking the specs from the [main
website](https://www.hetzner.com/)
2. head to the [order page](https://robot.your-server.de/order) and pick the right server. pay close
attention to the location, you might want to put it alongside
other TPO servers (or not!) depending on redundancy or traffic
requirements. Click `Add to shopping cart`, leaving all other
fields as default.
3. in the `Server login details` page, you should leave `Type` set to
`Public key`. If you do not recognize your public SSH key in
there, head to the [server list](https://robot.your-server.de/server) and click on [key
management](https://robot.your-server.de/key/index) to add your public keys
4. when you're certain of everything, click `Checkout` in the cart,
review the order again and click `Order in obligation`.
A confirmation email will be sent by Hetzner at the TPA alias when the
order is filed. Then you wait for the order to complete before being
able to proceed with the install.
Ordering physical servers from Hetzner can be very fast: we've seen 2
minutes turn around times.
Install
-------
At this point you should have received an email from Hetzner with a
subject like:
Subject: Your ordered SX62 server
It should contain the SSH fingerprint, and IP address of the new host
which we'll use below.
1. login to the server using the IP address and host key hash
provided above:
ssh -o FingerprintHash=md5 -o UserKnownHostsFile=/dev/null root@159.69.63.226
Note: the `FingerprintHash` parameter above is to make sure we
match the hashing algorithm used by Hetzner in their email, which
is, at the time of writing, MD5 (!). Newer versions of SSH will
also encode the hash as base64 instead of hexadecimal, so you
might want to decode the base64 into the latter using this: The
`UserKnownHostsFile` is to make sure we don't store the
(temporary) SSH host key.
perl -MMIME::Base64 -e '$h = unpack("H*", decode_base64(<>)); $h =~ s/(..)(?=.)/\1:/g; print $h, "\n"'
2. Partition disks. This might vary wildly between hosts, but in
general, we want:
* GPT partitionning, with space for a 8MB grub partition and
cleartext `/boot`
* software RAID (RAID-1 for two drives, RAID-5 for 3, RAID-10
for 4)
* crypto (LUKS)
* LVM, with separate volume groups for different medium (SSD vs
HDD)
This can be done with the `tor-install-format-disks` in the
`tsa-misc` repository, which should be carefully checked and
configured before running.
To get the scripts onto the host, you can clone them using
`git clone https://git.torproject.org/admin/tsa-misc`.
Check that the master hash matches what you expect
`(cd tsa-misc && git show-ref master)`.
3. Install the system. This can be done with `grml-debootstrap` which
will also configure grub, a root password and so on. This should
get you started, assuming the formatted root disk is mounted on
`/target`:
# make target/run stay clean
mkdir /target/run && mount -t tmpfs tgt-run /target/run
# grml-debootstrap hangs for weasel in vgs without this:
mkdir /target/run/udev && mount -o bind /run/udev /target/run/udev
ROOTPASSWORD=\$(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 30) \
echo -n "boot disk device: " && read bootdisk &&
apt-get install -y grml-debootstrap && \
sed -e 's/postfix//;
s/vlan//;
s/bridge-utils//;
s/ifenslave//;
s/resolvconf//;
s/zsh//;
s/strace//;
s/os-prober//;
s/bzip2//;
s/file//;
s/lsof//;
s/most//;
$adbus
$acryptsetup-initramfs
' /etc/debootstrap/packages > /root/grml-packages &&
grml-debootstrap --grub "$bootdisk" --target /target \
--hostname `hostname` --release buster \
--mirror https://mirror.hetzner.de/debian/packages/ \
--packages /root/grml-packages \
--password "$ROOTPASSWORD" \
--remove-configs --defaultinterfaces
4. Once the bootstrapping is complete, you still need to make sure
the system can boot as, the above doesn't (unfortunately)
configure everything for you. First, make a reasonable etc/fstab:
sed -e 's/^[[:space:]]*//' > /target/etc/fstab << EOF
/dev/$vg/root / ext4 errors=remount-ro 0 1
/dev/md/boot /boot ext4 defaults 0 2
/dev/$vg/swap none swap sw 0 0
tmpfs /tmp tmpfs defaults,size=512m 0 0
EOF
5. setup dropbear-initramfs to unlock the filesystem on boot. this
can be done with the `tor-install-luks-setup` in the `tsa-misc`
repository.
cd /target && bash /root/tsa-misc/scripts/tor-install-luks-setup
5. Review the crypto configuration:
cat /target/etc/crypttab
6. mount the helper filesystems once more
for fs in dev proc run sys ; do
mount -o bind /$fs "/target/$fs";
done
7. Do the same with the RAID configuration, probably with something like:
chroot /target sh -c "/usr/share/mdadm/mkconf > /etc/mdadm/mdadm.conf"
8. install grub on any secondary disk, for instance
chroot /target grub-install /dev/nvme1n1
9. Review the network configuration:
cat /target/etc/network/interfaces
An example safe configuration is:
iface lo inet loopback
allow-hotplug eth0
iface eth0 inet dhcp
10. Copy paste your key into the root's authorized keys, just to make
sure you can login:
mkdir -p /target/root/.ssh/ &&
cp /root/.ssh/authorized_keys /target/root/.ssh/authorized_keys
11. If any of those latter things changed, you need to regenerate the
initramfs:
chroot /target update-initramfs -u
chroot /target update-grub
12. umount things
umount /target/run/udev
for fs in dev proc run sys ; do
umount /target/$fs
done
umount /target/boot
cd / && umount /target
13. close things
vgchange -a n
cryptsetup luksClose cpv_nvme
for i in /dev/md/*; do mdadm --stop $i; done
14. Document the LUKS passphrase and root password in `tor-passwords`
15. Cross fingers and reboot:
reboot
Configuration
-------------
See [[new-machine]] for post-install configuration steps.