[[_TOC_]]

How to install a new bare metal server at Hetzner
=================================================

This is for setting up physical metal at Hetzner.

Order
-----

 1. get approval for the server, picking the specs from the [main
 website](https://www.hetzner.com/)

 2. head to the [order page](https://robot.your-server.de/order) and pick the right server. pay close
    attention to the location, you might want to put it alongside
    other TPO servers (or not!) depending on redundancy or traffic
    requirements. Click `Add to shopping cart`, leaving all other
    fields as default.

 3. in the `Server login details` page, you should leave `Type` set to
    `Public key`. If you do not recognize your public SSH key in
    there, head to the [server list](https://robot.your-server.de/server) and click on [key
    management](https://robot.your-server.de/key/index) to add your public keys

 4. when you're certain of everything, click `Checkout` in the cart,
    review the order again and click `Order in obligation`.

A confirmation email will be sent by Hetzner at the TPA alias when the
order is filed. Then you wait for the order to complete before being
able to proceed with the install.

Ordering physical servers from Hetzner can be very fast: we've seen 2
minutes turn around times, but it can also take a lot more time in
some situations, see their [status page](https://wiki.hetzner.de/index.php/Auftragsbearbeitung/en) for estimates.

Automated install procedure
---------------------------

At this point you should have received an email from Hetzner with a
subject like:

    Subject: Your ordered SX62 server

It should contain the SSH fingerprint, and IP address of the new host
which we'll use below. The machine can be bootstrapped with a basic
Debian installer with the [howto/Fabric](howto/Fabric) code in the `tsa-misc` git
repository. Here's an example of a commandline:

    ./install -H root@88.99.194.57 \
              --fingerprint 0d:4a:c0:85:c4:e1:fe:03:15:e0:99:fe:7d:cc:34:f7 \
              --verbose \
              hetzner-robot \
              --fqdn=HOSTNAME.torproject.org \
              --fai-disk-config=installer/disk-config/gnt-fsn-NVMe \
              --package-list=installer/packages \
              --post-scripts-dir=installer/post-scripts/ \
              --mirror=https://mirror.hetzner.de/debian/packages/

Taking that apart:

 * `-H root@88.99.194.57`: the IP address provided by Hetzner in the
   confirmation email
 * `--fingerprint`: the ed25519 MD5 fingerprint from the same email
 * `--verbose`: important for now because otherwise the job is silent
   except for errors, which could be confusing
 * `hetzner-robot`: the install job type (only robot supported for now)
 * `--fqdn=HOSTNAME.torproject.org`: the Fully Qualified Domain Name
   to set on the machine, it is used in a few places, but the
   `hostname` is correctly set to the `HOSTNAME` part only
 * `--fai-disk-config=installer/disk-config/gnt-fsn-NVMe`: the disk
   configuration, in [fai-setup-storage(8)](https://manpages.debian.org/setup-storage.8) format
 * `--package-list=installer/packages`: the base packages to isntall
 * `--post-scripts-dir=installer/post-scripts/`: post-install scripts,
   magic glue that does everything
 
The last two are passed to `grml-debootstrap` and should rarely be
changed (although they could be coverted in to Fabric tasks
themselves).

Note that the script will show you lines like:

    STEP 1: SSH into server with fingerprint ...

Those correspond to the manual install procedure, below. If the
procedure stops before the last step (currently STEP 12), there was a
problem in the procedure, but the remaining steps can still be
performed by hand.

If a problem occurs in the install, you can login to the rescue shell
with:

    ssh -o FingerprintHash=md5 -o UserKnownHostsFile=~/.ssh/authorized_keys.hetzner-rescue root@88.99.194.57

... and check the fingerprint against the email provided by Hetzner.

See [howto/new-machine](howto/new-machine) for post-install configuration steps, then follow
[howto/new-machine-mandos](howto/new-machine-mandos) for setting up the mandos client on this host.

Manual install procedure
------------------------

WARNING: this procedure is kept for historical reference, and in case
the automatic procedure above fails for some reason. It should not be
used.

At this point you should have received an email from Hetzner with a
subject like:

    Subject: Your ordered SX62 server

It should contain the SSH fingerprint, and IP address of the new host
which we'll use below.

 1. login to the server using the IP address and host key hash
    provided above:

        ssh -o FingerprintHash=md5 -o UserKnownHostsFile=~/.ssh/authorized_keys.hetzner-rescue root@159.69.63.226

    Note: the `FingerprintHash` parameter above is to make sure we
    match the hashing algorithm used by Hetzner in their email, which
    is, at the time of writing, MD5 (!). Newer versions of SSH will
    also encode the hash as base64 instead of hexadecimal, so you
    might want to decode the base64 into the latter using this: The
    `UserKnownHostsFile` is to make sure we don't store the
    (temporary) SSH host key.

        perl -MMIME::Base64 -e '$h = unpack("H*", decode_base64(<>)); $h =~ s/(..)(?=.)/\1:/g; print $h, "\n"'

 2. Set a hostname (short version, not the FQDN):
 
        echo -n 'New hostname: ' && read hn && hostname "$hn" && exec bash

    TODO: merge this with wrapper script below.

 3. Partition disks. This might vary wildly between hosts, but in
    general, we want:

      * GPT partitionning, with space for a 8MB grub partition and
        cleartext `/boot`
      * software RAID (RAID-1 for two drives, RAID-5 for 3, RAID-10
        for 4)
      * crypto (LUKS)
      * LVM, with separate volume groups for different medium (SSD vs
        HDD)

    We are experimenting with FAI's [setup-storage](https://manpages.debian.org/setup-storage) to partition
    disks instead of rolling our own scripts. You first need to
    checkout the installer's configuration:

            apt install git
            git clone https://git.torproject.org/admin/tsa-misc
            cd tsa-misc/installer
            git show-ref master

    Check that the above hashes match a trusted copy.

    Use the following to setup a Ganeti node, for example:

            apt install fai-setup-storage

            setup-storage -f "disk-config/gnt-fsn-NVMe" -X

    TODO: merge this with wrapper script below.

    TODO: convert the other existing `tor-install-format-disks-4HDDs`
    script into a `setup-storage` configuration.

    And finally mount the filesystems:

            . /tmp/fai/disk_var.sh &&
            mkdir /target &&
            mount "$ROOT_PARTITION" /target &&
            mkdir /target/boot &&
            mount "$BOOT_DEVICE" /target/boot

    TODO: test if we can skip that test by passing `$ROOT_PARTITION`
    as a `--target` to `grml-debootstrap`. Probably not.
    
    TODO: in any case, this could be all wrapper up in a single
    wrapper shell script in tsa-misc instead of this long
    copy-paste. Possibly merge with `tor-install-hetzner` from
    [howto/new-machine-hetzner-cloud](howto/new-machine-hetzner-cloud).

 4. Install the system. This can be done with `grml-debootstrap` which
    will also configure grub, a root password and so on. This should
    get you started, assuming the formatted root disk is mounted on
    `/target` and that the boot device is defined by `$BOOT_DEVICE`
    (populated above by FAI). Note that `BOOT_DISK` is the *disk* as
    opposed to the *partition* which is `$BOOT_DEVICE`.

        BOOT_DISK=/dev/nvme0n1 &&
        mkdir -p /target/run && mount -t tmpfs tgt-run /target/run &&
        mkdir /target/run/udev && mount -o bind /run/udev /target/run/udev &&
        apt-get install -y grml-debootstrap && \
        grml-debootstrap \
            --debopt "--no-merged-usr" \
            --grub "$BOOT_DISK" \
            --target /target \
            --hostname `hostname` \
            --release buster \
            --mirror https://mirror.hetzner.de/debian/packages/ \
            --packages /root/tsa-misc/installer/packages \
            --post-scripts /root/tsa-misc/installer/post-scripts/ \
            --nopassword \
            --remove-configs \
            --defaultinterfaces &&
        umount /target/run/udev /target/run

 5. setup dropbear-initramfs to unlock the filesystem on boot. this
    should already have been done by the `50-tor-install-luks-setup` hook
    deployed in the grml-debootstrap stage.

    TODO: in an install following the above procedure, a keyfile was
    left unprotected in `/etc`. Make sure we have strong mechanisms to
    avoid that ever happening again. For example:
    
        chmod 0 /etc/luks/
    
    TODO: the keyfiles deployed there can be used to bootstrap
    mandos. Document how to do this better.

 6. Review the crypto configuration:

        cat /target/etc/crypttab

    If the backing device is *NOT* an SSD, remove the  `,discard`
    option.
    
    TODO: remove this step, it is probably unnecessary.

 7. Review the network configuration, since it will end up in the
    installed instance:

        cat /target/etc/network/interfaces

    An example safe configuration is:

        auto lo
        iface lo inet loopback
        
        allow-hotplug eth0
        iface eth0 inet dhcp

    The latter two lines usually need to be added as they are missing
    from Hetzner rescue shells:
    
        cat >> /etc/network/interfaces <<EOF
        
        allow-hotplug eth0
        iface eth0 inet dhcp
        EOF

    TODO: fix this in a post-install debootstrap hook, or in
    grml-debootstrap already, see also [upstream issue 105](https://github.com/grml/grml-debootstrap/issues/105) and
    [issue 136](https://github.com/grml/grml-debootstrap/issues/136).
    
    Add the hostname, IP address and domain to `/etc/hosts` and
    `/etc/resolv.conf`:
    
        grep torproject.org /etc/resolv.conf || ( echo 'domain torproject.org'; echo 'nameserver 8.8.8.8' ) >> /etc/resolv.conf
        if ! hostname -f 2>/dev/null || [ "$(hostname)" = "$(hostname -f)" ]; then
            IPADDRESS=$(ip -br -color=never route get to 8.8.8.8 | head -1 | grep -v linkdown | sed 's/.*  *src  *\([^ ]*\)  *.*/\1/')
            echo "$IPADDRESS $(hostname).torproject.org $(hostname)" >> /etc/hosts
        fi

    TODO: add the above as a post-hook. possibly merge with
    `tor-puppet/3rdparty/modules/ganeti/files/instance-debootstrap/hooks/gnt-debian-interfaces`

    TODO: add IPv6 address configuration. look at how
    `tor-install-generate-ldap` guesses as well.

 8. If any of those latter things changed, you need to regenerate the
    initramfs:

        chroot /target update-initramfs -u
        chroot /target update-grub

    TODO: remove this step, if the above extra steps are removed.

 9. umount things:
 
        umount /target/run/udev || true &&
        for fs in dev proc run sys  ; do
            umount /target/$fs || true
        done &&
        umount /target/boot &&
        cd / && umount /target

    TODO: merge this with wrapper script.

 10. close things

        vgchange -a n
        cryptsetup luksClose crypt_dev_md1
        cryptsetup luksClose crypt_dev_md2
        mdadm --stop /dev/md*

     TODO: merge this with wrapper script.

 11. Document the LUKS passphrase and root password in `tor-passwords`

 12. Cross fingers and reboot:

        reboot

See [howto/new-machine](howto/new-machine) for post-install configuration steps, then
follow [howto/new-machine-mandos](howto/new-machine-mandos) for setting up the mandos client on this host.
