ganeti.mdwn

# Ganeti Cluster Operations

## Cluster Setup Preliminaries

- Make sure all nodes have the same LVM setup and the same network setup.  They want openvswitch.  Cf. host `fsn-node-01`'s /etc/network/interfaces.

- Prepare all the nodes by configuring them in puppet.  They should be in the class `roles::ganeti::fsn` if they
  are part of the fsn cluster.  If you make a new cluster, make a new role and add nodes.

## New Master

To create the fsn master, we added fsngnt to DNS, then ran

    gnt-cluster init \
      --master-netdev vlan-gntbe \
      --vg-name vg_ganeti \
      --secondary-ip 172.30.135.1 \
      --enabled-hypervisors kvm \
      --nic-parameters link=br0,vlan=4000 \
      --mac-prefix 00:66:37 \
      --no-ssh-init \
      --no-etc-hosts \
      fsngnt.torproject.org

## Add a new node

We did run the following on fsn-node-01:

    gnt-node add \
      --secondary-ip 172.30.135.2 \
      --no-ssh-key-check \
      --no-node-setup \
      fsn-node-02.torproject.org

## cluster config

These could probably be merged into the cluster init, but just to document what has been done:

    gnt-cluster modify --reserved-lvs vg_ganeti/root,vg_ganeti/swap
    gnt-cluster modify -H kvm:kernel_path=,initrd_path=,
    gnt-cluster modify -H kvm:security_model=pool
    gnt-cluster modify -H kvm:kvm_extra='-device virtio-rng-pci\,bus=pci.0\,addr=0x1e\,max-bytes=1024\,period=1000'
    gnt-cluster modify -H kvm:disk_cache=none
    gnt-cluster modify -H kvm:disk_discard=unmap
    gnt-cluster modify -H kvm:scsi_controller_type=virtio-scsi-pci
    gnt-cluster modify -H kvm:disk_type=scsi-hd
    gnt-cluster modify --uid-pool 4000-4019
    gnt-cluster modify --nic-parameters mode=openvswitch,link=br0,vlan=4000
    gnt-cluster modify -D drbd:c-plan-ahead=0,disk-custom='--c-plan-ahead 0'

### Network configuration

IP allocation is managed by Ganeti through the `gnt-network(8)`
system. Say we have `192.0.2.0/24` reserved for the cluster, with
the host IP `192.0.2.100`` and the gateway on `192.0.2.1`. You will
create this network with:

    gnt-network add --network 192.0.2.0/24 --gateway 192.0.2.1 --network6 2001:db8::/32 --gateway6 fe80::1 example-network

Then we associate the new network to the default node group:

    gnt-network connect --nic-parameters=link=br0,vlan=4000,mode=openvswitch example-network default

The arguments to `--nic-parameters` come from the values configured in
the cluster, above. The current values can be found with `gnt-cluster
info`.

TODO: create a private network.

## Listing instances and nodes

    gnt-instance list
    gnt-node list
    watch -n5 -d 'gnt-instance list -o pnode,name,be/vcpus,be/memory,status,disk_template  |  sort; echo; gnt-node list'

# Instance Operations

## Adding a new instance

This command creates a new guest, or "instance" in Ganeti's
vocabulary:

    gnt-instance add \
      -o debootstrap+buster \
      -t drbd --no-wait-for-sync \
      --node fsn-node-01:fsn-node-02 \
      --disk 0:size=10G \
      --disk 1:size=2G,name=swap \
      --disk 2:size=20G \
      --disk 3:size=800G,vg=vg_ganeti_hdd \
      --backend-parameters memory=8g,vcpus=2 \
      --net 0:ip=pool,network=gnt-fsn \
      --no-name-check \
      --no-ip-check \
      static-master-fsn.torproject.org

TODO: the above doesn't include the private network configuration.

This configures the following:

 * redundant disks in a DRBD mirror, use `-t plain` instead of `-t drbd` for
   tests as that avoids syncing of disks and will speed things up considerably
   (even with `--no-wait-for-sync` there are some operations that block on
   synced mirrors).  Only one node should be provided as the argument for
   `--node` then.
 * three partitions: one on the default VG (SSD), one on another (HDD)
   and a swap file on the default VG, if you don't specify a swap device,
   a 512MB swapfile is created in `/swapfile`
 * 2GB of RAM with 2 virtual CPUs
 * an IP allocated from the public gnt-fsn pool:
   `gnt-instance add` will print the IPv4 address it picked to stdout.  The
   IPv6 address can be found in `/var/log/ganeti/os/` on the primary node
   of the instance, see below.
 * with the `test01.torproject.org` hostname

To find the root password, ssh host key fingerprints, and the IPv6 address, run this on the node where the instance was created:

    egrep 'root password|configured eth0 with|SHA256' $(ls -tr /var/log/ganeti/os/* | tail -1) | grep -v $(hostname)

Note that you need to use the `--node` parameter to pick on which
machines you want the machine to end up, otherwise Ganeti will choose
for you.`

We copy root's authorized keys into the new instance, so you should be able to
log in with your token.  You will be required to change the root password immediately.
Pick something nice and document it in `tor-passwords`.

Also set reverse DNS for both IPv4 and IPv6 in [hetzner's robot](https://robot.your-server.de/).

Then follow [[new-machine]].

## Adding and removing addresses on instances

Say you created an instance but forgot to assign a private IP. You can
still do so with:

    gnt-instance modify --net -1:add,ip=172.30.135.3,network=internal test01.torproject.org

TODO: the internal network hasn't been created yet.

## Destroying an instance

This totally deletes the instance, including all mirrors and
everything, be very careful with it:

    gnt-instance remove test01.torproject.org

## Accessing serial console

Our instances do serial console, starting in grub.  To access it, run

    gnt-instance console test01.torproject.org

To exit, use `^]` -- that is, Control-&lt;Closing Bracket&gt;.

## Disk operations (DRBD)

Instances should be setup using the DRBD backend, in which case you
should probably take a look at [[drbd]] if you have problems with
that. Ganeti handles most of the logic there so that should generally
not be necessary.