ganeti.mdwn

# Ganeti setup

## Prepare

- Make sure all nodes have the same LVM setup and the same network setup.  They want openvswitch.  Cf. host `fsn-node-01`'s /etc/network/interfaces.

- Prepare all the nodes by configuring them in puppet.  They should be in the class `roles::ganeti::fsn` if they
  are part of the fsn cluster.  If you make a new cluster, make a new role and add nodes.

## New master

To create the fsn master, we added fsngnt to DNS, then ran

    gnt-cluster init \
      --master-netdev vlan-gntbe \
      --vg-name vg_ganeti \
      --secondary-ip 172.30.135.1 \
      --enabled-hypervisors kvm \
      --nic-parameters link=br0,vlan=4000 \
      --mac-prefix 00:66:37 \
      --no-ssh-init \
      --no-etc-hosts \
      fsngnt.torproject.org

## Add a new node

We did run the following on fsn-node-01:

    gnt-node add \
      --secondary-ip 172.30.135.2 \
      --no-ssh-key-check \
      --no-node-setup \
      fsn-node-02.torproject.org

## cluster config

These could probably be merged into the cluster init, but just to document what has been done:

    gnt-cluster modify --reserved-lvs vg_ganeti/root,vg_ganeti/swap
    gnt-cluster modify -H kvm:kernel_path=,initrd_path=,
    gnt-cluster modify -H kvm:security_model=pool
    gnt-cluster modify -H kvm:kvm_extra='-device virtio-rng-pci\,bus=pci.0\,addr=0x1e\,max-bytes=1024\,period=1000'
    gnt-cluster modify -H kvm:disk_cache=none
    gnt-cluster modify --uid-pool 4000-4019
    gnt-cluster modify --nic-parameters mode=openvswitch,link=br0,vlan=4000
    gnt-cluster modify -D drbd:c-plan-ahead=0,disk-custom='--c-plan-ahead 0'

### Network configuration

IP allocation is managed by Ganeti through the `gnt-network(8)`
system. Say we have `192.0.2.0/24` reserved for the cluster, with
the host IP `192.0.2.100`` and the gateway on `192.0.2.1`. You will
create this network with:

    gnt-network add --network 192.0.2.0/24 --gateway 192.0.2.1 --network6 2001:db8::/32 --gateway6 fe80::1 example-network

Then we associate the new network to the default node group:

    gnt-network connect --nic-parameters=link=br0,vlan=4000,mode=openvswitch example-network default

The arguments to `--nic-parameters` come from the values configured in
the cluster, above. The current values can be found with `gnt-cluster
info`.

TODO: create a private network.

# Regular maintenance

## Adding a new instance

This command creates a new guest, or "instance" in Ganeti's
vocabulary:

    gnt-instance add \
      -o debootstrap+buster \
      -t drbd --no-wait-for-sync \
      --disk 0:size=10G --disk 1:size=10G,vg=vg_ganeti_hdd --disk 2:size=1G,name=swap \
      --backend-parameters memory=2g,vcpus=2 \
      --net 0:ip=pool,network=gnt-fsn --no-name-check --no-ip-check \
      test01.torproject.org

TODO: the above doesn't include the private network configuration.

This configures the following:

 * redundant disks in a DRBD mirror, use `-t plain` for tests
 * three partitions: one on the default VG (SSD), one on another (HDD)
   and a swap file on the default VG, if you don't specify a swap device,
   a 512MB swapfile is created in `/swapfile`
 * 2GB of RAM with 2 virtual CPUs
 * an IP allocated from the public gnt-fsn pool
 * with the `test01.torproject.org` hostname

To find the root password, run this on the node where the instance was created:

    grep 'root password' $(ls -tr /var/log/ganeti/os/* | tail -1)

Note that you need to use the `--node` parameter to pick on which
machines you want the machine to end up, otherwise Ganeti will choose
for you.`

## Adding and removing addresses on instances

Say you created an instance but forgot to assign a private IP. You can
still do so with:

    gnt-instance modify --net -1:add,ip=172.30.135.3,network=internal test01.torproject.org

TODO: the internal network hasn't been created yet.

## Listing instances

    gnt-instance list

## Destroying an instance

This totally deletes the instance, including all mirrors and
everything, be very careful with it:

    gnt-instance remove test01.torproject.org

## Disk operations (DRBD)

Instances should be setup using the DRBD backend, in which case you
should probably take a look at [[drbd]] if you have problems with
that. Ganeti handles most of the logic there so that should generally
not be necessary.