ganeti.mdwn

[Ganeti](http://ganeti.org/) is software designed to facilitate the management of
virtual machines (KVM or Xen). It helps you move virtual machine
instances from one node to another, create an instance with DRBD
replication on another node and do the live migration from one to
another, etc.

[[!toc levels=3]]

# Tutorial

## Listing virtual machines (instances)

This will show the running guests, known as "instances":

    gnt-instance list

## Accessing serial console

Our instances do serial console, starting in grub.  To access it, run

    gnt-instance console test01.torproject.org

To exit, use `^]` -- that is, Control-<Closing Bracket>.

# How-to

## Glossary

In Ganeti, a physical machine is called a *node* and a virtual machine
is an *instance*. A node is elected to be the *master* where all
commands should be ran from. Nodes are interconnected through a
private network that is used to communicate commands and synchronise
disks (with [[drbd]]). Instances are normally assigned two nodes: a
*primary* and a *secondary*: the *primary* is where the virtual
machine actually runs and th *secondary* acts as a hot failover.

See also the more extensive [glossary in the Ganeti documentation](http://docs.ganeti.org/ganeti/2.15/html/glossary.html).

## Adding a new instance

WARNING: this creates a machine with usrmerge! See [bug 34115](https://trac.torproject.org/projects/tor/ticket/34115)
before proceeding.

This command creates a new guest, or "instance" in Ganeti's
vocabulary:

    gnt-instance add \
      -o debootstrap+buster \
      -t drbd --no-wait-for-sync \
      --net 0:ip=pool,network=gnt-fsn \
      --no-ip-check \
      --no-name-check \
      --disk 0:size=10G \
      --disk 1:size=2G,name=swap \
      --disk 2:size=20G \
      --disk 3:size=800G,vg=vg_ganeti_hdd \
      --backend-parameters memory=8g,vcpus=2 \
      test-01.torproject.org

WARNING: there is a bug in `ganeti-instance-debootstrap` which
misconfigures `ping` (among other things), see [bug #31781](https://bugs.torproject.org/31781). It's
currently patched in our version of the Debian package, but that patch
might disappear if Debian upgrade the package without [shipping our
patch](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=944538).

This configures the following:

 * redundant disks in a DRBD mirror, use `-t plain` instead of `-t drbd` for
   tests as that avoids syncing of disks and will speed things up considerably
   (even with `--no-wait-for-sync` there are some operations that block on
   synced mirrors).  Only one node should be provided as the argument for
   `--node` then.
 * three partitions: one on the default VG (SSD), one on another (HDD)
   and a swap file on the default VG, if you don't specify a swap device,
   a 512MB swapfile is created in `/swapfile`. TODO: configure disk 2
   and 3 automatically in installer. (`/var` and `/srv`?)
 * 8GB of RAM with 2 virtual CPUs
 * an IP allocated from the public gnt-fsn pool:
   `gnt-instance add` will print the IPv4 address it picked to stdout.  The
   IPv6 address can be found in `/var/log/ganeti/os/` on the primary node
   of the instance, see below.
 * with the `test-01.torproject.org` hostname

To find the root password, ssh host key fingerprints, and the IPv6
address, run this **on the node where the instance was created**, for
example:

    egrep 'root password|configured eth0 with|SHA256' $(ls -tr /var/log/ganeti/os/* | tail -1) | grep -v $(hostname)

Note that you need to use the `--node` parameter to pick on which
machines you want the machine to end up, otherwise Ganeti will choose
for you. Use, for example, `--node fsn-node-01:fsn-node-02` to use
`node-01` as primary and `node-02` as secondary. It might be better to
let the Ganeti allocator do its job since it will, eventually do this
during cluster rebalancing. The allocator can sometimes fail if the
allocator is upset about something in the cluster, for example:
    
    Can's find primary node using iallocator hail: Request failed: No valid allocation solutions, failure reasons: FailMem: 2, FailN1: 2

This situation is covered by [ticket 33785](https://trac.torproject.org/projects/tor/ticket/33785).

We copy root's authorized keys into the new instance, so you should be able to
log in with your token.  You will be required to change the root password immediately.
Pick something nice and document it in `tor-passwords`.

Also set reverse DNS for both IPv4 and IPv6 in [hetzner's robot](https://robot.your-server.de/).
(Chek under servers -> vSwitch -> IPs)

Then follow [[new-machine]].

## Modifying an instance

It's possible to change the IP, CPU, or memory allocation of an instance
using the [gnt-instance modify](http://docs.ganeti.org/ganeti/2.15/man/gnt-instance.html#modify) command:

    gnt-instance modify -B vcpus=2 test1.torproject.org
    gnt-instance modify -B memory=4g test1.torproject.org
    gnt-instance reboot test1.torproject.org

IP address changes require a full stop and will require manual changes
to the `/etc/network/interfaces*` files:

    gnt-instance modify --net 0:modify,ip=116.202.120.175 test1.torproject.org
    gnt-instance stop test1.torproject.org
    gnt-instance start test1.torproject.org
    gnt-instance console test1.torproject.org

The [gnt-instance grow-disk](http://docs.ganeti.org/ganeti/2.15/man/gnt-instance.html#grow-disk) command can be used to change the size
of the underlying device:

    gnt-instance grow-disk test1.torproject.org 0 16g
    gnt-instance reboot test1.torproject.org

The number `0` in this context, indicates the first disk of the
instance. Then the filesystem needs to be resized inside the VM:

    ssh root@test1.torproject.org resize2fs /dev/sda1

## Adding a network interface on the rfc1918 vlan

We have a vlan that some VMs that do not have public addresses sit on.
Its vlanid is 4002 and its backed by Hetzner vswitch vSwitch #11973 "fsn-gnt-rfc1918-traffic".
Note that traffic on this vlan will travel in the clear between nodes.

To add an instance to this vlan, give it a second network interface using

    gnt-instance modify --net add:link=br0,vlan=4002,mode=openvswitch test1.torproject.org

## Destroying an instance

This totally deletes the instance, including all mirrors and
everything, be very careful with it:

    gnt-instance remove test01.torproject.org

## Disk operations (DRBD)

Instances should be setup using the DRBD backend, in which case you
should probably take a look at [[drbd]] if you have problems with
that. Ganeti handles most of the logic there so that should generally
not be necessary.

## Evaluating cluster capacity

This will list instances repeatedly, but also show their assigned
memory, and compare it with the node's capacity:

    watch -n5 -d 'gnt-instance list -o pnode,name,be/vcpus,be/memory,disk_usage,disk_template,status  |  sort; echo; gnt-node list'

The latter does not show disk usage for secondary volume groups, for a
complete picture of disk usage, try this:

    gnt-cluster command vgs

The [gnt-cluster verify](http://docs.ganeti.org/ganeti/2.15/man/gnt-cluster.html#verify) command will also check to see if there's
enough space on secondaries to account for the failure of a
node. Healthy output looks like this:

    root@fsn-node-01:~# gnt-cluster verify
    Submitted jobs 48030, 48031
    Waiting for job 48030 ...
    Fri Jan 17 20:05:42 2020 * Verifying cluster config
    Fri Jan 17 20:05:42 2020 * Verifying cluster certificate files
    Fri Jan 17 20:05:42 2020 * Verifying hypervisor parameters
    Fri Jan 17 20:05:42 2020 * Verifying all nodes belong to an existing group
    Waiting for job 48031 ...
    Fri Jan 17 20:05:42 2020 * Verifying group 'default'
    Fri Jan 17 20:05:42 2020 * Gathering data (2 nodes)
    Fri Jan 17 20:05:42 2020 * Gathering information about nodes (2 nodes)
    Fri Jan 17 20:05:45 2020 * Gathering disk information (2 nodes)
    Fri Jan 17 20:05:45 2020 * Verifying configuration file consistency
    Fri Jan 17 20:05:45 2020 * Verifying node status
    Fri Jan 17 20:05:45 2020 * Verifying instance status
    Fri Jan 17 20:05:45 2020 * Verifying orphan volumes
    Fri Jan 17 20:05:45 2020 * Verifying N+1 Memory redundancy
    Fri Jan 17 20:05:45 2020 * Other Notes
    Fri Jan 17 20:05:45 2020 * Hooks Results

A sick node would have said something like this instead: