... | @@ -2056,12 +2056,56 @@ Look into logs on the relevant nodes (particularly |
... | @@ -2056,12 +2056,56 @@ Look into logs on the relevant nodes (particularly |
|
`/var/log/ganeti/node-daemon.log`, which shows all commands ran by
|
|
`/var/log/ganeti/node-daemon.log`, which shows all commands ran by
|
|
ganeti) when you have problems.
|
|
ganeti) when you have problems.
|
|
|
|
|
|
|
|
### Migrating a VM between clusters
|
|
|
|
|
|
|
|
The [export/import](https://docs.ganeti.org/docs/ganeti/3.0/html/admin.html#export-import) mechanism can also be used to export and import
|
|
|
|
VMs one at a time, if only a subset of the cluster needs to be
|
|
|
|
evacuated.
|
|
|
|
|
|
|
|
1. find nodes to host the exported VM on the source cluster and the
|
|
|
|
target cluster; it needs enough disk space in `/srv/ganeti` to
|
|
|
|
keep a copy of a snapshot of the VM:
|
|
|
|
|
|
|
|
df -h /srv/ganeti
|
|
|
|
|
|
|
|
2. have the right kernel modules loaded, which might require a
|
|
|
|
reboot of the source node:
|
|
|
|
|
|
|
|
modprobe dm_snapshot
|
|
|
|
|
|
|
|
3. on the master of the source Ganeti cluster, export the VM to the
|
|
|
|
source node, also use `--noshutdown` if you cannot afford to have
|
|
|
|
downtime on the VM *and* you are ready to lose data accumulated
|
|
|
|
after the snapshot:
|
|
|
|
|
|
|
|
gnt-backup export -n fsn-node-01.torproject.org test-01.torproject.org
|
|
|
|
|
|
|
|
4. copy the VM snapshot from the source node to node in the target
|
|
|
|
cluster:
|
|
|
|
|
|
|
|
scp /srv/ganeti/... root@chi-node-02.torproject.org:/srv/ganeti
|
|
|
|
|
|
|
|
5. on the master of the target Ganeti cluster, import the VM:
|
|
|
|
|
|
|
|
gnt-backup import -n chi-node-08.torproject.org --src-node=chi-node-02.torproject.org --src-dir=/srv/ganeti/??? test-01.torproject.org
|
|
|
|
|
|
|
|
### Mass migrating instances to a new cluster
|
|
|
|
|
|
|
|
The [move-instance](https://docs.ganeti.org/docs/ganeti/3.0/html/move-instance.html) command can do this.
|
|
|
|
|
|
|
|
TODO: document mass cluster migrations.
|
|
|
|
|
|
## Disaster recovery
|
|
## Disaster recovery
|
|
|
|
|
|
If things get completely out of hand and the cluster becomes too
|
|
If things get completely out of hand and the cluster becomes too
|
|
unreliable for service, the only solution is to rebuild another one
|
|
unreliable for service, the only solution is to rebuild another one
|
|
elsewhere. Since Ganeti 2.2, there is a [move-instance](http://docs.ganeti.org/ganeti/2.15/html/move-instance.html) command to
|
|
elsewhere. Since Ganeti 2.2, there is a [move-instance](https://docs.ganeti.org/docs/ganeti/3.0/html/move-instance.html) command to
|
|
move instances between cluster that can be used for that purpose.
|
|
move instances between cluster that can be used for that purpose. See
|
|
|
|
the [mass migration procedure](#mass-migrating-instances-to-a-new-cluster) above.
|
|
|
|
|
|
|
|
The [export/import](https://docs.ganeti.org/docs/ganeti/3.0/html/admin.html#export-import) mechanism can also be used to export and import
|
|
|
|
VMs one at a time, if only a subset of the cluster needs to be
|
|
|
|
evacuated. See the [migrating a VM between clusters](#migrating-a-vm-between-clusters) procedure above.
|
|
|
|
|
|
If Ganeti is completely destroyed and its APIs don't work anymore, the
|
|
If Ganeti is completely destroyed and its APIs don't work anymore, the
|
|
last resort is to restore all virtual machines from
|
|
last resort is to restore all virtual machines from
|
... | | ... | |