... | @@ -350,7 +350,7 @@ to the `/etc/network/interfaces*` files: |
... | @@ -350,7 +350,7 @@ to the `/etc/network/interfaces*` files: |
|
|
|
|
|
The renumbering can be done with Fabric, with:
|
|
The renumbering can be done with Fabric, with:
|
|
|
|
|
|
./ganeti -v -H test1.torproject.org renumber-instance --ganeti-node $PRIMARY_NODE
|
|
./ganeti -H test1.torproject.org renumber-instance --ganeti-node $PRIMARY_NODE
|
|
|
|
|
|
Note that the `$PRIMARY_NODE` must be passed here, not the "master"!
|
|
Note that the `$PRIMARY_NODE` must be passed here, not the "master"!
|
|
|
|
|
... | @@ -907,15 +907,15 @@ Import procedure: |
... | @@ -907,15 +907,15 @@ Import procedure: |
|
|
|
|
|
2. copy the disks, without downtime:
|
|
2. copy the disks, without downtime:
|
|
|
|
|
|
./ganeti -v -H $INSTANCE libvirt-import --ganeti-node $SPARE_NODE --libvirt-host $KVM_HOST
|
|
./ganeti -H $INSTANCE libvirt-import --ganeti-node $SPARE_NODE --libvirt-host $KVM_HOST
|
|
|
|
|
|
3. copy the disks again, this time suspending the machine:
|
|
3. copy the disks again, this time suspending the machine:
|
|
|
|
|
|
./ganeti -v -H $INSTANCE libvirt-import --ganeti-node $SPARE_NODE --libvirt-host $KVM_HOST --suspend --adopt
|
|
./ganeti -H $INSTANCE libvirt-import --ganeti-node $SPARE_NODE --libvirt-host $KVM_HOST --suspend --adopt
|
|
|
|
|
|
4. renumber the host:
|
|
4. renumber the host:
|
|
|
|
|
|
./ganeti -v -H $INSTANCE renumber-instance --ganeti-node $SPARE_NODE
|
|
./ganeti -H $INSTANCE renumber-instance --ganeti-node $SPARE_NODE
|
|
|
|
|
|
5. test services by changing your `/etc/hosts`, possibly warning
|
|
5. test services by changing your `/etc/hosts`, possibly warning
|
|
service admins:
|
|
service admins:
|
... | @@ -963,8 +963,8 @@ Import procedure: |
... | @@ -963,8 +963,8 @@ Import procedure: |
|
8. shutdown original instance and redo migration as in step 3 and 4:
|
|
8. shutdown original instance and redo migration as in step 3 and 4:
|
|
|
|
|
|
fab -H $INSTANCE reboot.halt-and-wait --delay-shutdown 60 --reason='migrating to new server' &&
|
|
fab -H $INSTANCE reboot.halt-and-wait --delay-shutdown 60 --reason='migrating to new server' &&
|
|
./ganeti -v -H $INSTANCE libvirt-import --ganeti-node $SPARE_NODE --libvirt-host $KVM_HOST --adopt &&
|
|
./ganeti -H $INSTANCE libvirt-import --ganeti-node $SPARE_NODE --libvirt-host $KVM_HOST --adopt &&
|
|
./ganeti -v -H $INSTANCE renumber-instance --ganeti-node $SPARE_NODE
|
|
./ganeti -H $INSTANCE renumber-instance --ganeti-node $SPARE_NODE
|
|
|
|
|
|
9. final test procedure
|
|
9. final test procedure
|
|
|
|
|
... | @@ -1289,7 +1289,7 @@ This command will reboot the entire Ganeti fleets, including the |
... | @@ -1289,7 +1289,7 @@ This command will reboot the entire Ganeti fleets, including the |
|
hosted VMs, use this when (for example) you have kernel upgrades to
|
|
hosted VMs, use this when (for example) you have kernel upgrades to
|
|
deploy everywhere:
|
|
deploy everywhere:
|
|
|
|
|
|
./reboot --skip-ganeti-empty -v --reason 'qemu flagged in needrestart' \
|
|
./reboot --skip-ganeti-empty --reason 'qemu flagged in needrestart' \
|
|
-H chi-node-0{1,2,3,4,5,6,7,8,9}.torproject.org \
|
|
-H chi-node-0{1,2,3,4,5,6,7,8,9}.torproject.org \
|
|
chi-node-1{0,1}.torproject.org \
|
|
chi-node-1{0,1}.torproject.org \
|
|
fsn-node-0{1,2,3,4,5,6,7,8}.torproject.org
|
|
fsn-node-0{1,2,3,4,5,6,7,8}.torproject.org
|
... | @@ -1306,7 +1306,7 @@ need a reboot, and not the instances. In that case, you want to reboot |
... | @@ -1306,7 +1306,7 @@ need a reboot, and not the instances. In that case, you want to reboot |
|
the nodes but before that, migrate the instances off the node and then
|
|
the nodes but before that, migrate the instances off the node and then
|
|
migrate it back when done. This incantation should do so:
|
|
migrate it back when done. This incantation should do so:
|
|
|
|
|
|
./reboot --ganeti-migrate-back -v --reason 'Open vSwitch upgrade' \
|
|
./reboot --ganeti-migrate-back --reason 'Open vSwitch upgrade' \
|
|
-H fsn-node-0{1,2,3,4,5,6,7,8}.torproject.org
|
|
-H fsn-node-0{1,2,3,4,5,6,7,8}.torproject.org
|
|
|
|
|
|
This should cause no user-visible disruption.
|
|
This should cause no user-visible disruption.
|
... | @@ -1319,7 +1319,7 @@ upgraded, for example `libxml` or OpenSSL. |
... | @@ -1319,7 +1319,7 @@ upgraded, for example `libxml` or OpenSSL. |
|
|
|
|
|
This will only migrate the VMs without rebooting the hosts:
|
|
This will only migrate the VMs without rebooting the hosts:
|
|
|
|
|
|
./reboot --ganeti-migrate-back --kind=cancel -v --reason 'qemu flagged in needrestart' \
|
|
./reboot --ganeti-migrate-back --kind=cancel --reason 'qemu flagged in needrestart' \
|
|
-H fsn-node-0{1,2,3,4,5,6,7,8}.torproject.org
|
|
-H fsn-node-0{1,2,3,4,5,6,7,8}.torproject.org
|
|
|
|
|
|
This should cause no user-visible disruption, as it migrates all the
|
|
This should cause no user-visible disruption, as it migrates all the
|
... | @@ -1331,7 +1331,7 @@ the libraries Qemu depends on. |
... | @@ -1331,7 +1331,7 @@ the libraries Qemu depends on. |
|
If you actually need to restart the instances in place (and not
|
|
If you actually need to restart the instances in place (and not
|
|
migrate them), you need to use the `--skip-ganeti-empty` flag instead:
|
|
migrate them), you need to use the `--skip-ganeti-empty` flag instead:
|
|
|
|
|
|
./reboot --skip-ganeti-empty --kind=cancel -v --reason 'qemu flagged in needrestart' \
|
|
./reboot --skip-ganeti-empty --kind=cancel --reason 'qemu flagged in needrestart' \
|
|
-H dal-node-0{1,2,3}.torproject.org
|
|
-H dal-node-0{1,2,3}.torproject.org
|
|
|
|
|
|
## Rebalancing a cluster
|
|
## Rebalancing a cluster
|
... | @@ -1789,7 +1789,7 @@ will move *one* VM, in this example the `test-01` VM from the |
... | @@ -1789,7 +1789,7 @@ will move *one* VM, in this example the `test-01` VM from the |
|
|
|
|
|
3. change the IP address inside the VM:
|
|
3. change the IP address inside the VM:
|
|
|
|
|
|
tsa-misc$ ./ganeti -H test-01.torproject.org -v renumber-instance dal-node-02.torproject.org
|
|
tsa-misc$ ./ganeti -H test-01.torproject.org renumber-instance dal-node-02.torproject.org
|
|
|
|
|
|
Note how we use the name of the Ganeti node where the VM resides,
|
|
Note how we use the name of the Ganeti node where the VM resides,
|
|
*not* the master.
|
|
*not* the master.
|
... | @@ -1806,7 +1806,7 @@ will move *one* VM, in this example the `test-01` VM from the |
... | @@ -1806,7 +1806,7 @@ will move *one* VM, in this example the `test-01` VM from the |
|
|
|
|
|
6. schedule destruction of the old VM (7 days)
|
|
6. schedule destruction of the old VM (7 days)
|
|
|
|
|
|
tsa-misc$ ./ganeti -v -H test-01.torproject.org retire --master-host=chi-node-01.torproject.org
|
|
tsa-misc$ ./ganeti -H test-01.torproject.org retire --master-host=chi-node-01.torproject.org
|
|
|
|
|
|
### Troubleshooting
|
|
### Troubleshooting
|
|
|
|
|
... | | ... | |