fix the reboot script to skip over plain hosts
when we reboot ganeti nodes, we carefully migrate all instances off the node before rebooting. this fails if there are plain instances.
make it so those instances are properly shutdown instead, and keep carrying on normally with the rest of the reboot procedure instead of crashing.
Activity
marked this issue as related to #41176 (closed)
assigned to @anarcat
i worked on a spike for this. the code is committed but untested, insofar as the current reboots required are global, so i'm doing the
--skip-ganeti-emptyroute, which doesn't test the main code path that is affected here.it tests the other code path that might have been broken by the change though, so it's still a worthwhile test...
added Needs Review label and removed Doing label
-
@lavamind is rebooting all machines today as well, but is also doing the
skip-ganeti-emptydance so we're still not testing the right codepath. surely, one day soon...alternatively, we could just test a single node reboot to confirm this works...
So there were some more upgrades this week so I ended up testing this:
Click to expand
$ ./reboot -H dal-node-01 --ganeti-migrate-back --kind=cancel --reason 'qemu flagged in needrestart' checking if host dal-node-01 needs a reboot NEEDRESTART-VER: 3.5 NEEDRESTART-KCUR: 5.10.0-23-amd64 NEEDRESTART-KEXP: 5.10.0-23-amd64 NEEDRESTART-KSTA: 1 NEEDRESTART-UCSTA: 1 NEEDRESTART-UCCUR: 0x0a0011d1 NEEDRESTART-UCEXP: 0x0a0011d1 NEEDRESTART-SVC: ganeti.service current kernel: 5.10.0-23-amd64, expected: 5.10.0-23-amd64 current microcode: 0x0a0011d1, expected: 0x0a0011d1 reboot required: ['ganeti.service'] rebooting host dal-node-01 checking for ganeti master on host dal-node-01.torproject.org ganeti node detected with master dal-node-01.torproject.org ganeti node detected, migrating 7 instances from dal-node-01.torproject.org: dangerzone-01.torproject.org donate-review-01.torproject.org forum-01.torproject.org minio-01.torproject.org static-gitlab-shim.torproje ct.org telegram-bot-01.torproject.org web-dal-07.torproject.org sending command gnt-node migrate -f dal-node-01.torproject.org to node dal-node-01.torproject.org Submitted jobs 85887, 85888, 85889, 85890, 85891, 85892, 85893 Waiting for job 85887 ... Tue Aug 1 20:51:45 2023 Migrating instance dangerzone-01.torproject.org Tue Aug 1 20:51:45 2023 * checking disk consistency between source and target Tue Aug 1 20:51:46 2023 * closing instance disks on node dal-node-03.torproject.org Tue Aug 1 20:51:47 2023 * changing into standalone mode Tue Aug 1 20:51:47 2023 * changing disks into dual-master mode Tue Aug 1 20:51:49 2023 * wait until resync is done Tue Aug 1 20:51:50 2023 * opening instance disks on node dal-node-01.torproject.org in shared mode Tue Aug 1 20:51:50 2023 * opening instance disks on node dal-node-03.torproject.org in shared mode Tue Aug 1 20:51:51 2023 * preparing dal-node-03.torproject.org to accept the instance Tue Aug 1 20:51:51 2023 * migrating instance to dal-node-03.torproject.org Tue Aug 1 20:51:51 2023 * starting memory transfer Tue Aug 1 20:52:02 2023 * memory transfer progress: 14.04 % Tue Aug 1 20:52:06 2023 * memory transfer has switched to postcopy Tue Aug 1 20:52:08 2023 * memory transfer complete Tue Aug 1 20:52:08 2023 * closing instance disks on node dal-node-01.torproject.org Tue Aug 1 20:52:09 2023 * wait until resync is done Tue Aug 1 20:52:10 2023 * changing into standalone mode Tue Aug 1 20:52:10 2023 * changing disks into single-master mode Tue Aug 1 20:52:12 2023 * wait until resync is done Tue Aug 1 20:52:12 2023 * done Waiting for job 85888 ... Tue Aug 1 20:52:13 2023 Migrating instance static-gitlab-shim.torproject.org Tue Aug 1 20:52:13 2023 * checking disk consistency between source and target Tue Aug 1 20:52:15 2023 * closing instance disks on node dal-node-03.torproject.org Tue Aug 1 20:52:16 2023 * changing into standalone mode Tue Aug 1 20:52:17 2023 * changing disks into dual-master mode Tue Aug 1 20:52:19 2023 * wait until resync is done Tue Aug 1 20:52:20 2023 * opening instance disks on node dal-node-01.torproject.org in shared mode Tue Aug 1 20:52:20 2023 * opening instance disks on node dal-node-03.torproject.org in shared mode Tue Aug 1 20:52:21 2023 * preparing dal-node-03.torproject.org to accept the instance Tue Aug 1 20:52:21 2023 * migrating instance to dal-node-03.torproject.org Tue Aug 1 20:52:21 2023 * starting memory transfer Tue Aug 1 20:52:32 2023 * memory transfer progress: 13.90 % Tue Aug 1 20:52:43 2023 * memory transfer progress: 28.19 % Tue Aug 1 20:52:53 2023 * memory transfer has switched to postcopy Tue Aug 1 20:52:53 2023 * memory transfer progress: 42.54 % Tue Aug 1 20:52:54 2023 * memory transfer complete Tue Aug 1 20:52:55 2023 * closing instance disks on node dal-node-01.torproject.org Tue Aug 1 20:52:56 2023 * wait until resync is done Tue Aug 1 20:52:56 2023 * changing into standalone mode Tue Aug 1 20:52:57 2023 * changing disks into single-master mode Tue Aug 1 20:53:00 2023 * wait until resync is done Tue Aug 1 20:53:00 2023 * done Waiting for job 85889 ... Tue Aug 1 20:53:01 2023 Migrating instance forum-01.torproject.org Tue Aug 1 20:53:01 2023 * checking disk consistency between source and target Tue Aug 1 20:53:03 2023 * closing instance disks on node dal-node-03.torproject.org Tue Aug 1 20:53:04 2023 * changing into standalone mode Tue Aug 1 20:53:05 2023 * changing disks into dual-master mode Tue Aug 1 20:53:07 2023 * wait until resync is done Tue Aug 1 20:53:08 2023 * opening instance disks on node dal-node-01.torproject.org in shared mode Tue Aug 1 20:53:09 2023 * opening instance disks on node dal-node-03.torproject.org in shared mode Tue Aug 1 20:53:09 2023 * preparing dal-node-03.torproject.org to accept the instance Tue Aug 1 20:53:10 2023 * migrating instance to dal-node-03.torproject.org Tue Aug 1 20:53:10 2023 * starting memory transfer Tue Aug 1 20:53:21 2023 * memory transfer progress: 14.17 % Tue Aug 1 20:53:31 2023 * memory transfer progress: 28.44 % Tue Aug 1 20:53:42 2023 * memory transfer progress: 42.66 % Tue Aug 1 20:53:52 2023 * memory transfer progress: 56.95 % Tue Aug 1 20:54:03 2023 * memory transfer progress: 71.16 % Tue Aug 1 20:54:14 2023 * memory transfer progress: 85.41 % Tue Aug 1 20:54:17 2023 * memory transfer has switched to postcopy Tue Aug 1 20:54:21 2023 * memory transfer complete Tue Aug 1 20:54:21 2023 * closing instance disks on node dal-node-01.torproject.org Tue Aug 1 20:54:22 2023 * wait until resync is done Tue Aug 1 20:54:23 2023 * changing into standalone mode Tue Aug 1 20:54:24 2023 * changing disks into single-master mode Tue Aug 1 20:54:26 2023 * wait until resync is done Tue Aug 1 20:54:27 2023 * done Waiting for job 85890 ... Job 85890 has failed: Failure: prerequisites not met for this operation: error type: wrong_state, error details: Instance's disk layout 'plain' does not allow migrations Waiting for job 85891 ... Tue Aug 1 20:54:27 2023 Migrating instance donate-review-01.torproject.org Tue Aug 1 20:54:27 2023 * checking disk consistency between source and target Tue Aug 1 20:54:29 2023 * closing instance disks on node dal-node-03.torproject.org Tue Aug 1 20:54:29 2023 * changing into standalone mode Tue Aug 1 20:54:30 2023 * changing disks into dual-master mode Tue Aug 1 20:54:32 2023 * wait until resync is done Tue Aug 1 20:54:32 2023 * opening instance disks on node dal-node-01.torproject.org in shared mode Tue Aug 1 20:54:33 2023 * opening instance disks on node dal-node-03.torproject.org in shared mode Tue Aug 1 20:54:33 2023 * preparing dal-node-03.torproject.org to accept the instance Tue Aug 1 20:54:34 2023 * migrating instance to dal-node-03.torproject.org Tue Aug 1 20:54:34 2023 * starting memory transfer Tue Aug 1 20:54:44 2023 * memory transfer progress: 28.32 % Tue Aug 1 20:54:49 2023 * memory transfer complete Tue Aug 1 20:54:49 2023 * closing instance disks on node dal-node-01.torproject.org Tue Aug 1 20:54:49 2023 * wait until resync is done Tue Aug 1 20:54:50 2023 * changing into standalone mode Tue Aug 1 20:54:51 2023 * changing disks into single-master mode Tue Aug 1 20:54:52 2023 * wait until resync is done Tue Aug 1 20:54:53 2023 * done Waiting for job 85892 ... Tue Aug 1 20:54:53 2023 Migrating instance web-dal-07.torproject.org Tue Aug 1 20:54:53 2023 * checking disk consistency between source and target Tue Aug 1 20:54:55 2023 * closing instance disks on node dal-node-03.torproject.org Tue Aug 1 20:54:56 2023 * changing into standalone mode Tue Aug 1 20:54:57 2023 * changing disks into dual-master mode Tue Aug 1 20:54:59 2023 * wait until resync is done Tue Aug 1 20:55:00 2023 * opening instance disks on node dal-node-01.torproject.org in shared mode Tue Aug 1 20:55:01 2023 * opening instance disks on node dal-node-03.torproject.org in shared mode Tue Aug 1 20:55:02 2023 * preparing dal-node-03.torproject.org to accept the instance Tue Aug 1 20:55:02 2023 * migrating instance to dal-node-03.torproject.org Tue Aug 1 20:55:02 2023 * starting memory transfer Tue Aug 1 20:55:13 2023 * memory transfer progress: 14.28 % Tue Aug 1 20:55:23 2023 * memory transfer progress: 28.31 % Tue Aug 1 20:55:34 2023 * memory transfer progress: 42.58 % Tue Aug 1 20:55:45 2023 * memory transfer progress: 56.85 % Tue Aug 1 20:55:55 2023 * memory transfer progress: 71.08 % Tue Aug 1 20:56:06 2023 * memory transfer progress: 85.40 % Tue Aug 1 20:56:15 2023 * memory transfer has switched to postcopy Tue Aug 1 20:56:16 2023 * memory transfer progress: 99.48 % Tue Aug 1 20:56:18 2023 * memory transfer complete Tue Aug 1 20:56:19 2023 * closing instance disks on node dal-node-01.torproject.org Tue Aug 1 20:56:20 2023 * wait until resync is done Tue Aug 1 20:56:20 2023 * changing into standalone mode Tue Aug 1 20:56:21 2023 * changing disks into single-master mode Tue Aug 1 20:56:23 2023 * wait until resync is done Tue Aug 1 20:56:24 2023 * done Waiting for job 85893 ... Tue Aug 1 20:56:24 2023 Migrating instance telegram-bot-01.torproject.org Tue Aug 1 20:56:25 2023 * checking disk consistency between source and target Tue Aug 1 20:56:25 2023 * closing instance disks on node dal-node-03.torproject.org Tue Aug 1 20:56:26 2023 * changing into standalone mode Tue Aug 1 20:56:26 2023 * changing disks into dual-master mode Tue Aug 1 20:56:28 2023 * wait until resync is done Tue Aug 1 20:56:28 2023 * opening instance disks on node dal-node-01.torproject.org in shared mode Tue Aug 1 20:56:28 2023 * opening instance disks on node dal-node-03.torproject.org in shared mode Tue Aug 1 20:56:29 2023 * preparing dal-node-03.torproject.org to accept the instance Tue Aug 1 20:56:29 2023 * migrating instance to dal-node-03.torproject.org Tue Aug 1 20:56:29 2023 * starting memory transfer Tue Aug 1 20:56:40 2023 * memory transfer progress: 14.04 % Tue Aug 1 20:56:42 2023 * memory transfer has switched to postcopy Tue Aug 1 20:56:43 2023 * memory transfer complete Tue Aug 1 20:56:43 2023 * closing instance disks on node dal-node-01.torproject.org Tue Aug 1 20:56:44 2023 * wait until resync is done Tue Aug 1 20:56:44 2023 * changing into standalone mode Tue Aug 1 20:56:44 2023 * changing disks into single-master mode Tue Aug 1 20:56:46 2023 * wait until resync is done Tue Aug 1 20:56:46 2023 * done There were 1 errors during the node migration. failed to empty node dal-node-01.torproject.org trying to find plain instances... failed to empty node dal-node-01.torproject.org, trying to shutdown plain instances: ['minio-01.torproject.org'] running shutdown -h +0 "rebooting parent ganeti node" on minio-01.torproject.org Exception (client): key cannot be used for signing Traceback (most recent call last): File "/usr/lib/python3/dist-packages/paramiko/transport.py", line 2164, in run handler(self.auth_handler, m) File "/usr/lib/python3/dist-packages/paramiko/auth_handler.py", line 395, in _parse_service_accept sig = self.private_key.sign_ssh_data(blob, algorithm) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/paramiko/agent.py", line 436, in sign_ssh_data raise SSHException("key cannot be used for signing") paramiko.ssh_exception.SSHException: key cannot be used for signingfailed to connect to minio-01.torproject.org, assuming down: No existing session scheduled reboots with 0 minute delay on minio-01.torproject.org waiting for minio-01.torproject.org to shutdown host minio-01.torproject.org was still up after 30 seconds, ignoring forcibly stopping all instances (1) from master dal-node-01.torproject.org Waiting for job 85901 for minio-01.torproject.org ... looking for pending upgrades...
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Listing... no pending package upgrades running shutdown -c +0 "qemu flagged in needrestart" on dal-node-01.torproject.org I: executing shutdown "-c" "+0" "qemu flagged in needrestart" regardless of check results. shutdown was just a cancel, not waiting for shutdown and proceeding immediately starting 1 instances Waiting for job 85902 for minio-01.torproject.org ... done with host dal-node-01
I probably missed the security key touch.
Edited by anarcat-
There were 1 errors during the node migration. failed to empty node dal-node-01.torproject.org trying to find plain instances... failed to empty node dal-node-01.torproject.org, trying to shutdown plain instances: ['minio-01.torproject.org'] running shutdown -h +0 "rebooting parent ganeti node" on minio-01.torproject.org Exception (client): key cannot be used for signing [...] failed to connect to minio-01.torproject.org, assuming down: No existing session scheduled reboots with 0 minute delay on minio-01.torproject.org waiting for minio-01.torproject.org to shutdown host minio-01.torproject.org was still up after 30 seconds, ignoring forcibly stopping all instances (1) from master dal-node-01.torproject.org Waiting for job 85901 for minio-01.torproject.org ... looking for pending upgrades... [...] starting 1 instances Waiting for job 85902 for minio-01.torproject.org ... done with host dal-node-01Okay, this is really valuable, thanks! It does seem like the orderly shutdown failed on minio-01, but it degenerated into a
gnt-instance stopwhich is, i guess, an acceptable compromise. And it did succeed in completing the work on the node, did it not?Or maybe it forgot to migrate the instances back here?
I probably missed the security key touch.
So that's a problem I see a lot now that i have touch enabled. We have some code to handle that failure, but only in one place... it seems we'd need to move that up the stack so we have a more generic retry method when SSH connects fail... I wonder if we could send that upstream to fabric/paramiko as well?
-
There's also an error when no
plainmachines need to migrate:Click to expand
./reboot -H dal-node-03 --ganeti-migrate-back --kind=cancel --reason 'qemu flagged in needrestart' [131/1693] checking if host dal-node-03 needs a reboot NEEDRESTART-VER: 3.5 NEEDRESTART-KCUR: 5.10.0-23-amd64 NEEDRESTART-KEXP: 5.10.0-23-amd64 NEEDRESTART-KSTA: 1 NEEDRESTART-UCSTA: 1 NEEDRESTART-UCCUR: 0x0a0011d1 NEEDRESTART-UCEXP: 0x0a0011d1 NEEDRESTART-SVC: ganeti.service current kernel: 5.10.0-23-amd64, expected: 5.10.0-23-amd64 current microcode: 0x0a0011d1, expected: 0x0a0011d1 reboot required: ['ganeti.service'] rebooting host dal-node-03 checking for ganeti master on host dal-node-03.torproject.org ganeti node detected with master dal-node-01.torproject.org ganeti node detected, migrating 7 instances from dal-node-03.torproject.org: ci-runner-x86-01.torproject.org crm-ext-01.torproject.org crm-int-01.torproject.org ns3.torproject.org probetelemetry-01.torproject.org rdsys-frontend-01.torproject.org tb-pkgstage-01.torproject.org sending command gnt-node migrate -f dal-node-03.torproject.org to node dal-node-01.torproject.org Submitted jobs 85967, 85968, 85969, 85970, 85971, 85972, 85973 Waiting for job 85967 ... Job 85967 has failed: Failure: prerequisites not met for this operation: error type: wrong_state, error details: Can't migrate, please use failover: Instance ci-runner-x86-01.torproject.org is not running Waiting for job 85968 ... Tue Aug 1 21:50:43 2023 Migrating instance rdsys-frontend-01.torproject.org Tue Aug 1 21:50:43 2023 * checking disk consistency between source and target Tue Aug 1 21:50:45 2023 * closing instance disks on node dal-node-02.torproject.org Tue Aug 1 21:50:46 2023 * changing into standalone mode Tue Aug 1 21:50:47 2023 * changing disks into dual-master mode Tue Aug 1 21:50:49 2023 * wait until resync is done Tue Aug 1 21:50:50 2023 * opening instance disks on node dal-node-03.torproject.org in shared mode Tue Aug 1 21:50:51 2023 * opening instance disks on node dal-node-02.torproject.org in shared mode Tue Aug 1 21:50:52 2023 * preparing dal-node-02.torproject.org to accept the instance Tue Aug 1 21:50:52 2023 * migrating instance to dal-node-02.torproject.org Tue Aug 1 21:50:52 2023 * starting memory transfer Tue Aug 1 21:51:03 2023 * memory transfer progress: 14.02 % Tue Aug 1 21:51:06 2023 * memory transfer has switched to postcopy Tue Aug 1 21:51:07 2023 * memory transfer complete Tue Aug 1 21:51:07 2023 * closing instance disks on node dal-node-03.torproject.org Tue Aug 1 21:51:08 2023 * wait until resync is done Tue Aug 1 21:51:09 2023 * changing into standalone mode Tue Aug 1 21:51:10 2023 * changing disks into single-master mode Tue Aug 1 21:51:12 2023 * wait until resync is done Tue Aug 1 21:51:13 2023 * done Waiting for job 85969 ... Tue Aug 1 21:51:13 2023 Migrating instance ns3.torproject.org Tue Aug 1 21:51:13 2023 * checking disk consistency between source and target Tue Aug 1 21:51:15 2023 * closing instance disks on node dal-node-02.torproject.org Tue Aug 1 21:51:15 2023 * changing into standalone mode Tue Aug 1 21:51:16 2023 * changing disks into dual-master mode Tue Aug 1 21:51:18 2023 * wait until resync is done Tue Aug 1 21:51:19 2023 * opening instance disks on node dal-node-03.torproject.org in shared mode Tue Aug 1 21:51:19 2023 * opening instance disks on node dal-node-02.torproject.org in shared mode Tue Aug 1 21:51:20 2023 * preparing dal-node-02.torproject.org to accept the instance Tue Aug 1 21:51:20 2023 * migrating instance to dal-node-02.torproject.org Tue Aug 1 21:51:20 2023 * starting memory transfer Tue Aug 1 21:51:31 2023 * memory transfer progress: 56.83 % Tue Aug 1 21:51:37 2023 * memory transfer has switched to postcopy Tue Aug 1 21:51:38 2023 * memory transfer complete Tue Aug 1 21:51:38 2023 * closing instance disks on node dal-node-03.torproject.org [71/1693] Tue Aug 1 21:51:39 2023 * wait until resync is done Tue Aug 1 21:51:39 2023 * changing into standalone mode Tue Aug 1 21:51:40 2023 * changing disks into single-master mode Tue Aug 1 21:51:42 2023 * wait until resync is done Tue Aug 1 21:51:42 2023 * done Waiting for job 85970 ... Tue Aug 1 21:51:43 2023 Migrating instance probetelemetry-01.torproject.org Tue Aug 1 21:51:43 2023 * checking disk consistency between source and target Tue Aug 1 21:51:45 2023 * closing instance disks on node dal-node-02.torproject.org Tue Aug 1 21:51:46 2023 * changing into standalone mode Tue Aug 1 21:51:47 2023 * changing disks into dual-master mode Tue Aug 1 21:51:49 2023 * wait until resync is done Tue Aug 1 21:51:50 2023 * opening instance disks on node dal-node-03.torproject.org in shared mode Tue Aug 1 21:51:51 2023 * opening instance disks on node dal-node-02.torproject.org in shared mode Tue Aug 1 21:51:51 2023 * preparing dal-node-02.torproject.org to accept the instance Tue Aug 1 21:51:52 2023 * migrating instance to dal-node-02.torproject.org Tue Aug 1 21:51:52 2023 * starting memory transfer Tue Aug 1 21:52:03 2023 * memory transfer progress: 28.65 % Tue Aug 1 21:52:13 2023 * memory transfer progress: 57.13 % Tue Aug 1 21:52:24 2023 * memory transfer progress: 85.42 % Tue Aug 1 21:52:26 2023 * memory transfer complete Tue Aug 1 21:52:26 2023 * closing instance disks on node dal-node-03.torproject.org Tue Aug 1 21:52:27 2023 * wait until resync is done Tue Aug 1 21:52:28 2023 * changing into standalone mode Tue Aug 1 21:52:29 2023 * changing disks into single-master mode Tue Aug 1 21:52:30 2023 * wait until resync is done Tue Aug 1 21:52:31 2023 * done Waiting for job 85971 ... Tue Aug 1 21:52:32 2023 Migrating instance crm-int-01.torproject.org Tue Aug 1 21:52:32 2023 * checking disk consistency between source and target Tue Aug 1 21:52:34 2023 * closing instance disks on node dal-node-02.torproject.org Tue Aug 1 21:52:34 2023 * changing into standalone mode Tue Aug 1 21:52:36 2023 * changing disks into dual-master mode Tue Aug 1 21:52:38 2023 * wait until resync is done Tue Aug 1 21:52:39 2023 * opening instance disks on node dal-node-03.torproject.org in shared mode Tue Aug 1 21:52:39 2023 * opening instance disks on node dal-node-02.torproject.org in shared mode Tue Aug 1 21:52:40 2023 * preparing dal-node-02.torproject.org to accept the instance Tue Aug 1 21:52:40 2023 * migrating instance to dal-node-02.torproject.org Tue Aug 1 21:52:41 2023 * starting memory transfer Tue Aug 1 21:52:51 2023 * memory transfer progress: 14.25 % Tue Aug 1 21:53:02 2023 * memory transfer progress: 28.56 % Tue Aug 1 21:53:12 2023 * memory transfer progress: 42.86 % Tue Aug 1 21:53:23 2023 * memory transfer progress: 57.18 % Tue Aug 1 21:53:34 2023 * memory transfer progress: 71.48 % Tue Aug 1 21:53:44 2023 * memory transfer progress: 85.76 % Tue Aug 1 21:53:51 2023 * memory transfer has switched to postcopy Tue Aug 1 21:53:52 2023 * memory transfer complete Tue Aug 1 21:53:52 2023 * closing instance disks on node dal-node-03.torproject.org Tue Aug 1 21:53:53 2023 * wait until resync is done Tue Aug 1 21:53:54 2023 * changing into standalone mode Tue Aug 1 21:53:55 2023 * changing disks into single-master mode Tue Aug 1 21:53:57 2023 * wait until resync is done Tue Aug 1 21:53:58 2023 * done Waiting for job 85972 ... Tue Aug 1 21:53:58 2023 Migrating instance crm-ext-01.torproject.org Tue Aug 1 21:53:58 2023 * checking disk consistency between source and target Tue Aug 1 21:54:00 2023 * closing instance disks on node dal-node-02.torproject.org Tue Aug 1 21:54:00 2023 * changing into standalone mode Tue Aug 1 21:54:01 2023 * changing disks into dual-master mode Tue Aug 1 21:54:03 2023 * wait until resync is done Tue Aug 1 21:54:04 2023 * opening instance disks on node dal-node-03.torproject.org in shared mode Tue Aug 1 21:54:04 2023 * opening instance disks on node dal-node-02.torproject.org in shared mode Tue Aug 1 21:54:05 2023 * preparing dal-node-02.torproject.org to accept the instance Tue Aug 1 21:54:05 2023 * migrating instance to dal-node-02.torproject.org Tue Aug 1 21:54:05 2023 * starting memory transfer Tue Aug 1 21:54:16 2023 * memory transfer progress: 56.99 % Tue Aug 1 21:54:22 2023 * memory transfer has switched to postcopy Tue Aug 1 21:54:23 2023 * memory transfer complete Tue Aug 1 21:54:23 2023 * closing instance disks on node dal-node-03.torproject.org Tue Aug 1 21:54:24 2023 * wait until resync is done Tue Aug 1 21:54:25 2023 * changing into standalone mode Tue Aug 1 21:54:25 2023 * changing disks into single-master mode Tue Aug 1 21:54:27 2023 * wait until resync is done Tue Aug 1 21:54:27 2023 * done Waiting for job 85973 ... Tue Aug 1 21:54:28 2023 Migrating instance tb-pkgstage-01.torproject.org Tue Aug 1 21:54:28 2023 * checking disk consistency between source and target Tue Aug 1 21:54:30 2023 * closing instance disks on node dal-node-02.torproject.org Tue Aug 1 21:54:31 2023 * changing into standalone mode Tue Aug 1 21:54:32 2023 * changing disks into dual-master mode Tue Aug 1 21:54:34 2023 * wait until resync is done Tue Aug 1 21:54:35 2023 * opening instance disks on node dal-node-03.torproject.org in shared mode Tue Aug 1 21:54:36 2023 * opening instance disks on node dal-node-02.torproject.org in shared mode Tue Aug 1 21:54:37 2023 * preparing dal-node-02.torproject.org to accept the instance Tue Aug 1 21:54:37 2023 * migrating instance to dal-node-02.torproject.org Tue Aug 1 21:54:37 2023 * starting memory transfer Tue Aug 1 21:54:48 2023 * memory transfer progress: 13.89 % Tue Aug 1 21:54:51 2023 * memory transfer complete Tue Aug 1 21:54:51 2023 * closing instance disks on node dal-node-03.torproject.org Tue Aug 1 21:54:52 2023 * wait until resync is done Tue Aug 1 21:54:53 2023 * changing into standalone mode Tue Aug 1 21:54:54 2023 * changing disks into single-master mode Tue Aug 1 21:54:56 2023 * wait until resync is done Tue Aug 1 21:54:56 2023 * done There were 1 errors during the node migration. failed to empty node dal-node-03.torproject.org trying to find plain instances... unexpected exception during reboot: [Exit('no plain instance found, failed to empty node dal-node-03.torproject.org, aborting')] no plain instance found, failed to empty node dal-node-03.torproject.org, aborting -
Can't migrate, please use failover: Instance ci-runner-x86-01.torproject.org is not runningokay, so that's one corner case i'm not handling properly... i guess that's a TODO then, but it is not related to plain hosts per se...
all in all, we seem to be doing the right thing here, don't you think?
This issue has been waiting for information two weeks or more. It needs attention. Please take care of this before the end of 2023-09-06. ~"Needs Information" tickets will be moved to the Icebox after that point.
(Any ticket left in Needs Review, Needs Information, Next, or Doing without activity for 14 days gets such notifications. Make a comment describing the current state of this ticket and remove the Stale label to fix this.)
added Stale label
removed Stale label
mentioned in commit fabric-tasks@32fe2858
Please register or sign in to reply