document this: Ganeti (temporary) disk errors after rebooting chi-node-08

While rebooting ch-node-08 for security updates, the migration back failed:

Wed Aug 10 21:48:22 2022 Migrating instance onionbalance-02.torproject.org
Wed Aug 10 21:48:22 2022 * checking disk consistency between source and target
Wed Aug 10 21:48:23 2022  - WARNING: Can't find disk on node chi-node-08.torproject.org
Failure: command execution error:
Disk 0 is degraded or not fully synchronized on target node, aborting migration
unexpected exception during reboot: [<UnexpectedExit: cmd='gnt-instance migrate -f onionbalance-02.torproject.org' exited=1>] Encountered a bad command exit code!

Command: 'gnt-instance migrate -f onionbalance-02.torproject.org'

I ran gnt-cluster verify to see what happened and

Submitted jobs 376732, 376733
Waiting for job 376732 ...
Wed Aug 10 21:49:31 2022 * Verifying cluster config
Wed Aug 10 21:49:31 2022 * Verifying cluster certificate files
Wed Aug 10 21:49:31 2022 * Verifying hypervisor parameters
Wed Aug 10 21:49:31 2022 * Verifying all nodes belong to an existing group
Waiting for job 376733 ...
Wed Aug 10 21:49:32 2022 * Verifying group 'default'
Wed Aug 10 21:49:32 2022 * Gathering data (11 nodes)
Wed Aug 10 21:49:32 2022 * Gathering information about nodes (11 nodes)
Wed Aug 10 21:49:36 2022 * Gathering disk information (11 nodes)
Wed Aug 10 21:49:37 2022 * Verifying configuration file consistency
Wed Aug 10 21:49:37 2022 * Verifying node status
Wed Aug 10 21:49:37 2022   - ERROR: node chi-node-08.torproject.org: drbd minor 0 of disk 1e713d4e-344c-4c39-9286-cb47bcaa8da3 (attached in instance 'probetelemetry-01.torproject.org') is not active
Wed Aug 10 21:49:37 2022   - ERROR: node chi-node-08.torproject.org: drbd minor 1 of disk 1948dcb7-b281-4ad3-a2e4-cdaf3fa159a0 (attached in instance 'probetelemetry-01.torproject.org') is not active
Wed Aug 10 21:49:37 2022   - ERROR: node chi-node-08.torproject.org: drbd minor 2 of disk 25986a9f-3c32-4f11-b546-71d432b1848f (attached in instance 'probetelemetry-01.torproject.org') is not active
Wed Aug 10 21:49:37 2022   - ERROR: node chi-node-08.torproject.org: drbd minor 3 of disk 7f3a5ef1-b522-4726-96cf-010d57436dd5 (attached in instance 'static-gitlab-shim.torproject.org') is not active
Wed Aug 10 21:49:37 2022   - ERROR: node chi-node-08.torproject.org: drbd minor 4 of disk bfd77fb0-b8ec-44dc-97ad-fd65d6c45850 (attached in instance 'static-gitlab-shim.torproject.org') is not active
Wed Aug 10 21:49:37 2022   - ERROR: node chi-node-08.torproject.org: drbd minor 5 of disk c1828d0a-87c5-49db-8abb-ee00ccabcb73 (attached in instance 'static-gitlab-shim.torproject.org') is not active
Wed Aug 10 21:49:37 2022   - ERROR: node chi-node-08.torproject.org: drbd minor 8 of disk 1f3f4f1e-0dfa-4443-aabf-0f3b4c7d2dc4 (attached in instance 'onionbalance-02.torproject.org') is not active
Wed Aug 10 21:49:37 2022   - ERROR: node chi-node-08.torproject.org: drbd minor 9 of disk bbd5b2e9-8dbb-42f4-9c10-ef0df7f59b85 (attached in instance 'onionbalance-02.torproject.org') is not active
Wed Aug 10 21:49:37 2022 * Verifying instance status
Wed Aug 10 21:49:37 2022   - WARNING: instance static-gitlab-shim.torproject.org: disk/0 on chi-node-04.torproject.org is degraded; local disk state is 'ok'
Wed Aug 10 21:49:37 2022   - WARNING: instance static-gitlab-shim.torproject.org: disk/1 on chi-node-04.torproject.org is degraded; local disk state is 'ok'
Wed Aug 10 21:49:37 2022   - WARNING: instance static-gitlab-shim.torproject.org: disk/2 on chi-node-04.torproject.org is degraded; local disk state is 'ok'
Wed Aug 10 21:49:37 2022   - ERROR: instance static-gitlab-shim.torproject.org: couldn't retrieve status for disk/0 on chi-node-08.torproject.org: Can't find device <DRBD8(hosts=0d8b8663-e2bd-42e7-9e8d-e4502fa621b8/3-3aa32c9d-c0a7-44bb-832d-851710d04765/8, port=11040, backend=<LogicalVolume(/dev/vg_ganeti/b1913b02-14f4-4c0e-9d78-970bd34f5291.disk0_data, visible as /dev/, size=10240m)>, metadev=<LogicalVolume(/dev/vg_ganeti/b1913b02-14f4-4c0e-9d78-970bd34f5291.disk0_meta, visible as /dev/, size=128m)>, visible as /dev/disk/0, size=10240m)>
Wed Aug 10 21:49:37 2022   - ERROR: instance static-gitlab-shim.torproject.org: couldn't retrieve status for disk/1 on chi-node-08.torproject.org: Can't find device <DRBD8(hosts=0d8b8663-e2bd-42e7-9e8d-e4502fa621b8/4-3aa32c9d-c0a7-44bb-832d-851710d04765/11, port=11041, backend=<LogicalVolume(/dev/vg_ganeti/5fc54069-ee70-499a-9987-8201a604ee77.disk1_data, visible as /dev/, size=2048m)>, metadev=<LogicalVolume(/dev/vg_ganeti/5fc54069-ee70-499a-9987-8201a604ee77.disk1_meta, visible as /dev/, size=128m)>, visible as /dev/disk/1, size=2048m)>
Wed Aug 10 21:49:37 2022   - ERROR: instance static-gitlab-shim.torproject.org: couldn't retrieve status for disk/2 on chi-node-08.torproject.org: Can't find device <DRBD8(hosts=0d8b8663-e2bd-42e7-9e8d-e4502fa621b8/5-3aa32c9d-c0a7-44bb-832d-851710d04765/12, port=11042, backend=<LogicalVolume(/dev/vg_ganeti/5d092bcf-d229-47cd-bb2b-04dfe241fb68.disk2_data, visible as /dev/, size=20480m)>, metadev=<LogicalVolume(/dev/vg_ganeti/5d092bcf-d229-47cd-bb2b-04dfe241fb68.disk2_meta, visible as /dev/, size=128m)>, visible as /dev/disk/2, size=20480m)>
Wed Aug 10 21:49:37 2022   - WARNING: instance probetelemetry-01.torproject.org: disk/0 on chi-node-06.torproject.org is degraded; local disk state is 'ok'
Wed Aug 10 21:49:37 2022   - WARNING: instance probetelemetry-01.torproject.org: disk/1 on chi-node-06.torproject.org is degraded; local disk state is 'ok'
Wed Aug 10 21:49:37 2022   - WARNING: instance probetelemetry-01.torproject.org: disk/2 on chi-node-06.torproject.org is degraded; local disk state is 'ok'
Wed Aug 10 21:49:37 2022   - ERROR: instance probetelemetry-01.torproject.org: couldn't retrieve status for disk/0 on chi-node-08.torproject.org: Can't find device <DRBD8(hosts=e2efd223-53e1-44f4-b84d-38f6eb26dcbb/3-0d8b8663-e2bd-42e7-9e8d-e4502fa621b8/0, port=11035, backend=<LogicalVolume(/dev/vg_ganeti/4b699f8a-ebde-4680-bfda-4e1a2e191b8f.disk0_data, visible as /dev/, size=10240m)>, metadev=<LogicalVolume(/dev/vg_ganeti/4b699f8a-ebde-4680-bfda-4e1a2e191b8f.disk0_meta, visible as /dev/, size=128m)>, visible as /dev/disk/0, size=10240m)>
Wed Aug 10 21:49:37 2022   - ERROR: instance probetelemetry-01.torproject.org: couldn't retrieve status for disk/1 on chi-node-08.torproject.org: Can't find device <DRBD8(hosts=e2efd223-53e1-44f4-b84d-38f6eb26dcbb/4-0d8b8663-e2bd-42e7-9e8d-e4502fa621b8/1, port=11036, backend=<LogicalVolume(/dev/vg_ganeti/e5f56f72-1492-4596-8957-ce442ef0fcd5.disk1_data, visible as /dev/, size=2048m)>, metadev=<LogicalVolume(/dev/vg_ganeti/e5f56f72-1492-4596-8957-ce442ef0fcd5.disk1_meta, visible as /dev/, size=128m)>, visible as /dev/disk/1, size=2048m)>
Wed Aug 10 21:49:37 2022   - ERROR: instance probetelemetry-01.torproject.org: couldn't retrieve status for disk/2 on chi-node-08.torproject.org: Can't find device <DRBD8(hosts=e2efd223-53e1-44f4-b84d-38f6eb26dcbb/5-0d8b8663-e2bd-42e7-9e8d-e4502fa621b8/2, port=11037, backend=<LogicalVolume(/dev/vg_ganeti/ee280ecd-78cb-46c6-aca4-db23a0ae1454.disk2_data, visible as /dev/, size=51200m)>, metadev=<LogicalVolume(/dev/vg_ganeti/ee280ecd-78cb-46c6-aca4-db23a0ae1454.disk2_meta, visible as /dev/, size=128m)>, visible as /dev/disk/2, size=51200m)>
Wed Aug 10 21:49:37 2022   - WARNING: instance onionbalance-02.torproject.org: disk/0 on chi-node-09.torproject.org is degraded; local disk state is 'ok'
Wed Aug 10 21:49:37 2022   - WARNING: instance onionbalance-02.torproject.org: disk/1 on chi-node-09.torproject.org is degraded; local disk state is 'ok'
Wed Aug 10 21:49:37 2022   - ERROR: instance onionbalance-02.torproject.org: couldn't retrieve status for disk/0 on chi-node-08.torproject.org: Can't find device <DRBD8(hosts=0d8b8663-e2bd-42e7-9e8d-e4502fa621b8/8-86e465ce-60df-4a6f-be17-c6abb33eaf88/4, port=11022, backend=<LogicalVolume(/dev/vg_ganeti/3b0e4300-d4c1-4b7c-970a-f20b2214dab5.disk0_data, visible as /dev/, size=10240m)>, metadev=<LogicalVolume(/dev/vg_ganeti/3b0e4300-d4c1-4b7c-970a-f20b2214dab5.disk0_meta, visible as /dev/, size=128m)>, visible as /dev/disk/0, size=10240m)>
Wed Aug 10 21:49:37 2022   - ERROR: instance onionbalance-02.torproject.org: couldn't retrieve status for disk/1 on chi-node-08.torproject.org: Can't find device <DRBD8(hosts=0d8b8663-e2bd-42e7-9e8d-e4502fa621b8/9-86e465ce-60df-4a6f-be17-c6abb33eaf88/5, port=11021, backend=<LogicalVolume(/dev/vg_ganeti/ec75f295-1e09-46df-b2c2-4fa24f064401.disk1_data, visible as /dev/, size=2048m)>, metadev=<LogicalVolume(/dev/vg_ganeti/ec75f295-1e09-46df-b2c2-4fa24f064401.disk1_meta, visible as /dev/, size=128m)>, visible as /dev/disk/1, size=2048m)>
Wed Aug 10 21:49:37 2022 * Verifying orphan volumes
Wed Aug 10 21:49:37 2022 * Verifying N+1 Memory redundancy
Wed Aug 10 21:49:37 2022 * Other Notes
Wed Aug 10 21:49:37 2022   - NOTICE: 1 offline node(s) found.
Wed Aug 10 21:49:37 2022 * Hooks Results

Shortly after starting this ticket, I ran gnt-cluster verify again and it found no issues

Submitted jobs 376743, 376744
Waiting for job 376743 ...
Wed Aug 10 21:55:20 2022 * Verifying cluster config
Wed Aug 10 21:55:20 2022 * Verifying cluster certificate files
Wed Aug 10 21:55:20 2022 * Verifying hypervisor parameters
Wed Aug 10 21:55:20 2022 * Verifying all nodes belong to an existing group
Waiting for job 376744 ...
Wed Aug 10 21:55:21 2022 * Verifying group 'default'
Wed Aug 10 21:55:21 2022 * Gathering data (11 nodes)
Wed Aug 10 21:55:21 2022 * Gathering information about nodes (11 nodes)
Wed Aug 10 21:55:24 2022 * Gathering disk information (11 nodes)
Wed Aug 10 21:55:25 2022 * Verifying configuration file consistency
Wed Aug 10 21:55:25 2022 * Verifying node status
Wed Aug 10 21:55:25 2022 * Verifying instance status
Wed Aug 10 21:55:25 2022 * Verifying orphan volumes
Wed Aug 10 21:55:25 2022 * Verifying N+1 Memory redundancy
Wed Aug 10 21:55:25 2022 * Other Notes
Wed Aug 10 21:55:25 2022   - NOTICE: 1 offline node(s) found.
Wed Aug 10 21:55:26 2022 * Hooks Results

I assume the errors I saw previously were because chi-node-08's disks were syncing, but that was a lot of errors so I'm opening this ticket anyway in case it's a symptom of a bigger issue.

added Ganeti label

changed title from Ganeti disk errors (temporary) after rebooting chi-node-08 to Ganeti (temporary) disk errors after rebooting chi-node-08

That is correct: I think the error was harmless and can be ignored. I'll just close this ticket, unless you think we should document this somewhere (or maybe better yet: fix the fabric script to handle this?)

...

On 2022-08-10 21:58:06, kezzle (@kez) wrote:

I assume the errors I saw previously were because chi-node-08's disks were syncing, but that was a lot of errors so I'm opening this ticket anyway in case it's a symptom of a bigger issue.

-- Antoine Beaupré torproject.org system administration

We should definitely do one or the other. I actually gave up trying to use migrate-back at some point because it was failing more often than not due to things like this so fixing it would be great, but I'll also settle for documenting it.

I've reopened then, feel free to reassign or deal with as you please. :)

...

On 2022-08-15 18:47:42, kezzle (@kez) wrote:

We should definitely do one or the other. I actually gave up trying to use migrate-back at some point because it was failing more often than not due to things like this so fixing it would be great, but I'll also settle for documenting it.

-- Antoine Beaupré torproject.org system administration

closed

added Backlog label

assigned to @anarcat

reopened

added Next label and removed Backlog label

changed title from Ganeti (temporary) disk errors after rebooting chi-node-08 to document this: Ganeti (temporary) disk errors after rebooting chi-node-08

closed

removed Next label

assigned to @kez and unassigned @anarcat

Closed in tpo/tpa/wiki commit 85ccd9f

i guess you meant wiki-replica@85ccd9f7

i'm considering moving the wiki here so that those hashes "just work"... see #29398

mentioned in commit wiki-replica@e9fc7d67