gnt-instance migrate fails after ganeti node upgrade to Bookworm

Trying to migrate a node between ganeti nodes that were recently upgraded to Bookworm fails:

$ fab -H fsn-node-01.torproject.org ganeti.migrate-instances -i idle-fsn-01.torproject.org
INFO: activating idle-fsn-01.torproject.org disks
fsn-node-01.torproject.org:disk/0:/dev/drbd18
fsn-node-01.torproject.org:disk/1:/dev/drbd19
INFO: migrating instance idle-fsn-01.torproject.org
Thu Dec  5 14:44:13 2024 Migrating instance idle-fsn-01.torproject.org
Thu Dec  5 14:44:13 2024 * checking disk consistency between source and target
Thu Dec  5 14:44:14 2024 * closing instance disks on node fsn-node-02.torproject.org
Thu Dec  5 14:44:15 2024 * changing into standalone mode
Thu Dec  5 14:44:15 2024 * changing disks into dual-master mode
Thu Dec  5 14:44:17 2024 * wait until resync is done
Thu Dec  5 14:44:17 2024 * opening instance disks on node fsn-node-01.torproject.org in shared mode
Thu Dec  5 14:44:18 2024 * opening instance disks on node fsn-node-02.torproject.org in shared mode
Thu Dec  5 14:44:18 2024 * preparing fsn-node-02.torproject.org to accept the instance
Thu Dec  5 14:44:18 2024 Pre-migration failed, aborting
Thu Dec  5 14:44:18 2024 * closing instance disks on node fsn-node-02.torproject.org
Thu Dec  5 14:44:19 2024 * changing into standalone mode
Thu Dec  5 14:44:19 2024 * changing disks into single-master mode
Thu Dec  5 14:44:21 2024 * wait until resync is done
Failure: command execution error:
Could not pre-migrate instance idle-fsn-01.torproject.org: Failed to accept instance: kvm: error executing the migrate-set-capabilities command: Postcopy is not supported (GenericError):

Failover works fine, though.

mentioned in issue #41254 (closed)

The first comment in this post mentions setting vm.unprivileged_userfaultfd=1 in sysctl.

@zen i think you can just flip that sysctl on two nodes, to confirm that it fixes the migration.

if that works, do it across the cluster. there's a base::sysctl resource you can use for this purpose.

changed milestone to %Debian 12 bookworm upgrade

added Doing label

assigned to @zen

marked this issue as related to #41254 (closed)

mentioned in commit wiki-replica@926e33e1

Summary:

• Live migrations with postcopy need vm.unprivileged_userfaultfd=1
• The default for that flag was changed in the kernel between bullseye and bookworm
• Flipping that flag back to 1 fixed live migrations.

closed