failure to create SAN-backed VM in gnt-chi

marked this issue as related to #40193 (closed)

marked this issue as related to #40131 (closed)

marked this issue as related to #40683 (closed)

i tried a multiple adopt option to setup the vm with:

gnt-instance add       -n chi-node-01.torproject.org       -o debootstrap+bullseye       -t blockdev --no-wait-for-sync       --net 0:ip=pool,network=gnt-chi-01       --no-ip-check       --no-name-check       --disk 0:adopt=/dev/disk/by-id/dm-name-telegram-bot-01-part1 --disk 1:adopt=/dev/disk/by-id/dm-name-telegram-bot-01-part2,name=swap --disk 2:adopt=/dev/disk/by-id/dm-name-telegram-bot-01-part3      --backend-parameters memory=8g,vcpus=2       telegram-bot-01.torproject.org

there's clearly a problem with the partition setup, probably again related to the partprobe thing i did:

/etc/kernel/postinst.d/zz-update-grub:
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.10.0-14-amd64
Found initrd image: /boot/initrd.img-5.10.0-14-amd64
Found linux image: /boot/vmlinuz-5.10.0-13-amd64
Found initrd image: /boot/initrd.img-5.10.0-13-amd64
Found Debian GNU/Linux 11 (bullseye) on /dev/sdaa1
Found Debian GNU/Linux 11 (bullseye) on /dev/sdac1
Found Debian GNU/Linux 11 (bullseye) on /dev/sdal1
Found Debian GNU/Linux 11 (bullseye) on /dev/sdam1
Found Debian GNU/Linux 11 (bullseye) on /dev/sdaq1
Found Debian GNU/Linux 11 (bullseye) on /dev/sdo1
Found Debian GNU/Linux 11 (bullseye) on /dev/sdr1
Found Debian GNU/Linux 11 (bullseye) on /dev/sdu1
Found Debian GNU/Linux 11 (bullseye) on /dev/sdy1
done

aka "not good". somehow this install also fails silently. this might be related to the partprobe, so i'll reboot chi-node-01 to clear things out.

okay, after a reboot, i found this error in the logs:

+ chroot /tmp/tmp.fZmlTQ6I1s grub-install /dev/loop0
Installing for i386-pc platform.
grub-install: warning: File system `ext2' doesn't support embedding.
grub-install: error: embedding is not possible, but this is required for cross-disk install.

so maybe PARTITION_STYLE=none is causing more problems than it solves, trying again without.

next failure:

+ chroot /tmp/tmp.DMBH8QvuuZ grub-install /dev/loop0
Installing for i386-pc platform.
grub-install: warning: Attempting to install GRUB to a disk with multiple partition labels.  This is not supported yet..
grub-install: error: embedding is not possible, but this is required for cross-disk install.

it looks like the procedure should basically be:

parted --script --align optimal /dev/mapper/telegram-bot-01 $(cat iscsi-parted-script.txt )
gnt-instance add       -n chi-node-01.torproject.org       -o debootstrap+bullseye       -t blockdev --no-wait-for-sync       --net 0:ip=pool,network=gnt-chi-01       --no-ip-check       --no-name-check       --disk 0:adopt=/dev/disk/by-id/dm-name-telegram-bot-01-part2 --disk 1:adopt=/dev/disk/by-id/dm-name-telegram-bot-01-part3,name=swap --disk 2:adopt=/dev/disk/by-id/dm-name-telegram-bot-01-part4      --backend-parameters memory=8g,vcpus=2       telegram-bot-01.torproject.org
gnt-instance shutdown --timeout=0 telegram-bot-01.torproject.org ; gnt-instance reinstall telegram-bot-01.torproject.org

i'll update the wiki and close this, but obviously we should have a better install procedure than this mess...

closed

fixed in wiki-replica@7a880aff

@lavamind i think the only remaining issue here is what to do about your patch in https://github.com/ganeti/instance-debootstrap/pull/17

i think we should consider reverting it and trying a new install to see if it still works. i bet it does and the problem i was having is more due to the way i was creating the VMs before (one LUN per part).

obviously, the current setup isn't great, but it works and doesn't require a patch so maybe we can just move on?

It feels really backward to me that we create these VMs with 3 disk "devices" that are actually partitions, then re-partition them within the VM itself, essentially just because ganeti-instance-bootstrap lacks any flexibility and stuffs the Debian OS in a partition that spans 100% of the drive we give it, rather than the standard 10G.

Once the instance is created with this geometry, I'm not even sure its possible to fix it: a new instance would have to be created if we wanted these VMs to have a sane block device disk layout.

Would it be possible to consider adjusting ganeti-instance-bootstrap slightly so that it creates a single 10G partition for the OS, and simply add the (simple) steps to add a swap and /srv partition inside the VM?

In the end we should really reconsider whether ganeti-instance-bootstrap is a good fit for our needs, but at least in the meantime we'll avoid creating VMs that sit on kaleidoscopes of partitions.

Another problem with the "partitions-as-disks" layout is that it makes resizing these disks much more complicated because we need to resize the underlying partitions on the ganeti nodes, ensure they're in sync across the cluster, and then also resize more partitions and/or filesystems inside the VM itself.

And if we really don't want to touch the ganeti-instance-bootstrap script then we should just deploy the instance as it creates them, with only 1 partition that spans the entire drive, and just use a simple procedure to reduce that filesystem/partition to 10G, before creating new partitions for swap and /srv.

I again tested the patch proposed by the upstream author and indeed, it doesn't work. What I found does work, is this: https://github.com/jcharaoui/instance-debootstrap/commit/6ee6c84d8eeb67fef061e05ccd4a5fde12206f06

Basically, instead of calling kpartx with the argument -p- (use a partition separator of -), call it with -part. This separator is what we see on the multipath devices when we manually create partitions on there and call partprobe on that device.

Why this works in our environment is not entirely clear to me. It might just be that we already use -NN suffixes in our device names, therefore confusing some setup command, or because of some multipath configuration?

Anyway, the important part is I've demonstrated with the test-01 instance, that we can bootstrap blockdev instances that adopt a single disk, and not have to rely on pre-partitioning multipath/iSCSI volumes on the nodes themselves.

It feels really backward to me that we create these VMs with 3 disk "devices" that are actually partitions, then re-partition them within the VM itself, essentially just because ganeti-instance-bootstrap lacks any flexibility and stuffs the Debian OS in a partition that spans 100% of the drive we give it, rather than the standard 10G.

I agree. Yet the way g-i-b works with the DRBD backend works fine, no? Or do you also question that system?

No judgement, I'm just asking.

Once the instance is created with this geometry, I'm not even sure its possible to fix it: a new instance would have to be created if we wanted these VMs to have a sane block device disk layout.

What do you mean here? "fix it", fix what?

Would it be possible to consider adjusting ganeti-instance-bootstrap slightly so that it creates a single 10G partition for the OS, and simply add the (simple) steps to add a swap and /srv partition?

Sure, although I'm always really hesitant in hacking at g-i-d, particularly do do partitionning, as this is normally handled at the ganeti storage backends level. For example, in DRBD, the storage backend does the partitionning, not the installer. This is why we do the partitioning by hand before handing off to gnt-instance add: it's the way other storage modules work.

In the end we should really reconsider whether ganeti-instance-bootstrap is a good fit for our needs,

That, I agree with.

but at least in the meantime we'll avoid creating VMs that sit on kaleidoscopes of partitions.

That, I'm not sure I have a clear idea of a solution for.

Another problem with the "partitions-as-disks" layout is that it makes resizing these disks much more complicated because we need to resize the underlying partitions on the ganeti nodes, ensure they're in sync across the cluster, and then also resize more partitions and/or filesystems inside the VM itself.

Yes. It's definitely more complicated than it should be. But it works.

And if we really don't want to touch the ganeti-instance-bootstrap then we should just deploy the instance as it creates them, with only 1 partition that pans the entire drive, and just use a simple procedure to reduce that filesystem/partition to 10G, before creating new partitions for swap and /srv.

What "simple procedure" do you have in mind here? Keep in mind you would need to repartition the disk (inside the running VM?) including the root partition.

I tested the patch proposed by the upstream author and indeed, it doesn't work. What I found does work, is this: https://github.com/jcharaoui/instance-debootstrap/commit/6ee6c84d8eeb67fef061e05ccd4a5fde12206f06

Basically, instead of calling kpartx with the argument -p- (use a partition separator of -), call it with -part. This separator is what we see on the multipath devices when we manually create partitions on there and call partprobe on that device.

Those that hack work both in the blockdev and drbd backends?

Why this works in our environment is not entirely clear to me. It might just be that we already use -NN suffixes in our device names, therefore confusing some setup command, or because of some multipath configuration?

One of the commend on that pull request is exactly about the multipath configuration, maybe we could look that way?

Or we could just treat this entire cluster as legacy and punt this problem forward.

The issue here is the incapacity of creating VMs in the the gnt-chi cluster, backed by the SAN. If we can fix that while not breaking DRBD, I will consider it fixed, which is why I closed this issue with the documentation patch in #40775 (comment 2808009)

If we're unhappy with the fix, maybe we could reopen this issue, or open a new one with a better description of what it is we are trying to do here?

...

On 2022-05-30 20:31:18, Jérôme Charaoui (@lavamind) wrote:

-- Antoine Beaupré torproject.org system administration

It feels really backward to me that we create these VMs with 3 disk "devices" that are actually partitions, then re-partition them within the VM itself, essentially just because ganeti-instance-bootstrap lacks any flexibility and stuffs the Debian OS in a partition that spans 100% of the drive we give it, rather than the standard 10G.

I agree. Yet the way g-i-b works with the DRBD backend works fine, no? Or do you also question that system?

We discussed it a bit in the past but I think the DRBD system might be better if it had less DRBD devices per VM. There's probably some amount of overhead there that could be avoided if each instance only had one DRBD device per backing-disks (of course having multiple DRBD devices in unavoidable for instances with both SSD and HDD backed disks). For example, fsn-node-03 has 35 DRBD devices even though only 6 instances are running on it.

Once the instance is created with this geometry, I'm not even sure its possible to fix it: a new instance would have to be created if we wanted these VMs to have a sane block device disk layout.

What do you mean here? "fix it", fix what?

I mean reconfigure the instance to use only 1 adopted block device instead of 3+.

And if we really don't want to touch the ganeti-instance-bootstrap then we should just deploy the instance as it creates them, with only 1 partition that pans the entire drive, and just use a simple procedure to reduce that filesystem/partition to 10G, before creating new partitions for swap and /srv.

What "simple procedure" do you have in mind here? Keep in mind you would need to repartition the disk (inside the running VM?) including the root partition.

I'm not suggesting we repartition inside the running VM: we can use Grml for that, via the grml-rescueboot package. Of course, I'd much prefer to avoid that and instead have our instance bootstrap script just create the 10G partition adequately from the get-go.

I tested the patch proposed by the upstream author and indeed, it doesn't work. What I found does work, is this: https://github.com/jcharaoui/instance-debootstrap/commit/6ee6c84d8eeb67fef061e05ccd4a5fde12206f06

Basically, instead of calling kpartx with the argument -p- (use a partition separator of -), call it with -part. This separator is what we see on the multipath devices when we manually create partitions on there and call partprobe on that device.

Does that hack work both in the blockdev and drbd backends?

Yes, I created a DRBD instance test-02 on gnt-chi with the patched code and it works.

One of the commend on that pull request is exactly about the multipath configuration, maybe we could look that way?

I did try to set partition_delimiter to "-" in /etc/multipath.conf, hoping it would make partitions appear with the - delimiter that the instance bootstrap scripts expects, but it didn't work. But I was also skeptical: if it did work, then it could mean that previously setup instances would need to be reconfigured since their adopted block device names include the -part delimiter that is actually used.

The issue here is the incapacity of creating VMs in the the gnt-chi cluster, backed by the SAN. If we can fix that while not breaking DRBD, I will consider it fixed, which is why I closed this issue with the documentation patch in #40775 (comment 2808009)

I think we can fix it better by using a different patch, described above. Note that I also did suggest that same fix (using a different partition delimiter) in the upstream issue tracker but it seems to have been ignored.

Jérôme Charaoui commented on a discussion: #40775 (comment 2808783)

It feels really backward to me that we create these VMs with 3 disk "devices" that are actually partitions, then re-partition them within the VM itself, essentially just because ganeti-instance-bootstrap lacks any flexibility and stuffs the Debian OS in a partition that spans 100% of the drive we give it, rather than the standard 10G.

I agree. Yet the way g-i-b works with the DRBD backend works fine, no? Or do you also question that system?

We discussed it a bit in the past but I think the DRBD system might be better if it had less DRBD devices per VM. There's probably some amount of overhead there that could be avoided if each instance only had one DRBD device per backing-disks (of course having multiple DRBD devices in unavoidable for instances with both SSD and HDD backed disks). For example, fsn-node-03 has 35 DRBD devices even though only 6 instances are running on it.

Well, that's a whole other ball game then, because that means patching ganeti itself, not "just" our wiki instructions or ganeti-instance-debootstrap.

Once the instance is created with this geometry, I'm not even sure its possible to fix it: a new instance would have to be created if we wanted these VMs to have a sane block device disk layout.

What do you mean here? "fix it", fix what?

I mean reconfigure the instance to use only 1 adopted block device instead of 3+.

Right. We don't have to create a completely "new instance" (as in go through the entire "new-machine" procedure). We can shutdown the instance, backup, destroy, restore with a new partition, and adopt the new disks. It's not that complex of a procedure.

And if we really don't want to touch the ganeti-instance-bootstrap then we should just deploy the instance as it creates them, with only 1 partition that pans the entire drive, and just use a simple procedure to reduce that filesystem/partition to 10G, before creating new partitions for swap and /srv.

What "simple procedure" do you have in mind here? Keep in mind you would need to repartition the disk (inside the running VM?) including the root partition.

I'm not suggesting we repartition inside the running VM: we can use Grml for that, via the grml-rescueboot package. Of course, I'd much prefer to avoid that and instead have our instance bootstrap script just create the 10G partition adequately from the get-go.

I'm not sure how we would run grml-rescueboot in a Ganeti context.

I tested the patch proposed by the upstream author and indeed, it doesn't work. What I found does work, is this: https://github.com/jcharaoui/instance-debootstrap/commit/6ee6c84d8eeb67fef061e05ccd4a5fde12206f06

Basically, instead of calling kpartx with the argument -p- (use a partition separator of -), call it with -part. This separator is what we see on the multipath devices when we manually create partitions on there and call partprobe on that device.

Does that hack work both in the blockdev and drbd backends?

Yes, I created a DRBD instance test-02 on gnt-chi with the patched code and it works.

Good!

One of the commend on that pull request is exactly about the multipath configuration, maybe we could look that way?

I did try to set partition_delimiter to "-" in /etc/multipath.conf, hoping it would make partitions appear with the - delimiter that the instance bootstrap scripts expects, but it didn't work. But I was also skeptical: if it did work, then it could mean that previously setup instances would need to be reconfigured since their adopted block device names include the -part delimiter that is actually used.

Could you report that in the upstream bug report please?

The issue here is the incapacity of creating VMs in the the gnt-chi cluster, backed by the SAN. If we can fix that while not breaking DRBD, I will consider it fixed, which is why I closed this issue with the documentation patch in #40775 (comment 2808009)

I think we can fix it better by using a different patch, described above. Note that I also did suggest that same fix (using a different partition delimiter) in the upstream issue tracker but it seems to have been ignored.

I received an email from GitHub about a patch like this, but I couldn't quite figure out what happened, to be honest. I think if you want to suggest a different patch, you should roll out a new PR instead.

And, if the current PR doesn't work, just close it.

I must say I'm also concerned about the delays in shipping the VM where we found that issue (#40683 (closed)). That ticket was opened 2 months ago, and a week ago we were pinged about it... Can you post an update there as well?

Let's not make perfect the ennemy of good and let's just ship something please. In the meantime, since you want to work on this, I'll reopen this ticket so that we have at least some tracking.

a.

...

On 2022-05-31 13:31:35, Jérôme Charaoui (@lavamind) wrote:

-- Antoine Beaupré torproject.org system administration

I'm not sure how we would run grml-rescueboot in a Ganeti context.

Ganeti boots Grub, so Grub can boot Grml. You just need to make sure it boots with both the "to ram" and "serial" boot options. I tested it and it works fine.

reopened

assigned to @lavamind

mentioned in commit wiki-replica@ad470551

we had a chat about this in the checkin.

i first stated that i would have liked to have this discussion about how we implement this before @lavamind went on implementing it. this morning, i stated this on IRC:

09:52:29 <anarcat> lavamind: also, re. the installer + SAN, i'd like you to make a new comment in the ticket clarifying more precisely what you intend to do as a solution
09:52:42 <anarcat> for me the solution is still unclear and spread over many comments, so i think it would be a good time to do a summary
09:52:42 <lavamind> anarcat: ack
09:52:49 <anarcat> restate the problem, and your step by step solution please

i was worried we were going in the wrong direction, and the proposed patches (the PARTITION_SIZE one in particular) seem problematic to me now. i worry the patch will not be accepted upstream and it's mixing concerns between the external storage: on the one hand we (typically) specify disk size at the storage provider level (in gnt-instance add --disk ...), but on the other hand here we are specifying a disk size in the installer. that won't work for other installers, and it will be difficult to customize. the (undocumented) way of doing this was to add a new PARTITION_SIZE line to the /etc/default/ganeti-instance-debootstrap file to assign that size inside the VM, but I found that was too error-prone.

on the other hand, doing the partitioning at the host/node level does't work either: those partitions won't be visible on other nodes unless some magic kpartx command is ran everywhere, which also seems dangerous. it seems like a bad idea to have the host kernel interpret the partitions inside the disk's VM.

so let's go back to multiple LUNs per vm (bleh). this involves:

tpo-create-san-disks creates the first virtual disk for /
list all the disk groups
let the user pick the disk group for the second/third parts
create those partitions
create the multipath configs for the created WWIDs on all nodes

then the operator passes those disks to --disk N:adopt=.

a few more decision points:

let's keep the swap for now
... but let's move teh iscsi::multipath::alias functionality into the tpo-create-san-disks, so we don't have that extra shit to carry arround into puppet (that's step 5 above)

if you can, at this point i'd seriously consider rewriting the shell script in python as well, because it's going to grow quite a bit, and will have to keep track of more state (and i don't want to see bash arrays flying around, please)

and let's finish the VM first, because it's been waiting long enough and it doesn't matter that much how it's setup. we can make tests with a test VM, and even convert this one later if we really want to keep things consistent (again, with --disk N:adopt=).

marked this issue as related to #40780 (closed)

mentioned in issue #40780 (closed)

changed due date to June 16, 2022

@lavamind has concerns about the plan I outlined in that previous comments, so we need to talk more about this. i set next thursday as a deadline for that conversation.

failure to create SAN-backed VM in gnt-chi

Designs

Child items ...

Activity