IO latency on new (bookworm?) hosts (on the gnt-dal cluster?)

I'm seeing a weird degradation when bootstrapping hosts in the gnt-dal cluster. I'm not sure it's specific to that cluster, nor to bookworm, but there's definitely something fishy there.

The symptom is that any disk access for a "new" thing gives me a hard 10 second delay. This affects tab completion, reading a file, writing a file, basically anything that involves IO. I am not sure if that involves networking, but apt update is affected, for example. Another example is the puppet bootstrap script hanging for ten seconds at:

missing torproject.org from /etc/resolv.conf, appending backup configuration...

Normally, that's instantaneous, it's just appending a line to a file, but there it took 10 seconds.

Next steps:

• test if this occurs on a bullseye host
• test if this occurs on a bookworm host on gnt-fsn
• test if this occurs on another host after reboot

added Ganeti Next labels

assigned to @anarcat

here's another example:

root@rdsys-test-01:~# reboot


W: molly-guard: SSH session detected!
Please type in hostname of the machine to reboot: Good thing I asked; I won't reboot rdsys-test-01 ...
W: aborting reboot due to 30-query-hostname exiting with code 1.
root@rdsys-test-01:~# 
root@rdsys-test-01:~# reboot
W: molly-guard: SSH session detected!
Please type in hostname of the machine to reboot: rdsys-test-01

those newlines before the W: line are me hitting enter because nothing was happening for 10 seconds.

mentioned in issue #41297 (closed)

this doesn't occur on a completely unrelated physical host, tb-build-03, running bookworm, on the gnt-dal site.

This issue has been waiting for information two weeks or more. It needs attention. Please take care of this before the end of 2023-10-05. ~"Needs Information" tickets will be moved to the Icebox after that point.

(Any ticket left in Needs Review, Needs Information, Next, or Doing without activity for 14 days gets such notifications. Make a comment describing the current state of this ticket and remove the Stale label to fix this.)

To make the bot ignore this ticket, add the bot-ignore label.

added Stale label

@lavamind did you see this while deploying the new puppetdb server (#41341 (closed))?

Nope, not this time, and not previous times either...

i wonder if this might not have been a fluke, possibly related to the gnt-dal cluster outage (#41176 (closed))...

... maybe that those checks have been disabled this is fixed?

in any case, if we can't reproduce, i'm just going to close this...

closed

removed Stale label