/srv filesystem failure on tb-build-03

today i heard this from @richard:

13:49:14 <+richard> anarcat, lavamind: so i'm trying to do a build on tb-build-03
13:49:23 <+richard> and i can't seem to create any files?
13:49:35 <+richard> eg touch foo -> touch: cannot touch 'foo': Read-only file system

that started happening "very shortly before" that message, according to richard. so this is recent.

filesystem was remounted read-only:

13:55:10 <+anarcat> /dev/sdc on /srv type ext4 (ro,relatime)
13:55:10 <+anarcat> /dev/sdc on /home type ext4 (ro,relatime)

and there were errors like this in the dmesg:

13:55:44 <+anarcat> [960186.849508] EXT4-fs warning (device sdc): ext4_end_bio:323: I/O error 5 writing to inode 28189202 (offset 8388608 size 4567040 starting block 159154688)

i naively rebooted the box, and now it fails to boot with this on the console:

[FAILED] Failed to start File Syste…5-6f79-40cf-8406-8d484132ffc6.
See 'systemctl status "systemd-fsck@dev…\x2d8d484132ffc6.service"' for details.
[DEPEND] Dependency failed for /srv.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for /home.

in the rescue, trying to mount the filesystems gives me:

root@tb-build-03:~# mount -a
mount: /srv: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error.
mount: /home: special device /srv/home does not exist.

(ignore the /home, it's a bind mount.)

so something is up with that filesystem. apparently:

14:04:12 <+richard> fwiw this happened in the middle of a build 
14:04:49 <+richard> so it just broke (or read/writing to disk exposed something that broke awhile ago)

action plan:

• reboot in case that just fixes it (nope, it won't boot now)
• investigate state on the SAN
• compare with the state on the ganeti node...
• ... and from inside the VM
• possibly: recreate the LUN on the SAN?
• alternative: recreate the box from scratch, it doesn't have backups anyways (and the drives were created weird anyways)

added Next label

i tried to boot the VM again and fsck the thing

root@tb-build-03:~# fsck /dev/sdc
fsck from util-linux 2.33.1
e2fsck 1.44.5 (15-Dec-2018)
/dev/sdc: recovering journal
Error writing block 157286402 (Invalid argument).  Ignore error<y>? yes
Error writing block 157286401 (Invalid argument).  Ignore error<y>? yes
Error writing block 157286403 (Invalid argument).  Ignore error<y>? yes
Error writing block 157286406 (Invalid argument).  Ignore error<y>? yes
Error writing block 157286407 (Invalid argument).  Ignore error<y>? yes
Error writing block 157286408 (Invalid argument).  Ignore error<y>? cancelled!
[...]
The filesystem size (according to the superblock) is 183500800 blocks
The physical size of the device is 157286400 blocks
Either the superblock or the partition table is likely to be corrupt!

i tried to say yes a few times, then hit control-c, which led to a lot of "cancelled", and, interestingly, that last error message that the device is not the right size.

could it be that the resize operation #40546 (closed) didn't work correctly, @lavamind? maybe we're somehow still using the old, not resized device?

I did the resize operation according to best practice: after resizing the LUN I rebooted the VM which allowed me to expand the logical volume and then resize the ext4 filesystem. All those operations went without a hitch, so I don't know what could possibly have been using the old device size while I was executing those operations.

Edit: there was no LVM or partitions under the filesystem (maybe that's related to the issue?)

changed the severity to High - S2

changed due date to March 08, 2022

assigned to @anarcat

changed the description

marked this issue as related to #40546 (closed)

I ran a consistency check on the virtual disk earlier today:

SMcli -n chi-san-01 -p $PASSWORD -c "check virtualdisk [\"tb-builder-03-srv\"] consistency verbose=TRUE;"

The process completed without errors, so I think we can perhaps reasonably exclude something being wrong with the SAN itself.

marked the checklist item investigate state on the SAN as completed

added Doing label and removed Next label

assigned to @lavamind and unassigned @anarcat

Tried another fsck from the node (host):

root@chi-node-01:~# fsck /dev/mapper/tb-build-03-srv
fsck from util-linux 2.36.1
e2fsck 1.46.2 (28-Feb-2021)
/dev/mapper/tb-build-03-srv contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  +(158512006--158512907) +(158513152--158515199) +(158525440--158535150) +(158564352--158603161) +(158603168--158603198) +(158603264--158613856) +(158613888--158613976) +(158614016--158614267) +(158615552--158625202) +(158625792--158633414) +(158633440--158633456) +(158633984--158643820) +(158644224--158647640) +(158648320--158668340) +(158670848--158753075) +(158754816--158859263) +(158867488--158867597) +(158892032--158908442) +(158910464--158964455) +(158965760--158977193) +(158978048--158981940) +(158982144--158985969) +(158986240--158991328) +(158992384--159012685) +(159012864--159024338) +(159025152--159074692) +(159076352--159091606) +(159092736--159112556) +(159113216--159138180) +(159139840--159167920) +(159168512--159181420) +(159182848--159192836) +(159193088--159227868) +(159227904--159252479) +(159318016--159322845) +(159324160--159326207) +(159350784--159369126) +(159369216--159375216) +(159375360--159383551) +(159391776--159393024) +(159416320--159427556) +(159428608--159432699) +(159432704--159501755) +(159502336--159528312) +(159528960--159547391) +(159549729--159551171) +(159551488--159564711) +(159565824--159575458) +(159576064--159647627) +(159647744--159657976) +(159657984--159660907) +(159662080--159676415) +(159678464--159682518) +(159682560--159686534) +(159686656--159694726) +(159694848--159705087)
Fix<y>? yes
Free blocks count wrong for group #4801 (1601, counted=30403).
Fix<y>? yes
Free blocks count wrong for group #4802 (2299, counted=11480).
Fix<y>? yes
Free blocks count wrong for group #4803 (2079, counted=10271).
Fix<y>? yes
Free blocks count wrong for group #4806 (1839, counted=16175).
Fix<y>? yes
Free blocks count wrong for group #4807 (1819, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4808 (1769, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4809 (158, counted=29012).
Fix<y>? yes
Free blocks count wrong for group #4810 (1826, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4811 (1851, counted=32768).
Fix ('a' enables 'yes' to all) <y>? yes
Free blocks count wrong for group #4812 (218, counted=32768).
Fix ('a' enables 'yes' to all) <y>? yes
Free blocks count wrong for group #4813 (1492, counted=32768).
Fix ('a' enables 'yes' to all) <y>? yes
Free blocks count wrong for group #4814 (0, counted=32768).
Fix ('a' enables 'yes' to all) <y>? yes
Free blocks count wrong for group #4815 (2023, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4816 (23333, counted=24544).
Fix<y>? yes
Free blocks count wrong for group #4817 (1854, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4818 (0, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4819 (0, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4825 (2547, counted=14835).
Fix<y>? yes
Free blocks count wrong for group #4826 (198, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4827 (2322, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4828 (1635, counted=29742).
Fix<y>? yes
Free blocks count wrong for group #4829 (1973, counted=31233).
Fix<y>? yes
Free blocks count wrong for group #4830 (16627, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4831 (2075, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4833 (1013, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4834 (2826, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4835 (5753, counted=32768).
Fix<y>? yes
Free blocks count wrong for group #4836 (27406, counted=30334).
Fix<y>? yes
Free blocks count wrong (45329358, counted=46160529).
Fix<y>? yes
Free inodes count wrong (40233601, counted=40233062).
Fix<y>? yes
/dev/mapper/tb-build-03-srv: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mapper/tb-build-03-srv: 5642138/45875200 files (0.1% non-contiguous), 137340271/183500800 blocks

I'm booting the VM to see if the filesystem is usable at all at this point.

Seems to be working.

Mar 08 15:58:40 tb-build-03 kernel: EXT4-fs (sdc): mounted filesystem with ordered data mode. Opts: (null)

I've also written a 40G file with data from /dev/zero to see if I could reproduce the prior I/O errors, and the test suceeded without issues.

root@tb-build-03:~# dd if=/dev/zero of=/srv/test bs=1M count=40960
40960+0 records in
40960+0 records out
42949672960 bytes (43 GB, 40 GiB) copied, 373.582 s, 115 MB/s

@richard I was able to fsck the filesystem and now it seems to be working correctly. Could you check if the system is still useable at this point? Thanks.

marked the checklist item compare with the state on the ganeti node... as completed

marked the checklist item ... and from inside the VM as completed

mentioned in issue #40656 (closed)

added Needs Review label and removed Doing label

@richard confirmed the machine has been working correctly lately, closing.

closed

changed the incident status to Resolved by closing the incident