failed disk on fsn-node-01

Date: Thu, 11 May 2023 07:29:41 +0000
From: mdadm monitoring <root@fsn-node-01.torproject.org>
To: root@fsn-node-01.torproject.org
Subject: Fail event on /dev/md/boot:fsn-node-01

This is an automatically generated mail message from mdadm
running on fsn-node-01

A Fail event had been detected on md device /dev/md/boot.

It could be related to component device /dev/nvme0n1p2.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md125 : active raid1 sda1[0] sdb1[1]
      9766303744 blocks super 1.2 [2/2] [UU]
      bitmap: 6/73 pages [24KB], 65536KB chunk

md126 : active raid1 nvme1n1p3[1]
      936512512 blocks super 1.2 [2/1] [_U]
      bitmap: 5/7 pages [20KB], 65536KB chunk

md127 : active raid1 nvme1n1p2[1] nvme0n1p2[2](F)
      523712 blocks super 1.2 [2/1] [_U]

unused devices: <none>

plan:

1. evacuate node
2. rotate master
3. verify with previous incidents if it's the same drive that failed (#40805 (closed), #40818 (closed))
4. shutdown node
5. ask hetzner for a replacement

/cc @lavamind

added Doing label

assigned to @anarcat

marked this issue as related to #40805 (closed)

marked this issue as related to #40818 (closed)

changed the description

root@fsn-node-01:~# gnt-node migrate fsn-node-01
Migrate instance(s) bacula-director-01.torproject.org,
staticiforme.torproject.org, submit-01.torproject.org, tb-
build-01.torproject.org, weather-01.torproject.org?
y/[n]/?: 

root@fsn-node-01:~# lsblk --nodeps  -o +SERIAL
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT SERIAL
sda       8:0    0   9.1T  0 disk            5950A00SFWEF
sdb       8:16   0   9.1T  0 disk            5950A015FWEF
nvme1n1 259:4    0 894.3G  0 disk            S437NX0M402699

looks like it's the same disk as #40805 (comment 2815778) that got kicked out.

All instances migrated successfully.

failed over to fsn-node-02:

root@fsn-node-02:~# gnt-cluster master-failover
root@fsn-node-02:~# gnt-node modify --offline yes fsn-node-01.torproject.org
Thu May 11 15:36:07 2023 Failed to stop KVM daemon on node 'fsn-node-01.torproject.org': Node is marked offline
Modified node fsn-node-01.torproject.org
 - master_candidate -> False
 - offline -> True
root@fsn-node-02:~# gnt-node list
Node                       DTotal  DFree MTotal MNode MFree Pinst Sinst
fsn-node-01.torproject.org      *      *      *     *     *     0    11
fsn-node-02.torproject.org 893.1G 378.0G  62.7G 24.0G 21.9G     6     4
fsn-node-03.torproject.org 893.6G 409.8G  62.7G 41.9G  7.0G     6     4
fsn-node-04.torproject.org 893.6G 321.8G  62.7G 31.9G 22.3G     6     4
fsn-node-05.torproject.org 893.6G 411.9G  62.7G 14.1G 41.2G     6     3
fsn-node-06.torproject.org 893.6G 449.9G  62.7G 11.9G 45.8G     5     5
fsn-node-07.torproject.org 893.6G  65.5G  62.7G  6.7G 52.1G     5     5
fsn-node-08.torproject.org 893.6G 461.4G  62.7G  9.8G 31.4G     6     4

marked the checklist item evacuate node as completed

marked the checklist item rotate master as completed

marked the checklist item verify with previous incidents if it's the same drive that failed (#40805 (closed), #40818 (closed)) as completed

I've sent the support request to Hetzner asking for a replacement NVMe drive.

marked the checklist item ask hetzner for a replacement as completed

@anarcat I think shutting down the node is just an extra safety blanket that helps ensure there's no confusion in the DC when they come to shut down one of our nodes.

@anarcat I think shutting down the node is just an extra safety blanket that helps ensure there's no confusion in the DC when they come to shut down one of our nodes.

i'm a little hesitant in shutting down the node because it's still the secondary for 11 instances... DRBD would take a while to catch back up...

maybe we could fully evacuate everything, but that would take much longer...

fsn-node-01 now down, i assume they are doing the replacement...

changed the incident status to Acknowledged

replacement done, arrays are rebuilding, i rotated the master back into place and the instances as well, closing.

closed

changed the incident status to Resolved by closing the incident