dead disk on moly
one of the hard drives on moly has died. this was spotted by cymru's staff and confirmed when smartd was installed (#29709 (moved)).
i have done some research on the machine to figure out what's up, and wrote the following reply to Cymru's people:
[...] I can confirm that one of the hard drives in Moly has failed, according to SMART metrics we have available.
According to smartd, that disk is:
[SEAGATE ST3600057SS 0008], lu id: 0x5000c5003b5bc36f, S/N: 6SL1G7Q60000N1497K0E, 600 GB
It's a 600GB SAS drive. It's part of a megaraid RAID-10 array that has marked the drive as "Firmware state: Failed". I'll go under the assertiont his means the drive is dead.
Being new here, I'm not familiar with the machine either. From what I can tell, it's a Supermicro X8DTU motherboard, and possibly an iXsystems iX1204-R700UB case. Does it look like this this picture?
https://static.ixsystems.co/uploads/2017/08/1204h-t_front.png
If so, the only datasheet I could find is this limited PDF:
https://www.ixsystems.com/wp-content/uploads/2017/09/Server_Line_2017_WEB.pdf
It does say the hard drives are hot-swappable, so in theory, it should just be a matter of replacing the hard drive.
It looks like each drive has its own LED, hopefully the one with the amber warning light should be the dead disk. I've issued a command to the RAID controller to make it "flash" the drive LED, so hopefully that will allow you to locate it better.
I think the disk controller is new enough for you to simply hot swap the drive with a new one without any other intervention on our part. But it might be better if we are available during the operation. [...]
I've created some documentation on the hardware RAID stuff here:
https://help.torproject.org/tsa/howto/raid/
we're at the waiting step now - we'll see if Cymru can do the replacement and when. i'm still not quite certain we can just hotswap the drive, but I'm hoping we can.