dead disk on moly

one of the hard drives on moly has died. this was spotted by cymru's staff and confirmed when smartd was installed (legacy/trac#29709 (moved)).

i have done some research on the machine to figure out what's up, and wrote the following reply to Cymru's people:

[...] I can confirm that one of the hard drives in Moly has failed, according to SMART metrics we have available.

According to smartd, that disk is:

[SEAGATE ST3600057SS 0008], lu id: 0x5000c5003b5bc36f, S/N: 6SL1G7Q60000N1497K0E, 600 GB

It's a 600GB SAS drive. It's part of a megaraid RAID-10 array that has marked the drive as "Firmware state: Failed". I'll go under the assertiont his means the drive is dead.

Being new here, I'm not familiar with the machine either. From what I can tell, it's a Supermicro X8DTU motherboard, and possibly an iXsystems iX1204-R700UB case. Does it look like this this picture?

https://static.ixsystems.co/uploads/2017/08/1204h-t_front.png

If so, the only datasheet I could find is this limited PDF:

https://www.ixsystems.com/wp-content/uploads/2017/09/Server_Line_2017_WEB.pdf

It does say the hard drives are hot-swappable, so in theory, it should just be a matter of replacing the hard drive.

It looks like each drive has its own LED, hopefully the one with the amber warning light should be the dead disk. I've issued a command to the RAID controller to make it "flash" the drive LED, so hopefully that will allow you to locate it better.

I think the disk controller is new enough for you to simply hot swap the drive with a new one without any other intervention on our part. But it might be better if we are available during the operation. [...]

I've created some documentation on the hardware RAID stuff here:

https://help.torproject.org/tsa/howto/raid/

we're at the waiting step now - we'll see if Cymru can do the replacement and when. i'm still not quite certain we can just hotswap the drive, but I'm hoping we can.