dal-rescue-01 and dal-rescue-02 disks are too small
Both servers keep stumbling upon themselves because their root filesystem fills up. We've past the mark where we could just hack our way around this problem, the OS is trimmed down to a pretty bare minimum... Next step would be to get rid of manpages for example.
right now the root filesystem on -02 is:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/crypt_dev_md1 3.3G 2.9G 277M 92% /
... and that's because there's a pending reboot, but even with the extra kernel removed, the disk is past 80% usage. the problem is the CF card is a measly 4GB thing:
root@dal-rescue-02:~# lsblk | grep -e disk -e NAME
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 111.8G 0 disk
mmcblk0 179:0 0 3.7G 0 disk
we can certainly afford a bigger CF and swap it in there. do the procedure on dal-rescue-02, ship the device to dallas, and have them ship the other back.