crm-int-01 disk space issues
crm-int-01 disk usage has been pretty flaky historically, but we've had especially bad situations recently, where it completely ran out of disk space two weeks ago, leading to a full crash (#34098 (moved)).
let's fix this situation.
disk usage history in the last year of crm-int-01:
AKA:
- 20GB was added to /srv in december
- space on / has always been tight
- cleanups were done in october and december, but it's still increasing linearily
- /srv is irregular and unpredictable
conclusion:
- let's double the size of
/
to give us another year or more - let's take up the available space in the PV for
/srv
and advise with service admin
also, regarding service-specific stuff (AKA /cc @eric), here are the big disk sinks i could find:
-
12GB: mysql watchdog table in the prod database (!): maybe it should be purged regularly? -
3.6GB: /srv/crm.torproject.org/htdocs-staging/sites/all/modules/custom/tor_donation/donation_processor.log
: is there a way we could rotate that logfile somehow? do you want me to hook a logrotate in there? what service generates this? it looks like we have records starting from 2021-05-27 in there, more than a year ago! -
3GB: /srv/openflows/opt/openflows/ebudsbacks/torcivicrm.torproject.org
-
1.4GB: /home/ericg
Edited by anarcat