backup server (bungei) volume filled up
Bacula backups have started failing today, amid warnings about free space on the server.
@anarcat investigated and found out that archive-01
that has grown seemingly out of control:
yep, so archive-01's /srv/archive.torproject.org/htdocs/tor-package-archive/torbrowser is 5TB
Space reservation was reduced from 0.5 to 0.01 percent which should give enough room for the upcoming backup to succeed.
Next steps:
-
reduce reserved space -
grow logical volume -
unlock mounts -
make sure the archive-01 job completes -
make sure all backups have ran in the last 24h (ie. make nagios green again) -
open a ticket for or just make a plan for the next time -
document this in the backup pager playbook
Edited by anarcat