lists-01 disk filling up abruptly

marked this incident as related to #41957 (closed)

added Emergency Lists Next lifecycle labels

assigned to @anarcat

changed the description

daemon.log is messing with us:

root@lists-01:/var/log# du -sch daemon.log*
1.3G    daemon.log
509M    daemon.log.1

marked this incident as related to #41851 (closed)

okay, so that was a bit of a ride. we started with:

root@lists-01:~# df -h                                                                                                                   
Filesystem      Size  Used Avail Use% Mounted on                                                                                         
udev            7.8G     0  7.8G   0% /dev                
tmpfs           1.6G  568K  1.6G   1% /run    
/dev/sda1       9.8G  8.3G 1007M  90% /        
tmpfs           7.9G  1.3M  7.9G   1% /dev/shm                                                                                                                                                                                                                                    
tmpfs           5.0M     0  5.0M   0% /run/lock                                                                                                                                                                                                                                   
tmpfs           7.9G     0  7.9G   0% /tmp       
/dev/sdb         49G   14G   33G  30% /srv       
tmpfs           1.6G     0  1.6G   0% /run/user/0

i found that uwsgi has basically a copy of the apache logs. i had found out about those logs in the mm3 deployment (#41851 (closed)), but didn't realize they were a copy, nor that they were using so much more disk space (about twice) than apache.

so i just turned those off. but before i could do so, i found a bunch of issues in the puppet hooks, which were effectively completely broken. so i cleaned that up.

now we're not logging uwsgi requests anymore at all, with the disable-logging option.

i've also rotated the system logs:

logrotate -f /etc/logrotate.d/syslog-ng
logrotate -f /etc/logrotate.d/syslog-ng

It was done twice to compress the daemon.log file properly.

now we're at:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       9.8G  7.5G  1.9G  81% /

we have burned through 1.4GB in the last 24h, so that should give us at least that, assuming the situation is unchanged. but we've at least halved that burn rate, so i estimate we should have at least 48h of disk space, assuming rotation can't handle the load.

i'm going to assume we can look at this again on monday safely. the alert itself was solved about 15 minutes ago:

22:32:37 -ALERTOR1:#tor-alerts- DiskWillFillSoon [resolved] Disk / on lists-01.torproject.org is almost full

(UTC-5 again)

Note that this is probably a sign that a crawler or attacker is abusing the service. For now we're treating this as outside of this incident, but it could explain the performance problems with the server as well (#41957 (closed)).

changed the severity to High - S2

added Needs Review label and removed Next label

changed the incident status to Acknowledged

added an incident timeline event

edited the event time/date on incident timeline event

this looks like it settled down, we can handle this issue again if it happens.

closed

changed the incident status to Resolved by closing the incident

changed the incident status to Resolved

mentioned in commit wiki-replica@f7523ef8

lists-01 disk filling up abruptly

Child items 0

Activity