ipsec VPN generates gigantic logs
Serious yak shaving night...
To try to silence this seemingly innocuous warning:
/etc/cron.daily/logrotate:
error: Compressing program wrote following message to stderr when compressing log /var/log/syslog.1:
gzip: stdin: file size changed while zipping
... I have looked at the logrotate configuration deployed through Puppet, and it seems slightly out of date compared to the one available in stretch. This is the configuration left over from the stretch upgrade on eugeni, for example:
/var/log/syslog
{
rotate 7
daily
missingok
notifempty
delaycompress
compress
postrotate
invoke-rc.d syslog-ng reload > /dev/null
endscript
}
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
/var/log/error
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
invoke-rc.d syslog-ng reload > /dev/null
endscript
}
Out of those, we're not doing the syslog-ng reload
, the delaycompress
, notifempty
and each logfile is in a separate block which makes it harder to read. So I looked at doing the postrotate action, but then I realized it was happening on the syslog logfile which is correctly reloaded. so then i figured the delaycompress
might be the bit missing.
but before enabling that blindly, I figured I would check if this would blow up the disk space on a server. how to do that you ask? well with our shiny new Cumin tool of course:
anarcat@curie:~(master)$ cumin -p 0 '*' 'for log in /var/log/*.log ; do if [ `du -b "$log" | cut -f1` -gt 1000000000 ] ; then echo "logfile $log larger than 1GB"; exit 1 ; fi; done'
74 hosts will be targeted:
alberti.torproject.org,arlgirdense.torproject.org,bracteata.torproject.org,brulloi.torproject.org,build-arm-[None..None](../compare/None...None).torproject.org,build-x86-[None..None](../compare/None...None).torproject.org,bungei.torproject.org,carinatum.torproject.org,cdn-backend-sunet-01.torproject.org,chamaemoly.torproject.org,chiwui.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crispum.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org,cupani.torproject.org,dictyotum.torproject.org,eugeni.torproject.org,fallax.torproject.org,forrestii.torproject.org,gayi.torproject.org,getulum.torproject.org,gitlab-01.torproject.org,henryi.torproject.org,hetzner-hel1-[None..None](../compare/None...None).torproject.org,hetzner-nbg1-01.torproject.org,hyalinum.torproject.org,iranicum.torproject.org,kvm[None..None](../compare/None...None).torproject.org,listera.torproject.org,macrum.torproject.org,majus.torproject.org,materculae.torproject.org,meronense.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,nova.torproject.org,nutans.torproject.org,omeiense.torproject.org,oo-hetzner-03.torproject.org,opacum.torproject.org,orestis.torproject.org,oschaninii.torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,rouyi.torproject.org,rude.torproject.org,savii.torproject.org,saxatile.torproject.org,scw-arm-ams-01.torproject.org,scw-arm-par-01.torproject.org,staticiforme.torproject.org,subnotabile.torproject.org,textile.torproject.org,togashii.torproject.org,troodi.torproject.org,unifolium.torproject.org,vineale.torproject.org,web-cymru-01.torproject.org,web-hetzner-01.torproject.org
Confirm to continue [y/n]? y
|██████████████▌ | 12% (9/74) [00:47<08:25, 7.78s/hosts]
===== NODE GROUP ===== |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
(3) build-arm-[None..None](../compare/None...None).torproject.org |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
----- OUTPUT of 'for log in /var/...xit 1 ; fi; done' ----- |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
Connection timed out during banner exchange |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
===== NODE GROUP ===== |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
(5) hetzner-hel1-01.torproject.org,kvm4.torproject.org,macrum.torproject.org,textile.torproject.org,unifolium.torproject.org |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
----- OUTPUT of 'for log in /var/...xit 1 ; fi; done' ----- |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
logfile /var/log/daemon.log larger than 1GB |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
===== NODE GROUP ===== |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
(1) hyalinum.torproject.org |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
----- OUTPUT of 'for log in /var/...xit 1 ; fi; done' ----- |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
ssh: Could not resolve hostname hyalinum.torproject.org: No address associated with hostname |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
================ PASS |████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 88% (65/74) [00:52<00:07, 1.23hosts/s]
FAIL |██████████████▌ | 12% (9/74) [00:52<08:25, 7.78s/hosts]
12.2% (9/74) of nodes failed to execute command 'for log in /var/...xit 1 ; fi; done': build-arm-[None..None](../compare/None...None).torproject.org,hetzner-hel1-01.torproject.org,hyalinum.torproject.org,kvm4.torproject.org,macrum.torproject.org,textile.torproject.org,unifolium.torproject.org
87.8% (65/74) success ratio (>= 0.0% threshold) for command: 'for log in /var/...xit 1 ; fi; done'.: alberti.torproject.org,arlgirdense.torproject.org,bracteata.torproject.org,brulloi.torproject.org,build-x86-[None..None](../compare/None...None).torproject.org,bungei.torproject.org,carinatum.torproject.org,cdn-backend-sunet-01.torproject.org,chamaemoly.torproject.org,chiwui.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crispum.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org,cupani.torproject.org,dictyotum.torproject.org,eugeni.torproject.org,fallax.torproject.org,forrestii.torproject.org,gayi.torproject.org,getulum.torproject.org,gitlab-01.torproject.org,henryi.torproject.org,hetzner-hel1-[None..None](../compare/None...None).torproject.org,hetzner-nbg1-01.torproject.org,iranicum.torproject.org,kvm5.torproject.org,listera.torproject.org,majus.torproject.org,materculae.torproject.org,meronense.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,nova.torproject.org,nutans.torproject.org,omeiense.torproject.org,oo-hetzner-03.torproject.org,opacum.torproject.org,orestis.torproject.org,oschaninii.torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,rouyi.torproject.org,rude.torproject.org,savii.torproject.org,saxatile.torproject.org,scw-arm-ams-01.torproject.org,scw-arm-par-01.torproject.org,staticiforme.torproject.org,subnotabile.torproject.org,togashii.torproject.org,troodi.torproject.org,vineale.torproject.org,web-cymru-01.torproject.org,web-hetzner-01.torproject.org
87.8% (65/74) success ratio (>= 0.0% threshold) of nodes successfully executed all commands.: alberti.torproject.org,arlgirdense.torproject.org,bracteata.torproject.org,brulloi.torproject.org,build-x86-[None..None](../compare/None...None).torproject.org,bungei.torproject.org,carinatum.torproject.org,cdn-backend-sunet-01.torproject.org,chamaemoly.torproject.org,chiwui.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crispum.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org,cupani.torproject.org,dictyotum.torproject.org,eugeni.torproject.org,fallax.torproject.org,forrestii.torproject.org,gayi.torproject.org,getulum.torproject.org,gitlab-01.torproject.org,henryi.torproject.org,hetzner-hel1-[None..None](../compare/None...None).torproject.org,hetzner-nbg1-01.torproject.org,iranicum.torproject.org,kvm5.torproject.org,listera.torproject.org,majus.torproject.org,materculae.torproject.org,meronense.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,nova.torproject.org,nutans.torproject.org,omeiense.torproject.org,oo-hetzner-03.torproject.org,opacum.torproject.org,orestis.torproject.org,oschaninii.torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,rouyi.torproject.org,rude.torproject.org,savii.torproject.org,saxatile.torproject.org,scw-arm-ams-01.torproject.org,scw-arm-par-01.torproject.org,staticiforme.torproject.org,subnotabile.torproject.org,togashii.torproject.org,troodi.torproject.org,vineale.torproject.org,web-cymru-01.torproject.org,web-hetzner-01.torproject.org
This might not be very easy to read, but the important bit is this:
(5) hetzner-hel1-01.torproject.org,kvm4.torproject.org,macrum.torproject.org,textile.torproject.org,unifolium.torproject.org
----- OUTPUT of 'for log in /var/...xit 1 ; fi; done' -----
|logfile /var/log/daemon.log larger than 1GB
So I looked at the first one of those (hetzner-hel1-01) and lo and behold, the daemon.log
is gigantic:
1,4G /var/log/daemon.log
I looked into the file briefly and it looks like a lot of information from ipsec. But before I start shaving another yak, I figured I would just file this as a ticket to document how far I went and let this one rest for a while.
(I did end up setting delaycompress after doing more investigations in Prometheus about free disk space, but that's documented in the tor-puppet commit 44f86c7d and previous.)