inconsistent systemd-journald storage policies
It seems we don't have a consistent journald storage policy. By default, before bullseye, the default is
auto, which means that we have journald persistent storage (in
/var/log/journal) if the directory exists. but in bullseye and later, this is enabled by default.
on the other hand, new machines created in bullseye do have the persistent journal enabled, and I have just enabled persistent journaling on polyanthum for #40414 (closed).
we need to decide what we do about this as part of the buster upgrade.
right now, the situation is as follows:
(4) chi-node-[05,08].torproject.org,polyanthum.torproject.org,tb-pkgstage-01.torproject.org ----- OUTPUT of 'file /var/log/journal' ----- /var/log/journal: setgid, directory
that is, 4 servers have persistent journals. all other servers do not have /var/log/journal, so, in theory, should not have persistent journals as well.
do note that polyanthum explicitely had that disabled in
/etc/systemd/journald.conf.d/volatile.conf, but this wasn't in puppet so I couldn't trace why this was done so. this needs to be revised, along with the journald retention policies.
i'm also worried about systemd's general lack of attention to PII retention. in journal ip anonymization #2447 for example, maintainers have showed they do not want to implement log mangling to remove PII.
and on the other hand, persistent journals are required for some operations. for example, user journals need it. there is a patch to support runtime (volatile) user journals #12263 , but it's been stalled for years.
one way to resolve this would be to enable persistent journaling, but keep it in a tmpfs.
we should also consider the duplication between journald and our regular syslog and whether we want to completely ditch one or the other.