Set up a loghost

Just do it.

added component::internal services/tor sysadmin team in Legacy / Trac owner::tpa in Legacy / Trac priority::medium in Legacy / Trac resolution::fixed in Legacy / Trac severity::normal in Legacy / Trac status::closed in Legacy / Trac type::task in Legacy / Trac labels

how do we do that? what's a loghost? just a syslog central server that receives everything? what about anonymisation? that's done on the remote hosts? what about non-syslog logs like apache?

if we go the prometheus route for monitoring, we are also probably going to use grafana for graphing/trending, and they have an interesting project called loki to aggregate and parse logs that we might want to look into.

another common alternative to syslog is logstash which, combined with ElasticSearch and Kibana makes for the acronym "ELK" that's commonly deployed as a stack, with Granafa sometimes replacing Kibana...

so just to formalize this, here are the questions we should answer first here:

1. what is the purpose of setting up a log host? I can imagine a few reasons myself, but would prefer if that was stated in the request
2. do we use syslog or something else that's more searchable? (ELK, Loki, etc)
3. do we still log on the individual hosts? or do we forward all the logs on the central server and keep nothing locally? (because that could break stuff like the postfix exporter)
4. what about non-syslog logs? should those be centralized as well?
5. which hardware?

I'd be down for setting up something like this and, in the infrared working groups, there's been talk of looking at this problem specifically. I know a fellow sysadmin has been experimenting with "log forwarding" that is, a simple syslogd running on a central server, and all other syslogd forward their logs to the server, and write nothing locally. They are worried about disks being overloaded with I/O and things relying on logs on the remote servers being present, but so far things go well.

weasel provided some answers on IRC, here's what I gathered:

1. purpose: redundancy and security (if a host crashes or is compromised, we have traces somewhere that's more realtime than backups)
2. syslogd is fine, DSA has syslog-ng code that does this
3. we still log on the nodes
4. no non-syslog stuff for now
5. hetzner, probably like prometheus?

we have loghost01 now. Yay.

Trac:
Resolution: N/A to fixed
Status: new to closed

thanks!

closed

moved from legacy/trac#29393 (moved)

added Task label and removed 1 deleted label

added Sysadmin label and removed 1 deleted label