document logs and email routing

howto/gitlab.md

@@ -620,6 +620,31 @@ To limit this to `job.log`, of course, you can do:
 
     find -name "job.log" -mtime +14 -print0 | du --files0-from=- -c -h | tee find-mtime+14-joblog-du.log
 
+### Email routing
+
+Incoming email get routed through either eugeni or the submission
+service, then end up on the Postfix server on `gitlab-02`, and from
+there, to a dovecot mailbox. You can use `postfix-trace` to confirm
+the message correctly ended up there.
+
+Normally, GitLab should be picking mails from the mailbox
+(`/srv/mail/git@gitlab.torproject.org/Maildir/`) regularly, and
+deleting them when done. If that is not happening, look at the
+mailroom logs:
+
+    tail -f /var/log/gitlab/mailroom/mail_room_json.log | jq -c
+
+A working run will look something like this:
+
+```
+{"severity":"INFO","time":"2022-08-29T20:15:57.734+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Processing started"}
+{"severity":"INFO","time":"2022-08-29T20:15:57.734+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"uid":7788,"action":"asking arbiter to deliver","arbitrator":"MailRoom::Arbitration::Redis"}.734+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Getting new messages","unread":{"count":1,"ids":[7788]},"to_be_delivered":{"count":1,"ids":[7788]}}ext":{"email":"git@gitlab.torproject.org","name":"inbox"},"uid":7788,"action":"sending to deliverer","deliverer":"MailRoom::Delivery::Sidekiq","byte_size":4162}","delivery_method":"Sidekiq","action":"message pushed"}
+{"severity":"INFO","time":"2022-08-29T20:15:57.744+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Processing started"}
+{"severity":"INFO","time":"2022-08-29T20:15:57.744+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Getting new messages","unread":{"count":0,"ids":[]},"to_be_delivered":{"count":0,"ids":[]}}0","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Idling"}
+```
+
+Emails should be processed every minute or so.
+
 ## Disaster recovery
 
 In case the entire GitLab machine is destroyed, a new server should be
@@ -1163,8 +1188,27 @@ the production server.
 
 ## Logs and metrics
 
-<!-- TODO: where are the logs? how long are they kept? any PII? -->
-<!-- what about performance metrics? same questions -->
+GitLab keeps an extensive (excessive?) amount of logs, in
+`/var/log/gitlab`, which includes PII, including IP addresses. 
+
+To see live logs, you can type the handy command:
+
+    gitlab-ctl tail
+
+... but that is sort of like drinking from a fire hose. You can
+inspect the logs of a specific component by passing it as an argument,
+for example to inspect the mail importer:
+
+    gitlab-ctl tail mailroom
+
+Each component is in his own directory, so the equivalent to the above
+is:
+
+    tail -f /var/log/gitlab/mailroom/{current,mail_room_json.log}
+
+Notice how both regular and JSON logs are kept.
+
+Logs seem to be kept for a month.
 
 ## Backups