Create an inventory of running services and make quick health checks

changed due date to April 30, 2021

@gk @gaba Let me know if you think anything is missing from this list.

changed the description

added Task label

added Doing label

It seems complete to me. I'm tagging @anarcat in case he can see if anything is missing.

changed the description

I've checked the latest descriptors/results/indexes for all the services on both CollecTors, and it looks like all the jobs are running. I've not yet got access to login to the hosts to check the logs but they appear to be functioning just fine.

For Onionoo, both the relays and bridges timestamp are within the expected range indicating that the updater is running. Again, I do not have access to log in so this will only be able to tell me about one of the updaters and one of the frontend caches (there are two of each).

For metrics-web statistics, all the modules have data within the expected recent date range.

It seems complete to me. I'm tagging @anarcat in case he can see if anything is missing.

i have no idea. :) i can barely keep an up to date list of all our services here:

https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service

what I would really like is if people started using that list as the canonical service list, even internally. for example, i don't think i have "Relay Search" in my list...

then we could add the team as the "maintainer" so that it would be easy to grep for them.

feel free to submit a pr for that! :)

Relay Search is actually a part of metrics-web in the way it's deployed, except that it's a completely different tech stack only bolted on to the JAR/WAR build+release step independent of everything else in metrics-web. It probably makes sense to talk about it as a service, because people have traditionally talked about it as a service and it used to be completely separate (Atlas), but the link would probably go into the metrics-web documentation. Either way, yes, a canonical list of services is a thing that should exist and I'll be happy to submit PRs there as documentation solidifies.

marked the checklist item CollecTor as completed

marked the checklist item collector.torproject.org as completed

marked the checklist item collector2.torproject.org as completed

CollecTor has a reasonable amount of disk space for the foreseeable future and is not showing anything worrying in the logs.

@gk @gaba Let me know if you think anything is missing from this list.

From what I can see this list looks good to me. (I am, however, not that well-versed in the metrics services jungle...)

CollecTor has a reasonable amount of disk space for the foreseeable future and is not showing anything worrying in the logs.

it actually triggered a few nagios warnings recently, now that you mention it. seems like it kicks a WARNING every day or so, e.g. yesterday:

From: nagios@hetzner-hel1-01.torproject.org
Subject: ** PROBLEM Service Alert: colchicifolium/disk usage - all is WARNING **
To: anarcat+rapports@orangeseeds.org
Date: Sun, 11 Apr 2021 03:55:05 +0000

***** Icinga *****

Notification Type: PROBLEM

Service: disk usage - all
Host: colchicifolium
Address: 49.12.57.145
State: WARNING

Date/Time: Sun Apr 11 03:55:05 UTC 2021

Additional Info:

DISK WARNING - free space: / 6390 MB (67% inode=85%): /dev 3973 MB (100% inode=99%): /dev/shm 3989 MB (99% inode=99%): /run 717 MB (89% inode=99%): /run/lock 5 MB (100% inode=99%): /run/user/1550 797 MB (100% inode=99%): /sys/fs/cgroup 3989 MB (100% inode=99%): /tmp 3950 MB (99% inode=99%): /srv 29416 MB (8% inode=86%): /home 29416 MB (8% inode=86%): /var/tmp 6390 MB (67% inode=85%): /run/user/2098 797 MB (100% inode=99%):

another example, this time a CRITICAL, a week earlier:

From: nagios@hetzner-hel1-01.torproject.org
Subject: ** PROBLEM Service Alert: colchicifolium/disk usage on /srv is CRITICAL **
To: anarcat+rapports@orangeseeds.org
Date: Wed, 07 Apr 2021 06:54:15 +0000

***** Icinga *****

Notification Type: PROBLEM

Service: disk usage on /srv
Host: colchicifolium
Address: 49.12.57.145
State: CRITICAL

Date/Time: Wed Apr 7 06:54:15 UTC 2021

Additional Info:

DISK CRITICAL - free space: /srv 15887 MB (4% inode=86%):

from what I understand, garbage collection on this was improved quite a bit and those actually never run out. maybe i should just bump up the disk allocation on this one?

removed due date

@irl can you add the metrics.torproject.org website to it?

We will also need documentation on how is deployed and how to troubleshoot any issues there.

mentioned in issue #15 (closed)

Create an inventory of running services and make quick health checks

Child items ...

Activity