I increased the number of vCPUs from 2 to 4 in an attempt to see if more resources could help here, but it did not. The load remains too high to service requests in a timely manner.
I have suspected exonerator to be a target for a bit. I have also noticed it has caused issues on metrics.tpo too since the web interface from metrics calls exonerator json api. Last time I checked though it seemed like apache was not getting a lot of requests, so I am left wondering what is causing this.
it's odd: even before the postgresql upgrade, PosgreSQL 13 got killed by the OOM:
Jan 31 12:14:17 materculae systemd[1]: postgresql@13-main.service: A process of this unit has been killed by the OOM killer.
the last WAL file written by 13 was:
Jan 31 13:41:18 materculae pg-backup-file[55212]: Archiving to torbackup@bungei.torproject.org: (materculae,main.WAL.00000001000005D300000097,16777216,da72c>
... then 15 took over:
Jan 31 15:32:15 materculae pg-backup-file[7306]: Archiving to torbackup@bungei.torproject.org: (materculae,main.WAL.00000001000005D3000000AE,16777216,ed18ce>
... and got killed in turn:
Feb 01 12:30:49 materculae systemd[1]: postgresql@15-main.service: A process of this unit has been killed by the OOM killer.
@lavamind bumped the CPU count on Jan 30th at 22:00UTC, and believes that might be the cause of the OOMs. the apt full-upgrade to bookworm occured on 2024-01-31 13:45:24, but it looks from Grafana that the memory usage "cliff" started before that, at around 4:15 on that day:
I had another look at ExoneraTor today while looking into why meronense was filling up its disk. It seems that the exonerator-web service was stopped on materculae, causing the metrics-web service on meronense to spam logs like:
ERROR o.t.m.e.ExoneraTorServlet:319 Backend query failed.java.io.IOException: Server returned HTTP response code: 503 for URL: https://exonerator.torproject.org/query.json?ip=[...]
Seeing that the service in question was controlled by a @reboot cron entry to materculae, I converted it to a systemd user service in the exonerator-web account so that we can more easily detect when it stops or fails. Starting it up again stopped the log spamming on meronense.
So seeing it's likely Denyal of service attacks I want to list a few scripts I made for my own server that was attacked by bot swarms(in my case it was SSH brute force but they likely also do DDoS ones)
Here is the github repository including logs,block scripts and IPs for Russian/Chinese botnets and
Here the IPs of set botnets that survived till now
It might be the same Ips if possible try blocking all Ip ranges belonging to Alicloud/AS45102(most of them are from this host) these are all of them
103.183.154.0/23
103.81.186.0/23
110.76.21.0/24
110.76.23.0/24
116.251.64.0/18
139.95.0.0/23
139.95.10.0/23
139.95.12.0/23
139.95.14.0/23
139.95.16.0/23
This issue has been waiting for information two
weeks or more. It needs attention. Please take care of
this before the end of
2024-03-14. ~"Needs
Information" tickets will be moved to the Icebox after
that point.
(Any ticket left in Needs Review, Needs Information, Next, or Doing
without activity for 14 days gets such
notifications. Make a comment describing the current state
of this ticket and remove the Stale label to fix this.)
To make the bot ignore this ticket, add the bot-ignore label.