GetTor is not replying to emails

Users from Iran reported that GetTor is not replying to them. I have tried myself and I didn't get a reply too.

assigned to @meskio

cc @meskio

anarcat and I verified that the mails are coming in to eugeni properly, and getting forwarded to gettor@rdsys-frontend properly, and delivered to an inbox on rdsys-frontend properly.

Past that, we're not really sure what the service is supposed to do with it.

We looked around for a survival guide for rdsys-frontend but did not find one. :(

Yes, I have to write a survival guide, I totally forgot, sorry.

assigned to @meskio

Restarting the service seems to fixed it:

Oct 06 08:46:49 rdsys-frontend-01 rdsys-distributors[1377043]: 2022/10/06 08:46:49 fetch 1059 messages from the imap server

I think its being 2 days without sending email, this is the last imap fetch I find the logs:

Oct 04 15:24:56 rdsys-frontend-01 rdsys-distributors[1182515]: 2022/10/04 15:24:56 fetch 3 messages from the imap server

Which is ~3hours after the last restart:

Oct 04 11:28:35 rdsys-frontend-01 systemd[464]: Started Rdsys's gettor distributor process.

I don't see any errors in the logs.

This is weird, in my tests it was responding email several days after I restarted, maybe I broke something in my latest fixes. I'm investigating.

1059 messages in 2 days, in a fast look in prometheus I think 2/3 of those produced a help email and 1/3 did provide proper response with links. So maybe a big part of those are actually spam.

I need to create a graphana dashboard for it.

moved from tpo/anti-censorship/gettor-project/gettor#90 (moved)

added Doing label

added Gettor label

I can't see any clear bug introduced in the recent code or any obvious bug on the email processing code. I'm adding some more log output and I'll be watching the logs to see if hapens again.

I updated the imap library we use, just in case was a bug there (!54 (merged)). They've included in the main library the extensions we were using to idle on the imap server.

rdsys survival guide updated: https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Rdsys-Survival-Guide

It might be possible to improve it a lot, but at least now we have something about it.

changed due date to October 13, 2022

added Next label and removed Doing label

Thanks!

When you restart it, does it automatically go back and handle the mails it hadn't yet handled? That is, did the outage result in dropped replies, or just delayed replies?

Delayed replies. After the restart it does reply to all the emails that were not handled over that period of time (1059).

mentioned in issue tpo/tpa/team#40915 (closed)

The service is still working without problems since I restarted it 5 days ago. I don't see any problems in the logs.

I'm giving it 10 more days, if it doesn't fail again I'll close this issue and hope that is being fixed by the changes I've done. We'll hold the retirement of gettor-01 until then (tpo/tpa/team#40915 (closed))

changed due date to October 25, 2022

I have created an alert so we'll get notified if no email arrives to gettor in 1 hour: tpo/tpa/prometheus-alerts!14 (merged)

mentioned in merge request tpo/tpa/prometheus-alerts!14 (merged)