Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-22T11:55:46Zhttps://gitlab.torproject.org/legacy/trac/-/issues/32908Only re-build missing incremental .mar files2020-06-22T11:55:46ZGeorg KoppenOnly re-build missing incremental .mar filesSometimes we only want to re-build a subset of incremental .mar files but that does not seem to be possible right now.
The way this could handled is like proceed with regular artifact builds: only the missing parts are (re-)built. Thus,...Sometimes we only want to re-build a subset of incremental .mar files but that does not seem to be possible right now.
The way this could handled is like proceed with regular artifact builds: only the missing parts are (re-)built. Thus, for incremental .mar files this would mean we'd delete the ones we want to rebuilt and `make incrementals-release` (assuming we are in a release build process) would be smart enough to just re-build the deleted ones.https://gitlab.torproject.org/legacy/trac/-/issues/34423Implement GetTor for mobile users2020-06-21T18:06:15ZCecylia BocovichImplement GetTor for mobile usersWhat happens if Tor Browser downloads through traditional app stores are blocked in certain regions?
Right now GetTor only distributes for windows, osx, and linux. If we upload .apks to our link download providers, is this a usable way ...What happens if Tor Browser downloads through traditional app stores are blocked in certain regions?
Right now GetTor only distributes for windows, osx, and linux. If we upload .apks to our link download providers, is this a usable way to install Tor Browser on Android?https://gitlab.torproject.org/legacy/trac/-/issues/34350Stop logging all successful databse queries in GetTor2020-06-21T18:06:14ZCecylia BocovichStop logging all successful databse queries in GetTorThis is another log message that isn't helpful and fills up our logs. Here's a patch: https://gitlab.torproject.org/tpo/anti-censorship/gettor-project/gettor/-/merge_requests/12This is another log message that isn't helpful and fills up our logs. Here's a patch: https://gitlab.torproject.org/tpo/anti-censorship/gettor-project/gettor/-/merge_requests/12Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/legacy/trac/-/issues/34300Ensure GetTor's email unit tests are properly formatted2020-06-21T18:06:14ZCecylia BocovichEnsure GetTor's email unit tests are properly formattedFrom #34286:
[comment:3 phw]:
> Looks good to me!
>
> On a slightly related note: I believe that an email's body is supposed to be separated by two (rather than one) newlines from its header. GetTor's unit tests are using only one (and...From #34286:
[comment:3 phw]:
> Looks good to me!
>
> On a slightly related note: I believe that an email's body is supposed to be separated by two (rather than one) newlines from its header. GetTor's unit tests are using only one (and mix \n with \r\n). Python's email module is also confused by this and thinks that the body is part of the `To` field:
>
> {{{
> In [1]: from email import message_from_string
> In [3]: m=message_from_string("From: MAILER-DAEMON@mx1.riseup.net\nSubject: Undelivered Mail Returned to Sender\r\nTo: gettor@torproject.org\n osx en\n")
> In [6]: m.items()
> Out[6]:
> [('From', 'MAILER-DAEMON@mx1.riseup.net'),
> ('Subject', 'Undelivered Mail Returned to Sender'),
> ('To', 'gettor@torproject.org\n osx en')]
> }}}
>
> This seems like something we should fix.https://gitlab.torproject.org/legacy/trac/-/issues/34286gettor appears to be in an email loop war with a .sk address2020-06-21T18:06:13ZRoger Dingledinegettor appears to be in an email loop war with a .sk addressI happened to be looking at eugeni's mail.log for other debugging, and saw that approximately 25% of the lines in mail.log contain the string gettor.
(Yesterday, eugeni's postfix had 460k lines in it, and 101k of them said "gettor" in t...I happened to be looking at eugeni's mail.log for other debugging, and saw that approximately 25% of the lines in mail.log contain the string gettor.
(Yesterday, eugeni's postfix had 460k lines in it, and 101k of them said "gettor" in them. Today in the first hour or so, it's 7k out of 25k.)
Does gettor get into fights with external addresses, where it replies to the bounce, gets another bounce and replies to that, etc?
There are probably smart guidelines for avoiding mail loop wars, like not answering names that start with mailer-domain, checking for the presence of an X-Something-Something header, or rate limiting responses to a given address.
And this is a great case where unifying how bridgedb handles its email answers, and how gettor does it, will save a lot of headache.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/legacy/trac/-/issues/34253GetTor should set In-Reply-To when responding to email2020-06-21T18:06:12ZPhilipp Winterphw@torproject.orgGetTor should set In-Reply-To when responding to emailGetTor currently doesn't set the `In-Reply-To` header when responding to an email. That breaks threading in the user's mailbox and it also makes it slightly more difficult to test our autoresponder.
It's not high priority but let's add ...GetTor currently doesn't set the `In-Reply-To` header when responding to an email. That breaks threading in the user's mailbox and it also makes it slightly more difficult to test our autoresponder.
It's not high priority but let's add the `In-Reply-To` header at some point.https://gitlab.torproject.org/legacy/trac/-/issues/34062Gracefully shutdown services in GetTor2020-06-21T18:06:11ZCecylia BocovichGracefully shutdown services in GetTorI get these errors when I shutdown or restart GetTor. Seems like something we can easily implement:
```
2020-04-29T15:24:03+0000 [gettor#debug] SERVICE:: Calling shutdown on sendmail
2020-04-29T15:24:03+0000 [twisted.internet.defer#crit...I get these errors when I shutdown or restart GetTor. Seems like something we can easily implement:
```
2020-04-29T15:24:03+0000 [gettor#debug] SERVICE:: Calling shutdown on sendmail
2020-04-29T15:24:03+0000 [twisted.internet.defer#critical] Unhandled error in Deferred:
2020-04-29T15:24:03+0000 [twisted.internet.defer#critical]
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 428, in fireEvent
result = callable(*args, **kwargs)
File "/usr/lib/python3/dist-packages/twisted/application/service.py", line 296, in stopServ
ice
l.append(defer.maybeDeferred(service.stopService))
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 151, in maybeDeferred
result = f(*args, **kw)
File "/usr/lib/python3/dist-packages/twisted/application/service.py", line 296, in stopServ
ice
l.append(defer.maybeDeferred(service.stopService))
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 151, in maybeDeferred
result = f(*args, **kw)
File "/srv/gettor.torproject.org/home/gettor/gettor/services/__init__.py", line 60, in stop
Service
self.instance.shutdown()
builtins.AttributeError: 'Sendmail' object has no attribute 'shutdown'
```https://gitlab.torproject.org/legacy/trac/-/issues/34061Reduce amount of GetTor logging2020-06-21T18:06:11ZCecylia BocovichReduce amount of GetTor loggingWe're logging at a very high level (looks like at debug), and outputting frequent successes when we only really need to be logging errors.
For example: a successfully processed email in `log/email_parser.log` outputs:
```
2020-04-27 23:...We're logging at a very high level (looks like at debug), and outputting frequent successes when we only really need to be logging errors.
For example: a successfully processed email in `log/email_parser.log` outputs:
```
2020-04-27 23:18:53+0000 [-] Log opened.
2020-04-27 23:18:53+0000 [process email] New email request received.
2020-04-27 23:18:53+0000 [process email] Reading new email.
2020-04-27 23:18:53+0000 [-] Database query executed successfully.
2020-04-27 23:18:53+0000 [email parser] Building email message from string.
2020-04-27 23:18:53+0000 [email parser] Normalizing and validating FROM email address.
2020-04-27 23:18:53+0000 [email parser] Email address normalized and validated.
2020-04-27 23:18:53+0000 [email parser] Request from [hid]
2020-04-27 23:18:53+0000 [email parser] Found request for links.
2020-04-27 23:18:53+0000 [-] Database query executed successfully.
2020-04-27 23:18:53+0000 [-] Main loop terminated.
2020-04-27 23:18:53+0000 [process email] Email request processed.
```
and in `log/gettor.log`:
```
2020-04-29T14:46:51+0000 [gettor#info] Getting links for windows is.
2020-04-29T14:46:51+0000 [-] Database query executed successfully.
2020-04-29T14:46:51+0000 [gettor#info] Sending links to [hid].
2020-04-29T14:46:51+0000 [gettor#debug] Creating plain text email
2020-04-29T14:46:51+0000 [gettor#debug] Calling asynchronous sendmail.
2020-04-29T14:46:51+0000 [twisted.mail.smtp.ESMTPSenderFactory#info] Starting factory <twisted.mail.smtp.ESMTPSenderFactory object at 0x7f0bba74b780>
2020-04-29T14:46:51+0000 [gettor#info] Email sent successfully.
2020-04-29T14:46:51+0000 [twisted.mail.smtp.ESMTPSenderFactory#info] Stopping factory <twisted.mail.smtp.ESMTPSenderFactory object at 0x7f0bba74b780>
2020-04-29T14:46:51+0000 [-] Database query executed successfully.
2020-04-29T14:46:51+0000 [-] Database query executed successfully.
```
We could reduce this to one log message at most. Especially since this information *should* be captured in the stats database.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/legacy/trac/-/issues/34060Errors thrown in Gettor if "To:" address field doesn't match gettor@tp.o2020-06-21T18:06:11ZCecylia BocovichErrors thrown in Gettor if "To:" address field doesn't match gettor@tp.oI was checking the logs for errors and found a bunch of the following failures:
```
2020-04-29 07:57:51+0000 [email parser] Error while parsing email content: [Failure instance: Traceba
ck: <class 'KeyError'>: 'command'
/usr/lib...I was checking the logs for errors and found a bunch of the following failures:
```
2020-04-29 07:57:51+0000 [email parser] Error while parsing email content: [Failure instance: Traceba
ck: <class 'KeyError'>: 'command'
/usr/lib/python3/dist-packages/twisted/internet/defer.py:311:addCallbacks
/usr/lib/python3/dist-packages/twisted/internet/defer.py:654:_runCallbacks
/usr/lib/python3/dist-packages/twisted/internet/defer.py:1613:unwindGenerator
/usr/lib/python3/dist-packages/twisted/internet/defer.py:1529:_cancellableInlineCallbacks
--- <exception caught here> ---
/usr/lib/python3/dist-packages/twisted/internet/defer.py:1418:_inlineCallbacks
/srv/gettor.torproject.org/home/gettor/gettor/parse/email.py:250:parse_callback
].
```
I noticed this is caused by `parse` returning an empty request [here](https://gitweb.torproject.org/gettor.git/tree/gettor/parse/email.py#n220) which only happens if the `"To:"` address doesn't match `gettor@torproject.org` exactly. After doing more looking, I found the following mismatched addresses:
- "To:" address is just blank
- gettor+[lang code]@torproject.org (e.g., gettor+en@torproject.org).
- [random user]@gmail.com
- [user]@[random domain].[random tld]
(where random = no known connection to gettor, not cryptographically random :))
For the blank and random addresses, I wonder how this is happening. Perhaps we're relying on information that's not consistently configured correctly on user email clients?
For addresses of the form gettor+[lang code]@torproject.org, it looks like gettor used to work by accepting emails of this form to determine localization (see https://twitter.com/get_tor/status/754126179506982912). Perhaps we shouldn't be throwing these out, even though we no longer do localization this way. We could use these language codes once we get around to localizing gettor messages as an optional step.https://gitlab.torproject.org/legacy/trac/-/issues/34058Make sure gettor logs are scrubbed of personal info2020-06-21T18:06:10ZCecylia BocovichMake sure gettor logs are scrubbed of personal infoSome error messages in the gettor logs contain personal info from gettor queries. We should make sure we're not logging this information (particularly for email addresses in SMTP errors).Some error messages in the gettor logs contain personal info from gettor queries. We should make sure we're not logging this information (particularly for email addresses in SMTP errors).https://gitlab.torproject.org/legacy/trac/-/issues/34035Dry out GetTor's sendmail function2020-06-21T18:06:10ZCecylia BocovichDry out GetTor's sendmail functionThis patch refactors the sendmail function in GetTor to avoid code duplication.
https://gitlab.torproject.org/torproject/anti-censorship/gettor-project/gettor/-/merge_requests/6This patch refactors the sendmail function in GetTor to avoid code duplication.
https://gitlab.torproject.org/torproject/anti-censorship/gettor-project/gettor/-/merge_requests/6Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/legacy/trac/-/issues/12802BridgeDB needs Nagios checks for the Email Distributor2020-06-21T18:06:10ZIsis LovecruftBridgeDB needs Nagios checks for the Email DistributorBridgeDB needs Nagios checks that the Email Distributor is working. The best way to do this would be to send an email to bridges@torproject.org which say "get help".BridgeDB needs Nagios checks that the Email Distributor is working. The best way to do this would be to send an email to bridges@torproject.org which say "get help".Philipp Winterphw@torproject.orgPhilipp Winterphw@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/34028Monitor GetTor's email autoresponder2020-06-21T18:06:09ZPhilipp Winterphw@torproject.orgMonitor GetTor's email autoresponderThe GetTor process occasionally dies without anyone noticing. See #34027 for the latest incident. We should set up a script that periodically emails the autoresponder and raises an alert if it doesn't get a response.
We already created ...The GetTor process occasionally dies without anyone noticing. See #34027 for the latest incident. We should set up a script that periodically emails the autoresponder and raises an alert if it doesn't get a response.
We already created [such a script](https://github.com/NullHypothesis/bridgedb/blob/enhancement/12802/scripts/nagios-email-check) for BridgeDB as part of #12802. Here's how it works:
0. We created a new Gmail address that's used by this script to email the autoresponder.
1. Polyanthum (the host BridgeDB runs on) runs the script in a cron job every three hours.
2. The script sends an email to bridges@torproject.org and, after waiting for a minute, checks for a response.
3. Depending on if there's a response, the script writes a status code to disk, which is read by Nagios.
4. Nagios should then alert us if the script's output says that BridgeDB's autoresponder is offline.
It shouldn't be too hard to adapt BridgeDB's monitoring script for GetTor. In fact, to avoid code duplication, we should move this script into a separate repository and parameterise it, so it can work for both GetTor and BridgeDB.https://gitlab.torproject.org/legacy/trac/-/issues/34027GetTor not responding to emails2020-06-21T18:06:08ZCecylia BocovichGetTor not responding to emailsGetTor isn't responding to emails. I just checked the logs and found the following errors:
```
2020-04-27T18:00:33+0000 [gettor#debug] Could not send email.
2020-04-27T18:00:33+0000 [gettor#info] Error sending email: [Failure instance: ...GetTor isn't responding to emails. I just checked the logs and found the following errors:
```
2020-04-27T18:00:33+0000 [gettor#debug] Could not send email.
2020-04-27T18:00:33+0000 [gettor#info] Error sending email: [Failure instance: Traceback (failure with no frames): <class 'twisted.mail._except.SMTPDeliveryError'>: 501 No recipients accepted
czjeff5655@!63.com: 501 5.1.3 Bad recipient address syntax
>>> MAIL FROM:<gettor@torproject.org>
<<< 250 2.1.0 Ok
>>> RCPT TO:<redacted>
<<< 501 5.1.3 Bad recipient address syntax
].
2020-04-27T18:00:33+0000 [twisted.mail.smtp.ESMTPSenderFactory#info] Stopping factory <twisted.mail.smtp.ESMTPSenderFactory object at 0x7fa3ac156940>
```
Looks like this started at `2020-04-17T16:08:57+0000`. Perhaps there was a twisted update that broke it?Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/legacy/trac/-/issues/33893Track down old GetTor mirrors and decide what to do with them2020-06-21T18:06:08ZCecylia BocovichTrack down old GetTor mirrors and decide what to do with themJust talked to alwayslivid on IRC and found out that there's a list of mirrors in `config/tor-mirrors.json`. Some of these are serving very old versions of Tor Browser. We should find them and reach out to the operators and figure out wh...Just talked to alwayslivid on IRC and found out that there's a list of mirrors in `config/tor-mirrors.json`. Some of these are serving very old versions of Tor Browser. We should find them and reach out to the operators and figure out what we want to do about mirrors going forward.https://gitlab.torproject.org/legacy/trac/-/issues/33836Require Twisted 20.3.0 in gettor's requirements.txt2020-06-21T18:06:07ZteorRequire Twisted 20.3.0 in gettor's requirements.txtTwisted has a HTTP request splitting vulnerability, GetTor is probably affected.
Please update your requirements.txt to depend on Twisted 20.3.0 or later.
(And any downstream packages.)
The GitHub alert is:
https://github.com/torprojec...Twisted has a HTTP request splitting vulnerability, GetTor is probably affected.
Please update your requirements.txt to depend on Twisted 20.3.0 or later.
(And any downstream packages.)
The GitHub alert is:
https://github.com/torproject/gettor/network/alert/requirements.txt/Twisted/open
The relevant CVEs are:
CVE-2020-10108
https://github.com/advisories/GHSA-h96w-mmrf-2h6v
CVE-2020-10109
https://github.com/advisories/GHSA-p5xh-vx83-mxcjhttps://gitlab.torproject.org/legacy/trac/-/issues/33543GetTor fails to respond if you don't include a proper command2020-06-21T18:06:07ZCecylia BocovichGetTor fails to respond if you don't include a proper commandRight now GetTor will only respond to emails in the following circumstances:
- With a help message if the email (or subject) contains the keyword "help"
- With links if the email (or subject) contains the keyword "links" or a valid pla...Right now GetTor will only respond to emails in the following circumstances:
- With a help message if the email (or subject) contains the keyword "help"
- With links if the email (or subject) contains the keyword "links" or a valid platform
We should fix this to **always** send an email, defaulting to a help message if a platform or links aren't provided.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/legacy/trac/-/issues/33343Update requirements.txt in GetTor2020-06-21T18:06:06ZCecylia BocovichUpdate requirements.txt in GetTorOur requirements.txt file is out of date which makes it difficult for devs to test things locally.
It's also worth looking at the packages in the README, since those seem to be out of date as well.Our requirements.txt file is out of date which makes it difficult for devs to test things locally.
It's also worth looking at the packages in the README, since those seem to be out of date as well.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/legacy/trac/-/issues/33191Write database tests for twisted adbapi for GetTor2020-06-21T18:06:06ZCecylia BocovichWrite database tests for twisted adbapi for GetTorWe discussed moving to a different sqlite3 api, but after talking with @meejah, it's a bad idea to mix and match twisted with other I/O operations.
So, we're opting to stay with twisted, write some tests that work, and see whether we ne...We discussed moving to a different sqlite3 api, but after talking with @meejah, it's a bad idea to mix and match twisted with other I/O operations.
So, we're opting to stay with twisted, write some tests that work, and see whether we need to make changes to how dbpool is called and used.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/legacy/trac/-/issues/33185Create a pipeline for gettor statistics2020-06-21T18:06:05ZCecylia BocovichCreate a pipeline for gettor statisticsWe have some functionality in place for collecting statistics for GetTor (see https://dip.torproject.org/torproject/anti-censorship/gettor-project/gettor/issues/10), but there is some work to be done in collecting these stats at regular ...We have some functionality in place for collecting statistics for GetTor (see https://dip.torproject.org/torproject/anti-censorship/gettor-project/gettor/issues/10), but there is some work to be done in collecting these stats at regular intervals and exporting them to CollecTor (if we decide that's what we want).
This ticket is for regularly collecting and putting GetTor statistics into a format that's useful for us and others.