In the previous commit, we forgot to remove requests that trigger
exceptions other than SMTPClientError.

This solves gettor#77 where a non-ascii email address was causing our
SMTP client to throw an error. We need to remove the request from the
database in order to move on and process the rest of the requests.

hanneloresx authored 4 years ago and Cecylia Bocovich committed 4 years ago

Addresses issue #34058 to scrub logs of personal info.

Closes issue #75 where the headers were being parsed for locale
instead of the message body.

If GetTor deals with an autoresponder, it returns an empty request,
which is then passed to parse_callback, which raises a KeyError because
of request["command"]:

2020-05-29 19:04:43+0000 [email parser] Error while parsing email content: [Failure instance: Traceback: <class 'KeyError'>: 'command'
	/usr/lib/python3/dist-packages/twisted/internet/defer.py:311:addCallbacks
	/usr/lib/python3/dist-packages/twisted/internet/defer.py:654:_runCallbacks
	/usr/lib/python3/dist-packages/twisted/internet/defer.py:1613:unwindGenerator
	/usr/lib/python3/dist-packages/twisted/internet/defer.py:1529:_cancellableInlineCallbacks
	--- <exception caught here> ---
	/usr/lib/python3/dist-packages/twisted/internet/defer.py:1418:_inlineCallbacks
	/srv/gettor.torproject.org/home/gettor/gettor/parse/email.py:256:parse_callback
	].

This is a fixup of: <https://bugs.torproject.org/34286>

This checks to see if an incoming request is from a known autoresponder.
GetTor can get into infinite loops with itself or postmaster addresses
(Bug #34286)

This ignores emails with the username "MAILER-DAEMON", a common email
for mail delivery systems. The goal of this is to avoid getting into a
loop with mail delivery system autoresponders after a bounced email.

A lot of these are debug messages and don't need to be logged by
default.

Before we were calling a non-existent shutdown function. These instances
already have __del__ functions so we can just call those instead.

Instead of processing link and help requests separately, we can process
them together and use the command to determine the different subject and
body messages.

Removed unused code and simplified database command.

This adds an extra check on SMTP errors to see if it's due to a bad
recipient address and then removes the request from the database. This
prevents one bad request from hanging up the processing of other
requests (see https://bugs.torproject.org/34027).

This isn't called by any other code and hasn't worked for a long time.

These files are empty or were accidentially added to the repository

This used to be the location of the code for gettor.torproject.org, but
that has been moved into a separate repository. While this old code is
suitable for GetTor mirror sites, it is very out of date and unclear
whether we want to support this going forward.

These have been replaced by newer scripts located in the top-level
scripts/ directory.

This makes the following changes to the links message:
- Breaks user actions into steps
- Simplifies wordings
- Includes signature verification and bridge instructions

This restructures the GetTor help message. The message now includes:
- instructions to reply to the message for links
- available locales
- a valid example links request

This will make the locale parsing more restrictive so that, for example,
sending "i like fries" will not result in the locale "fr".

teor mentioned in ticket #23225 that some clients will top-post replies
without quoting the the text. This commit adds a test case for that.
Since the command detector is greedy, gettor parses the reply first.

Also added a test for bottom-posting.

This commit makes some changes and adds some tests to ensure that
valid replies to the gettor help message will contain links.
Non-valid replies should still return the GetTor help message.

This fixes an issue caused by gitlab repositories soft-delete for 7 days
before they are fully deleted. If we remove a repository in order
to upload new binaries, we need to be able to create a new one right
away. See https://bugs.torproject.org/33034

This fixes a bug where improperly formatted emails were not receiving
gettor replies. We should always reply to emails and default to sending
a help message if the request was invalid.

This also prioritizes sending links if both a valid links request and
the word help were sent. Several tests were added.

We were leaving connections to the database open, which causes some
calls to hang. This adds destructors to the SQLite3 class and the
classes that use it, and refactors some code to make one database
connection per class object. This also makes sure that the destructors
for objects that use databases are actually called.

The rest of GetTor uses the python sqlite3 package, and it's much
simpler to use and to test.

This will remove gettor requests from the requests database when they
are completed (rather than changing the status to 'SENT'). There's no
need to keep this information around, and we don't want to store
hashed email addresses as identifiers long term.

The only help message urged users to only specify 'en' as a locale. We
now support all locales for tor browser binaries.