[rt.torproject.org](https://rt.torproject.org/) is an installation of [Request Tracker](http://bestpractical.com/rt/) used
for support. Users (of the Tor software, not of the TPA
infrastructure) write emails, support assistants use web
interface. 

Note that support requests for the infrastructure should *not* go to
RT and instead be directed at our usual [support channels](support).

[[_TOC_]]

# How-to

## Creating a queue

On the RT web interface:

 1. authenticate to <https://rt.torproject.org/>
 2. head to the [Queue creation form](https://rt.torproject.org/Admin/Queues/Modify.html?Create=1) (Admin -> Queues -> Create)
 3. pick a Queue Name, set the `Reply Address` to
    `QUEUENAME@rt.torproject.org` and leave the `Comment Address`
    blank
 4. hit the `Create` button
 5. grant a group access to the queue, in the `Group rights` tab
    ([create a group](https://rt.torproject.org/Admin/Groups/Modify.html?Create=1) if necessary) - you want to grant the
    following to the group
    * all "General rights"
    * in "Rights for staff":
      * Delete tickets (`DeleteTicket`)
      * Forward messages outside of RT (`ForwardMessage`)
      * Modify ticket owner on owned tickets (`ReassignTicket`)
      * Modify tickets (`ModifyTicket`)
      * Own tickets (`OwnTicket`)
      * Sign up as a ticket or queue AdminCc (`WatchAsAdminCc`)
      * Take tickets (`TakeTicket`)
      * View exact outgoing email messages and their recipients (`ShowOutgoingEmail`)
      * View ticket private (`commentary`)
      That is, everything but:
      * Add custom field values only at object creation time (`SetInitialCustomField`)
      * Modify custom field values (`ModifyCustomField`)
      * Steal tickets (`StealTicket`)
 6. if the queue is public (and it most likely is), grant the
    following to the `Everyone`, `Privileged`, and `Unprivileged`
    groups:
      * Create tickets (`CreateTicket`)
      * Reply to tickets (`ReplyToTicket`)

On the RT server (currently `rude`):

 1. edit the `/etc/aliases` file to add a line like:

        rt-QUEUENAME: rtmailarchive+QUEUENAME,      "|/usr/bin/rt-mailgate --queue QUEUENAME --action correspond --url https://rt.torproject.org/"

 2. regenerate the alias database:
    
        newaliases

 3. add an entry in the virtual table (`/etc/postfix/virtual`):
    
        QUEUENAME@rt.torproject.org rt-QUEUENAME

 4. regenerate the virtual database:
 
        postmap /etc/postfix/virtual

In Puppet:

 1. add an entry in the main mail server virtual file (currently
    `tor-puppet/modules/postfix/files/virtual`) like:
    
        QUEUENAME@torproject.org         QUEUENAME@rt.torproject.org

TODO: the above should be automated. Ideally,
`QUEUENAME@rt.torproject.org` should be an alias that automatically
sends the message to the relevant `QUEUENAME`. That way, RT admins can
create Queues without requiring the intervention of a sysadmin.

## Using the commandline client

RT has a neat little commandline client that can be used to operate on tickets. To install it, in Debian:

    sudo apt install rt4-clients

Then add this to your `~/.rtrc`:

    server https://rt.torproject.org/

If your local UNIX username is different than your user on RT, you'll also need:

    user anarcat

Then just run, say:

    rt ls

... which will prompt you for your RT password and list the open tickets! This will, for example, move the tickets 1 and 2 to the `Spam` queue:

    rt edit set queue=Spam 1 2

This will mark all tickets older than 3 weeks as deleted in the `roots` queue:

    rt ls -i -q roots "Status=new and LastUpdated < '3 weeks ago'" | parallel --progress --pipe -N50 -j1  -v --halt 1 rt edit - set status=deleted

See also `rt help` for more information.

This page describes the role of the help desk coordinator. This role is currently handled by Colin "Phoul" Childs.

## Maintenance

For maintainance, the service can be shut down by stopping the mail server:

```
sudo service postfix stop
```

Then uncomment lines related to authentication in `/etc/apache2/sites-staging/rt.torproject.org`, then update Apache by:

```
sudo apache2-vhost-update rt.torproject.org
```

Once the maintainance is down, comment the lines again in `/etc/apache2/sites-staging/rt.torproject.org` and update the config again:

```
sudo apache2-vhost-update rt.torproject.org
```

Don't forget to restart the mail server:

```
sudo service postfix start
```

## Support Tasks

The support help desk coordinator handles the following tasks:

 * Listowner of the `support-team-private` mailing list.
 * Administrator for the Request Tracker installation at [rt.torproject.org](./org/operations/Infrastructure/rt.torproject.org).
 * Keeping the list of known issues at https://help.torproject.org/ up to date.
 * Sending monthly reports on the [tor-reports](https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-reports) mailing list.
 * Make the life of support assistants as good as it can be.
 * Be the contact point for other parts of the project regarding help desk matters.
 * Lead discussions about non-technical aspects of help requests to conclusions.
 * Maintain the [`support-tools` Git repository](https://gitweb.torproject.org/support-tools).
 * Keep an eye on the [calendar](./doc/support/Calendar) for the 'help' queue.

## Create accounts for webchat / stats

 * Login to the VM "moschatum"
 * Navigate to `/srv/support.torproject.org/pups`
 * Run `sudo -u support python manage.py createuser username password`
 * Open a Trac ticket for a new account on moschatum's Prosody installation (same username as pups)
 * Send credentials for pups / prosody to support assistant

## Manage the private mailing list

Administration of the private mailing list is done through
[Mailman web interface](https://lists.torproject.org/cgi-bin/mailman/admin/support-team-private).

## Create the monthly report

To create the monthly report chart, one should use the script
`rude.torproject.org:/srv/rtstuff/support-tools/monthly-report/monthly_stats.py`.

Also, each month data need to be added for the quarterly reports for the [business graph](https://gitweb.torproject.org/support-tools.git/tree/HEAD:/queue-graphs/data) and for the [time graph](https://gitweb.torproject.org/support-tools.git/tree/HEAD:/response-time/dataresponse).

Data for the business graph is generated by `monthly_stats`. Data for the response time graph is generated by running `rude.torproject.org:/srv/rtstuff/support-tools/response-time/response_time.py`.

## Read/only access to the RT database

Member of the `rtfolks` group can have read-only access to the RT database. The password
can be found in `/srv/rtstuff/db-info`.

To connect to the database, one can use:

```
psql "host=drobovi.torproject.org sslmode=require user=rtreader dbname=rt"
```

### Number of tickets per week

```
    SELECT COUNT(tickets.id),
           CONCAT_WS(' ', DATE_PART('year', tickets.created),
                          TO_CHAR(date_part('week', tickets.created), '99')) AS d
     FROM tickets
     JOIN queues ON (tickets.queue = queues.id)
    WHERE queues.name LIKE 'help%'
    GROUP BY d
    ORDER BY d;
```

## Extract the most frequently used articles

**Replace the dates.**

```
   SELECT COUNT(tickets.id) as usage, articles.name as article
     FROM queues, tickets, links, articles
    WHERE queues.name = 'help'
      AND tickets.queue = queues.id
      AND tickets.lastupdated >= '2014-02-01'
      AND tickets.created < '2014-03-01'
      AND links.type = 'RefersTo'
      AND links.base = CONCAT('fsck.com-rt://torproject.org/ticket/', tickets.id)
      AND articles.id = TO_NUMBER(SUBSTRING(links.target from '[0-9]+$'), '9999999')
    GROUP BY articles.id
    ORDER BY usage DESC;
```

### Graphs of activity for the past month

Using Gnuplot:

```
set terminal pngcairo enhanced size 600,400
set style fill solid 1.0 border
set border linewidth 1.0
set bmargin at screen 0.28
set tmargin at screen 0.9
set key at screen 0.9,screen 0.95
set xtics rotate
set yrange [0:]
set output "month.png"
plot "<                                                                                                      \
  echo \"SELECT COUNT(tickets.id),                                                                           \
                TO_CHAR(tickets.created, 'YYYY-MM-DD') AS d                                                  \
     FROM tickets                                                                                            \
     JOIN queues ON (tickets.queue = queues.id)                                                              \
    WHERE queues.name LIKE 'help%'                                                                           \
      AND tickets.created >= TO_DATE(TO_CHAR(NOW() - INTERVAL '1 MONTH', 'YYYY-MM-01'), 'YYYY-MM-DD')        \
      AND tickets.created <  TO_DATE(TO_CHAR(NOW(), 'YYYY-MM-01'), 'YYYY-MM-DD')                             \
    GROUP BY d                                                                                               \
    ORDER BY d;\" |                                                                                          \
  ssh rude.torproject.org psql \\\"host=drobovi.torproject.org sslmode=require user=rtreader dbname=rt\\\" | \
  sed 's/|//'                                                                                                \
" using 1:xtic(2) with boxes title "new tickets"
```

### Get the most recent version of each RT articles

```
SELECT classes.name AS class,
       articles.name AS title,
       CASE WHEN objectcustomfieldvalues.content != '' THEN objectcustomfieldvalues.content
            ELSE objectcustomfieldvalues.largecontent
       END AS content,
       objectcustomfieldvalues.lastupdated,
       articles.id
  FROM classes, articles, objectcustomfieldvalues
 WHERE articles.class = classes.id
   AND objectcustomfieldvalues.objecttype = 'RT::Article'
   AND objectcustomfieldvalues.objectid = articles.id
   AND objectcustomfieldvalues.id = (
           SELECT objectcustomfieldvalues.id
             FROM objectcustomfieldvalues
            WHERE objectcustomfieldvalues.objectid = articles.id
              AND objectcustomfieldvalues.disabled = 0
            ORDER BY objectcustomfieldvalues.lastupdated DESC
            LIMIT 1)
 ORDER BY classes.id, articles.id;
```

## Granting access to a support help desk coordinator

The support help desk coordinator needs the following assets to perform their duties:

 * Administration password for the support-team-private mailing list.
 * Being owner in the support-team-private mailing list configuration.
 * Commit access to [help wiki Git repository](https://gitweb.torproject.org/project/help/wiki.git).
 * Shell access to [rude.torproject.org](https://db.torproject.org/machines.cgi?host=rude).
 * LDAP account member of the `rtfolks` group.
 * LDAP account member of the `support` group.
 * `root` password for Request Tracker.
 * Being owner of the “Tor Support” component in Trac.

## New RT admin

This task is typically done by TPA, but can technically be done by any
RT admin.

 1. find the RT admin password in `hosts-extra-info` in the TPA password
    manager and login as root OR login as your normal RT admin user

 2. create an account member of `rt-admin`

## Pager playbook

### Ticket creation failed / No permission to create tickets in the queue

If you receive an email like this:

> From: rt@rt.torproject.org
> Subject: Ticket creation failed: \[ORIGINAL SUBJECT\]
> To: root@rude.torproject.org
> Date: Tue, 05 Jan 2021 01:01:21 +0000
>
> No permission to create tickets in the queue 'help'
>
> \[ORIGINAL EMAIL\]

It is because a disabled user tried to send an email to RT. It is
unclear how to handle those issues in the future, but to remove such
warnings, the user could just be re-enabled.

Alternatively, sysadmins can block the user completely by adding the
user to the `/etc/postfix/access-2-sender` map and reload it with:

    postmap /etc/postfix/access-2-sender

See Also [issue 33314](https://gitlab.torproject.org/tpo/tpa/team/-/issues/33314) for more information.

### Email delivery fails

If no email is coming into RT, it might be one of the spam filters
that failed to start. Nagios would complain with a message like:

    Subject: ** PROBLEM Service Alert: rude/process - spampd - master is CRITICAL **

    [...]

    Service: process - spampd - master
    Host: rude
    Address: 116.202.120.187
    State: CRITICAL

    Date/Time: Thu Feb 11 14:53:44 UTC 2021

    Additional Info:

    PROCS CRITICAL: 0 processes with UID = 112 (spampd), command name spampd, PPID = 1

It's still unclear why this happens (see e.g. [issue 40175](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40175)), but a
workaround is to either reboot the server or restart `spampd`:

    systemctl restart spampd

# Reference

## Installation

No clue. Probably Debian-ish?

Configuration is readable in `/etc/request-tracker4/RT_SiteConfig.d/`.

There's no logs. No clue why.

## Auto-reply to new requestors

When an unknown email address sends an email to the support, it will be automatically replied to warn users about the data retention policy.

A global _Scrip_ is responsible for this. It will be default use the global template named “Initial reply”. It is written in English. In each queue except _help_, a template named exactly “Initial reply” is defined in order to localize the message.

## Expiration of old tickets

Tickets (and affiliated users) get erased from the RT database after 100 days. This is done by the [expire-old-tickets script](https://gitweb.torproject.org/support-tools.git/blob/HEAD:/expire-old-tickets/expire-old-tickets). The script is run everyday at 06:02 UTC through a _cronjob_ run as user `colin`.

Encrypted SQL dumps of the data removed from the database will be written to `/srv/rtstuff/shredded` and must be put away regularily.

## Dump of RT templates

RT articles are dumped into text files and then pushed to the [rt-articles Git repository](https://people.torproject.org/~colin/rt-articles.git). An email is sent each time there's a new commit, so collective reviews can happen by the rest of the support team.

The machinery is [spread through several scripts](https://gitweb.torproject.org/support-tools.git/tree/HEAD:/rt-articles). The one run on `rude` is [dump_rt_articles](https://gitweb.torproject.org/support-tools.git/blob/HEAD:/rt-articles/dump_rt_articles), and it will run everyday through a _cronjob_ as user `colin`.

# Discussion

## Spam filter training design

RT is designed to be trained for spam filtering. RT users put spam in
the "Spam" queue and then a set of scripts run in the background to
train spamassassin, based on a mail archive that procmail keeps of
every incoming mail.

This runs as a cronjob in the `rtmailarchive` user, which looks like
this:

    /srv/rtstuff/support-tools/train-spam-filters/train_spam_filters && bin/spam-learn && find Maildir/.spam.learned Maildir/.xham.learned -type f -delete

The [train_spam_filters](https://gitweb.torproject.org/support-tools.git/tree/train-spam-filters/train_spam_filters) script basically does this:

 1. for each mail in the `Maildir/.help*` archive
 2. find its Message-Id header
 3. load the equivalent message from RT:
    * if it is in the Spam queue, marked as "Rejected", it is spam.
    * if it is in a help-* queue, marked as "Resolved", it is ham.
 4. move the email in the right directory mail folder (`.spam.learn`,
    `.xham.learn`) depending on status
 5. if the file is more than 100 days old, delete it.

Then the rest of the cron job continues. `spam-learn` is this shell
script:

    #!/bin/bash

    dbpath="/var/cache/spampd"

    learn() {
        local what="$1"; shift;
        local whence="$1"; shift;
        local whereto="$1"; shift;

        (
            cd "$whence"
            find -type f | \
              while read f; do
                sudo -u spampd -H sa-learn --dbpath "$dbpath" --"$what" < "$f"
                mv "$f" "$whereto/$f"
            done
        )
    }

    set -e

    learn spam /srv/rtmailarchive/Maildir/.spam.learn /srv/rtmailarchive/Maildir/.spam.learned
    learn ham /srv/rtmailarchive/Maildir/.xham.learn /srv/rtmailarchive/Maildir/.xham.learned

    # vim:set et:
    # vim:set ts=4:
    # vim:set shiftwidth=4:

which, basically, calls `sa-learn` on each individual email in the
folder, moving it to `.spam.learned` or `.xham.learned` when done.

Then, interestingly, those emails are destroyed. It's unclear why that
is not done in the `spam-learn` step directly.

### Possible improvements

The above design has a few problems:

 1. it assumes "ham" queues are named "help-*" - but there are other
    queues in the system
 2. it might be slow: if there are lots of emails to process, it will
    do an SQL query for each and a move, and not all at once
 3. it is split over multiple shell scripts, not versioned

I would recommend the following:

 1. reverse the logic of the queue checks: instead of checking for
    folders and queues named `help-*`, check if the folders or queues
    are *not* named `spam*` or `xham*`
 2. batch jobs: use a generator to yield Message-Id, then pick a
    certain number of emails and batch-send them to psql and the
    rename
 3. do all operations at once: look in psql, move the files in the
    learning folder, and train, possibly in parallel, but at least all
    in the same script
 4. sa-learn can read from a folder now, so there's no need for that
    wrapper shell script in any case
 5. commit the script to version control and, even better, puppet

We could also [add a CAPTCHA](https://metacpan.org/pod/RT::Extension::Captcha) and look at the [RT::Extension::ReportSpam](https://metacpan.org/pod/RT::Extension::ReportSpam)...

# Alternatives

 * [Zammad](https://zammad.com/) has a [free version](https://zammad.org/), but it's unclear which
   features the latter has... the .com site has a [bunch of
   features](https://zammad.com/en/product/features) that could prove interesting for us, particularly the
   [GitLab integration](https://zammad.com/en/product/features/gitlab-integration)
