Skip to content
Snippets Groups Projects

rt.torproject.org is an installation of Request Tracker used for support. Users (of the Tor software, not of the TPA infrastructure) write emails, support assistants use web interface.

Note that support requests for the infrastructure should not go to RT and instead be directed at our usual support channels.

How-to

Creating a queue

On the RT web interface:

  1. authenticate to https://rt.torproject.org/
  2. head to the Queue creation form (Admin -> Queues -> Create)
  3. pick a Queue Name, set the Reply Address to QUEUENAME@rt.torproject.org and leave the Comment Address blank
  4. hit the Create button
  5. grant a group access to the queue, in the Group rights tab (create a group if necessary) - you want to grant the following to the group
    • all "General rights"
    • in "Rights for staff":
      • Delete tickets (DeleteTicket)
      • Forward messages outside of RT (ForwardMessage)
      • Modify ticket owner on owned tickets (ReassignTicket)
      • Modify tickets (ModifyTicket)
      • Own tickets (OwnTicket)
      • Sign up as a ticket or queue AdminCc (WatchAsAdminCc)
      • Take tickets (TakeTicket)
      • View exact outgoing email messages and their recipients (ShowOutgoingEmail)
      • View ticket private (commentary) That is, everything but:
      • Add custom field values only at object creation time (SetInitialCustomField)
      • Modify custom field values (ModifyCustomField)
      • Steal tickets (StealTicket)
  6. if the queue is public (and it most likely is), grant the following to the Everyone, Privileged, and Unprivileged groups:
    • Create tickets (CreateTicket)
    • Reply to tickets (ReplyToTicket)

On the RT server (currently rude):

  1. edit the /etc/aliases file to add a line like:

    rt-QUEUENAME: rtmailarchive+QUEUENAME,      "|/usr/bin/rt-mailgate --queue QUEUENAME --action correspond --url https://rt.torproject.org/"
  2. regenerate the alias database:

    newaliases
  3. add an entry in the virtual table (/etc/postfix/virtual):

    QUEUENAME@rt.torproject.org rt-QUEUENAME
  4. regenerate the virtual database:

    postmap /etc/postfix/virtual

In Puppet:

  1. add an entry in the main mail server virtual file (currently tor-puppet/modules/postfix/files/virtual) like:

    QUEUENAME@torproject.org         QUEUENAME@rt.torproject.org

TODO: the above should be automated. Ideally, QUEUENAME@rt.torproject.org should be an alias that automatically sends the message to the relevant QUEUENAME. That way, RT admins can create Queues without requiring the intervention of a sysadmin.

Using the commandline client

RT has a neat little commandline client that can be used to operate on tickets. To install it, in Debian:

sudo apt install rt4-clients

Then add this to your ~/.rtrc:

server https://rt.torproject.org/

If your local UNIX username is different than your user on RT, you'll also need:

user anarcat

Then just run, say:

rt ls

... which will prompt you for your RT password and list the open tickets! This will, for example, move the tickets 1 and 2 to the Spam queue:

rt edit set queue=Spam 1 2

This will mark all tickets older than 3 weeks as deleted in the roots queue:

rt ls -i -q roots "Status=new and LastUpdated < '3 weeks ago'" | parallel --progress --pipe -N50 -j1  -v --halt 1 rt edit - set status=deleted

See also rt help for more information.

This page describes the role of the help desk coordinator. This role is currently handled by Colin "Phoul" Childs.

Maintenance

For maintainance, the service can be shut down by stopping the mail server:

sudo service postfix stop

Then uncomment lines related to authentication in /etc/apache2/sites-staging/rt.torproject.org, then update Apache by:

sudo apache2-vhost-update rt.torproject.org

Once the maintainance is down, comment the lines again in /etc/apache2/sites-staging/rt.torproject.org and update the config again:

sudo apache2-vhost-update rt.torproject.org

Don't forget to restart the mail server:

sudo service postfix start

Support Tasks

The support help desk coordinator handles the following tasks:

  • Listowner of the support-team-private mailing list.
  • Administrator for the Request Tracker installation at https://rt.torproject.org.
  • Keeping the list of known issues at https://help.torproject.org/ up to date.
  • Sending monthly reports on the tor-reports mailing list.
  • Make the life of support assistants as good as it can be.
  • Be the contact point for other parts of the project regarding help desk matters.
  • Lead discussions about non-technical aspects of help requests to conclusions.
  • Maintain the support-tools Git repository.
  • Keep an eye on the calendar for the 'help' queue.

Create accounts for webchat / stats

  • Login to the VM "moschatum"
  • Navigate to /srv/support.torproject.org/pups
  • Run sudo -u support python manage.py createuser username password
  • Open a Trac ticket for a new account on moschatum's Prosody installation (same username as pups)
  • Send credentials for pups / prosody to support assistant

Manage the private mailing list

Administration of the private mailing list is done through Mailman web interface.

Create the monthly report

To create the monthly report chart, one should use the script rude.torproject.org:/srv/rtstuff/support-tools/monthly-report/monthly_stats.py.

Also, each month data need to be added for the quarterly reports for the business graph and for the time graph.

Data for the business graph is generated by monthly_stats. Data for the response time graph is generated by running rude.torproject.org:/srv/rtstuff/support-tools/response-time/response_time.py.

Read/only access to the RT database

Member of the rtfolks group can have read-only access to the RT database. The password can be found in /srv/rtstuff/db-info.

To connect to the database, one can use:

psql "host=drobovi.torproject.org sslmode=require user=rtreader dbname=rt"

Number of tickets per week

    SELECT COUNT(tickets.id),
           CONCAT_WS(' ', DATE_PART('year', tickets.created),
                          TO_CHAR(date_part('week', tickets.created), '99')) AS d
     FROM tickets
     JOIN queues ON (tickets.queue = queues.id)
    WHERE queues.name LIKE 'help%'
    GROUP BY d
    ORDER BY d;

Extract the most frequently used articles

Replace the dates.

   SELECT COUNT(tickets.id) as usage, articles.name as article
     FROM queues, tickets, links, articles
    WHERE queues.name = 'help'
      AND tickets.queue = queues.id
      AND tickets.lastupdated >= '2014-02-01'
      AND tickets.created < '2014-03-01'
      AND links.type = 'RefersTo'
      AND links.base = CONCAT('fsck.com-rt://torproject.org/ticket/', tickets.id)
      AND articles.id = TO_NUMBER(SUBSTRING(links.target from '[0-9]+$'), '9999999')
    GROUP BY articles.id
    ORDER BY usage DESC;

Graphs of activity for the past month

Using Gnuplot:

set terminal pngcairo enhanced size 600,400
set style fill solid 1.0 border
set border linewidth 1.0
set bmargin at screen 0.28
set tmargin at screen 0.9
set key at screen 0.9,screen 0.95
set xtics rotate
set yrange [0:]
set output "month.png"
plot "<                                                                                                      \
  echo \"SELECT COUNT(tickets.id),                                                                           \
                TO_CHAR(tickets.created, 'YYYY-MM-DD') AS d                                                  \
     FROM tickets                                                                                            \
     JOIN queues ON (tickets.queue = queues.id)                                                              \
    WHERE queues.name LIKE 'help%'                                                                           \
      AND tickets.created >= TO_DATE(TO_CHAR(NOW() - INTERVAL '1 MONTH', 'YYYY-MM-01'), 'YYYY-MM-DD')        \
      AND tickets.created <  TO_DATE(TO_CHAR(NOW(), 'YYYY-MM-01'), 'YYYY-MM-DD')                             \
    GROUP BY d                                                                                               \
    ORDER BY d;\" |                                                                                          \
  ssh rude.torproject.org psql \\\"host=drobovi.torproject.org sslmode=require user=rtreader dbname=rt\\\" | \
  sed 's/|//'                                                                                                \
" using 1:xtic(2) with boxes title "new tickets"

Get the most recent version of each RT articles

SELECT classes.name AS class,
       articles.name AS title,
       CASE WHEN objectcustomfieldvalues.content != '' THEN objectcustomfieldvalues.content
            ELSE objectcustomfieldvalues.largecontent
       END AS content,
       objectcustomfieldvalues.lastupdated,
       articles.id
  FROM classes, articles, objectcustomfieldvalues
 WHERE articles.class = classes.id
   AND objectcustomfieldvalues.objecttype = 'RT::Article'
   AND objectcustomfieldvalues.objectid = articles.id
   AND objectcustomfieldvalues.id = (
           SELECT objectcustomfieldvalues.id
             FROM objectcustomfieldvalues
            WHERE objectcustomfieldvalues.objectid = articles.id
              AND objectcustomfieldvalues.disabled = 0
            ORDER BY objectcustomfieldvalues.lastupdated DESC
            LIMIT 1)
 ORDER BY classes.id, articles.id;

Granting access to a support help desk coordinator

The support help desk coordinator needs the following assets to perform their duties:

  • Administration password for the support-team-private mailing list.
  • Being owner in the support-team-private mailing list configuration.
  • Commit access to help wiki Git repository.
  • Shell access to rude.torproject.org.
  • LDAP account member of the rtfolks group.
  • LDAP account member of the support group.
  • root password for Request Tracker.
  • Being owner of the “Tor Support” component in Trac.

New RT admin

This task is typically done by TPA, but can technically be done by any RT admin.

  1. find the RT admin password in hosts-extra-info in the TPA password manager and login as root OR login as your normal RT admin user

  2. create an account member of rt-admin

Pager playbook

Ticket creation failed / No permission to create tickets in the queue

If you receive an email like this:

From: rt@rt.torproject.org Subject: Ticket creation failed: [ORIGINAL SUBJECT] To: root@rude.torproject.org Date: Tue, 05 Jan 2021 01:01:21 +0000

No permission to create tickets in the queue 'help'

[ORIGINAL EMAIL]

It is because a disabled user tried to send an email to RT. It is unclear how to handle those issues in the future, but to remove such warnings, the user could just be re-enabled.

Alternatively, sysadmins can block the user completely by adding the user to the /etc/postfix/access-2-sender map and reload it with:

postmap /etc/postfix/access-2-sender

See Also issue 33314 for more information.

Email delivery fails

If no email is coming into RT, it might be one of the spam filters that failed to start. Nagios would complain with a message like:

Subject: ** PROBLEM Service Alert: rude/process - spampd - master is CRITICAL **

[...]

Service: process - spampd - master
Host: rude
Address: 116.202.120.187
State: CRITICAL

Date/Time: Thu Feb 11 14:53:44 UTC 2021

Additional Info:

PROCS CRITICAL: 0 processes with UID = 112 (spampd), command name spampd, PPID = 1

It's still unclear why this happens (see e.g. issue 40175), but a workaround is to either reboot the server or restart spampd:

systemctl restart spampd

Reference

Installation

No clue. Probably Debian-ish?

Configuration is readable in /etc/request-tracker4/RT_SiteConfig.d/.

There's no logs. No clue why.

Auto-reply to new requestors

When an unknown email address sends an email to the support, it will be automatically replied to warn users about the data retention policy.

A global Scrip is responsible for this. It will be default use the global template named “Initial reply”. It is written in English. In each queue except help, a template named exactly “Initial reply” is defined in order to localize the message.

Expiration of old tickets

Tickets (and affiliated users) get erased from the RT database after 100 days. This is done by the expire-old-tickets script. The script is run everyday at 06:02 UTC through a cronjob run as user colin.

Encrypted SQL dumps of the data removed from the database will be written to /srv/rtstuff/shredded and must be put away regularily.

Dump of RT templates

RT articles are dumped into text files and then pushed to the rt-articles Git repository. An email is sent each time there's a new commit, so collective reviews can happen by the rest of the support team.

The machinery is spread through several scripts. The one run on rude is dump_rt_articles, and it will run everyday through a cronjob as user colin.

Issues

There is no issue tracker specifically for this project, File or search for issues in the team issue tracker with the ~RT label.

Discussion

Spam filter training design

RT is designed to be trained for spam filtering. RT users put spam in the "Spam" queue and then a set of scripts run in the background to train spamassassin, based on a mail archive that procmail keeps of every incoming mail.

This runs as a cronjob in the rtmailarchive user, which looks like this:

/srv/rtstuff/support-tools/train-spam-filters/train_spam_filters && bin/spam-learn && find Maildir/.spam.learned Maildir/.xham.learned -type f -delete

The train_spam_filters script basically does this:

  1. for each mail in the Maildir/.help* archive
  2. find its Message-Id header
  3. load the equivalent message from RT:
    • if it is in the Spam queue, marked as "Rejected", it is spam.
    • if it is in a help-* queue, marked as "Resolved", it is ham.
  4. move the email in the right directory mail folder (.spam.learn, .xham.learn) depending on status
  5. if the file is more than 100 days old, delete it.

Then the rest of the cron job continues. spam-learn is this shell script:

#!/bin/bash

dbpath="/var/cache/spampd"

learn() {
    local what="$1"; shift;
    local whence="$1"; shift;
    local whereto="$1"; shift;

    (
        cd "$whence"
        find -type f | \
          while read f; do
            sudo -u spampd -H sa-learn --dbpath "$dbpath" --"$what" < "$f"
            mv "$f" "$whereto/$f"
        done
    )
}

set -e

learn spam /srv/rtmailarchive/Maildir/.spam.learn /srv/rtmailarchive/Maildir/.spam.learned
learn ham /srv/rtmailarchive/Maildir/.xham.learn /srv/rtmailarchive/Maildir/.xham.learned

# vim:set et:
# vim:set ts=4:
# vim:set shiftwidth=4:

which, basically, calls sa-learn on each individual email in the folder, moving it to .spam.learned or .xham.learned when done.

Then, interestingly, those emails are destroyed. It's unclear why that is not done in the spam-learn step directly.

Possible improvements

The above design has a few problems:

  1. it assumes "ham" queues are named "help-*" - but there are other queues in the system
  2. it might be slow: if there are lots of emails to process, it will do an SQL query for each and a move, and not all at once
  3. it is split over multiple shell scripts, not versioned

I would recommend the following:

  1. reverse the logic of the queue checks: instead of checking for folders and queues named help-*, check if the folders or queues are not named spam* or xham*
  2. batch jobs: use a generator to yield Message-Id, then pick a certain number of emails and batch-send them to psql and the rename
  3. do all operations at once: look in psql, move the files in the learning folder, and train, possibly in parallel, but at least all in the same script
  4. sa-learn can read from a folder now, so there's no need for that wrapper shell script in any case
  5. commit the script to version control and, even better, puppet

We could also add a CAPTCHA and look at the RT::Extension::ReportSpam...

Alternatives