- How-to
- Creating a queue
- Using the commandline client
- Maintenance
- Support Tasks
- Create accounts for webchat / stats
- Manage the private mailing list
- Create the monthly report
- Read/only access to the RT database
- Number of tickets per week
- Extract the most frequently used articles
- Graphs of activity for the past month
- Get the most recent version of each RT articles
- Granting access to a support help desk coordinator
- New RT admin
- Pager playbook
- Ticket creation failed / No permission to create tickets in the queue
- Email delivery fails
- Reference
- Installation
- Auto-reply to new requestors
- Expiration of old tickets
- Dump of RT templates
- Issues
- Discussion
- Spam filter training design
- Possible improvements
- Alternatives
rt.torproject.org is an installation of Request Tracker used for support. Users (of the Tor software, not of the TPA infrastructure) write emails, support assistants use web interface.
Note that support requests for the infrastructure should not go to RT and instead be directed at our usual support channels.
-
How-to
- Creating a queue
- Using the commandline client
- Maintenance
- Support Tasks
- Create accounts for webchat / stats
- Manage the private mailing list
- Create the monthly report
- Read/only access to the RT database
- Extract the most frequently used articles
- Granting access to a support help desk coordinator
- New RT admin
- Pager playbook
- Reference
- Discussion
- Alternatives
How-to
Creating a queue
On the RT web interface:
- authenticate to https://rt.torproject.org/
- head to the Queue creation form (Admin -> Queues -> Create)
- pick a Queue Name, set the
Reply Address
toQUEUENAME@rt.torproject.org
and leave theComment Address
blank - hit the
Create
button - grant a group access to the queue, in the
Group rights
tab (create a group if necessary) - you want to grant the following to the group- all "General rights"
- in "Rights for staff":
- Delete tickets (
DeleteTicket
) - Forward messages outside of RT (
ForwardMessage
) - Modify ticket owner on owned tickets (
ReassignTicket
) - Modify tickets (
ModifyTicket
) - Own tickets (
OwnTicket
) - Sign up as a ticket or queue AdminCc (
WatchAsAdminCc
) - Take tickets (
TakeTicket
) - View exact outgoing email messages and their recipients (
ShowOutgoingEmail
) - View ticket private (
commentary
) That is, everything but: - Add custom field values only at object creation time (
SetInitialCustomField
) - Modify custom field values (
ModifyCustomField
) - Steal tickets (
StealTicket
)
- Delete tickets (
- if the queue is public (and it most likely is), grant the
following to the
Everyone
,Privileged
, andUnprivileged
groups:- Create tickets (
CreateTicket
) - Reply to tickets (
ReplyToTicket
)
- Create tickets (
On the RT server (currently rude
):
-
edit the
/etc/aliases
file to add a line like:rt-QUEUENAME: rtmailarchive+QUEUENAME, "|/usr/bin/rt-mailgate --queue QUEUENAME --action correspond --url https://rt.torproject.org/"
-
regenerate the alias database:
newaliases
-
add an entry in the virtual table (
/etc/postfix/virtual
):QUEUENAME@rt.torproject.org rt-QUEUENAME
-
regenerate the virtual database:
postmap /etc/postfix/virtual
In Puppet:
-
add an entry in the main mail server virtual file (currently
tor-puppet/modules/postfix/files/virtual
) like:QUEUENAME@torproject.org QUEUENAME@rt.torproject.org
TODO: the above should be automated. Ideally,
QUEUENAME@rt.torproject.org
should be an alias that automatically
sends the message to the relevant QUEUENAME
. That way, RT admins can
create Queues without requiring the intervention of a sysadmin.
Using the commandline client
RT has a neat little commandline client that can be used to operate on tickets. To install it, in Debian:
sudo apt install rt4-clients
Then add this to your ~/.rtrc
:
server https://rt.torproject.org/
If your local UNIX username is different than your user on RT, you'll also need:
user anarcat
Then just run, say:
rt ls
... which will prompt you for your RT password and list the open tickets! This will, for example, move the tickets 1 and 2 to the Spam
queue:
rt edit set queue=Spam 1 2
This will mark all tickets older than 3 weeks as deleted in the roots
queue:
rt ls -i -q roots "Status=new and LastUpdated < '3 weeks ago'" | parallel --progress --pipe -N50 -j1 -v --halt 1 rt edit - set status=deleted
See also rt help
for more information.
This page describes the role of the help desk coordinator. This role is currently handled by Colin "Phoul" Childs.
Maintenance
For maintainance, the service can be shut down by stopping the mail server:
sudo service postfix stop
Then uncomment lines related to authentication in /etc/apache2/sites-staging/rt.torproject.org
, then update Apache by:
sudo apache2-vhost-update rt.torproject.org
Once the maintainance is down, comment the lines again in /etc/apache2/sites-staging/rt.torproject.org
and update the config again:
sudo apache2-vhost-update rt.torproject.org
Don't forget to restart the mail server:
sudo service postfix start
Support Tasks
The support help desk coordinator handles the following tasks:
- Listowner of the
support-team-private
mailing list. - Administrator for the Request Tracker installation at https://rt.torproject.org.
- Keeping the list of known issues at https://help.torproject.org/ up to date.
- Sending monthly reports on the tor-reports mailing list.
- Make the life of support assistants as good as it can be.
- Be the contact point for other parts of the project regarding help desk matters.
- Lead discussions about non-technical aspects of help requests to conclusions.
- Maintain the
support-tools
Git repository. - Keep an eye on the calendar for the 'help' queue.
Create accounts for webchat / stats
- Login to the VM "moschatum"
- Navigate to
/srv/support.torproject.org/pups
- Run
sudo -u support python manage.py createuser username password
- Open a Trac ticket for a new account on moschatum's Prosody installation (same username as pups)
- Send credentials for pups / prosody to support assistant
Manage the private mailing list
Administration of the private mailing list is done through Mailman web interface.
Create the monthly report
To create the monthly report chart, one should use the script
rude.torproject.org:/srv/rtstuff/support-tools/monthly-report/monthly_stats.py
.
Also, each month data need to be added for the quarterly reports for the business graph and for the time graph.
Data for the business graph is generated by monthly_stats
. Data for the response time graph is generated by running rude.torproject.org:/srv/rtstuff/support-tools/response-time/response_time.py
.
Read/only access to the RT database
Member of the rtfolks
group can have read-only access to the RT database. The password
can be found in /srv/rtstuff/db-info
.
To connect to the database, one can use:
psql "host=drobovi.torproject.org sslmode=require user=rtreader dbname=rt"
Number of tickets per week
SELECT COUNT(tickets.id),
CONCAT_WS(' ', DATE_PART('year', tickets.created),
TO_CHAR(date_part('week', tickets.created), '99')) AS d
FROM tickets
JOIN queues ON (tickets.queue = queues.id)
WHERE queues.name LIKE 'help%'
GROUP BY d
ORDER BY d;
Extract the most frequently used articles
Replace the dates.
SELECT COUNT(tickets.id) as usage, articles.name as article
FROM queues, tickets, links, articles
WHERE queues.name = 'help'
AND tickets.queue = queues.id
AND tickets.lastupdated >= '2014-02-01'
AND tickets.created < '2014-03-01'
AND links.type = 'RefersTo'
AND links.base = CONCAT('fsck.com-rt://torproject.org/ticket/', tickets.id)
AND articles.id = TO_NUMBER(SUBSTRING(links.target from '[0-9]+$'), '9999999')
GROUP BY articles.id
ORDER BY usage DESC;
Graphs of activity for the past month
Using Gnuplot:
set terminal pngcairo enhanced size 600,400
set style fill solid 1.0 border
set border linewidth 1.0
set bmargin at screen 0.28
set tmargin at screen 0.9
set key at screen 0.9,screen 0.95
set xtics rotate
set yrange [0:]
set output "month.png"
plot "< \
echo \"SELECT COUNT(tickets.id), \
TO_CHAR(tickets.created, 'YYYY-MM-DD') AS d \
FROM tickets \
JOIN queues ON (tickets.queue = queues.id) \
WHERE queues.name LIKE 'help%' \
AND tickets.created >= TO_DATE(TO_CHAR(NOW() - INTERVAL '1 MONTH', 'YYYY-MM-01'), 'YYYY-MM-DD') \
AND tickets.created < TO_DATE(TO_CHAR(NOW(), 'YYYY-MM-01'), 'YYYY-MM-DD') \
GROUP BY d \
ORDER BY d;\" | \
ssh rude.torproject.org psql \\\"host=drobovi.torproject.org sslmode=require user=rtreader dbname=rt\\\" | \
sed 's/|//' \
" using 1:xtic(2) with boxes title "new tickets"
Get the most recent version of each RT articles
SELECT classes.name AS class,
articles.name AS title,
CASE WHEN objectcustomfieldvalues.content != '' THEN objectcustomfieldvalues.content
ELSE objectcustomfieldvalues.largecontent
END AS content,
objectcustomfieldvalues.lastupdated,
articles.id
FROM classes, articles, objectcustomfieldvalues
WHERE articles.class = classes.id
AND objectcustomfieldvalues.objecttype = 'RT::Article'
AND objectcustomfieldvalues.objectid = articles.id
AND objectcustomfieldvalues.id = (
SELECT objectcustomfieldvalues.id
FROM objectcustomfieldvalues
WHERE objectcustomfieldvalues.objectid = articles.id
AND objectcustomfieldvalues.disabled = 0
ORDER BY objectcustomfieldvalues.lastupdated DESC
LIMIT 1)
ORDER BY classes.id, articles.id;
Granting access to a support help desk coordinator
The support help desk coordinator needs the following assets to perform their duties:
- Administration password for the support-team-private mailing list.
- Being owner in the support-team-private mailing list configuration.
- Commit access to help wiki Git repository.
- Shell access to rude.torproject.org.
- LDAP account member of the
rtfolks
group. - LDAP account member of the
support
group. -
root
password for Request Tracker. - Being owner of the “Tor Support” component in Trac.
New RT admin
This task is typically done by TPA, but can technically be done by any RT admin.
-
find the RT admin password in
hosts-extra-info
in the TPA password manager and login as root OR login as your normal RT admin user -
create an account member of
rt-admin
Pager playbook
Ticket creation failed / No permission to create tickets in the queue
If you receive an email like this:
From: rt@rt.torproject.org Subject: Ticket creation failed: [ORIGINAL SUBJECT] To: root@rude.torproject.org Date: Tue, 05 Jan 2021 01:01:21 +0000
No permission to create tickets in the queue 'help'
[ORIGINAL EMAIL]
It is because a disabled user tried to send an email to RT. It is unclear how to handle those issues in the future, but to remove such warnings, the user could just be re-enabled.
Alternatively, sysadmins can block the user completely by adding the
user to the /etc/postfix/access-2-sender
map and reload it with:
postmap /etc/postfix/access-2-sender
See Also issue 33314 for more information.
Email delivery fails
If no email is coming into RT, it might be one of the spam filters that failed to start. Nagios would complain with a message like:
Subject: ** PROBLEM Service Alert: rude/process - spampd - master is CRITICAL **
[...]
Service: process - spampd - master
Host: rude
Address: 116.202.120.187
State: CRITICAL
Date/Time: Thu Feb 11 14:53:44 UTC 2021
Additional Info:
PROCS CRITICAL: 0 processes with UID = 112 (spampd), command name spampd, PPID = 1
It's still unclear why this happens (see e.g. issue 40175), but a
workaround is to either reboot the server or restart spampd
:
systemctl restart spampd
Reference
Installation
No clue. Probably Debian-ish?
Configuration is readable in /etc/request-tracker4/RT_SiteConfig.d/
.
There's no logs. No clue why.
Auto-reply to new requestors
When an unknown email address sends an email to the support, it will be automatically replied to warn users about the data retention policy.
A global Scrip is responsible for this. It will be default use the global template named “Initial reply”. It is written in English. In each queue except help, a template named exactly “Initial reply” is defined in order to localize the message.
Expiration of old tickets
Tickets (and affiliated users) get erased from the RT database after 100 days. This is done by the expire-old-tickets script. The script is run everyday at 06:02 UTC through a cronjob run as user colin
.
Encrypted SQL dumps of the data removed from the database will be written to /srv/rtstuff/shredded
and must be put away regularily.
Dump of RT templates
RT articles are dumped into text files and then pushed to the rt-articles Git repository. An email is sent each time there's a new commit, so collective reviews can happen by the rest of the support team.
The machinery is spread through several scripts. The one run on rude
is dump_rt_articles, and it will run everyday through a cronjob as user colin
.
Issues
There is no issue tracker specifically for this project, File or search for issues in the team issue tracker with the ~RT label.
Discussion
Spam filter training design
RT is designed to be trained for spam filtering. RT users put spam in the "Spam" queue and then a set of scripts run in the background to train spamassassin, based on a mail archive that procmail keeps of every incoming mail.
This runs as a cronjob in the rtmailarchive
user, which looks like
this:
/srv/rtstuff/support-tools/train-spam-filters/train_spam_filters && bin/spam-learn && find Maildir/.spam.learned Maildir/.xham.learned -type f -delete
The train_spam_filters script basically does this:
- for each mail in the
Maildir/.help*
archive - find its Message-Id header
- load the equivalent message from RT:
- if it is in the Spam queue, marked as "Rejected", it is spam.
- if it is in a help-* queue, marked as "Resolved", it is ham.
- move the email in the right directory mail folder (
.spam.learn
,.xham.learn
) depending on status - if the file is more than 100 days old, delete it.
Then the rest of the cron job continues. spam-learn
is this shell
script:
#!/bin/bash
dbpath="/var/cache/spampd"
learn() {
local what="$1"; shift;
local whence="$1"; shift;
local whereto="$1"; shift;
(
cd "$whence"
find -type f | \
while read f; do
sudo -u spampd -H sa-learn --dbpath "$dbpath" --"$what" < "$f"
mv "$f" "$whereto/$f"
done
)
}
set -e
learn spam /srv/rtmailarchive/Maildir/.spam.learn /srv/rtmailarchive/Maildir/.spam.learned
learn ham /srv/rtmailarchive/Maildir/.xham.learn /srv/rtmailarchive/Maildir/.xham.learned
# vim:set et:
# vim:set ts=4:
# vim:set shiftwidth=4:
which, basically, calls sa-learn
on each individual email in the
folder, moving it to .spam.learned
or .xham.learned
when done.
Then, interestingly, those emails are destroyed. It's unclear why that
is not done in the spam-learn
step directly.
Possible improvements
The above design has a few problems:
- it assumes "ham" queues are named "help-*" - but there are other queues in the system
- it might be slow: if there are lots of emails to process, it will do an SQL query for each and a move, and not all at once
- it is split over multiple shell scripts, not versioned
I would recommend the following:
- reverse the logic of the queue checks: instead of checking for
folders and queues named
help-*
, check if the folders or queues are not namedspam*
orxham*
- batch jobs: use a generator to yield Message-Id, then pick a certain number of emails and batch-send them to psql and the rename
- do all operations at once: look in psql, move the files in the learning folder, and train, possibly in parallel, but at least all in the same script
- sa-learn can read from a folder now, so there's no need for that wrapper shell script in any case
- commit the script to version control and, even better, puppet
We could also add a CAPTCHA and look at the RT::Extension::ReportSpam...
Alternatives
- Zammad has a free version, but it's unclear which features the latter has... the .com site has a bunch of features that could prove interesting for us, particularly the GitLab integration UPDATE: note that the community team has been experimenting with an external Zammad instance (not managed by TPA), see tpo/tpa/team#40578 for details.
- Freescout