crm.md

CRM stands for "Customer Relationship Management" but we actually use it to manage contacts and donations. It is how we send our massive newsletter once in a while.

• Tutorial
  • Basic access
• Howto
  • Monitoring mailings
  • Handling abuse complains
  • Pager playbook
  • Disaster recovery
• Reference
  • Installation
  • SLA
  • Design
    • Services
      • Access
      • Components
  • Issues
  • Maintainer, users, and upstream
  • Monitoring and testing
  • Logs and metrics
  • Backups
  • Other documentation
• Discussion
  • Overview
  • Goals
    • Must have
    • Nice to have
    • Non-Goals
  • Approvals required
  • Proposed Solution
  • Cost
  • Alternatives considered

Tutorial

Basic access

The main website is at:

https://crm.torproject.org/

It is protected by basic authentication and the site's login as well, so you actually need two sets of password to get in.

Howto

Monitoring mailings

The CiviCRM server can generate large mailings, in the order of hundreds of thousands of unique email addresses. Those can create significant load on the server if mishandled, and worse, trigger blocking at various providers if not correctly rate-limited.

For this, we have various knobs and tools:

• Grafana dashboard watching the two main mailservers
• Place to enable/disable mailing (grep for Send sched...)
• Where the batches are defined
• The Civimail interface should show the latest mailings (when clicking twice on "STARTED", from there click the Report button to see how many mails have been sent, bounced, etc

The Grafana dashboard is based on metrics from Prometheus, which can be inspected live with the following command:

curl -s localhost:3903/metrics | grep -v -e ^go_ -e '^#' -e '^mtail' -e ^process -e _tls_; postfix-queues-sizes

Using lnav can also be useful to monitor logs in real time, as it provides per-queue ID navigation, marks warnings (deferred messages) in yellow and errors (bounces) in red.

A few commands to inspect the email queue:

• list the queue, with more recent entries first

   postqueue -j | jq -C .recipients[] | tac

• find how many emails in the queue, per domain:

   postqueue -j | jq -r .recipients[].address | sed 's/.*@//' | sort | uniq -c | sort -n

  Note that the qshape deferred command gives a similar (and actually better) output.

In case of a major problem, you can stop the mailing in CiviCRM and put all emails on hold with:

postsuper -h ALL

Then the postfix-trickle script can be used to slowly release emails:

postfix-trickle 10 5

When an email bounces, it should go to civicrm@crm.torproject.org, which is an IMAP mailbox periodically checked by CiviCRM. It will ingest bounces landing in that mailbox and disable them for the next mailings. It's also how users can unsubscribe from those mailings, so it is critical that this service runs correctly.

A lot of those notes come from the issue where we enabled CiviCRM to receive its bounces.

Handling abuse complains

Our postmaster alias can receive emails like this:

Subject: Abuse Message [AbuseID:809C16:27]: AbuseFBL: UOL Abuse Report

Those emails usually contain enough information to figure out which email address filed a complaint. The action to take is to remove them from the mailing. Here's an example email sample:

Received: by crm-int-01.torproject.org (Postfix, from userid 33)
        id 579C510392E; Thu, 4 Feb 2021 17:30:12 +0000 (UTC)
[...]
Message-Id: <20210204173012.579C510392E@crm-int-01.torproject.org>
[...]
List-Unsubscribe: <mailto:civicrm+u.2936.7009506.26d7b951968ebe4b@crm.torproject.org>
job_id: 2936
Precedence: bulk
[...]
X-CiviMail-Bounce: civicrm+b.2936.7009506.26d7b951968ebe4b@crm.torproject.org
[...]

Your bounce might have only some of those. Possible courses of action to find the victim's email:

1. Grep for the queue ID (579C510392E) in the mail logs
2. Grep for the Message-Id (20210204173012.579C510392E@crm-int-01.torproject.org) in mail logs (with postfix-trace)

Once you have the email address:

1. head for the CiviCRM search interface to find that user
2. remove the from the "Tor News" group, in the Group tab

You can also just send email to the List-Unsubscribe address or click the "unsubscribe" links at the bottom of the email.

Pager playbook

Disaster recovery

Reference

Installation

Full documentation on the installation of this system is somewhat out of scope for TPA: sysadmins only installed the servers and setup basic services like a VPN (using IPsec) and an Apache, PHP, MySQL stack.

The Puppet classes used on the two servers are roles::civicrm_int_2018 and roles::civicrm_ext_2018.

SLA

This service is critical, as it is used to host donations, and should be as highly available as possible. Unfortunately, its design has multiple single point of failures, which, in practice, makes this target difficult to fulfill at this point.

Design

Services

The CRM service is built with two distinct servers:

• crm-int-01.torproject.org, AKA crm-int-01
  • software:
    • CiviCRM on top of Drupal
    • Apache with PHP FPM
    • MariaDB (MySQL) database (Drupal storage backend)
    • Redis cache (?)
    • Dovecot IMAP server (to handle bounces)
  • sites:
    • crm.torproject.org: production CiviCRM site
    • staging.crm.torproject.org: staging site
    • test.crm.torproject.org: testing site
• crm-ext-01.torproject.org, AKA crm-ext-01. Runs:
  • software:
    • Apache with PHP FPM
  • sites:
    • donate-api.torproject.org: production donation API middleware
    • staging.donate-api.torproject.org: staging API
    • test.donate-api.torproject.org: testing API
    • api.donate.torproject.org: not live yet
    • staging-api.donate.torproject.org: not live yet
    • test-api.donate.torproject.org: test site to rename the API middleware (see issue 40123)

Access

The CRM doesn't talk to the outside internet and can be accessed only via http authentication.

Users that need to access the CRM need both to be added on Civi and on the apache on crm-ext-01.tpo.

Components

The Civi CRM instance is used for:

• Mass emailing via the Newsletter
• Donation campaigns

The monthly newsletter is configured on Civi and an archive is created on https://newsletter.torproject.org.

Donation campaigns are setup both on Civi and the donate website (donate.torproject.org.

https://donate.torproject.org is a static website built with Lektor like all the other tpo websites.

https://donate.torproject.org doesn't talk to Civi. The donate-api php app is the component that allows communications via the donate.torproject.org site and Civi.

The donate.tpo site talks with the donation API middleware through Javascript - a react component was written for this and is part of the donate-static repository.

Issues

There is no issue tracker specifically for this project, File or search for issues in the team issue tracker.

Maintainer, users, and upstream

Monitoring and testing

Logs and metrics

Backups

Other documentation

Discussion

Overview

Goals

Must have

Nice to have

Non-Goals

Approvals required

Proposed Solution

Cost

Alternatives considered