document CRM token rotation procedures (tpo/tpa/team#41530)

service/crm.md

@@ -140,11 +140,68 @@ accessible only by TPA. To add a user, on the backend server
 
     htdigest /etc/apache2/htdigest 'Tor CRM' $USERNAME
 
+## Rotating API tokens
+
+If we feel our API tokens might have been exposed, or staff leaves and
+we would feel more comfortable replacing those secrets, we need to
+rotate API tokens. There are two to replace: Stripe and PayPal keys.
+
+### Stripe rotation procedure
+
+Stripe has an excellent [Stripe roll key](https://docs.stripe.com/keys#rolling-keys) procedure. You first need
+to have a [developer account](https://docs.stripe.com/payments/account/teams/roles#developer) (ask accounting) then head over to
+the [test API keys page](https://dashboard.stripe.com/test/apikeys). You will first rotate the API keys,
+test that staging still works, then rotate the live keys. Here's the
+full procedure.
+
+ 1. test that [staging](https://donate.staging.torproject.net/) still works *before* the change (see the
+    [test procedure](#tests)), as it's possible it's broken for other
+    reasons. if it *is* broken, fix that first.
+
+ 2. roll the API key, with a 24h expiration
+
+ 3. deploy the new secret on the middleware, on
+    `tordonate@crm-ext-01.torproject.org`, in the file
+    `/srv/donate.torproject.org/htdocs-staging/private/settings.local.php`
+
+ 4. test donations on staging, again: the transaction should show up
+    in the staging CiviCRM server and the "test" Stripe environment
+ 
+ 5. wait 24h
+
+ 6. test staging again (since the old key is now expired)
+
+ 7. run steps 1-6 with the production site, except with a 1h delay
+
+Note that the "public" part of the key is stored in multiple
+places. It's possible this was changed (in staging, in particular) but
+not correctly updated everywhere. On top of the above
+`private/settings.local.php`, the key is also in `databags/donate.ini`
+on the [donate-static](https://gitlab.torproject.org/tpo/web/donate-static/) site.
+
+### PayPal rotation procedure
+
+A similar procedure can be followed for PayPal, but has not been
+documented thoroughly.
+
+To the best of our best knowledge right now, if you log in to the
+developer dashboard and select "apps & credentials" there should be a
+section labeled "REST API Apps" which contains the application we're
+using for the live site - it should have a listing for the client ID
+and app secret (as well as a separate section somewhere for the
+sandbox client id and app secret)."
+
 ## Pager playbook
 
-<!-- information about common errors from the monitoring system and -->
-<!-- how to deal with them. this should be easy to follow: think of -->
-<!-- your future self, in a stressful situation, tired and hungry. -->
+### Security breach
+
+If there's a major security breach on the service, the first thing to
+do is probably to shutdown the CiviCRM server completely. Halt the
+`crm-int-01` and `crm-ext-01` machines completely, and remove access
+to the underlying storage from the attacker.
+
+Then API keys secrets should probably be rotated, follow the [Rotating
+API tokens procedure](#rotating-api-tokens).
 
 ## Disaster recovery
 
@@ -376,12 +433,13 @@ below.
 
 The donation process can be tested without a real credit card. When the
 frontend (donate.torproject.org static website) is updated, GitLab CI builds and
-deploys a staging version at https://donate.staging.torproject.net
+deploys a staging version at <https://donate.staging.torproject.net>.
 
 It's possible to fill in the donation form on this page, and use [Stripe test
 credit card numbers][] for the payment information. When a donation is
 submitted on this form, it should be processed by the PHP middleware and
-inserted into the staging CiviCRM instance.
+inserted into the staging CiviCRM instance. It should also be visible
+in the "test" Stripe interface.
 
 [Stripe test credit card numbers]: https://stripe.com/docs/testing?testing-method=card-numbers#cards
 

service/password-manager.md

@@ -295,6 +295,12 @@ runners need manual configuration.
 
 The bearer token also needs to be reset for Prometheus monitoring.
 
+### Other services
+
+Each item in the [service list](service) is also probably affected and might
+warrant a review. In particular, you may want to [rotate the CRM
+keys](service/crm#rotating-api-tokens).
+
 ## Pager playbook
 
 This service is likely not going to alert or require emergency