Merge remote-tracking branch 'stephen/donate-docs-update-post-launch'

Closes: tpo/web/donate-neo#79

service/donate.md

@@ -506,23 +506,68 @@ There is a staging server and development "review apps"
 (`donate-review-01`) that is managed by a `gitlab-runner` and driven
 by GitLab CI.
 
-The Django app is designed to be simple: all it's really doing is
-making an API request to CiviCRM, doing some templating, validating a
-form, and using the paypal/stripe API.
+The Django app is designed to be simple: all it's really doing is some
+templating, validating a form, implementing the payment vendor APIs, and
+sending donation information to CiviCRM.
 
-Here is a sequence diagram built by @kez in January 2023
-([tpo/web/donate-static#107](https://gitlab.torproject.org/tpo/web/donate-static/-/issues/107#note_2872652)):
+This simplicity is powered, in part, by a dependency injection framework
+which more straightforwardly allows Django apps to leverage data or methods
+from parallel apps without constantly instantiating transient instances of
+those other apps.
+
+Here is a relationship diagram by @stephen outlining this dependency tree:
+
+```mermaid
+erDiagram
+    Redis ||--|{ CiviCRM : "Redis/Resque DAL"
+    CiviCRM ||--|{ "Main app (donation form model & view)": "Perk & minimum-donation data"
+    CiviCRM ||--|{ "Stripe app": "Donation-related CRM methods"
+    CiviCRM ||--|{ "PayPal app": "Donation-related CRM methods"
+```
+
+Despite this simplicity, `donate-neo`'s final design is more complex than its
+original thumbnailed design. This is largely due to the differential between
+`donate-paleo`'s implementation of Stripe and PayPal payments, which have
+changed and become more strictly implemented over time.
+
+In particular, earlier designs for the donate page treated the time-of-transaction
+result of a donation attempt as canonical. However, both Stripe and PayPal
+now send webhook messages post-donation intended to serve as the final word on
+whether a transaction was accepted or rejected. `donate-neo` therefore requires
+confirmation of a transaction via webhook before sending donation data to CiviCRM.
+
+Also of note is the way CiviCRM-held perk information and donation minimums
+are sent to `donate-neo`. In early design discussions between @mathieu and @kez,
+this data was intended to be retrieved via straightforward HTTP requests to
+CiviCRM's API. However, this turned out to be at cross-purposes with the server
+architecture design, in which communication between the Django server and the
+CiviCRM server would only occur via [IPsec](howto/ipsec) tunnel. 
+
+As a result, perk and donation minimum data is exported from CiviCRM and stored
+in the `donate-neo` repository as a JSON file. (Note that as of this writing,
+the raw export of that data by CiviCRM is not valid JSON and must be massaged
+by hand before `donate-neo` can read it.)
+
+Following is a sequence diagram by @stephen describing the donation flow from
+user-initiated page request to receipt by CiviCRM:
 
 ```mermaid
 sequenceDiagram
-    user->>donate.tpo: visits the donation site
-    donate.tpo->>civicrm: requests the current perks, and prices
-    civicrm->>donate.tpo: stickers: 25, t-shirt: 75...
-    donate.tpo->>user: responds with a fully-rendered donation form
-    user->>donate.tpo: submits the donation form with stripe/paypal details
-    donate.tpo->>donate.tpo: validates form, creates payment contract with stripe/paypal
-    donate.tpo->>civicrm: submits donation and perk info
-    donate.tpo->>user: redirects to donation thank you page
+    actor user
+    participant donate as donate tpo
+    participant civi as civicrm
+    participant pp as payment processor
+    Note left of civi: Perk data periodically exported from CiviCRM
+    Note right of donate: Perk data added to donate repo as JSON
+    user->>donate: Visits the donation site
+    donate->>user: Responds with a fully-rendered donation form
+    pp->>user: Embeds payment interface on page via vendor-hosted JS
+    user->>donate: Completes and submits donation form
+    donate->>donate: Validates form, creates payment contract with Stripe/PayPal
+    donate->>pp: Initiates payment process
+    donate->>user: Redirects to donation thank you page
+    pp->>donate: Sends webhook confirming results of transaction
+    donate->>civi: Submits donation and perk info
 ```
 
 Another possible implementation was this:
@@ -539,8 +584,7 @@ graph TD
     F --> H(civi gets the donation info from the django backend, and adds it to the civi database without trying to validate the donation amount or perks/swag)
 ```
 
-As of this writing (August 2024), it's unclear whether the actual
-design follows this, see [tpo/web/donate-neo#79](https://gitlab.torproject.org/tpo/web/donate-neo/-/issues/79) for the task of
+See [tpo/web/donate-neo#79](https://gitlab.torproject.org/tpo/web/donate-neo/-/issues/79) for the task of
 clarifying those docs.
 
 ### Review apps
@@ -620,8 +664,8 @@ the CiviCRM backend. It handles those types of transactions:
 The Redis server runs on the CiviCRM server, and is accessed through
 an IPsec tunnel, see the [authentication](#authentication) section below as
 well. The Django application reimplements the [resque](https://resque.github.io/) queue
-(originally written in Ruby, but ported to PHP by GiantRabbit) to pass
-messages to the CiviCRM backend.
+(originally written in Ruby, ported to PHP by GiantRabbit, and here ported to Python)
+to pass messages to the CiviCRM backend.
 
 Both types of donations and mailing list subscriptions are confirmed before
 they are queued for processing by CiviCRM. In both cases, unconfirmed data
@@ -643,6 +687,10 @@ but they will not pair with any stored form data, so they are passed along to Ci
 with a `recurring_billing_id` that CiviCRM uses to group them with a
 recurring donation series. 
 
+Recurring PayPal donations first made on `donate-paleo` also issue legacy IPN messages,
+and have a separate handler and validator from webhooks, but contain data conforming
+to the Resque handler and so are passed to CiviCRM and processed in the same manner.
+
 Confirming mailing list subscriptions works similarly to confirming donations,
 but we also coordinate the confirmation process ourselves.
 Donors who check the "subscribe me!" box in the donation form generate
@@ -658,9 +706,10 @@ outside of the purview of this writeup).
 
 ## Interfaces
 
-Most of the interactions with donate happen over HTTP. Payment
-providers ping back the site with webhook endpoints which have to
-bypass CSRF protections.
+Most of the interactions with donate happen over HTTP. Payment providers
+ping back the site with webhook endpoints (and, in the case of legacy
+`donate-paleo` NVP/SOAP API recurring payments, a PayPal-specific "IPN" endpoint)
+which have to bypass CSRF protections.
 
 The views handling these endpoints are designed to only reply with HTTP
 status codes (200 or 400). If the message is legitimate but was malformed
@@ -748,7 +797,7 @@ documented in the [pyproject.toml](https://gitlab.torproject.org/tpo/web/donate-
 poetry.lock](https://gitlab.torproject.org/tpo/web/donate-neo/-/blob/main/poetry.lock?ref_type=heads) file contains actual versions currently deployed.
 
 Poetry is used to manage dependencies and builds. The frontend CSS /
-JS code is managed with NVM. The [README file](https://gitlab.torproject.org/tpo/web/donate-neo/-/tree/main?ref_type=heads) has more information
+JS code is managed with NPM. The [README file](https://gitlab.torproject.org/tpo/web/donate-neo/-/tree/main?ref_type=heads) has more information
 about the development setup.
 
 ## Related services
@@ -818,6 +867,10 @@ Also note that the CiviCRM side of things has its own metrics, see the
 
 ## Tests
 
+The `pytest` test suite can be run by entering a `poetry shell` and running:
+
+    coverage run manage.py test
+
 To test donations after upgrades or to confirm everything works, see
 the [Testing the donation site](#testing-the-donation-site) section.
 
@@ -960,6 +1013,18 @@ ensure no untrusted party is allowed to gain this privilege.
 
 ## Technical debt and next steps
 
+### PII handling and Stripe Radar
+
+`donate-neo` is severely opinionated about user PII; it attempts to handle
+it as little as is necessary and discard it as soon as possible. This is
+at odds with Stripe Radar's fraud detection algorithm, which weights 
+a given transaction as "less fraudulent" the more user PII is attached to it.
+This clash is compounded by the number of well-intended donors using Tor
+exit node IPs - some of which which bear low reputation scores with Stripe
+due to bad behavior by prior users. This results in some transactions
+being rejected due to receiving insufficient signals of legitimacy.
+See Stripe's docs [here](https://docs.stripe.com/disputes/prevention/advanced-fraud-detection) and [here](https://docs.stripe.com/radar/integration).
+
 ### Dependencies chase
 
 The `renovate-cron` project should be used on the donate-neo codebase