Changes

Page history
docs: Hydrating donate docs in underpopulated sections authored by stephen's avatar stephen 
Major documentation updates made to:
- Design and architecture
  - Written description regarding how and why donate-neo design diverged between initial planning and execution
  - Mermaid diagrams outlining both DI-driven data relationships between Django apps and user donation checkout flow
- Technical debt, outlining the friction between donate-neo's PII minimal handling and Stripe Radar's fraud detection algorithm

Minor documentation updates made to:
- Queues, regarding the custom Resque handler and PayPal IPN handling
- Interfaces, regarding PayPal IPN handling
- Implementation, clarifying that CSS and JS are managed via NPM rather than NVM (which itself simply manages local NPM versioning)
- Tests, listing the command for running unit tests
Hide whitespace changes
Inline Side-by-side
service/donate.md
View page @ e2d4449f
...@@ -452,28 +452,69 @@ There is a staging server and development "review apps" ...@@ -452,28 +452,69 @@ There is a staging server and development "review apps"
(`donate-review-01`) that is managed by a `gitlab-runner` and driven (`donate-review-01`) that is managed by a `gitlab-runner` and driven
by GitLab CI. by GitLab CI.
The Django app is designed to be simple: all it's really doing is The Django app is designed to be simple: all it's really doing is some
making an API request to CiviCRM, doing some templating, validating a templating, validating a form, implementing the payment vendor APIs, and
form, and using the paypal/stripe API. sending donation information to CiviCRM.
Here is a sequence diagram built by @kez in January 2023 This simplicity is powered, in part, by a dependency injection framework
([tpo/web/donate-static#107](https://gitlab.torproject.org/tpo/web/donate-static/-/issues/107#note_2872652)): which more straightforwardly allows Django apps to leverage data or methods
from parallel apps without constantly instantiating transient instances of
those other apps.
Here is a relationship diagram by @stephen outlining this dependency tree:
```mermaid ```mermaid
sequenceDiagram erDiagram
user->>donate.tpo: visits the donation site Redis ||--|{ CiviCRM : "Redis/Resque DAL"
donate.tpo->>civicrm: requests the current perks, and prices CiviCRM ||--|{ "Main app (donation form model & view)": "Perk & minimum-donation data"
civicrm->>donate.tpo: stickers: 25, t-shirt: 75... CiviCRM ||--|{ "Stripe app": "Donation-related CRM methods"
donate.tpo->>user: responds with a fully-rendered donation form CiviCRM ||--|{ "PayPal app": "Donation-related CRM methods"
user->>donate.tpo: submits the donation form with stripe/paypal details
donate.tpo->>donate.tpo: validates form, creates payment contract with stripe/paypal
donate.tpo->>civicrm: submits donation and perk info
donate.tpo->>user: redirects to donation thank you page
``` ```
As of this writing (August 2024), it's unclear whether the actual Despite this simplicity, `donate-neo`'s final design is more complex than its
design follows this, see [tpo/web/donate-neo#79](https://gitlab.torproject.org/tpo/web/donate-neo/-/issues/79) for the task of original thumbnailed design. This is largely due to the differential between
clarifying those docs. `donate-paleo`'s implementation of Stripe and PayPal payments, which have
changed and become more strictly implemented over time.
In particular, earlier designs for the donate page treated the time-of-transaction
result of a donation attempt as canonical. However, both Stripe and PayPal
now send webhook messages post-donation intended to serve as the final word on
whether a transaction was accepted or rejected. `donate-neo` therefore requires
confirmation of a transaction via webhook before sending donation data to CiviCRM.
Also of note is the way CiviCRM-held perk information and donation minimums
are sent to `donate-neo`. In early design discussions between @mathieu and @kez,
this data was intended to be retrieved via straightforward HTTP requests to
CiviCRM's API. However, this turned out to be at cross-purposes with the server
architecture design, in which communication between the Django server and the
CiviCRM server would only occur via [IPsec](howto/ipsec) tunnel.
As a result, perk and donation minimum data is exported from CiviCRM and stored
in the `donate-neo` repository as a JSON file. (Note that as of this writing,
the raw export of that data by CiviCRM is not valid JSON and must be massaged
by hand before `donate-neo` can read it.)
Following is a sequence diagram by @stephen describing the donation flow from
user-initiated page request to receipt by CiviCRM:
```mermaid
sequenceDiagram
actor user
participant donate as donate tpo
participant civi as civicrm
participant pp as payment processor
Note left of civi: Perk data periodically exported from CiviCRM
Note right of donate: Perk data added to donate repo as JSON
user->>donate: Visits the donation site
donate->>user: Responds with a fully-rendered donation form
pp->>user: Embeds payment interface on page via vendor-hosted JS
user->>donate: Completes and submits donation form
donate->>donate: Validates form, creates payment contract with Stripe/PayPal
donate->>pp: Initiates payment process
donate->>user: Redirects to donation thank you page
pp->>donate: Sends webhook confirming results of transaction
donate->>civi: Submits donation and perk info
```
### Review apps ### Review apps
...@@ -547,8 +588,8 @@ the CiviCRM backend. It handles those types of transactions: ...@@ -547,8 +588,8 @@ the CiviCRM backend. It handles those types of transactions:
The Redis server runs on the CiviCRM server, and is accessed through The Redis server runs on the CiviCRM server, and is accessed through
an IPsec tunnel, see the [authentication](#authentication) section below as an IPsec tunnel, see the [authentication](#authentication) section below as
well. The Django application reimplements the [resque](https://resque.github.io/) queue well. The Django application reimplements the [resque](https://resque.github.io/) queue
(originally written in Ruby, but ported to PHP by GiantRabbit) to pass (originally written in Ruby, ported to PHP by GiantRabbit, and here ported to Python)
messages to the CiviCRM backend. to pass messages to the CiviCRM backend.
Both types of donations and mailing list subscriptions are confirmed before Both types of donations and mailing list subscriptions are confirmed before
they are queued for processing by CiviCRM. In both cases, unconfirmed data they are queued for processing by CiviCRM. In both cases, unconfirmed data
...@@ -570,6 +611,10 @@ but they will not pair with any stored form data, so they are passed along to Ci ...@@ -570,6 +611,10 @@ but they will not pair with any stored form data, so they are passed along to Ci
with a `recurring_billing_id` that CiviCRM uses to group them with a with a `recurring_billing_id` that CiviCRM uses to group them with a
recurring donation series. recurring donation series.
Recurring PayPal donations first made on `donate-paleo` also issue legacy IPN messages,
and have a separate handler and validator from webhooks, but contain data conforming
to the Resque handler and so are passed to CiviCRM and processed in the same manner.
Confirming mailing list subscriptions works similarly to confirming donations, Confirming mailing list subscriptions works similarly to confirming donations,
but we also coordinate the confirmation process ourselves. but we also coordinate the confirmation process ourselves.
Donors who check the "subscribe me!" box in the donation form generate Donors who check the "subscribe me!" box in the donation form generate
...@@ -585,9 +630,10 @@ outside of the purview of this writeup). ...@@ -585,9 +630,10 @@ outside of the purview of this writeup).
## Interfaces ## Interfaces
Most of the interactions with donate happen over HTTP. Payment Most of the interactions with donate happen over HTTP. Payment providers
providers ping back the site with webhook endpoints which have to ping back the site with webhook endpoints (and, in the case of legacy
bypass CSRF protections. `donate-paleo` NVP/SOAP API recurring payments, a PayPal-specific "IPN" endpoint)
which have to bypass CSRF protections.
The views handling these endpoints are designed to only reply with HTTP The views handling these endpoints are designed to only reply with HTTP
status codes (200 or 400). If the message is legitimate but was malformed status codes (200 or 400). If the message is legitimate but was malformed
...@@ -673,7 +719,7 @@ documented in the [pyproject.toml](https://gitlab.torproject.org/tpo/web/donate- ...@@ -673,7 +719,7 @@ documented in the [pyproject.toml](https://gitlab.torproject.org/tpo/web/donate-
poetry.lock](https://gitlab.torproject.org/tpo/web/donate-neo/-/blob/main/poetry.lock?ref_type=heads) file contains actual versions currently deployed. poetry.lock](https://gitlab.torproject.org/tpo/web/donate-neo/-/blob/main/poetry.lock?ref_type=heads) file contains actual versions currently deployed.
Poetry is used to manage dependencies and builds. The frontend CSS / Poetry is used to manage dependencies and builds. The frontend CSS /
JS code is managed with NVM. The [README file](https://gitlab.torproject.org/tpo/web/donate-neo/-/tree/main?ref_type=heads) has more information JS code is managed with NPM. The [README file](https://gitlab.torproject.org/tpo/web/donate-neo/-/tree/main?ref_type=heads) has more information
about the development setup. about the development setup.
## Related services ## Related services
...@@ -724,7 +770,10 @@ website, and through Discord, a forum, and GitHub issues. ...@@ -724,7 +770,10 @@ website, and through Discord, a forum, and GitHub issues.
<!-- how the service can be tested, for example after major changes --> <!-- how the service can be tested, for example after major changes -->
<!-- like IP address changes or upgrades. describe CI, test suites, linting --> <!-- like IP address changes or upgrades. describe CI, test suites, linting -->
TODO: how do we run the test suite or test that things are still working after an upgrade? (@stephen) also cross-ref with @mattlav's checklist
The `pytest` test suite can be run by entering a `poetry shell` and running:
coverage run manage.py test
To test donations after upgrades or to confirm everything works, see To test donations after upgrades or to confirm everything works, see
the [Testing the donation site](#testing-the-donation-site) section. the [Testing the donation site](#testing-the-donation-site) section.
...@@ -854,6 +903,18 @@ ensure no untrusted party is allowed to gain this privilege. ...@@ -854,6 +903,18 @@ ensure no untrusted party is allowed to gain this privilege.
## Technical debt and next steps ## Technical debt and next steps
### PII handling and Stripe Radar
`donate-neo` is severely opinionated about user PII; it attempts to handle
it as little as is necessary and discard it as soon as possible. This is
at odds with Stripe Radar's fraud detection algorithm, which weights
a given transaction as "less fraudulent" the more user PII is attached to it.
This clash is compounded by the number of well-intended donors using Tor
exit node IPs - some of which which bear low reputation scores with Stripe
due to bad behavior by prior users. This results in some transactions
being rejected due to receiving insufficient signals of legitimacy.
See Stripe's docs [here](https://docs.stripe.com/disputes/prevention/advanced-fraud-detection) and [here](https://docs.stripe.com/radar/integration).
### Dependencies chase ### Dependencies chase
The `renovate-cron` project should be used on the donate-neo codebase The `renovate-cron` project should be used on the donate-neo codebase
... ...
......