feat: Send CiviCRM only complete and uncomplicated messages
This MR ties together several disparate threads to accomplish one goal - Sending CiviCRM exactly one message for every webhook-receivable donation-related event about which it might expect to know.
These events are:
- Successful one-time contributions, and the donation form data related to them (mailing address to send perks to, particular perk-options, etc)
- Successful subscription payments, with related form data if this was the initial subscription kicked off by a donation form submission
- Unsuccessful payments, especially subscription payments which have failed
The state of this codebase prior to this MR gestured at this functionality in several ways but did not actually fully perform it. Two outstanding needs had to be addressed for this to be possible:
- Outbound data headed to CiviCRM must include donation form data where relevant
- Outbound data headed to CiviCRM must be straightforward for CiviCRM to parse
The following details how needs were addressed, giving context for each code change in the MR.
NEED: Resque messages about donations did not contain donation form data
SOLUTION: Successful donation form submissions now hand off form data to civicrm.repository.handle_donation_form_data()
, where data is pruned and normalized before being temporarily queued in Redis via civicrm.repository.queue_donation()
. When payment gateways message our webhook endpoints with information about donations, we validate and normalize that message data as normal, then hand off to civicrm.repository.report_donation()
, which now additionally calls new method civicrm.repository.retrieve_donation()
to pull any donation form data matching a unique shared identifier available to both an in-progress transaction and a payment gateway-issued webhook message confirming said transaction's validity. This form data is added to the incoming webhook message data relevant to CiviCRM before being sent to redis.resque_send()
to be enqueued in Redis as a Resque message.
RELEVANT FILES MODIFIED: static/js/modules/paypal.js
-
setupPaypalOneTimeButtons()
andsetupPaypalSubscriptionButtons()
modified to pass valid form data topaypal.views.create_order()
andpaypal.views.capture_subscription()
civicrm/repository.py - handle_donation_form_data() added
- queue_donation() added
- retrieve_donation() added stripe/controller.py
- send_donation_info_to_crm() was removed, as it was not used before now and ultimately became
civicrm.repository.handle_donation_form_data()
- validate_webhook() adjusted to handle cases where HTTP signatures are missing more gracefully
- process_webhook() adjusted to ensure
args["trxn_id"]
contains the value expected bycivicrm.repository.queue_donation()
andcivicrm.repository.retrieve_donation()
, and to ensurecivicrm.repository.donation_exists()
is called with args["trxn_id"], which is sure to be there, rather than event["id"], which is not paypal/controller.py - Expanded webhook handling in
paypal.controller.process_webhook()
to cover one-time contributions as well as recurring donations - Refactored
paypal.controller.process_webhook()
'smatch
block to ensure each case contained correctmessage
andargs["trxn_id"]
values before passing tocivicrm.repository.report_donation()
- Refactored
paypal.controller.process_webhook()
to ensurecivicrm.repository.donation_exists()
is called with args["trxn_id"], which is sure to be there, rather than event["id"], which is not - and that we only report webhook messages which we whitelist as having a Resque route (sent tocivicrm.repository.report_donation()
asmessage
) paypal/urls.py -
/capture-subscription/
was added to send form data topaypal.views.capture_subscription()
paypal/views.py - create_order() was modified to call
civicrm.repository.handle_donation_form_data()
- capture_subscription() was added in order to call
civicrm.repository.handle_donation_form_data()
with data mappable to subscription-related Paypal webhook messages - The webhook handler was adjusted slightly to more explicitly return a 400 status in cases where a webhook message fails to earn a status 200 response stripe/views.py
- process() was updated to call
civicrm.repository.handle_donation_form_data()
for both one-time and recurring donations
NEED: Perk information in Resque messages must be straightforward for Resque to programatically interpret
SOLUTION: The donate form HTML was modified so perk option select
fields would always have names prepended with perk_option--
. (Javascript related to these fields which refers to some by name was adjusted to match.) When civicrm.repository.handle_donation_form_data()
it prunes and normalizes donation form data to remove duplicate fields or fields unrelated to CiviCRM's own interests. As well, any selected perk options are assigned to their own new key in that form data, perk_option
, so that CiviCRM doesn't have to fuzzy-match fields on its end looking for that perk option.
RELEVANT FILES MODIFIED: templates/donate.html.jinja
- Classes, IDs, and names changed in HTML related to option groups, such that resultant form field values will have semantically-useful names and values static/js/script.js
- Option group toggling code improved and updated to match new tag IDs civicrm/repository.py
- handle_donation_form_data() added
ADDITIONALLY stripe/views.py
- webhook() was modified to explicity handle
KeyError
s thrown bystripe.controller.validate_webhook()
, as well asValueError
s, so as to prevent people from being able to crash the server by sending the Stripe webhook endpoint a message without astripe-signature
header. (We now log the occurrence instead.) various testing files - Various tests have been updated to reflect above changes