Skip to content

feat: Send CiviCRM only complete and uncomplicated messages

stephen requested to merge send-civicrm-only-complete-notifications into main

This MR ties together several disparate threads to accomplish one goal - Sending CiviCRM exactly one message for every webhook-receivable donation-related event about which it might expect to know.

These events are:

  • Successful one-time contributions, and the donation form data related to them (mailing address to send perks to, particular perk-options, etc)
  • Successful subscription payments, with related form data if this was the initial subscription kicked off by a donation form submission
  • Unsuccessful payments, especially subscription payments which have failed

The state of this codebase prior to this MR gestured at this functionality in several ways but did not actually fully perform it. Two outstanding needs had to be addressed for this to be possible:

  • Outbound data headed to CiviCRM must include donation form data where relevant
  • Outbound data headed to CiviCRM must be straightforward for CiviCRM to parse

The following details how needs were addressed, giving context for each code change in the MR.

NEED: Resque messages about donations did not contain donation form data

SOLUTION: Successful donation form submissions now hand off form data to civicrm.repository.handle_donation_form_data(), where data is pruned and normalized before being temporarily queued in Redis via civicrm.repository.queue_donation(). When payment gateways message our webhook endpoints with information about donations, we validate and normalize that message data as normal, then hand off to civicrm.repository.report_donation(), which now additionally calls new method civicrm.repository.retrieve_donation() to pull any donation form data matching a unique shared identifier available to both an in-progress transaction and a payment gateway-issued webhook message confirming said transaction's validity. This form data is added to the incoming webhook message data relevant to CiviCRM before being sent to redis.resque_send() to be enqueued in Redis as a Resque message.

RELEVANT FILES MODIFIED: static/js/modules/paypal.js

  • setupPaypalOneTimeButtons() and setupPaypalSubscriptionButtons() modified to pass valid form data to paypal.views.create_order() and paypal.views.capture_subscription() civicrm/repository.py
  • handle_donation_form_data() added
  • queue_donation() added
  • retrieve_donation() added stripe/controller.py
  • send_donation_info_to_crm() was removed, as it was not used before now and ultimately became civicrm.repository.handle_donation_form_data()
  • validate_webhook() adjusted to handle cases where HTTP signatures are missing more gracefully
  • process_webhook() adjusted to ensure args["trxn_id"] contains the value expected by civicrm.repository.queue_donation() and civicrm.repository.retrieve_donation(), and to ensure civicrm.repository.donation_exists() is called with args["trxn_id"], which is sure to be there, rather than event["id"], which is not paypal/controller.py
  • Expanded webhook handling in paypal.controller.process_webhook() to cover one-time contributions as well as recurring donations
  • Refactored paypal.controller.process_webhook()'s match block to ensure each case contained correct message and args["trxn_id"] values before passing to civicrm.repository.report_donation()
  • Refactored paypal.controller.process_webhook() to ensure civicrm.repository.donation_exists() is called with args["trxn_id"], which is sure to be there, rather than event["id"], which is not - and that we only report webhook messages which we whitelist as having a Resque route (sent to civicrm.repository.report_donation() as message) paypal/urls.py
  • /capture-subscription/ was added to send form data to paypal.views.capture_subscription() paypal/views.py
  • create_order() was modified to call civicrm.repository.handle_donation_form_data()
  • capture_subscription() was added in order to call civicrm.repository.handle_donation_form_data() with data mappable to subscription-related Paypal webhook messages
  • The webhook handler was adjusted slightly to more explicitly return a 400 status in cases where a webhook message fails to earn a status 200 response stripe/views.py
  • process() was updated to call civicrm.repository.handle_donation_form_data() for both one-time and recurring donations

NEED: Perk information in Resque messages must be straightforward for Resque to programatically interpret

SOLUTION: The donate form HTML was modified so perk option select fields would always have names prepended with perk_option--. (Javascript related to these fields which refers to some by name was adjusted to match.) When civicrm.repository.handle_donation_form_data() it prunes and normalizes donation form data to remove duplicate fields or fields unrelated to CiviCRM's own interests. As well, any selected perk options are assigned to their own new key in that form data, perk_option, so that CiviCRM doesn't have to fuzzy-match fields on its end looking for that perk option.

RELEVANT FILES MODIFIED: templates/donate.html.jinja

  • Classes, IDs, and names changed in HTML related to option groups, such that resultant form field values will have semantically-useful names and values static/js/script.js
  • Option group toggling code improved and updated to match new tag IDs civicrm/repository.py
  • handle_donation_form_data() added

ADDITIONALLY stripe/views.py

  • webhook() was modified to explicity handle KeyErrors thrown by stripe.controller.validate_webhook(), as well as ValueErrors, so as to prevent people from being able to crash the server by sending the Stripe webhook endpoint a message without a stripe-signature header. (We now log the occurrence instead.) various testing files
  • Various tests have been updated to reflect above changes

Merge request reports