Skip to content

feat: Deduplicate IPN and webhook messages carrying the same legacy-series recurring donation data

stephen requested to merge webhook-ipn-dedupe-coordination into main

The further we go into accomodating the legacy IPN message service which PayPal still operates for legacy recurring-transaction series, the more mysteries we uncover. The latest wrinkle in this saga appears to be that - after a tremendous amount of effort spent to receive, validate, and handle IPN messages - a system which is set up to properly handle IPN messages may also start receiving webhook messages containing the same data.

In a world where PayPal was interested in documenting this behavior, rather than letting their IPN documentation and tooling slowly go to rot, it might be a nifty trick to implement IPN support up until the point at which we return 200, then ignore the contents of those IPN messages, and wait for duplicate webhook messages to arrive and process those instead. However - given the thick curtain of mystery draped over the entire legacy messaging infrastructure - limiting the amount we lean on the mysterious workings of these legacy systems seems most prudent, and while the implementation of baseline IPN messaging support is barely documented, "knock-on downstream webhook behavior" is completely undocumented and potentially unintended behavior. Therefore, to play it safe, this MR implements a method to deduplicate messages regarding legacy recurring transactions, whether we get a webhook message about it first, or an IPN message.

This is accomplished by ensuring both the webhook message and the IPN message surface the same value for use in deduplication - first by placing a transient entry into Redis with tordonate.civicrm.note_donation, and later by checking for that same entry with tordonate.civicrm.donation_exists.

Both IPN messages and webhook messages carry their own bespoke ID, but they do not match each other. Therefore, deduping between the two requires two small and similar tweaks:

  1. We change the trxn_id value of the IPN message's arguments dict to the donation series' ID, rather than the bespoke per-IPN-message ID.
  2. We change the trxn_id value of the webhook message's arguments dict to the donation series' ID, rather than the bespoke per-webhook-message ID.

These values are not used further down the pipeline for any particular proof of uniqueness, and they should still serve well as a helpful identifier since they are still unique to the transaction series, so changing them in this manner should allow us to have our cake and eat it too.

This PR also adjusts the PayPal controller tests to conform to this change.

Merge request reports

Loading