feat: Validates IPN messages via Paypal and processes IPN data
The overview of MR !156 (merged) describes the history of the Paypal IPN message and details our need to handle those messages and pass along the subscription information they detail to CiviCRM, as we do with webhook messages from both Paypal and Stripe. This MR seeks to fulfill that need by validating these messages against Paypal's own IPN-validation endpoints, then processing the messages' data, just as we do with webhook message data.
An IPN validation method, tordonate.paypal.controller.validate_ipn(), receives the content of POST messages sent to /paypal-ipn/, formats it in the specific manner which Paypal's IPN validation specs require, and POSTs back to an IPN-validation Paypal URL.
Paypal will respond to these POSTs with text reading either "VERIFIED" or "INVALID," and validate_ipn() returns this information as a boolean back to the view. Based on these results, tordonate.paypal.views.ipn then invokes the method process_ipn(), which as of this commit is merely a stub.
!156 (merged) mistakenly implemented the IPN message parser as though the IPN data was being passed along as a URL parameter; it is in fact delivered as a POST message body. Due to Paypal being persnickety about receiving IPN messages back in exactly the same format, we now handle that POST body as a bytestring directly.
PayPal's own requirements for handling IPNs as detailed in their legacy documentation is no longer completely correct, and several tools mentioned optimistically in !156 (merged) no longer function as originally intended. This scenario is discussed in a large comment block within validate_ipn() under the presumption that anyone nosing around in this part of the codebase will want all the context they can get.
IPN messages, once validated, contain all of the same information we
take from webhook messages to construct outgoing Resque messages to
CiviCRM. This commit adds the tordonate.paypal.controller method
process_ipn(), which is invoked by tordonate.paypal.views.ipn()
after successfully validating an IPN message.
process_ipn(), like validate_ipn() before it, accepts a single
paramater - the body of the original POST request, in bytes. Since all
IPN messages are strings containing data expressed in key-value pairs,
formatted as a URL querystring, process_ipn() converts those bytes
into a dict, and from it, constructs the data necessary to call
tordonate.civicrm.repository.donation_exists() and
tordonate.civicrm.repository.report_donation(), just as we would with
data pulled from a webhook message. We similarly use this data to manage
the relevant Prometheus vendor transaction datapoint.
The only tricky part of this exercise is (unsurprisingly) the timestamp; IPN messages use a bespoke format for their timestamps, and unlike all timestamps used by webhook messages, they are set to US Pacific time; a certain amount of conversion is necessary to have them match the timestamps generated from webhook messages.
Comprehensive testing for IPN validation
and IPN processing has additionally been added, as well as testing against general interaction with the /paypal-ipn
URL endpoint. Because IPN message test data is expressed in pure text
rather than in a data format, the /json subfolder of the Paypal test
files folder has been renamed /data, and IPN message test data has
been stored as .txt files in /data. All relevant Paypal tests have
been updated to reflect this change.