Create pager playbook for when redis is unavailable authored by lelutin's avatar lelutin
......@@ -383,6 +383,41 @@ those logs and metrics do not have access to _why_ the rate-limit event
was fired, or _what_ it fired upon. (The IP address can be scraped from the
originating HTTP request, at least.)
### Redis is unreachable from the frontend server
The frontend server depends on being able to contact redis on the CiviCRM
server. Transactions need to interact with redis in order to complete
successfully.
If redis is unreachable, first check if the VPN is disconnected:
root@donate-01:~# ipsec status
Routed Connections:
civicrm::crm-int-01{1}: ROUTED, TUNNEL, reqid 1
civicrm::crm-int-01{1}: 49.12.57.139/32 172.30.136.4/32 2a01:4f8:fff0:4f:266:37ff:fe04:d2bd/128 === 172.30.136.1/32 204.8.99.142/32 2620:7:6002:0:266:37ff:fe4d:f883/128
Security Associations (1 up, 0 connecting):
civicrm::crm-int-01[10]: ESTABLISHED 2 hours ago, 49.12.57.139[49.12.57.139]...204.8.99.142[204.8.99.142]
civicrm::crm-int-01{42}: INSTALLED, TUNNEL, reqid 1, ESP SPIs: c644b828_i cd819116_o
civicrm::crm-int-01{42}: 49.12.57.139/32 172.30.136.4/32 2a01:4f8:fff0:4f:266:37ff:fe04:d2bd/128 === 172.30.136.1/32 204.8.99.142/32 2620:7:6002:0:266:37ff:fe4d:f883/128
If the command shows something else than the status above, then try to reconnect
the tunnel:
ipsec up civicrm::crm-int-01
If still unsuccessful, check the output from that command, or logs from
strongswan.
If the tunnel is up, you can check that you can reach the service from the
frontend server:
curl 'http://localhost:9115/probe?target=crm-int-01-priv:6379&module=redis_banner&debug=true'
# The following is equivalent but with less debugging info:
echo PING | nc -w 1 crm-int-01-priv 6379
If you can't reach the service, check on the crm-int-01.tpo server that the
redis service is correctly running.
## Disaster recovery
A disaster, for the donation site, can take two major forms:
......
......