finish documenting locking issues

howto/ldap.md

@@ -288,21 +288,39 @@ workaround is to run this command on the primary DNS server (currently
 
     sudo -u dnsadm /srv/dns.torproject.org/bin/update --force
 
-### ud-replicate failures
+### Deadlocks in ud-replicate
 
-TODO: i seem to recall `ud-replicate` failing somehow, possibly
-because of SSH multiplexing or something?
+The `ud-replicate` process keeps a "reader" lock on the LDAP
+server. If for some reason the network transport fails, that lock
+might be held on forever. This happened in the past on hosts with
+flaky network or ipsec problems that null-routed packets between ipsec
+nodes.
 
-Example:
+There is a Nagios check that will detect stale
+synchronisations. Example:
 
     Subject: ** PROBLEM Service Alert: palmeri/setup - ud-ldap freshness is WARNING **
 
-### Locking
+Note that this can generate a *lot* of warnings because one per server
+will be sent!
 
-`ud-generate` holds a lock in
-`/var/cache/userdir-ldap/hosts/ud-generate.lock` when running. If
-something bad happens and it can't run, it might be because of such a
-stale lock file.
+The fix is to find the offending locked process and kill it. In
+desperation:
+
+    pkill -u sshdist rsync
+
+... but really, you should carefully review the rsync processes before
+killing them all like that. And obviously, fixing the underlying
+network issue would be important to avoid such problems in the future.
+
+Also note that the lock file is in
+`/var/cache/userdir-ldap/hosts/ud-generate.lock`, and `ud-generate`
+tries to get a *write* lock on the file. This implies that a deadlock
+will also affect file generation and keep `ud-generate` from
+generating fresh config files.
+
+Finally, `ud-replicate` also holds a lock on `/var/lib/misc` on the
+*client* side, but that rarely causes problems.
 
 ### Troubleshooting changes@ failures
 
@@ -592,7 +610,7 @@ Those are the configuration files shipped with the package:
  * [userdir-ldap source code](https://salsa.debian.org/dsa-team/mirror/userdir-ldap)
  * [userdir-ldap-cgi source code](https://salsa.debian.org/dsa-team/mirror/userdir-ldap-pylons)
  * [ud](https://github.com/Debian/ud) - a partial ud-ldap rewrite in Django from 2013-2014, no
-   change since 2017
+   change since 2017, the [announcement for the rewrite](https://wiki.debian.org/Teams/DSA/UserdirLdapRewrite)
  * [userdir-ldap-pylons](https://salsa.debian.org/dsa-team/mirror/userdir-ldap-pylons) - a partial ud-ldap rewrite in Pylons from
    2011, abandoned