puppet: disaster recovery docs

howto/puppet.md

@@ -768,11 +768,22 @@ role account instead.
 
 ## Disaster recovery
 
-<!-- what to do if all goes to hell. e.g. restore from backups? -->
-<!-- rebuild from scratch? not necessarily those procedures (e.g. see -->
-<!-- "Installation" below but some pointers. -->
-
-TODO.
+Ideally, the main Puppet server would be deployable from Puppet
+bootstrap code and the [main installer](new-machine). But in practice, much of
+its configuration was done manually over the years and it MUST be
+restored from [backups](backup) in case of failure.
+
+This probably includes a restore of the [PostgreSQL](postgresql) database
+backing the PuppetDB server as well. It's *possible* this step *could*
+be skipped in an emergency, because most of the information in
+PuppetDB is a cache of exported resources, reports and facts. But it
+could also break hosts and make converging the infrastructure
+impossible, as there might be dependency loops in exported resources
+(for example, the Puppet server needs access to the LDAP server, and
+that is configured in Puppet). 
+
+So it is strongly encouraged to restore the PuppetDB server database
+as well in case of disaster.
 
 # Reference