more director restore docs

tsa/howto/backup.mdwn

@@ -509,6 +509,15 @@ director.
  
         killall sleep
 
+ 16. switch the nagios checks over the new director: grep for the old
+     director name in the nagios configuration and fix up some of the
+     checks
+
+        git -C tor-nagios grep dictyotum
+
+ 17. you will also need to restore the password file for the nagios
+     check in `/etc/nagios/bacula-database`
+
 The new scheduler and director should now have completely taken over
 the new one, and backups should resume. The old server can now be
 [[decommissioned|retire-a-host]], if it's still around, when you feel
@@ -516,6 +525,44 @@ comfortable the new setup is working.
 
 TODO: `15:19:55 <weasel> and once that's up and running, it'd probably be smart to upgrade it to 11.  pg_upgradecluster -m upgrade --link`
 
+TODO: some psql users still refer to host-specific usernames like
+`bacula-dictyotum-reader`, maybe they should refer to role-specif
+names instead?
+
+### Troubleshooting
+
+If you get this error:
+
+    psycopg2.OperationalError: definition of service "bacula" not found
+
+It's probably the scheduler failing to connect to the database server,
+because the `/etc/dsa/bacula-reader-database` refers to a non-existent
+"service", as defined in
+`/etc/postgresql-common/pg_service.conf`. Either add something like:
+
+    [bacula]
+    dbname=bacula
+    port=5433
+
+to that file, or specify the `dbname` and `port` manually in the
+config file.
+
+If the scheduler is sending you an email every three minutes with this
+error:
+
+    FileNotFoundError: [Errno 2] No such file or directory: '/etc/dsa/bacula-reader-database'
+
+It's because you forgot to create that file, in step 8. Similar errors
+may occur if you forgot to change that password.
+
+If the director takes a long time to start and ultimately fails with:
+
+    oct 10 18:19:41 bacula-director-01 bacula-dir[31276]: bacula-dir JobId 0: Fatal error: Could not open Catalog "MyCatalog", database "bacula".
+    oct 10 18:19:41 bacula-director-01 bacula-dir[31276]: bacula-dir JobId 0: Fatal error: postgresql.c:332 Unable to connect to PostgreSQL server. Database=bacula User=bac
+    oct 10 18:19:41 bacula-director-01 bacula-dir[31276]: Possible causes: SQL server not running; password incorrect; max_connections exceeded.
+
+It's because you forgot to reset the director password, in step 9.
+
 Get files without a director
 ----------------------------
 

tsa/howto/postgresql.mdwn

@@ -180,11 +180,12 @@ server to the database server and restores from those.
         service postgresql start
         tail -f /var/log/postgresql/*
 
- 11. re-enable Puppet, which should clean up the extra SSH key:
+ 11. re-enable Puppet, which should clean up the extra SSH key and
+     firewall rules:
 
         puppet agent -t
     
-     make sure it's okay in `/etc/ssh/userkeys/root`
+     make sure it's okay in `/etc/ssh/userkeys/root` and `iptables -L`.
 
 When the restore succeeds, the `recovery.conf` file will be renamed to
 `recovery.done` and you will see something like:
@@ -284,11 +285,16 @@ information on how to pick a restore procedure.
     beginning, but it should eventually get started and complete. The
     "hang" is because `retrieve-file` sends a header which includes a
     `sha512sum` and it takes a while to compute. The `sed` command in
-    that pipeline is there to skip that header.
+    that pipeline is there to skip that header. Example of a
+    successful restore operated in [#31786][]:
+   
+        ssh torbackup@bungei.torproject.org bacula-director-01 retrieve-file pg dictyotum bacula.BASE.bungei.torproject.org-20191010-184205-dictyotum.torproject.org-bacula-9.6-backup.tar.gz | pv -s 33G | sed '0,/^$/d' | tar -C /var/lib/postgresql/9.6/main -x -z -f -
 
     If the above doesn't work, use the indirect procedure to restore
     the `BASE` file.
 
+[#31786]: https://bugs.torproject.org/31786
+
  6. Make sure the `pg_xlog` directory doesn't contain any files.
  
         rm -rf -- /var/lib/postgresql/9.6/main/pg_xlog/*