figured out upgrades, restarts, some diagnostics

service/BTCpayserver.md

@@ -24,7 +24,33 @@ TODO: think of a few basic use cases
 
 ## Upgrades
 
-TODO: document how upgrades work
+Upgrades work by updating all container images and restarting the
+right one. The [upstream procedure](https://docs.btcpayserver.org/FAQ/ServerSettings/#how-to-update-btcpay-server) recommends using a wrapper
+script that takes care of this. It does some weird stuff with git, so
+the way to run it is better:
+
+    cd /root/BTCPayServer/btcpayserver-docker
+    git pull --ff-only
+    ./btcpay-update.sh --skip-git-pull
+
+This will basically:
+
+ 1. pull a new version of the repository
+ 2. rebuild the configuration files (by calling `build.sh`, but also
+    by calling a `helper.sh` function to regenerate the env file)
+ 3. reinstall dependencies if missing (docker, `/usr/local/bin`
+    symlinks, etc)
+ 4. run `docker-compose up` to reload the running containers, if their
+    images changed
+ 5. cleanup old container images
+
+We could, in theory, do something like this to do the upgrade instead:
+
+    ./build.sh # to generate the new docker-compose file
+    docker-compose -f $BTCPAY_DOCKER_COMPOSE up -d
+
+... but that won't take into account all the ... uh... subtleties of
+the full upgrade process.
 
 ## Restart
 
@@ -37,26 +63,77 @@ Since the server is hooked into systemd, this should be sufficient:
 
     systemctl restart btcpayserver
 
+Given that this is managed through `docker-compose`, it's also
+possible to restart the containers directly, with:
+
+    docker-compose -f $BTCPAY_DOCKER_COMPOSE restart
+
+That gives better progress information than the systemd restart.
+
 ## Inspecting status
 
-Given that this is managed through `docker-compose`, it's actually
-possible to inspect the server status manually, with:
+This will show the running containers:
 
     docker-compose -f $BTCPAY_DOCKER_COMPOSE ps
 
-Note the `$BTCPAY_DOCKER_COMPOSE` environment there. It should
-normally be sourced from `/etc/profile.d/btcpay-env.sh` on login, but
-failing that, the actual file should generally be found in:
+This will tail the logs of all the containers:
 
-    /root/BTCPayServer/btcpayserver-docker/Generated/docker-compose.generated.yml
+    docker-compose -f $BTCPAY_DOCKER_COMPOSE logs -f --tail=10
 
 ## Pager playbook
 
-<!-- information about common errors from the monitoring system and -->
-<!-- how to deal with them. this should be easy to follow: think of -->
-<!-- your future self, in a stressful situation, tired and hungry. -->
+When you're lost, look at the variables in
+`/etc/profile.d/btcpay-env.sh`. Three important settings:
+
+    export BTCPAY_DOCKER_COMPOSE="/root/BTCPayServer/btcpayserver-docker/Generated/docker-compose.generated.yml"
+    export BTCPAY_BASE_DIRECTORY="/root/BTCPayServer"
+    export BTCPAY_ENV_FILE="/root/BTCPayServer/.env"
+
+Spelling those out:
 
-TODO: pager playbook.
+ * `BTCPAY_DOCKER_COMPOSE` file can be used to talk with
+   `docker-compose` (see above for examples)
+
+ * `BTCPAY_BASE_DIRECTORY` is where the source code was checked out
+   (basically)
+ 
+ * `BTCPAY_ENV_FILE` is the environment file passed to docker-compose
+
+### containers not starting
+
+If the containers fail to start with this error:
+
+    btcpayserver_1                       | fail: PayServer:      Error on the MigrationStartupTask
+    btcpayserver_1                       | System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known
+
+Take a look at disk space.
+
+We've had situations like this where the containers would fail with
+the above error when running out of disk space.
+
+### Stuck at "node is starting"
+
+If you get this message in the web UI:
+
+> Your nodes are synching...
+> 
+> Your node is synching the entire blockchain and validating the consensus rules...
+> BTC
+> 
+>     NBXplorer headers height: 0
+>     The node is starting...
+
+Look at the logs of the containers. If you see this:
+
+    NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 40 seconds
+
+That's a [known problem](https://github.com/btcpayserver/btcpayserver-docker/issues/206) with NBXplorer corrupting its database
+when it runs out of disk space. The fix is to stop the container,
+delete the data, and restart:
+
+    docker-compose -f $BTCPAY_DOCKER_COMPOSE stop nbxplorer
+    rm -r /var/lib/docker/volumes/generated_nbxplorer_datadir/_data/Main
+    docker-compose -f $BTCPAY_DOCKER_COMPOSE start nbxplorer
 
 ## Disaster recovery
 
@@ -241,6 +318,29 @@ understand how all those pieces fit together.
 
 This audit was performed by anarcat in the beginning of 2022.
 
+### General architecture
+
+The [Docker install documentation](https://docs.btcpayserver.org/Docker/) (?) has an [architecture
+overview](https://docs.btcpayserver.org/Docker/#architecture) that has this image:
+
+![Image of BTCPay talking with Postgres and NBXplorer, itself talking with Bitcoin Core](https://github.com/btcpayserver/btcpayserver-doc/raw/master/docs/img/Architecture.png)
+
+Upstream says:
+
+> As you can see, BTCPay depends on several pieces of infrastructure, mainly:
+> 
+>  * A lightweight block explorer (NBXplorer),
+>  * A database (PostgreSQL or SQLite),
+>  * A full node (eg. Bitcoin Core)
+> 
+> There can be more dependencies if you support more than just standard Bitcoin transactions, including:
+> 
+>  * C-Lightning
+>  * LitecoinD
+>  * and other coin daemons
+>
+> And more...
+
 ### Docker containers
 
 BTCpayserver is a bunch of shell scripts built on top of a bunch of
@@ -323,10 +423,12 @@ configuration and glue things together, see above.
 ### Storage and queues
 
 It's unclear what is stored where. Transactions, presumably, get
-recorded in the blockchain, but they *may* also be recorded in the
-PostgreSQL database?
+recorded in the blockchain, but they are also certainly recorded in
+the PostgreSQL database.
 
-Unclear.
+Transactions can be held in PostgreSQL for a while until a
+verification comes in, presumably through NBXplorer. Old transactions
+seem to stick around, presumably forever.
 
 ### Authentication
 
@@ -387,10 +489,21 @@ TODO: how do we test BTC payments works?
 
 ## Logs and metrics
 
-<!-- where are the logs? how long are they kept? any PII? -->
-<!-- what about performance metrics? same questions -->
+BTCpay actually configures the Docker daemon to keep only 5m (5MB?) of
+logs in `/etc/docker/daemon.json`:
+
+    {
+    "log-driver": "json-file",
+    "log-opts": {"max-size": "5m", "max-file": "3"}
+    }
+
+Container logs can be inspected with:
 
-TODO: logs and metrics
+    docker-compose -f $BTCPAY_DOCKER_COMPOSE logs -f --tail=10
+
+Those include PII information like IP addresses, recorded by the Nginx
+webserver. It is unclear how long that configuration will actually
+keep data for, considering it's size-based.
 
 ## Backups
 
@@ -455,9 +568,7 @@ current state.
 
 ## Other documentation
 
-<!-- references to upstream documentation, if relevant -->
-
-TODO: link to upstream manual
+Upstream has [documentation](https://docs.btcpayserver.org/).
 
 # Discussion