From df9fa73bfc9f4c3c64d84f72b09a86cabe8bfbb1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= <anarcat@debian.org>
Date: Thu, 21 Apr 2022 14:50:20 -0400
Subject: [PATCH] figured out upgrades, restarts, some diagnostics
---
service/BTCpayserver.md | 151 ++++++++++++++++++++++++++++++++++------
1 file changed, 131 insertions(+), 20 deletions(-)
diff --git a/service/BTCpayserver.md b/service/BTCpayserver.md
index 311dc634..7ed894d5 100644
--- a/service/BTCpayserver.md
+++ b/service/BTCpayserver.md
@@ -24,7 +24,33 @@ TODO: think of a few basic use cases
## Upgrades
-TODO: document how upgrades work
+Upgrades work by updating all container images and restarting the
+right one. The [upstream procedure](https://docs.btcpayserver.org/FAQ/ServerSettings/#how-to-update-btcpay-server) recommends using a wrapper
+script that takes care of this. It does some weird stuff with git, so
+the way to run it is better:
+
+ cd /root/BTCPayServer/btcpayserver-docker
+ git pull --ff-only
+ ./btcpay-update.sh --skip-git-pull
+
+This will basically:
+
+ 1. pull a new version of the repository
+ 2. rebuild the configuration files (by calling `build.sh`, but also
+ by calling a `helper.sh` function to regenerate the env file)
+ 3. reinstall dependencies if missing (docker, `/usr/local/bin`
+ symlinks, etc)
+ 4. run `docker-compose up` to reload the running containers, if their
+ images changed
+ 5. cleanup old container images
+
+We could, in theory, do something like this to do the upgrade instead:
+
+ ./build.sh # to generate the new docker-compose file
+ docker-compose -f $BTCPAY_DOCKER_COMPOSE up -d
+
+... but that won't take into account all the ... uh... subtleties of
+the full upgrade process.
## Restart
@@ -37,26 +63,77 @@ Since the server is hooked into systemd, this should be sufficient:
systemctl restart btcpayserver
+Given that this is managed through `docker-compose`, it's also
+possible to restart the containers directly, with:
+
+ docker-compose -f $BTCPAY_DOCKER_COMPOSE restart
+
+That gives better progress information than the systemd restart.
+
## Inspecting status
-Given that this is managed through `docker-compose`, it's actually
-possible to inspect the server status manually, with:
+This will show the running containers:
docker-compose -f $BTCPAY_DOCKER_COMPOSE ps
-Note the `$BTCPAY_DOCKER_COMPOSE` environment there. It should
-normally be sourced from `/etc/profile.d/btcpay-env.sh` on login, but
-failing that, the actual file should generally be found in:
+This will tail the logs of all the containers:
- /root/BTCPayServer/btcpayserver-docker/Generated/docker-compose.generated.yml
+ docker-compose -f $BTCPAY_DOCKER_COMPOSE logs -f --tail=10
## Pager playbook
-<!-- information about common errors from the monitoring system and -->
-<!-- how to deal with them. this should be easy to follow: think of -->
-<!-- your future self, in a stressful situation, tired and hungry. -->
+When you're lost, look at the variables in
+`/etc/profile.d/btcpay-env.sh`. Three important settings:
+
+ export BTCPAY_DOCKER_COMPOSE="/root/BTCPayServer/btcpayserver-docker/Generated/docker-compose.generated.yml"
+ export BTCPAY_BASE_DIRECTORY="/root/BTCPayServer"
+ export BTCPAY_ENV_FILE="/root/BTCPayServer/.env"
+
+Spelling those out:
-TODO: pager playbook.
+ * `BTCPAY_DOCKER_COMPOSE` file can be used to talk with
+ `docker-compose` (see above for examples)
+
+ * `BTCPAY_BASE_DIRECTORY` is where the source code was checked out
+ (basically)
+
+ * `BTCPAY_ENV_FILE` is the environment file passed to docker-compose
+
+### containers not starting
+
+If the containers fail to start with this error:
+
+ btcpayserver_1 | fail: PayServer: Error on the MigrationStartupTask
+ btcpayserver_1 | System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known
+
+Take a look at disk space.
+
+We've had situations like this where the containers would fail with
+the above error when running out of disk space.
+
+### Stuck at "node is starting"
+
+If you get this message in the web UI:
+
+> Your nodes are synching...
+>
+> Your node is synching the entire blockchain and validating the consensus rules...
+> BTC
+>
+> NBXplorer headers height: 0
+> The node is starting...
+
+Look at the logs of the containers. If you see this:
+
+ NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 40 seconds
+
+That's a [known problem](https://github.com/btcpayserver/btcpayserver-docker/issues/206) with NBXplorer corrupting its database
+when it runs out of disk space. The fix is to stop the container,
+delete the data, and restart:
+
+ docker-compose -f $BTCPAY_DOCKER_COMPOSE stop nbxplorer
+ rm -r /var/lib/docker/volumes/generated_nbxplorer_datadir/_data/Main
+ docker-compose -f $BTCPAY_DOCKER_COMPOSE start nbxplorer
## Disaster recovery
@@ -241,6 +318,29 @@ understand how all those pieces fit together.
This audit was performed by anarcat in the beginning of 2022.
+### General architecture
+
+The [Docker install documentation](https://docs.btcpayserver.org/Docker/) (?) has an [architecture
+overview](https://docs.btcpayserver.org/Docker/#architecture) that has this image:
+
+
+
+Upstream says:
+
+> As you can see, BTCPay depends on several pieces of infrastructure, mainly:
+>
+> * A lightweight block explorer (NBXplorer),
+> * A database (PostgreSQL or SQLite),
+> * A full node (eg. Bitcoin Core)
+>
+> There can be more dependencies if you support more than just standard Bitcoin transactions, including:
+>
+> * C-Lightning
+> * LitecoinD
+> * and other coin daemons
+>
+> And more...
+
### Docker containers
BTCpayserver is a bunch of shell scripts built on top of a bunch of
@@ -323,10 +423,12 @@ configuration and glue things together, see above.
### Storage and queues
It's unclear what is stored where. Transactions, presumably, get
-recorded in the blockchain, but they *may* also be recorded in the
-PostgreSQL database?
+recorded in the blockchain, but they are also certainly recorded in
+the PostgreSQL database.
-Unclear.
+Transactions can be held in PostgreSQL for a while until a
+verification comes in, presumably through NBXplorer. Old transactions
+seem to stick around, presumably forever.
### Authentication
@@ -387,10 +489,21 @@ TODO: how do we test BTC payments works?
## Logs and metrics
-<!-- where are the logs? how long are they kept? any PII? -->
-<!-- what about performance metrics? same questions -->
+BTCpay actually configures the Docker daemon to keep only 5m (5MB?) of
+logs in `/etc/docker/daemon.json`:
+
+ {
+ "log-driver": "json-file",
+ "log-opts": {"max-size": "5m", "max-file": "3"}
+ }
+
+Container logs can be inspected with:
-TODO: logs and metrics
+ docker-compose -f $BTCPAY_DOCKER_COMPOSE logs -f --tail=10
+
+Those include PII information like IP addresses, recorded by the Nginx
+webserver. It is unclear how long that configuration will actually
+keep data for, considering it's size-based.
## Backups
@@ -455,9 +568,7 @@ current state.
## Other documentation
-<!-- references to upstream documentation, if relevant -->
-
-TODO: link to upstream manual
+Upstream has [documentation](https://docs.btcpayserver.org/).
# Discussion
--
GitLab