Skip to content

Changes

Page history
figured out upgrades, restarts, some diagnostics authored by anarcat's avatar anarcat
Hide whitespace changes
Inline Side-by-side
service/BTCpayserver.md
View page @ df9fa73b
......@@ -24,7 +24,33 @@ TODO: think of a few basic use cases
## Upgrades
TODO: document how upgrades work
Upgrades work by updating all container images and restarting the
right one. The [upstream procedure](https://docs.btcpayserver.org/FAQ/ServerSettings/#how-to-update-btcpay-server) recommends using a wrapper
script that takes care of this. It does some weird stuff with git, so
the way to run it is better:
cd /root/BTCPayServer/btcpayserver-docker
git pull --ff-only
./btcpay-update.sh --skip-git-pull
This will basically:
1. pull a new version of the repository
2. rebuild the configuration files (by calling `build.sh`, but also
by calling a `helper.sh` function to regenerate the env file)
3. reinstall dependencies if missing (docker, `/usr/local/bin`
symlinks, etc)
4. run `docker-compose up` to reload the running containers, if their
images changed
5. cleanup old container images
We could, in theory, do something like this to do the upgrade instead:
./build.sh # to generate the new docker-compose file
docker-compose -f $BTCPAY_DOCKER_COMPOSE up -d
... but that won't take into account all the ... uh... subtleties of
the full upgrade process.
## Restart
......@@ -37,26 +63,77 @@ Since the server is hooked into systemd, this should be sufficient:
systemctl restart btcpayserver
Given that this is managed through `docker-compose`, it's also
possible to restart the containers directly, with:
docker-compose -f $BTCPAY_DOCKER_COMPOSE restart
That gives better progress information than the systemd restart.
## Inspecting status
Given that this is managed through `docker-compose`, it's actually
possible to inspect the server status manually, with:
This will show the running containers:
docker-compose -f $BTCPAY_DOCKER_COMPOSE ps
Note the `$BTCPAY_DOCKER_COMPOSE` environment there. It should
normally be sourced from `/etc/profile.d/btcpay-env.sh` on login, but
failing that, the actual file should generally be found in:
This will tail the logs of all the containers:
/root/BTCPayServer/btcpayserver-docker/Generated/docker-compose.generated.yml
docker-compose -f $BTCPAY_DOCKER_COMPOSE logs -f --tail=10
## Pager playbook
<!-- information about common errors from the monitoring system and -->
<!-- how to deal with them. this should be easy to follow: think of -->
<!-- your future self, in a stressful situation, tired and hungry. -->
When you're lost, look at the variables in
`/etc/profile.d/btcpay-env.sh`. Three important settings:
export BTCPAY_DOCKER_COMPOSE="/root/BTCPayServer/btcpayserver-docker/Generated/docker-compose.generated.yml"
export BTCPAY_BASE_DIRECTORY="/root/BTCPayServer"
export BTCPAY_ENV_FILE="/root/BTCPayServer/.env"
Spelling those out:
TODO: pager playbook.
* `BTCPAY_DOCKER_COMPOSE` file can be used to talk with
`docker-compose` (see above for examples)
* `BTCPAY_BASE_DIRECTORY` is where the source code was checked out
(basically)
* `BTCPAY_ENV_FILE` is the environment file passed to docker-compose
### containers not starting
If the containers fail to start with this error:
btcpayserver_1 | fail: PayServer: Error on the MigrationStartupTask
btcpayserver_1 | System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known
Take a look at disk space.
We've had situations like this where the containers would fail with
the above error when running out of disk space.
### Stuck at "node is starting"
If you get this message in the web UI:
> Your nodes are synching...
>
> Your node is synching the entire blockchain and validating the consensus rules...
> BTC
>
> NBXplorer headers height: 0
> The node is starting...
Look at the logs of the containers. If you see this:
NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 40 seconds
That's a [known problem](https://github.com/btcpayserver/btcpayserver-docker/issues/206) with NBXplorer corrupting its database
when it runs out of disk space. The fix is to stop the container,
delete the data, and restart:
docker-compose -f $BTCPAY_DOCKER_COMPOSE stop nbxplorer
rm -r /var/lib/docker/volumes/generated_nbxplorer_datadir/_data/Main
docker-compose -f $BTCPAY_DOCKER_COMPOSE start nbxplorer
## Disaster recovery
......@@ -241,6 +318,29 @@ understand how all those pieces fit together.
This audit was performed by anarcat in the beginning of 2022.
### General architecture
The [Docker install documentation](https://docs.btcpayserver.org/Docker/) (?) has an [architecture
overview](https://docs.btcpayserver.org/Docker/#architecture) that has this image:
![Image of BTCPay talking with Postgres and NBXplorer, itself talking with Bitcoin Core](https://github.com/btcpayserver/btcpayserver-doc/raw/master/docs/img/Architecture.png)
Upstream says:
> As you can see, BTCPay depends on several pieces of infrastructure, mainly:
>
> * A lightweight block explorer (NBXplorer),
> * A database (PostgreSQL or SQLite),
> * A full node (eg. Bitcoin Core)
>
> There can be more dependencies if you support more than just standard Bitcoin transactions, including:
>
> * C-Lightning
> * LitecoinD
> * and other coin daemons
>
> And more...
### Docker containers
BTCpayserver is a bunch of shell scripts built on top of a bunch of
......@@ -323,10 +423,12 @@ configuration and glue things together, see above.
### Storage and queues
It's unclear what is stored where. Transactions, presumably, get
recorded in the blockchain, but they *may* also be recorded in the
PostgreSQL database?
recorded in the blockchain, but they are also certainly recorded in
the PostgreSQL database.
Unclear.
Transactions can be held in PostgreSQL for a while until a
verification comes in, presumably through NBXplorer. Old transactions
seem to stick around, presumably forever.
### Authentication
......@@ -387,10 +489,21 @@ TODO: how do we test BTC payments works?
## Logs and metrics
<!-- where are the logs? how long are they kept? any PII? -->
<!-- what about performance metrics? same questions -->
BTCpay actually configures the Docker daemon to keep only 5m (5MB?) of
logs in `/etc/docker/daemon.json`:
{
"log-driver": "json-file",
"log-opts": {"max-size": "5m", "max-file": "3"}
}
Container logs can be inspected with:
TODO: logs and metrics
docker-compose -f $BTCPAY_DOCKER_COMPOSE logs -f --tail=10
Those include PII information like IP addresses, recorded by the Nginx
webserver. It is unclear how long that configuration will actually
keep data for, considering it's size-based.
## Backups
......@@ -455,9 +568,7 @@ current state.
## Other documentation
<!-- references to upstream documentation, if relevant -->
TODO: link to upstream manual
Upstream has [documentation](https://docs.btcpayserver.org/).
# Discussion
......