[btcpayserver][] is a collection of Docker containers that enables us to
process cryptocurrency (currently only Bitcoin) payment.

[BTCpayserver]: https://btcpayserver.org/

This page shouldn't be misconstrued as an approval of the use of the
BTCpayserver project or, indeed, any cryptocurrency whatsoever. In
fact, our experience with BTCpayserver makes us encourage you to look
at alternatives instead, including not taking cryptocurrency payments
at all, see [TPA-RFC-25: BTCpayserver replacement](policy/tpa-rfc-25-btcpay-replacement) for that
discussion.

[[_TOC_]]

# Tutorial

<!-- simple, brainless step-by-step instructions requiring little or -->
<!-- no technical background -->

TODO: think of a few basic use cases

# How-to

<!-- more in-depth procedure that may require interpretation -->

## Upgrades

Upgrades work by updating all container images and restarting the
right one. The [upstream procedure](https://docs.btcpayserver.org/FAQ/ServerSettings/#how-to-update-btcpay-server) recommends using a wrapper
script that takes care of this. It does some weird stuff with git, so
the way to run it is better:

    cd /root/BTCPayServer/btcpayserver-docker
    git pull --ff-only
    ./btcpay-update.sh --skip-git-pull

This will basically:

 1. pull a new version of the repository
 2. rebuild the configuration files (by calling `build.sh`, but also
    by calling a `helper.sh` function to regenerate the env file)
 3. reinstall dependencies if missing (docker, `/usr/local/bin`
    symlinks, etc)
 4. run `docker-compose up` to reload the running containers, if their
    images changed
 5. cleanup old container images

We could, in theory, do something like this to do the upgrade instead:

    ./build.sh # to generate the new docker-compose file
    docker-compose -f $BTCPAY_DOCKER_COMPOSE up -d

... but that won't take into account all the ... uh... subtleties of
the full upgrade process.

## Restart

Restarting BTCpayserver shouldn't generally be necessary. It is hooked
with systemd on boot and should start normally on reboots. It has,
however, been necessary to restart the server to generate a new TLS
certificate, for example.

Since the server is hooked into systemd, this should be sufficient:

    systemctl restart btcpayserver

Given that this is managed through `docker-compose`, it's also
possible to restart the containers directly, with:

    docker-compose -f $BTCPAY_DOCKER_COMPOSE restart

That gives better progress information than the systemd restart.

## Inspecting status

This will show the running containers:

    docker-compose -f $BTCPAY_DOCKER_COMPOSE ps

This will tail the logs of all the containers:

    docker-compose -f $BTCPAY_DOCKER_COMPOSE logs -f --tail=10

## Manual backup and restore

A manual backup/restore procedure might look like this:

    systemctl stop btcpayserver
    tar cfz backup.tgz /var/lib/docker/volumes/
    systemctl start btcpayserver

A restore, then, would look like this:

    systemctl stop btcpayserver
    mv /var/lib/docker/volumes/ /var/lib/docker/volumes.old # optional
    tar -C / -x -f -z backup.tgz
    systemctl start btcpayserver

> If you're worried about the backup clobbering other files on restore
> (for example you're not sure about the backup source or file
> structure), this should restore only `volumes/` in the
> `/var/lib/docker` directory:
>
>     systemctl stop btcpayserver
>     mv /var/lib/docker/volumes/ /var/lib/docker/volumes.old # optional
>     tar -C /var/lib/docker/ -x -f backup.tar.gz -z --strip-components=3
>     systemctl start btcpayserver

The `mv` step should be turned into a `rm -rf
/var/lib/docker/volumes/` command if we are likely to run out of disk
space on restore and we're confident in the backup's integrity.

Note that the upstream backup procedure does *not* keep a copy of the
blockchain, so this will be regenerated on startup. That, in turn, can
take a *long* time (30 hours on last count). In that case, keeping a
copy of the blockchain on restore might make sense, it is stored in:

    /var/lib/docker/volumes/generated_bitcoin_datadir/_data/

Finally, also note that if you *rename* the server (e.g. we moved from
btcpay.torproject.net to btcpayserver.torproject.org in the past), you
*also* need to perform a [rename procedure](https://docs.btcpayserver.org/Deployment/ChangeDomain/#how-to-change-the-domain-name-of-your-btcpay-server), which is basically:

    /root/BTCPayServer/btcpayserver-docker/changedomain.sh btcpay.torproject.org

### Full migration procedure

Back from the top, migrating from server A to server B, with a rename,
should be like this. This assumes server B followed the [installation
procedure](#installation) and has an up to date blockchain.

On server A:

    systemctl stop btcpayserver
    tar -c -z -f backup.tgz /var/lib/docker/volumes/

Copy `backup.tgz` to server B.

On server B:

    systemctl stop btcpayserver
    tar -C / -x -f -z backup.tgz
    systemctl start btcpayserver

Note that this is likely to run out of disk space because it
(deliberately) includes the blockchain.

Another option is to stream the content between the two servers, if
you have a fast link:

    ssh old.example.net 'systemctl stop btcpayserver'
    ssh new.example.net 'systemctl stop btcpayserver'
    ssh old.example.net 'tar cf - /var/lib/docker/volumes/' | pv -s 49G | ssh new.example.net tar -C / -x -f -
    ssh new.example.net 'systemctl start btcpayserver'

Or, alternatively, you can also create an SSH key on the new server,
copy it on the old one, and just use `rsync`, which is what ended up
being used in the [actual migration](https://gitlab.torproject.org/tpo/tpa/team/-/issues/33750):

    ssh old.example.net 'systemctl stop btcpayserver'
    ssh new.example.net 'systemctl stop btcpayserver'
    ssh new.example.net 'ssh-keygen -t ed25519'
    ssh new.example.net 'cat .ssh/id_ed25519.pub' | ssh old.example.net 'cat >> .ssh/authorized_keys'
    ssh new.example.net 'rsync -a --info=progress2 --delete old.example.net:/var/lib/docker/volumes/ /var/lib/docker/volumes/'

It's important that the Docker volumes are synchronized: for example,
if the NBXplorer volume is ahead or behind the bitcoind volume, it
will get confused and will not be able to synchronize with the
blockchain. This is why we copy the full blockchain which, anyways, is
faster than copying it from the network.

Also, if you are changing to a new hostname, do not forget to change
it on the new server:

    ssh new.example.net /root/BTCPayServer/btcpayserver-docker/changedomain.sh btcpay.torproject.org

In any case, make sure to update the target of the donation form on
`donate.torproject.org`. See for example [merge request
tpo/web/donate-static!76](https://gitlab.torproject.org/tpo/web/donate-static/-/merge_requests/76).

### Faulty upstream procedure

Upstream has a [backup procedure](https://docs.btcpayserver.org/Docker/#how-can-i-back-up-my-btcpay-server) but, oddly, [no restore
procedure](https://github.com/btcpayserver/btcpayserver-docker/issues/629). It seems like, anyways, what the backup script does is:

 1. dump the database (in `$backup_volume/postgres.sql`)
 2. stops the server
 3. tar the Docker volumes (`/var/lib/docker/volumes/`) into a tar
    file in the backup directory (`$backup_volume/backup.tar.gz`),
    excluding the `generated_bitcoin_datadir` volume,
    `generated_litecoin_datadir` **and** the `$backup_volume` (?!)
 4. start the server
 5. delete the database dump

In the above, `$backup_volume` is
`/var/lib/docker/volumes/backup_datadir/_data/`. And no, the
`postgres.sql` database dump is not in the backups. I filed [upstream
issue 628](https://github.com/btcpayserver/btcpayserver-docker/issues/628) about this as well.

We do not recommend using the upstream backup procedures in their
current state.

## Pager playbook

When you're lost, look at the variables in
`/etc/profile.d/btcpay-env.sh`. Three important settings:

    export BTCPAY_DOCKER_COMPOSE="/root/BTCPayServer/btcpayserver-docker/Generated/docker-compose.generated.yml"
    export BTCPAY_BASE_DIRECTORY="/root/BTCPayServer"
    export BTCPAY_ENV_FILE="/root/BTCPayServer/.env"

Spelling those out:

 * `BTCPAY_DOCKER_COMPOSE` file can be used to talk with
   `docker-compose` (see above for examples)

 * `BTCPAY_BASE_DIRECTORY` is where the source code was checked out
   (basically)
 
 * `BTCPAY_ENV_FILE` is the environment file passed to docker-compose

### containers not starting

If the containers fail to start with this error:

    btcpayserver_1                       | fail: PayServer:      Error on the MigrationStartupTask
    btcpayserver_1                       | System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known

Take a look at disk space. We've had situations like this where the
containers would fail with the above error when running out of disk
space.

### Stuck at "node is starting"

If you get this message in the web UI:

> Your nodes are synching...
> 
> Your node is synching the entire blockchain and validating the consensus rules...
> BTC
> 
>     NBXplorer headers height: 0
>     The node is starting...

Look at the logs of the containers. If you see this:

    NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 40 seconds

That's a [known problem](https://github.com/btcpayserver/btcpayserver-docker/issues/206) with NBXplorer corrupting its database
when it runs out of disk space. The fix is to stop the container,
delete the data, and restart:

    docker-compose -f $BTCPAY_DOCKER_COMPOSE stop nbxplorer
    rm -r /var/lib/docker/volumes/generated_nbxplorer_datadir/_data/Main
    docker-compose -f $BTCPAY_DOCKER_COMPOSE start nbxplorer

## Disaster recovery

In theory, it should be possible to rebuild this service from scratch
by following our [install procedures](#installation) and then hooking up the
hardware wallet to the server. In practice, that is undocumented and
hasn't been tested.

Normally, you should be able to restore parts (or the entirety) of
this service using the normal [backup procedures](howto/backup). But those
backups may be inconsistent. If an emergency server migration is
possible (ie. the old server is still online), follow the [manual
backup and restore procedure](#manual-backup-and-restore).

# Reference

## Installation

### TPA deployment

Before the install, a CNAME must be added to the DNS to point to the
actual machine, for example, in `dns.git`'s `domains/torproject.org`
file:

    btcpayserver	IN	CNAME	btcpayserver-02.torproject.org

We are following the [full installation manual](https://docs.btcpayserver.org/Docker/#full-installation-for-technical-users), which is basically
this questionable set of steps:

```
mkdir BTCPayServer
cd BTCPayServer
git clone https://github.com/btcpayserver/btcpayserver-docker
cd btcpayserver-docker
```

Then the procedure wants us to declare those:

```
export BTCPAY_HOST="btcpayserver.torproject.org"
export BTCPAY_ADDITIONAL_HOSTS="btcpayserver-02.torproject.org"
export NBITCOIN_NETWORK="mainnet"
export BTCPAYGEN_CRYPTO1="btc"
export BTCPAYGEN_ADDITIONAL_FRAGMENTS="opt-save-storage-s"
export BTCPAYGEN_REVERSEPROXY="nginx"
export BTCPAYGEN_LIGHTNING=""
export BTCPAY_ENABLE_SSH=false
```

We explicitly changed those settings from upstream:

 * `BTCPAY_HOST` and `BTCPAY_ADDITIONAL_HOSTS`
 * `BTCPAY_ENABLE_SSH` (WTF?!)
 * `BTCPAYGEN_LIGHTNING="clightning"` disabled, see [tpo/web/donate-static#63](https://gitlab.torproject.org/tpo/web/donate-static/-/issues/63)

Then we launch the setup script, skipping the docker install because
that's already done by Puppet:

```
root@btcpayserver-02:~/BTCPayServer/btcpayserver-docker# . btcpay-setup.sh --docker-unavailable

-------SETUP-----------
Parameters passed:
BTCPAY_PROTOCOL:https
BTCPAY_HOST:btcpayserver.torproject.org
BTCPAY_ADDITIONAL_HOSTS:btcpayserver-02.torproject.org
REVERSEPROXY_HTTP_PORT:80
REVERSEPROXY_HTTPS_PORT:443
REVERSEPROXY_DEFAULT_HOST:none
LIBREPATRON_HOST:
ZAMMAD_HOST:
WOOCOMMERCE_HOST:
BTCTRANSMUTER_HOST:
CHATWOOT_HOST:
BTCPAY_ENABLE_SSH:false
BTCPAY_HOST_SSHKEYFILE:
LETSENCRYPT_EMAIL:
NBITCOIN_NETWORK:mainnet
LIGHTNING_ALIAS:
BTCPAYGEN_CRYPTO1:btc
BTCPAYGEN_CRYPTO2:
BTCPAYGEN_CRYPTO3:
BTCPAYGEN_CRYPTO4:
BTCPAYGEN_CRYPTO5:
BTCPAYGEN_CRYPTO6:
BTCPAYGEN_CRYPTO7:
BTCPAYGEN_CRYPTO8:
BTCPAYGEN_CRYPTO9:
BTCPAYGEN_REVERSEPROXY:nginx
BTCPAYGEN_LIGHTNING:none
BTCPAYGEN_ADDITIONAL_FRAGMENTS:opt-save-storage-s
BTCPAYGEN_EXCLUDE_FRAGMENTS:
BTCPAY_IMAGE:
ACME_CA_URI:production
TOR_RELAY_NICKNAME: 
TOR_RELAY_EMAIL: 
PIHOLE_SERVERIP: 
FIREFLY_HOST: 
----------------------
Additional exported variables:
BTCPAY_DOCKER_COMPOSE=/root/BTCPayServer/btcpayserver-docker/Generated/docker-compose.generated.yml
BTCPAY_BASE_DIRECTORY=/root/BTCPayServer
BTCPAY_ENV_FILE=/root/BTCPayServer/.env
BTCPAYGEN_OLD_PREGEN=false
BTCPAY_SSHKEYFILE=
BTCPAY_SSHAUTHORIZEDKEYS=
BTCPAY_HOST_SSHAUTHORIZEDKEYS:
BTCPAY_SSHTRUSTEDFINGERPRINTS:
BTCPAY_CRYPTOS:btc
BTCPAY_ANNOUNCEABLE_HOST:btcpayserver.torproject.org
----------------------

BTCPay Server environment variables successfully saved in /etc/profile.d/btcpay-env.sh

BTCPay Server docker-compose parameters saved in /root/BTCPayServer/.env

Adding btcpayserver.service to systemd
Setting limited log files in /etc/docker/daemon.json
BTCPay Server systemd configured in /etc/systemd/system/btcpayserver.service

Created symlink /etc/systemd/system/multi-user.target.wants/btcpayserver.service → /etc/systemd/system/btcpayserver.service.
Installed bitcoin-cli.sh to /usr/local/bin: Command line for your Bitcoin instance
Installed btcpay-clean.sh to /usr/local/bin: Command line for deleting old unused docker images
Installed btcpay-down.sh to /usr/local/bin: Command line for stopping all services related to BTCPay Server
Installed btcpay-restart.sh to /usr/local/bin: Command line for restarting all services related to BTCPay Server
Installed btcpay-setup.sh to /usr/local/bin: Command line for restarting all services related to BTCPay Server
Installed btcpay-up.sh to /usr/local/bin: Command line for starting all services related to BTCPay Server
Installed btcpay-admin.sh to /usr/local/bin: Command line for some administrative operation in BTCPay Server
Installed btcpay-update.sh to /usr/local/bin: Command line for updating your BTCPay Server to the latest commit of this repository
Installed changedomain.sh to /usr/local/bin: Command line for changing the external domain of your BTCPay Server
```

Then starting the server with `systemctl start btcpayserver` pulls a lot *more* docker containers (which takes time). and things seem to work:

    systemctl restart btcpayserver

and now the server is up. it asks me to create an account (!) so I did and stored the password in the password manager. now it's doing:

> Your nodes are synching...
> 
> Your node is synching the entire blockchain and validating the consensus rules...
> BTC
> 
>     NBXplorer headers height: 732756
>     Node headers height: 732756
>     Validated blocks: 185982
> 
> 0%
> 
> Watch this video to understand the importance of blockchain synchronization.
> 
> If you really don't want to sync and you are familiar with the command line, check FastSync.

In theory, the blocks should now sync and the node is ready to go. 

TODO: document how to hook into the hardware wallet, possibly see:
https://docs.btcpayserver.org/ConnectWallet/

Last time we followed this procedure, instead of hooking up the
wallet, we restored from backup. See [this comment and following](https://gitlab.torproject.org/tpo/tpa/team/-/issues/33750#note_2797196)
and the [full migration procedure](#full-migration-procedure).

### Lunanode Deployment

The machine was temporarily hosted at Lunanode before being moved to
TPA. This procedure was followed:

- https://docs.btcpayserver.org/LunaNodeWebDeployment/

Lunanode was chosen as a cheap and easy temporarily solution, but was
eventually retired in favor of a normal TPA machines so that we would
have it hooked in Puppet to have the normal system-level backups,
monitoring, and so on.

## SLA

There is no official SLA for this service, but it should generally be
up so that we can take donations.

## Design

According to the upstream website, "BTCPay Server is a self-hosted,
open-source cryptocurrency payment processor. It's secure, private,
censorship-resistant and free."

In practice, BTCpay is a rather complicated stack made of Docker,
Docker Compose, C# .net, bitcoin, PostgreSQL, Nginx, lots of shell
scripts and more, through plugins. It's actually pretty hard to
understand how all those pieces fit together.

This audit was performed by anarcat in the beginning of 2022.

### General architecture

The [Docker install documentation](https://docs.btcpayserver.org/Docker/) (?) has an [architecture
overview](https://docs.btcpayserver.org/Docker/#architecture) that has this image:

![Image of BTCPay talking with Postgres and NBXplorer, itself talking with Bitcoin Core](https://github.com/btcpayserver/btcpayserver-doc/raw/master/docs/img/Architecture.png)

Upstream says:

> As you can see, BTCPay depends on several pieces of infrastructure, mainly:
> 
>  * A lightweight block explorer (NBXplorer),
>  * A database (PostgreSQL or SQLite),
>  * A full node (eg. Bitcoin Core)
> 
> There can be more dependencies if you support more than just standard Bitcoin transactions, including:
> 
>  * C-Lightning
>  * LitecoinD
>  * and other coin daemons
>
> And more...

### Docker containers

BTCpayserver is a bunch of shell scripts built on top of a bunch of
Docker images. At the time of writing (~2022), we seemed to have the
following components setup (looking at
`/root/BTCPayServer/btcpayserver-docker/Generated/docker-compose.generated.yml`):

 * nginx 1.16 (from the [official docker image](https://hub.docker.com/_/nginx))
 * nginx-gen (which is some container based on [docker-gen](https://hub.docker.com/r/btcpayserver/docker-gen), which ... generates config files?)
 * btcpayserver (from [their image](https://hub.docker.com/r/btcpayserver/btcpayserver) of course)
 * bitcoind (from their [btcpayserver/bitcoin](https://hub.docker.com/r/btcpayserver/bitcoin) Docker image)
 * NBXplorer 2.2.5 (from the [nicolasdorier/nbxplorer](https://hub.docker.com/r/nicolasdorier/nbxplorer) Docker
   image)
 * postgresql 13.6 (from their [btcpayserver/postgres](https://hub.docker.com/r/btcpayserver/postgres) image)
 * [btcpayserver/letsencrypt-nginx-proxy-companion](https://hub.docker.com/r/btcpayserver/letsencrypt-nginx-proxy-companion) (presumably to
   generate LE certs automatically?)
 * [btcpayserver/tor](https://hub.docker.com/r/btcpayserver/tor) (yes, they have a tor container image)
 * tor-gen, also based on [docker-gen](https://hub.docker.com/r/btcpayserver/docker-gen) to generate a config for the
   above container

On the previous server, this also included:

 * lnd_bitcoin (for the "lighting network", based on [their image](https://hub.docker.com/r/btcpayserver/lnd))
 * bitcoin_rtl (based on [shahanafarooqui/rtl](https://hub.docker.com/r/shahanafarooqui/rtl), a webapp for the
   lightning network)
 * postgresql 9.6.20 (severely out of date!)

In theory, it should be possible to operate this using standard Docker
(or `docker-compose` to be more precise) commands. In practice,
there's a `build.sh` shell script that generate the
`docker-compose.yml` file from scratch. That process is itself done
through *another* container
[btcpayserver/letsencrypt-nginx-proxy-companion](https://hub.docker.com/r/btcpayserver/docker-compose-generator).

Basically, BTCpayserver folks wrote something like a home-made
[Kubernetes operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) for people familiar with that concept. Except
it doesn't run in Kubernetes, and it only *partly* runs inside
containers, being mostly managed through shell (and Powershell!)
scripts.

### Programming languages

Moving on, it seems like the [BTCpayserver server](https://github.com/btcpayserver/btcpayserver) itself and
[NBXplorer](https://github.com/dgarage/NBXplorer) are mostly written in C# (oh yes).

Their [docker-gen](https://github.com/btcpayserver/docker-gen) thing is actually a *fork* of
[nginx-proxy/docker-gen](https://github.com/nginx-proxy/docker-gen), obviously out of date. That's written in
Golang. Same with
[btcpayserver/docker-letsencrypt-nginx-proxy-companion](https://github.com/btcpayserver/docker-letsencrypt-nginx-proxy-companion), an out of
date fork of [nginx-proxy/acme-companion](https://github.com/nginx-proxy/acme-companion), built with docker-gen
and lots of shell glue.

Nginx, PostgreSQL, bitcoin, and Tor are, of course, written in
C.

### Services

It's hard to figure out exactly how this thing works at all, but it
seems there are at those major components working underneath
here:

 * an Nginx web proxy with TLS support managed by a sidecar container
   (`btcpayserver/letsencrypt-nginx-proxy-companion`)
 * btcpayserver, the web interface which processes payments
 * NBXplorer, "A minimalist UTXO tracker for HD Wallets. The goal is
   to have a flexible, .NET based UTXO tracker for HD wallets. The
   explorer supports P2SH,P2PKH,P2WPKH,P2WSH and Multi-sig
   derivation." I challenge any cryptobro to explain this to me
   without a single acronym, from first principles, in a single
   sentence that still makes sense. Probably strictly internal?
 * an SQL database (PostgreSQL), presumably to keep track of
   administrator accounts
 * bitcoind, the bitcoin daemon which actually records transactions in
   the global ledger that is the blockchain, eventually, maybe, if you
   ask nicely?

There's a bunch of Docker containers around this that generate
configuration and glue things together, see above.

### Storage and queues

It's unclear what is stored where. Transactions, presumably, get
recorded in the blockchain, but they are also certainly recorded in
the PostgreSQL database.

Transactions can be held in PostgreSQL for a while until a
verification comes in, presumably through NBXplorer. Old transactions
seem to stick around, presumably forever.

### Authentication

A simple username and password gives access to the administrative
interface. An admin password is stored in `tor-passwords.git`, either
in `external-services` (old server) or `hosts-extra-info` (new
server). There's support for 2FA, but it hasn't been enabled.

### Integration with CiviCRM/donate.tpo

The [cryptocurrency donations page](https://donate.torproject.org/cryptocurrency/) on [donate.torproject.org](https://donate.torproject.org)
actually simply does a `POST` request to either the hidden service or
the normal site. The form has a hidden `storeId` tag that matches it
to the "store" on the BTCpayserver side, and from there the
btcpayserver side takes over.

The server doesn't appear to do anything special with the payment:
users are supposed to report their donations themselves.

## Issues

There is no issue tracker specifically for this project, [File][] or
[search][] for issues in the [team issue tracker][search] with the
~BTCpayserver label.

 [File]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/new
 [search]: https://gitlab.torproject.org/tpo/tpa/team/-/issues?label_name%5B%5D=BTCpayserver

Upstream has *set* of [GitHub repositories](https://github.com/btcpayserver/) with its own issues.

## Maintainer, users, and upstream

hiro did the first deployment of this service at lunanode, anarcat did
the second deployment, managed by TPA.

The finance team is fundamentally the people responsible or at least
dependent on this service, alongside anyone who needs to donate
cryptocurrency to the Tor project.

Upstream is the [BTCpayserver project][btcpayserver] itself ([GitHub org](https://github.com/btcpayserver/)) and
are fairly active. Their [support channel is on Mattermost](https://chat.btcpayserver.org/btcpayserver/channels/support) and
they *eventually* answer (~24h latency last time).

## Monitoring and testing

There is no application-specific monitoring of this service. Users are
expected to try to make a donation with Bitcoin (!) to see if payments
go through. Money machine team is responsible for testing.

Rudimentary tests can be performed by going to the main domain website
(<https://btcpay.torproject.org>) and logging in with the credentials
from the TPA password manager. When someone does a payment, it should
show up as an invoice.

## Logs and metrics

BTCpay actually configures the Docker daemon to keep only 5m (5MB?) of
logs in `/etc/docker/daemon.json`:

    {
    "log-driver": "json-file",
    "log-opts": {"max-size": "5m", "max-file": "3"}
    }

Container logs can be inspected with:

    docker-compose -f $BTCPAY_DOCKER_COMPOSE logs -f --tail=10

Those include PII information like IP addresses, recorded by the Nginx
webserver. It is unclear how long that configuration will actually
keep data for, considering it's size-based.

## Backups

This service is made up of multiple Docker containers that are
technically hard to backup. Upstream has the approach of just stopping
the server (i.e. all containers) then performing the backup (badly,
see below).

So we're going to just pretend this is not a problem and let Bacula
backup `/var/lib/docker` as is. Yes, including the blockchain crap,
because that actually takes a long time to recover. Consistency might
be a problem. Sorry.

Full backup restore procedures are visible in the [backup and restore
section](#manual-backup-and-restore).

## Other documentation

Upstream has [documentation](https://docs.btcpayserver.org/).

# Discussion

This section aims at documenting more in-depth issues with the current
setup and possible solutions.

## Overview

BTCpay has a somewhat obscure and complicated history at Tor, and is
in itself a rather complicated project, as explained above in the
[design section](#design).

## Deployment history

The BTCpay server was originally setup, hosted, and managed by the
BTCpay people themselves. Back in March 2020, they suggested we host
it ourselves and, [in November 2020](https://gitlab.torproject.org/tpo/tpa/team/-/issues/33750#note_2714987), hiro had it deployed on a
Lunanode.com VM, at the recommendation of the BTCPay people.

Since then, an effort was made to move the VM inside TPA-managed
infrastructure, which is the setup that is documented in this
page. That effort is tracked in the above ticket,
[tpo/tpa/team#33750](https://gitlab.torproject.org/tpo/tpa/team/-/issues/33750).

The VM at Lunanode was setup with Ubuntu 16.04 which became EOL
(except for extended support) on 2021-04-30 (extended support stops in
2026). A quick audit in February 2022 showed that it didn't actually
have the extended support enabled, so that was done with anarcat's
personal Ubuntu credentials (it's not free).

Around April 2022, more effort was done to finally move the VM to TPA
infrastructure, but in doing so, significant problems were found with
BTCpay in particular, but also with our cryptocurrency handling in
general.

## Security review

There was never a security review performed on BTCpay by Tor
people. As far as we can tell, there was no security audit performed
on BTCpay by anyone.

The core of BTCpayserver is written in C# should should generally be a
safer language than some others, that said.

The state of the old VM is concerning, as it's basically EOL. We also
don't have good mechanisms for automating upgrades. We need to
remember to go in the machine and run the magic commands to update the
containers. It's unclear if this could be automated, considering the
upgrade procedure upstream proposes actually involves dynamically
regenerating the docker-compose file. It's also noisy so not a good
fit for a cron job.

Part of the reason this machine was migrated to TPA infrastructure was
to at least resolve the OS part of that technical debt, so that OS
upgrades, backups, and basic security (e.g. firewalls) would be
covered. This still leaves a gaping hole for the update and
maintenance of BTCpay itself.

## PII concerns

There are no efforts in BTCpay to redact PII from logs. Nginx logs
keep second-granular timestamps with full IP address and user agent
information. It's unclear how long invoices are retained in the
PostgreSQL database nor what information they contain.

BTCpay correctly generates a one-time Bitcoin address for
transactions, so that is done correctly at least. But right next to
the BTCpay button on https://donate.torproject.org/cryptocurrency,
there are static addresses for various altcoins (including bitcoin)
that are a serious liability, see [tpo/web/donate-static#74](https://gitlab.torproject.org/tpo/web/donate-static/-/issues/74) for
details.

## Alternatives considered

See [TPA-RFC-25: BTCpay replacement](policy/tpa-rfc-25-btcpay-replacement) for an evaluation of
alternatives.
