BTCPayServer is Down

I am unable to connect to the btcpay.torproject.org. It says the site cannot be reached. I believe this means that donors cannot use it to donate either.

you're right, @susan i actually had noticed it in monitoring but we have so much noise there that i forgot about it. i'll take a look immediately.

added BTCpayserver Doing Emergency labels

assigned to @anarcat

changed title from BTCPayServer if Down to BTCPayServer is Down

service should be restored, sorry for the inconvenience. i have also tried to implement a mitigation to keep this from happening again, but i'm not exactly sure why it failed in the first place yet, so this might happen again.

it had already failed on February 13, and it was restored on the 14, but then promptly fell over again on the 14, until the 21st when it was fixed until it crashed again, on Wednesday:

  Wed. 22:14 [1/1]   nagios@hetzner-hel1-01.torproject.org  ** PROBLEM Service Alert: btcpayserver-02/network service - https cert is CRITICAL ** (nagios rapports tor)
 February 21 [1/1]   nagios@hetzner-hel1-01.torproject.org  ** RECOVERY Service Alert: btcpayserver-02/network service - https cert is OK ** (nagios rapports tor)
 February 14 [1/1]   nagios@hetzner-hel1-01.torproject.org  ** PROBLEM Service Alert: btcpayserver-02/network service - https cert is CRITICAL ** (nagios rapports tor)
 February 14 [1/1]   nagios@hetzner-hel1-01.torproject.org  ** RECOVERY Service Alert: btcpayserver-02/network service - https cert is OK ** (nagios rapports tor)
 February 13 [1/1]   nagios@hetzner-hel1-01.torproject.org  ** PROBLEM Service Alert: btcpayserver-02/network service - https cert is CRITICAL ** (nagios rapports tor)

do let me know asap if you see it failing again.

thanks for the report!

closed

So now I can't get on because the certificate is not valid on the site. Is that me or is something wrong on the server?

it's not you, checking.

reopened

there's something wrong with their btcpayserver thing. i'm not sure what's happening, but it can't fetch is HTTPS certificate, and is therefore failing to provide users with a proper one.

this one a bit harder than usual, even for that confusing service. i'll need to investigate further.

Creating/renewal btcpay.torproject.org certificates... (btcpay.torproject.org btcpayserver-02.torproject.org)
[Fri Mar  1 21:07:02 UTC 2024] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 28
[Fri Mar  1 21:07:02 UTC 2024] Can not init api for: https://acme-v02.api.letsencrypt.org/directory.
Sleep for 3600s

i'm considering ditching their nginx proxy and using our own instead, i'll need to look more deeply at how all of this is setup. their documentation is... rather convoluted, but i think i need to switch to a no reverse proxy configuration, as their nginx container stuff keeps failing weirdly like this.

i started the conversion operation, but need to step out for a bit, i'm hoping to finish this by EODAOE.

@susan i think i nailed it about right, can you try again?

internal note: i've seen docker-compose fail to restart the service a few times, i think because the DOCKER chain didn't exist... maybe this is a case of "ferm needs to restart docker" kind of problem?

added Needs Review label and removed Doing label

this needs documentation too, i need to clarify in the wiki that the install procedure changed a bit, and tweak the pager playbook that refers to the old container-based cert issuance.

i added some tweaks from this guide https://docs.btcpayserver.org/FAQ/Deployment/#can-i-use-an-existing-nginx-server-as-a-reverse-proxy-with-ssl-termination

i also followed

https://github.com/btcpayserver/btcpayserver-docker/blob/master/Production-NoReverseProxy/README.md

Now it is say invalid login attempt.

Did this all change my login info?

the btcpay server works; the issue with @susan's password was https://gitlab.torproject.org/tpo/tpa/team/-/issues/41530 and has been fixed out of band.

closed

@susan had issues with websockets. i thought i had done something to make those work in #41549 (comment 3002411) but it seems it still doesn't work. she also needs a password reset and some help moving stuff around computers, we'll reconvene on monday.

i tested with susan in a live session, and it does seem like the web browser is hitting some weird websocket issue. the btcpayserver people don't seem to have any idea of what's involved here, instead suggesting we upgrade btcpay vault or switch to Electrum, which doesn't seem like a related issue.

so next step is for me to rollback the standalone nginx configuration to see if that fixes the issue, and, failing that, at least get a sample configuration to work from.

added Next label and removed Needs Review label

(looks like i forgot to post this comment yesterday:)

so, uh. i have reverted to the previous configuration and against all odds, it looks like the server is working again. i didn't expect that because I didn't expect the HTTPS certs to be valid because we've remove the server from CAA records.

anyways, now it seems online, and without our nginx frontend, so it should work as it did before.

i pinged @susan so she tests the site. in the meantime, puppet is disabled on the host to keep it from trying to deploy nginx again.

Click to expand

here's a dump of the generated nginx configuration:

root@btcpayserver-02:~/BTCPayServer/btcpayserver-docker# docker exec -it nginx /bin/bash
root@3198582b3e86:/# cat /etc/nginx/nginx.conf 

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}
root@3198582b3e86:/# cat /etc/nginx/conf.d/default.conf 
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
  default $http_x_forwarded_port;
  ''      $server_port;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Prevent Nginx Information Disclosure
server_tokens off;
# Default dhparam
# Set appropriate X-Forwarded-Ssl header
map $scheme $proxy_x_forwarded_ssl {
  default off;
  https on;
}
gzip on;
gzip_min_length 1000;
gzip_types image/svg+xml text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';
access_log off;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
proxy_buffer_size          128k;
proxy_buffers              4 256k;
proxy_busy_buffers_size    256k;
client_header_buffer_size 500k;
large_client_header_buffers 4 500k;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
	server_name _; # This is just an invalid value which will never trigger on a real hostname.
	listen 80;
	access_log /var/log/nginx/access.log vhost;
	return 503;
}
server {
	server_name _; # This is just an invalid value which will never trigger on a real hostname.
	listen 443 ssl;
	http2  on;
	access_log /var/log/nginx/access.log vhost;
	return 503;
	ssl_session_tickets off;
	ssl_certificate /etc/nginx/certs/default.crt;
	ssl_certificate_key /etc/nginx/certs/default.key;
}
upstream btcpay {
	## Can be connected with "generated_default" network
	# generated_btcpayserver_1
	server 172.18.0.2:49392;
}
server {
	client_max_body_size 100M;
	server_name 3rk3juzfjgs6tpn57betff4yrnuprd72xdzn6tmgf4rbdwnjy3dapoqd.onion;
	listen 80 ;
	proxy_set_header X-Forwarded-Host  $http_host;
	proxy_set_header Host $http_host;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection $proxy_connection;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
	proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
	proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
	access_log /var/log/nginx/access.log vhost;
	include /etc/nginx/vhost.d/default;
	location / {
		proxy_pass http://btcpay;
	}
}
server {
	server_name btcpay.torproject.org;
	listen 80 ;
	access_log /var/log/nginx/access.log vhost;
	return 301 https://$host$request_uri;
}
server {
	proxy_set_header X-Forwarded-Host  $http_host;
	proxy_set_header Host $http_host;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection $proxy_connection;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
	proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
	proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
	client_max_body_size 100M;
	server_name btcpay.torproject.org;
	listen 443 ssl ;
	http2  on;
	access_log /var/log/nginx/access.log vhost;
	ssl_protocols TLSv1.2;
	ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
	ssl_prefer_server_ciphers on;
	ssl_session_timeout 5m;
	ssl_session_cache shared:SSL:50m;
	ssl_session_tickets off;
	ssl_certificate /etc/nginx/certs/btcpay.torproject.org.crt;
	ssl_certificate_key /etc/nginx/certs/btcpay.torproject.org.key;
	ssl_dhparam /etc/nginx/certs/btcpay.torproject.org.dhparam.pem;
	ssl_stapling on;
	ssl_stapling_verify on;
	ssl_trusted_certificate /etc/nginx/certs/btcpay.torproject.org.chain.pem;
	add_header Strict-Transport-Security "max-age=31536000" always;
	include /etc/nginx/vhost.d/default;
	location / {
		proxy_pass http://btcpay;
	}
}
server {
	server_name btcpayserver-02.torproject.org;
	listen 80 ;
	access_log /var/log/nginx/access.log vhost;
	return 301 https://$host$request_uri;
}
server {
	proxy_set_header X-Forwarded-Host  $http_host;
	proxy_set_header Host $http_host;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection $proxy_connection;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
	proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
	proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
	client_max_body_size 100M;
	server_name btcpayserver-02.torproject.org;
	listen 443 ssl ;
	http2  on;
	access_log /var/log/nginx/access.log vhost;
	ssl_protocols TLSv1.2;
	ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
	ssl_prefer_server_ciphers on;
	ssl_session_timeout 5m;
	ssl_session_cache shared:SSL:50m;
	ssl_session_tickets off;
	ssl_certificate /etc/nginx/certs/btcpayserver-02.torproject.org.crt;
	ssl_certificate_key /etc/nginx/certs/btcpayserver-02.torproject.org.key;
	ssl_dhparam /etc/nginx/certs/btcpayserver-02.torproject.org.dhparam.pem;
	ssl_stapling on;
	ssl_stapling_verify on;
	ssl_trusted_certificate /etc/nginx/certs/btcpayserver-02.torproject.org.chain.pem;
	add_header Strict-Transport-Security "max-age=31536000" always;
	include /etc/nginx/vhost.d/default;
	location / {
		proxy_pass http://btcpay;
	}
}
root@3198582b3e86:/# cat /etc/nginx/vhost.d/default;
## Start of configuration add by letsencrypt container
location ^~ /.well-known/acme-challenge/ {
    auth_basic off;
    auth_request off;
    allow all;
    root /usr/share/nginx/html;
    try_files $uri =404;
    break;
}
## End of configuration add by letsencrypt container

reopened

added Doing label and removed Next label

so i'm re-deploying our standard nginx configuration in the hope that the tweaks i've done to it will be sufficient to make the hardware wallet works. and if it fails, i have a good idea of the nginx configuration that should be running, above.

the extra configuration that needs to be done for this is, in /etc/profile.d/btcpay-env.sh:

export BTCPAYGEN_REVERSEPROXY="none"
export BTCPAYGEN_EXCLUDE_FRAGMENTS="$BTCPAYGEN_EXCLUDE_FRAGMENTS;nginx-https"

and in /root/BTCPayServer/.env:

#REVERSEPROXY_HTTP_PORT=80
#REVERSEPROXY_HTTPS_PORT=443
NOREVERSEPROXY_HTTP_PORT=127.0.0.1:8080
BTCPAYGEN_REVERSEPROXY="none"

then to deploy:

. /etc/profile.d/btcpay-env.sh &&
cd BTCPayServer/btcpayserver-docker/ &&
. /etc/profile.d/btcpay-env.sh && 
. btcpay-setup.sh -i &&
./build.sh &&
docker-compose -f $BTCPAY_DOCKER_COMPOSE up --remove-orphans -d &&
puppet agent --enable &&
puppet agente --test

mentioned in commit wiki-replica@cc4ac49d

sue was able to test the modified service and it now works properly with our plain nginx! whoohoo!

i've also documented the changes in the wiki.

so i think this is done!

closed