... | ... | @@ -41,10 +41,73 @@ provider. This information is available in [Nagios][]: |
|
|
a hosting provider (e.g. `hetzner-hel1-01` is Hetzner, `gw-cymru`
|
|
|
is Cymru, `gw-scw-*` are at Scaleway, `gw-sunet` is Sunet)
|
|
|
|
|
|
Check if the parent server is online. If you still can't figure out
|
|
|
which server that is, use `traceroute` or `mtr` to see the hops to the
|
|
|
server. Normally, you should see a reverse DNS matching one of our
|
|
|
point of presence. This will also show you whether or not the upstream
|
|
|
routers are responsive. This is an example of a healthy trace to
|
|
|
`fsn-node-01`, hosted at Hetzner robot:
|
|
|
|
|
|
anarcat@curie:~$ mtr -c 10 -r fsn-node-01.torproject.org
|
|
|
Start: 2022-11-22T14:42:43-0500
|
|
|
HOST: curie Loss% Snt Last Avg Best Wrst StDev
|
|
|
1.|-- belleville.lan 0.0% 10 4.8 1.1 0.5 4.8 1.3
|
|
|
2.|-- 49.119.137.216.v.cable.or 0.0% 10 13.4 14.9 12.1 20.9 3.0
|
|
|
3.|-- 10.170.192.58 0.0% 10 23.5 16.3 10.8 24.6 5.1
|
|
|
4.|-- 21.189.18.64.static.orico 0.0% 10 58.6 25.9 15.8 58.6 13.8
|
|
|
5.|-- 10.0.4.15 0.0% 10 11.0 15.9 11.0 32.5 6.5
|
|
|
6.|-- 10gigabitethernet1-1-24.s 50.0% 10 11.8 18.1 11.8 32.5 8.6
|
|
|
7.|-- port-channel1.core1.ymq1. 30.0% 10 23.2 20.9 11.6 37.8 9.8
|
|
|
8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
|
|
|
9.|-- port-channel11.core3.lon2 80.0% 10 200.9 192.7 184.4 200.9 11.7
|
|
|
10.|-- port-channel12.core2.ams1 90.0% 10 111.7 111.7 111.7 111.7 0.0
|
|
|
11.|-- amsix-gw.hetzner.de 0.0% 10 106.1 104.4 100.3 110.5 3.9
|
|
|
12.|-- core4.fra.hetzner.com 0.0% 10 150.5 151.5 135.5 193.6 18.2
|
|
|
13.|-- core23.fsn1.hetzner.com 0.0% 10 124.8 122.3 111.9 134.5 7.4
|
|
|
14.|-- ex9k2.dc13.fsn1.hetzner.c 0.0% 10 126.0 120.8 116.3 130.7 4.5
|
|
|
15.|-- fsn-node-01.torproject.or 0.0% 10 134.2 134.7 129.0 143.6 5.0
|
|
|
|
|
|
Here's a healthy trace to a `hetzner-he1-01`, hosted in Hetzner cloud:
|
|
|
|
|
|
anarcat@curie:~$ mtr -c 10 -r hetzner-hel1-01.torproject.org
|
|
|
Start: 2022-11-22T14:43:46-0500
|
|
|
HOST: curie Loss% Snt Last Avg Best Wrst StDev
|
|
|
1.|-- belleville.lan 0.0% 10 0.6 0.7 0.5 0.9 0.1
|
|
|
2.|-- 49.119.137.216.v.cable.or 0.0% 10 49.9 23.6 12.1 53.4 16.1
|
|
|
3.|-- 10.170.192.58 0.0% 10 11.9 22.9 11.7 64.9 16.6
|
|
|
4.|-- 17.189.18.64.static.orico 0.0% 10 30.4 24.1 13.4 36.1 6.9
|
|
|
5.|-- 10.0.4.15 0.0% 10 19.6 22.0 9.8 61.8 19.2
|
|
|
6.|-- 10gigabitethernet1-1-24.s 30.0% 10 13.6 15.7 10.8 25.5 5.2
|
|
|
7.|-- port-channel1.core1.ymq1. 20.0% 10 13.9 13.3 10.2 18.2 2.5
|
|
|
8.|-- 100ge0-71.core2.nyc4.he.n 90.0% 10 49.4 49.4 49.4 49.4 0.0
|
|
|
9.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
|
|
|
10.|-- 100ge0-35.core1.ewr5.he.n 70.0% 10 32.7 33.1 26.7 39.9 6.6
|
|
|
11.|-- ve951.core2.cph1.he.net 20.0% 10 110.4 107.2 101.2 129.4 9.5
|
|
|
12.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
|
|
|
13.|-- port-channel4.core2.hel1. 70.0% 10 117.5 118.3 112.7 124.8 6.1
|
|
|
14.|-- hetzner.msk.piter-ix.net 0.0% 10 116.2 118.1 112.7 127.2 4.2
|
|
|
15.|-- core31.hel1.hetzner.com 0.0% 10 113.4 117.7 113.1 131.3 5.2
|
|
|
16.|-- spine2.cloud1.hel1.hetzne 0.0% 10 122.6 127.9 113.2 155.2 15.3
|
|
|
17.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
|
|
|
18.|-- 10620.your-cloud.host 0.0% 10 118.6 118.4 112.9 128.5 5.4
|
|
|
19.|-- hetzner-hel1-01.torprojec 0.0% 10 122.6 119.4 114.0 139.5 7.5
|
|
|
|
|
|
What follows are per-provider instructions:
|
|
|
|
|
|
### Hetzner robot (physical servers)
|
|
|
|
|
|
If you're not sure yet whether it's the server or Hetzner, you can
|
|
|
use location-specific Hetzner targets:
|
|
|
|
|
|
ash.icmp.hetzner.com
|
|
|
fsn.icmp.hetzner.com
|
|
|
hel.icmp.hetzner.com
|
|
|
nbg.icmp.hetzner.com
|
|
|
|
|
|
... and so on.
|
|
|
|
|
|
If all fails, you can try to reset or reboot the server remotely:
|
|
|
|
|
|
1. Visit the [Heztner Robot server page](https://robot.your-server.de/server) (password in
|
|
|
`tor-passwords/hosts-extra-info`)
|
|
|
2. Select the right server (hostname is the second column)
|
... | ... | |