Relay health work · Wiki · The Tor Project / Network Health / Team

There are several indicators available for detecting whether a relay is in a bad or a healthy shape. This page documents our current efforts in this area.

Relay overload metrics

Starting with the implementation of proposal 328 (which landed in its first version in the 0.4.6 series) there are several overload metrics available both for external consumers and just for relay operators.

Exposed metrics

Overload metrics exposed in server and extra-info descriptors (and via Onionoo) are:

overload-general: indicates any of the following issues got hit during the last 72 hours:
- Tor OOM due to memory pressure
- ntor onionskins are dropped
- TCP port exhaustion

overload-ratelimits: contains the number of times BandwidthBurst and BandwidthRate limits got reached in the past 24 hours
overload-fd-exhausted: indicates the relay did have enough file descriptors available at some point during the last 72 hours

Internal metrics

Overload metrics accessible over the MetricsPort are (we omit any congestion control related metrics as we still need to stabilize that interface):

# HELP tor_relay_connections Total number of opened connections
# TYPE tor_relay_connections gauge
tor_relay_connections{type="OR listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="OR listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="OR listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="OR listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="OR",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="OR",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="OR",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="OR",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Exit",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Exit",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Exit",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Exit",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Socks listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Socks listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Socks listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Socks listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Socks",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Socks",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Socks",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Socks",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Directory listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Directory listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Directory listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Directory listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Directory",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Directory",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Directory",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Directory",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Control listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Control listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Control listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Control listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Control",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Control",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Control",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Control",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Transparent pf/netfilter listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Transparent pf/netfilter listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Transparent pf/netfilter listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Transparent pf/netfilter listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Transparent natd listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Transparent natd listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Transparent natd listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Transparent natd listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="DNS listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="DNS listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="DNS listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="DNS listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Extended OR",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Extended OR",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Extended OR",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Extended OR",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Extended OR listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Extended OR listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Extended OR listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Extended OR listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="HTTP tunnel listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="HTTP tunnel listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="HTTP tunnel listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="HTTP tunnel listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Metrics listener",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Metrics listener",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Metrics listener",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Metrics listener",direction="received",state="opened",family="ipv6"} 0
tor_relay_connections{type="Metrics",direction="initiated",state="opened",family="ipv4"} 0
tor_relay_connections{type="Metrics",direction="initiated",state="opened",family="ipv6"} 0
tor_relay_connections{type="Metrics",direction="received",state="opened",family="ipv4"} 0
tor_relay_connections{type="Metrics",direction="received",state="opened",family="ipv6"} 0
# HELP tor_relay_connections_total Total number of created/rejected connections
# TYPE tor_relay_connections_total counter
tor_relay_connections_total{type="OR listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="OR listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="OR listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="OR listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="OR listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="OR listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="OR",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="OR",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="OR",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="OR",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="OR",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="OR",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Exit",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Exit",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Exit",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Exit",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Exit",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Exit",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Socks listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Socks listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Socks listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Socks listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Socks listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Socks listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Socks",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Socks",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Socks",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Socks",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Socks",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Socks",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Directory listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Directory listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Directory listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Directory listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Directory listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Directory listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Directory",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Directory",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Directory",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Directory",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Directory",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Directory",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Control listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Control listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Control listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Control listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Control listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Control listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Control",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Control",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Control",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Control",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Control",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Control",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Transparent pf/netfilter listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Transparent pf/netfilter listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Transparent pf/netfilter listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Transparent pf/netfilter listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Transparent pf/netfilter listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Transparent pf/netfilter listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Transparent natd listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Transparent natd listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Transparent natd listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Transparent natd listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Transparent natd listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Transparent natd listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="DNS listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="DNS listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="DNS listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="DNS listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="DNS listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="DNS listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Extended OR",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Extended OR",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Extended OR",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Extended OR",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Extended OR",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Extended OR",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Extended OR listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Extended OR listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Extended OR listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Extended OR listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Extended OR listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Extended OR listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="HTTP tunnel listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="HTTP tunnel listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="HTTP tunnel listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="HTTP tunnel listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="HTTP tunnel listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="HTTP tunnel listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Metrics listener",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Metrics listener",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Metrics listener",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Metrics listener",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Metrics listener",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Metrics listener",direction="received",state="rejected",family="ipv6"} 0
tor_relay_connections_total{type="Metrics",direction="initiated",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Metrics",direction="initiated",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Metrics",direction="received",state="created",family="ipv4"} 0
tor_relay_connections_total{type="Metrics",direction="received",state="created",family="ipv6"} 0
tor_relay_connections_total{type="Metrics",direction="received",state="rejected",family="ipv4"} 0
tor_relay_connections_total{type="Metrics",direction="received",state="rejected",family="ipv6"} 0
# HELP tor_relay_flag Relay flags from consensus
# TYPE tor_relay_flag gauge
tor_relay_flag{type="Fast"} 0
tor_relay_flag{type="Exit"} 0
tor_relay_flag{type="Authority"} 0
tor_relay_flag{type="Stable"} 0
tor_relay_flag{type="HSDir"} 0
tor_relay_flag{type="Running"} 0
tor_relay_flag{type="V2Dir"} 0
tor_relay_flag{type="Sybil"} 0
tor_relay_flag{type="Guard"} 0
# HELP tor_relay_circuits_total Total number of circuits
# TYPE tor_relay_circuits_total gauge
tor_relay_circuits_total{state="opened"} 0
# HELP tor_relay_streams_total Total number of streams
# TYPE tor_relay_streams_total counter
tor_relay_streams_total{type="BEGIN"} 0
tor_relay_streams_total{type="BEGIN_DIR"} 0
tor_relay_streams_total{type="RESOLVE"} 0
# HELP tor_relay_traffic_bytes Traffic related counters
# TYPE tor_relay_traffic_bytes counter
tor_relay_traffic_bytes{direction="read"} 0
tor_relay_traffic_bytes{direction="written"} 0
# HELP tor_relay_dos_total Denial of Service defenses related counters
# TYPE tor_relay_dos_total counter
tor_relay_dos_total{type="circuit_rejected"} 0
tor_relay_dos_total{type="circuit_killed_max_cell"} 0
tor_relay_dos_total{type="circuit_killed_max_cell_outq"} 0
tor_relay_dos_total{type="marked_address"} 0
tor_relay_dos_total{type="marked_address_maxq"} 0
tor_relay_dos_total{type="conn_rejected"} 0
tor_relay_dos_total{type="concurrent_conn_rejected"} 0
tor_relay_dos_total{type="single_hop_refused"} 0
tor_relay_dos_total{type="introduce2_rejected"} 0
# HELP tor_relay_load_onionskins_total Total number of onionskins handled
# TYPE tor_relay_load_onionskins_total counter
tor_relay_load_onionskins_total{type="tap",action="processed"} 0
tor_relay_load_onionskins_total{type="tap",action="dropped"} 0
tor_relay_load_onionskins_total{type="fast",action="processed"} 0
tor_relay_load_onionskins_total{type="fast",action="dropped"} 0
tor_relay_load_onionskins_total{type="ntor",action="processed"} 0
tor_relay_load_onionskins_total{type="ntor",action="dropped"} 0
tor_relay_load_onionskins_total{type="ntor_v3",action="processed"} 0
tor_relay_load_onionskins_total{type="ntor_v3",action="dropped"} 0
# HELP tor_relay_exit_dns_query_total Total number of DNS queries done by this relay
# TYPE tor_relay_exit_dns_query_total counter
tor_relay_exit_dns_query_total 0
# HELP tor_relay_exit_dns_error_total Total number of DNS errors encountered by this relay
# TYPE tor_relay_exit_dns_error_total counter
tor_relay_exit_dns_error_total{reason="success"} 0
tor_relay_exit_dns_error_total{reason="format"} 0
tor_relay_exit_dns_error_total{reason="serverfailed"} 0
tor_relay_exit_dns_error_total{reason="notexist"} 0
tor_relay_exit_dns_error_total{reason="notimpl"} 0
tor_relay_exit_dns_error_total{reason="refused"} 0
tor_relay_exit_dns_error_total{reason="truncated"} 0
tor_relay_exit_dns_error_total{reason="unknown"} 0
tor_relay_exit_dns_error_total{reason="tor_timeout"} 0
tor_relay_exit_dns_error_total{reason="shutdown"} 0
tor_relay_exit_dns_error_total{reason="cancel"} 0
tor_relay_exit_dns_error_total{reason="nodata"} 0
# HELP tor_relay_load_oom_bytes_total Total number of bytes the OOM has freed by subsystem
# TYPE tor_relay_load_oom_bytes_total counter
tor_relay_load_oom_bytes_total{subsys="cell"} 0
tor_relay_load_oom_bytes_total{subsys="dns"} 0
tor_relay_load_oom_bytes_total{subsys="geoip"} 0
tor_relay_load_oom_bytes_total{subsys="hsdir"} 0
# HELP tor_relay_load_socket_total Total number of sockets
# TYPE tor_relay_load_socket_total gauge
tor_relay_load_socket_total{state="opened"} 0
tor_relay_load_socket_total 0
# HELP tor_relay_load_tcp_exhaustion_total Total number of times we ran out of TCP ports
# TYPE tor_relay_load_tcp_exhaustion_total counter
tor_relay_load_tcp_exhaustion_total 0
# HELP tor_relay_load_global_rate_limit_reached_total Total number of global connection bucket limit reached
# TYPE tor_relay_load_global_rate_limit_reached_total counter
tor_relay_load_global_rate_limit_reached_total{side="read"} 0
tor_relay_load_global_rate_limit_reached_total{side="write"} 0

E-Mail templates for contacting operators of overloaded relays or bridges

overload-fd-exhausted

File descriptor exhaustion at your "$nickname" Tor $node_type

Hello!

It seems your Tor ${node_type}[1] is suffering from not enough file
descriptors being available. Can you check how many file descriptors are
currently available on your machine (be sure to check on the shell that
runs the Tor relay) and raise the limit if it is low (lower than 65536)?
We have some documentation on how to do that and how to mitigate the
problem.[2]

Thanks!
$name

[1] https://metrics.torproject.org/rs.html#details/$fingerprint
[2] https://gitlab.torproject.org/tpo/core/tor/-/blob/main/doc/TUNING

Reachability check for exit relays

Besides making the above metrics available for relays in general there is the option to check exit relays for various misconfigurations by using them to reach a destination. Right now we mainly check for:

DNS misconfiguration

Scanners

For detecting DNS misconfigurations we use the DNS resolution module in our exitmap tool as we already use that for other exit scanning purposes.

Possible future work

In case the relay overload metrics we currently make available are not sufficient for our network-health purposes we can think about exposing further information, visible in Tor's heartbeat, in an aggregated way.

Additionally, we might want to think about implementing the missing pieces of proposal 328 related to the overload-general line: CPU utilization of Tor's mainloop (CPU core above 90% for 60 seconds) and potential control port overload.

Comments

Please register or sign in to add a comment.