metricsdb-03 has 100k UDPsockets open on port 53, overwhelming the node exporter

metricsdb03 started making noises last night

Steps to reproduce

unclear. TBD

What is the current bug behavior?

it seems like the node exporter is failing to report to prometheus, triggering a HostDown alert, even though the host is actually responsive and accessible over SSH.

furthermore, it's flapping: we've had a bunch of on/off alerts that are quite noisy.

What is the expected correct behavior?

we shouldn't have such alerts.

When did this start?

23:25:22UTC is the first alert.

Relevant logs and/or screenshots

#tor-alerts logs, UTC-4.

19:25:22 -ALERTOR3^:#tor-alerts- HostDown [firing] Host metricsdb-03.torproject.org is not responding CRITICAL!
20:55:22 -ALERTOR3^:#tor-alerts- HostDown [resolved] Host metricsdb-03.torproject.org is not responding CRITICAL!
21:19:22 -ALERTOR3^:#tor-alerts- HostDown [firing] Host metricsdb-03.torproject.org is not responding CRITICAL!
21:34:22 -ALERTOR3^:#tor-alerts- HostDown [resolved] Host metricsdb-03.torproject.org is not responding CRITICAL!
22:08:22 -ALERTOR3^:#tor-alerts- HostDown [firing] Host metricsdb-03.torproject.org is not responding CRITICAL!
23:08:22 -ALERTOR3^:#tor-alerts- HostDown [resolved] Host metricsdb-03.torproject.org is not responding CRITICAL!
23:23:22 -ALERTOR3^:#tor-alerts- HostDown [firing] Host metricsdb-03.torproject.org is not responding CRITICAL!
Day changed to 26 Mar 2026
00:38:22 -ALERTOR3^:#tor-alerts- HostDown [resolved] Host metricsdb-03.torproject.org is not responding CRITICAL!
00:51:22 -ALERTOR3^:#tor-alerts- HostDown [firing] Host metricsdb-03.torproject.org is not responding CRITICAL!
01:06:22 -ALERTOR3^:#tor-alerts- HostDown [resolved] Host metricsdb-03.torproject.org is not responding CRITICAL!
01:21:22 -ALERTOR3^:#tor-alerts- HostDown [firing] Host metricsdb-03.torproject.org is not responding CRITICAL!
01:36:22 -ALERTOR3^:#tor-alerts- HostDown [resolved] Host metricsdb-03.torproject.org is not responding CRITICAL!
01:52:22 -ALERTOR3^:#tor-alerts- HostDown [firing] Host metricsdb-03.torproject.org is not responding CRITICAL!
04:12:22 -ALERTOR3^:#tor-alerts- HostDown [resolved] Host metricsdb-03.torproject.org is not responding CRITICAL!
04:28:22 -ALERTOR3^:#tor-alerts- HostDown [firing] Host metricsdb-03.torproject.org is not responding CRITICAL!
05:48:22 -ALERTOR3^:#tor-alerts- HostDown [resolved] Host metricsdb-03.torproject.org is not responding CRITICAL!
06:01:22 -ALERTOR3^:#tor-alerts- HostDown [firing] Host metricsdb-03.torproject.org is not responding CRITICAL!

Possible fixes

for now, we deployed a hotfix (https://gitlab.torproject.org/tpo/tpa/puppet-control/-/commit/95b73c870896cedd1d470eaaf19bd4fed737f929) that disables the collector at fault (udp_queues), but we need to investigate why this is happening in the first place.