snowflake-01: Connect the other NIC and move wireguard and sshd to it

This will allow for

• skipping conntracking on the 10G NIC
• ethtool tuning of the 10G NIC without risking losing the host

changed the description

assigned to @linus

This will probably happen within 4-6 hours.

DNS records for snowflake-01-mgmt.torproject.net and corresponding reverse records have been added, for sshd and wg.

Didn't happen yesterday. I have reasonably high hopes for today though.

I expect this to happen today sometime between 18:00 and 21:00 UTC.

mentioned in issue #40188 (closed)

Seems like we will most likely not get going any earlier than 20:00 UTC.

marked this issue as related to #40191 (closed)

Running sshd on eno2 now.

Down between 22:17-22:24.

closed

This turned out less successful than expected. Due to misconfiguration (two default routes (per address family), one per NIC) there were much confusion for a long period of time. Ended up rebooting to get rid of strange things. Current situation is that snowflake binds to IPv4 only (port 443) and the tor processes bind to .42 (on eno1) explicitly (with some noise in the logs).

Timeline of some events during deployment.

time	event
2022-09-29 22:17	reboot
2022-09-29 22:49	restart snowflake-server with IPv4-only bind
2022-09-29 23:22	delete eno2 default route
2022-09-29 23:27	restart snowflake-server with [::] bind
2022-09-29 23:31	restart snowflake-server with IPv4-only bind again
2022-09-29 23:35	restart snowflake-server
2022-09-29 23:43	reboot again

I have been watching the bandwidth charts and incoming traffic has been switching sometimes to use network interface eno2, then switching back. I caught the approximate start and end of two such periods when eno2 was used. They'll also be visible in the bandwidth graph because eno2 is only a 1 Gbps network interface.

start	end	interface
2022-09-30 03:31	2022-09-30 03:47	eno2
2022-09-30 04:27	2022-09-30 05:17	eno2

During these times, there was still no default route for eno2.

I have been watching the bandwidth charts and incoming traffic has been switching sometimes to use network interface eno2, then switching back.

Before restarting the eno2 interface on a separate subnet, we disabled it at 2022-09-30 05:44:28. Immediately following that there was an outage of IPv4 traffic that lasted until 2022-09-30 05:50:20.

Current situation is that snowflake binds to IPv4 only

We restarted snowflake-server with its usual dual-stack IPv4 and IPv6 binding at 2022-09-30 07:22:23.

The "new NIC", eno2 with a 1G uplink, is configured with a separate network since this morning. There are a few static routes to where we can reach it from. Let me know if you want your network added to this list. sshd is still binding to both addresses so this is not crucial for ssh access.

We (ie me) have console access since earlier today. This means that we can be more bold with network configuration and less stressed out if we mess up while optimising some part of the network stack. Let me know if you need access to the console.

mentioned in issue #40196 (closed)

mentioned in issue snowflake-webext#59 (closed)