dal-rescue-01 deployment

marked this issue as related to #41058 (closed)

changed due date to May 16, 2023

assigned to @anarcat

changed milestone to %trusted high performance cluster (gnt-dal migration)

added Next label and removed Backlog label

changed the description

update on this: i started contacting quintex about this on april 17th, got a response same day with a proposed date of may 5th... i asked to see if we could settle on some date that wasn't the weekend, no response since April 18th... sent a ping today.

changed the description

got a response from quintex, they're available, we just need to ship the device first to see how much time it will take to arrive.

added Doing label and removed Next label

marked the checklist item ship dal-rescue-01 to the datacenter by TPA as completed

device shipped

added Next label and removed Doing label

changed the description

marked the checklist item dal-rescue-01 delivered to Quintex as completed

marked the checklist item agree on a plan with quintex as completed

device arrived, confirmed the plan to deploy this at 13:00 central (14:00 eastern) on May 16th.

ping'd quintex to confirm deploy tomorrow.

added Doing label and removed Next label

quintex is at the datacenter and we'll proceed to the deployment earlier than planned

marked the checklist item dal-rescue-01 online, connected to the three VLANs by Quintex as completed

marked the checklist item access to dal-rescue-01 confirmed by TPA as completed

changed the description

marked the checklist item dal-node-03 as completed

dal-node-03 was evacuated and renumbered, going to use this occasion to reboot it as well since there's a pending reboot

root@dal-node-01:~# gnt-node migrate dal-node-03.torproject.org
Migrate instance(s) dangerzone-01.torproject.org, ns3.torproject.org,
probetelemetry-01.torproject.org, static-gitlab-shim.torproject.org,
tb-pkgstage-01.torproject.org, telegram-bot-01.torproject.org?
y/[n]/?: y

there was a critical warning in the BIOS, asked @lavamind to add a ticket to add monitoring for that. quintex reseated the second power supply, clearing the first warning, and i reset the chassis intrusion flag, clearing the second.

Ticket created at #41172 (closed)

dal-rescue-01 online, connected to the three VLANs by Quintex

i had to allocate two ports in the dal-sw-01 switch for this:

port 47: public VLAN, dal-rescue-01 eth0
port 48: storage VLAN, dal-rescue-02 eth1

dal-rescue-01 was also misconfigured so it didn't fully come up on boot: the default route was set to .1 instead of the iconoclastic .254

marked the checklist item dal-node-02 as completed

marked the checklist item dal-node-01 as completed

changed the description

marked the checklist item chi-node-14 as completed

marked the checklist item dal-sw-01 as completed

marked the checklist item change the IP address of each OOB interface, one by one: as completed

i believe all interfaces were renumbered properly and we're go to remove the OOB switch from the old VPN network.

changed the description

marked the checklist item Quintex disconnects the OOB switch from the VPN network, now completely isolated behind dal-rescue-01 as completed

marked the checklist item do one final round of testing as completed

all clear, dal-node-01 shipped and running.

changed the description

note that dal-rescue-01 didn't properly setup networking after boot, because the gateway was misconfigured. i managed to login over IPv6 with:

ssh -J people.torproject.org  2620:7:6002::20d:b9ff:fe60:cc30 -l root

phew!

i tested a reboot and, while that worked, i noticed the host is not hooked into mandos, so i'll fix that now.

configured mandos, doing one more reboot

reboot succeeded, all done.

closed

marked this issue as related to #41174 (closed)

mentioned in issue #41174 (closed)

marked the checklist item one last dal-rescue-01 reboot as completed

dal-rescue-01 deployment

Designs

Child items ...

Activity