retire textile

Trac:
Child Ticket(s): #32281 (moved), #31700 (moved), #31676 (moved)

added component::internal services/tor sysadmin team owner::anarcat priority::high resolution::fixed severity::normal status::closed type::project labels

i'm all for making use of the new cluster, but I'm thinking that we should also keep space on the fsn* cluster to alleviate the load on kvm4, which is suffering a bit.

i'd also like to move the bacula director off of moly, which i distrust, see also #29974 (moved).

one of those machines is chamaemoly, which will be decommissioned altogether and shouldn't be migrated, see #31700 (moved).

it's actually ~~one~~ ~~two~~ ~~three~~ four (!) machines that are are to eventually be retired so we shouldn't migrate those either:

gayi (SVN) in #17202 (moved)
chiwui (check.tpo/tordnsel) in #29399 (moved)
togashii (testnet) in #31676 (moved)
chamaemoly (jabber) in #31700 (moved)

this leaves us with only those machines to migrate before textile can be decommissioned, yaay!

 	iranicum.torproject.org:
	 - shell/irc box
	saxatile.torproject.org:
	 - www.torproject.org
	 - static content rotation
	weissii.torproject.org:
	 - Windows buildbox
	winklerianum.torproject.org:
	 - Windows buildbox

derotated saxatile

windows VMs moved to kvm4. (winklerianum is not defined in virsh, but the data is there)

saxatile removed from ldap, disks queued for removal in 7 days, removed from puppet, nothing left in DNS, entry removed from spreadsheet, nagios, tor-passwords, and scheduled backup deletion in 30 days.

saxatile is done.

i'll fire this up next week and retire textile so we can loop in another gnt node already.

Trac:
Owner: tpa to anarcat
Status: new to assigned
Priority: Medium to High

starting work on this, will document a migration procedure as i go along.

Trac:
Status: assigned to accepted

i've disconnected the SVN and chiwui shutdown from this procedure because I don't want to wait for those projects to complete before decommissionning textile.

chiwui has been imported on fsn-node-03, but not converted yet. still need to do some functional tests and DNS changes before the switchover. documentation in https://help.torproject.org/tsa/howto/ganeti/#Importing_external_instances

i've done an import of gayi as well and improved the procedure to support the multiple disks in use there as well as the swap initialization. i've also reversed the sync path so we don't have to trust the old node.

unfortunately, fsn-node-03 has shown (HDD) disk problems which is (obviously) slowing down this work. they replaced the drive but the problem came up again, so i'll put this on hold for now.

the base images are all stored in /srv for now, a logical volume created for that purpose. i'll wait for news from hetzner before going any further.

worst case we setup a second ganeti node or just use the SSDs, which haven't shown sign of problems so far.

opened #33098 (moved) to followup on the disk issue, which is blocking this deployment.

gayi migrated to gnt-fsn, DNS records updated. next step is gayi decom.

gayi decom procedure done:

N/A
shutdown
undefined
scheduled removal in 7 days: echo 'rm -r /srv/vmstore/gayi.torproject.org/' | at now + 7 days
N/A: gayi remains for now, because it's still on gnt-fsn
N/A
N/A
N/A
N/A
move to gnt-fsn in the spreadsheet
N/A
N/A
N/A
N/A
N/A

chiwui specs:

20G, 4G swap
2G RAM
2CPU
SWAP_UUID=484b554e-fb17-4330-94e6-9a3fa3f8e1ed
NEW_IP=116.202.120.176

add:mac=00:66:37:24:f1:63,ip=116.202.120.177,mode=openvswitch,link=br0,network=gnt-fsn

There are two old IP addresses, strangely: 138.201.14.212 and 138.201.14.213, respectively chiwui2 and chiwui4 in DNS, not sure why that is.

i migrated a first version of the machine over and things still seem to work, although it's hard to tell if TBB is pinging the old server or the new (probably the former, unfortunately). will start the final migration now.

correct interfaces file:

auto eth0
iface eth0 inet static
    address 116.202.120.176/27
    gateway 116.202.120.161
iface eth0 inet6 static
    accept_ra 0
    address 2a01:4f8:fff0:4f:266:37ff:fe5a:8583/64
    gateway 2a01:4f8:fff0:4f::1

auto eth1
iface eth1 inet static
    address 116.202.120.177/32
iface eth1 inet6 static
    address 2a01:4f8:fff0:4f:266:37ff:fe24:f163/64

the two IP addresses are necessary for check to operate, because there are two services on port 80 (the normal webserver and tordnssel). the latter also requires IP changes in /srv, which should be grepped on top of /etc for IP address in the final run.

we're at step 11 for chiwui.

instance was recreated, new interfaces snippet:

auto eth0
iface eth0 inet static
    address 116.202.120.176/27
    gateway 116.202.120.161
iface eth0 inet6 static
    accept_ra 0
    address 2a01:4f8:fff0:4f:266:37ff:fe69:3bda/64
    gateway 2a01:4f8:fff0:4f::1

auto eth1
iface eth1 inet static
    address 116.202.120.177/27
iface eth1 inet6 static
    address 2a01:4f8:fff0:4f:266:37ff:fea4:3cf3/64

changed IP in LDAP
changed the firewall rules in puppet and deployed
changed IP in DNS

remaining work:

change IP in nagios
change reverse DNS

in the meantime, the TTL is hurting us: there are 1h records on that thing, so everything is timing out.... https://check.torproject.org/ works in a regular browser when hacking at my /etc/hosts however, so presumably that part will work once the tor network catches up (through DNS propagation)

similarly, tordnsel seems to work:

$ dig +short 62.62.129.185.80.4.3.2.1.ip-port.exitlist.torproject.org @116.202.120.176
127.0.0.2

so let's wait. i flipped DNS at around Tue Feb 4 14:49:00 2020 -0500 (commit time, pushed shortly after). so things should coverge in maximum ~32 minutes now.

changed IP in nagios and changed reverse DNS records (i had forgotten gayi too!).

next step is to decom chiwui completely, then we wait 7 days until we decom textile!

chiwui host retirement procedure:

N/A
done
"done" (only removed from autostart but kept the xml file in case we want to restore this in a pinch)
done:

root@textile:/etc/libvirt# echo 'rm -r /srv/vmstore/chiwui.torproject.org/' | at now + 7 days warning: commands will be executed using /bin/sh job 6 at Tue Feb 11 20:31:00 2020

 5. N/A
 6. N/A
 7. N/A
 8. N/A
 9. N/A
 10. moved to the gnt-fsn cluster in the spreadsheet
 11. N/A
 12. N/A
 13. N/A
 14. N/A
 15. N/A

chiwui can be considered fully migrated now. next step is to decomission textile, on february 11th.

posted a notification about the change to the mailing list here : https://lists.torproject.org/pipermail/tor-project/2020-February/002696.html

retire textile

Child items 0

Activity