migrate gitlab-02 to new gnt-dal cluster

marked this issue as related to #41425 (closed)

marked this issue as related to #41429 (closed)

quick bandwidth test between the two sites:

root@fsn-node-01:~# pv /dev/zero | ssh -i /etc/ssh/ssh_host_ed25519_key root@d
al-node-01.torproject.org 'cat > /dev/null'
^C 919MiB 0:01:01 [14.9MiB/s] [                                  <=>           ]

that would mean 18 hours transfer, during which time we'd probably need gitlab to be completely offline unless we start doing fancy things with snapshots and so on.

changed title from migrate gitlab-02 to new gnt-fsn cluster to migrate gitlab-02 to new gnt-dal cluster

added Backlog label

mentioned in issue #40950

marked this issue as related to #41677 (closed)

mentioned in issue #41677 (closed)

marked this issue as related to #41765 (closed)

mentioned in issue #41765 (closed)

added Roadmap::Future label and removed Backlog label

mentioned in issue #41696 (closed)

@lavamind want to take this on? or perhaps @lelutin could get his hands dirty in this nasty piece of ganeti?

I'm happy either way. @lelutin if you want to take the lead on this, I'll be happy to help out flashing out the plan and be there for assistance on the day-of.

Just making a note however that this will need coordination across teams considering the downtime this will require, which I estimate to at least several hours.

It might even be a case for a special Friday workday...

It might even be a case for a special Friday workday...

hell, we could make that a work party... i am not sure i'd do it on a friday though, because if it breaks then we need to work weekends... if anything, i'd rather set that on a sunday if we really want to avoid the downtime. or a bank holiday or something.

I'm up for taking the steering wheel. yeah I'm inclined to think that this transfer should be started on thursday end of day and finish it up on a friday.

@lavamind I'd be happy to hop on a video call soon (maybe today?) to start carving up a plan for the migration.

@anarcat do we have a script on hand that can do the inter-cluster migration for us? I seem to remember seeing that, maybe in fabric scripts?

@anarcat never mind, I found https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/ganeti#cross-cluster-migrations which answers my question. so not one unified script but some documented instructions on how to accomplish this. great. and ganeti seems to have some script for moving the VM

@anarcat do we have a script on hand that can do the inter-cluster migration for us? I seem to remember seeing that, maybe in fabric scripts?

it's basically built-in to ganeti, but there's some glue work that needs to happen around it. for now it's a series of copy-paste from the ganeti wiki page, see if you can find it on your own! (and if not, tell me where you look so we can hotlink it)

I think this would be a good use-case for the "idle" machines, you could practice migrating idle-fsn-01 over to the gnt-dal cluster.

migrating idle-fsn-01 over to the gnt-dal cluster.

that could become confusing quick though...

i think we can trust @lelutin to follow the process correctly, same way you migrated donate-01 without training wheels. :)

heh yeah I was actually thinking of maybe staging a practice run since ganeti "works but has quirks".. but idle-fsn-01 being in the dal cluster would be confusing :P

Oh yeah you would need to migrate it back of course... it's a 2-for-1!

added Backlog label and removed Roadmap::Future label

assigned to @lelutin

one thing i wonder about this: do we want to keep gitlab-02 on the gnt-fsn cluster? we could use it as a "dev server"...

changed the description

I've added a quick summary of the documented move procedure to the description so that we have a flight-checklist for this particular operation. I've added communication to the rest of tor and moved reconfigure of grub-pc above the DNS change and added a reboot test after that. I'll update our documentation now to reflect that. better to consider the grub-pc detail as part of our tests and since we're messing with the boot procedure we're better off testing a reboot after that.

added Gitlab label

changed the description

one concern that @lavamind raised on IRC is the sheer size of this VM and the time it would take to transfer between the two clusters.

i've ran some tests to see what we're dealing with. a simple bandwidth tests between fsn-node-01 and dal-node-01 seems to say we have about 160mbps between the two clusters, which matches my experience with past test:

root@fsn-node-01:~# iperf3 -4 -c dal-node-01.torproject.org
Connecting to host dal-node-01.torproject.org, port 5201
[  5] local 88.198.8.185 port 34636 connected to 204.8.99.101 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.12 MBytes  34.5 Mbits/sec    0    595 KBytes       
[  5]   1.00-2.00   sec  17.5 MBytes   147 Mbits/sec    0   8.81 MBytes       
[  5]   2.00-3.00   sec  21.2 MBytes   178 Mbits/sec    0   8.83 MBytes       
[  5]   3.00-4.00   sec  22.5 MBytes   189 Mbits/sec    0   8.83 MBytes       
[  5]   4.00-5.00   sec  21.2 MBytes   178 Mbits/sec    0   8.83 MBytes       
[  5]   5.00-6.00   sec  22.5 MBytes   189 Mbits/sec    0   8.83 MBytes       
[  5]   6.00-7.00   sec  21.2 MBytes   178 Mbits/sec    0   8.83 MBytes       
[  5]   7.00-8.00   sec  21.2 MBytes   178 Mbits/sec    0   8.83 MBytes       
[  5]   8.00-9.00   sec  22.5 MBytes   189 Mbits/sec    0   8.83 MBytes       
[  5]   9.00-10.00  sec  21.2 MBytes   178 Mbits/sec    0   8.83 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   195 MBytes   164 Mbits/sec    0             sender
[  5]   0.00-10.13  sec   195 MBytes   162 Mbits/sec                  receiver

in #40917 (comment 2840145), i did some tests to use "zerofree" on the disks to improve compression. one trick is that the filesystem layout on gitlab-02 is a tad more complicated: we have LVM in there, so zerofree won't cut it for that, we need... "something else" to zero out the unallocated LVM space. it looks like this could be as simple as create a new LV taking up all empty space and zeroing it, though:

https://superuser.com/questions/363639/lvm-zero-out-empty-space-in-a-volume-group

i'm testing zerofree on the 300G gitlab-backup volume now. it's a simple, low hanging fruit because it can just be unmounted without stopping anything and should give us a good idea of how long zerofree will take.

update: it did 2% in about 2 minutes, so it's going to take 100 minutes for 300GB, or what, 400 minutes for 1.25TB? that's about 7 hours.

transfering 1.25TB at 164Mbit/s, on the other hand, will take about 17 hours (1.25terabyte/(164Mbit/s) in qalculate)... which is more, but not that much more... would we gain that much from the zerofree? not sure it's worth it.

zeroing out the unallocated volumes, however, seems like a smart move.

to benefit from this, however, we'll need to use the --compress flag to move-instance, which we haven't tested before. it seems the valid values for that are IEC_ALL = ["gzip","gzip-fast","gzip-slow","lzop","none"] with none being the likely default.

also note that zerofree can operate on readonly filesystems, which could be used to zero out the backup volume more intelligently than what i'm doing now... i ended up interrupting the process because it would have taken too long.

started this to actually, correctly zero out the backup volume:

date ; mount -o remount,ro /srv/gitlab-backup && time zerofree -v /dev/mapper/vg_gitlab--02_hdd-gitlab--backup ; mount -o remount,rw /s
rv/gitlab-backup; date

started at Thu Nov 14 21:11:59 UTC 2024, currently running in a screen(1).

update: that completed:

root@gitlab-02:~# date ; mount -o remount,ro /srv/gitlab-backup && time zerofree -v /dev/mapper/vg_gitlab--02_hdd-gitlab--backup ; mount -o remount,rw /srv/gitlab-backup; date
Thu Nov 14 21:11:59 UTC 2024
61968855/66249791/78642176

real    94m2.569s
user    1m35.577s
sys     5m18.201s
Thu Nov 14 22:46:01 UTC 2024

that's ... only 94 minutes! maybe it's worth it after all... i wonder how well gitlab behaves with a readonly filesystem?

@anarcat @lavamind should we plan for a migration date now that anarcat has some results wrt zerofree?

I would throw a first date proposition on the table: would november 28th/29th be a good time for this maintenance?

I would throw a first date proposition on the table: would november 28th/29th be a good time for this maintenance?

lgtm

mentioned in commit status-site@83135ffd

status maintenance planned: https://status.torproject.org/issues/2024-11-28-gitlab-migration/

sent an email to tor-project@lists.tpo and it's now awaiting moderation.

marked the checklist item communicate date of outage to all of tor as completed

marked the checklist item add planned maintenance item on status.tpo as completed

I looked into lowering down the TTL. all of the hostnames other than gitlab-02 and pages are CNAMEs to gitlab-02, so they don't need to move.

I've changed the TTL on *pages.tpo to 10 minutes

For gitlab-02 however the IP comes from LDAP. So I've added a dnsTTL field to the gitlab-02 host entry in ldap and after a little while the change is visible on external dns servers.

marked the checklist item lower TTL for subdomains that need it to 5mins -- gitlab-02, gitlab, containers, gitaly, pages as completed

changed the description

checking for the prep for migration to gnt-dal cluster:

rapi user was already present on dal-node-01
all nodes within each cluster share the same password. but the password is not the same for both clusters. I'm guessing this is what's intended by the procedure and so nothing needs to be done? @anarcat can you confirm my understanding of point 0 in https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/ganeti#move-instance-configuration ?
- If this corresponds to what I've described above, I'll change the documentation because the documentation's formulation seems to imply that we should have the same password on both clusters.
the RAPI cert for the fsn cluster was alredy present on dal-node-01 (under /root)
the rapi passwords for both clusters were already present in files on dal-node-01
skipped over points 5 and 6. we'll have to do those just before we start the transfer -- adding to list in description

marked the checklist item prepare dal cluster for accepting instances from fsn, if needed (e.g. RAPI cert, RAPI passwords, firewall) as completed

Prep is now done. The next step is the migration next week, so moving to Next instead

added Next label and removed Backlog label

changed the description

marked the checklist item zerofree on the backup partition (@anarcat) as completed

marked the checklist item zerofree on the other partitions (not done because requires downtime, as it requires readonly partition) as completed

i added items to the checklist that can be done before the maintenance window, i've zerofree'd the backup partion, but i don't think we can zerofree the other parts without downtime.

we should, however, wipe the free space in LVM, see #41431 (comment 3130680)

changed the description

marked the checklist item wipe free space on volume group (create lv that covers all the remaining free space and wipe it out with dd) (@lelutin) as completed

changed the description

marked the checklist item run steps 5 and 6 of prep as completed

here's what the documentation of --compress is not telling us: the available values for the option can be found out with:

# gnt-cluster info
[...]
Cluster parameters:
[...]
  compression tools: 
    - gzip
    - gzip-fast
    - gzip-slow

you can enable more tools by modifying both clusters: gnt-cluster modify --compression-tools=gzip,gzip-fast,gzip-slow,lzop

we ran two transfer tests in preparation, to make sure that we have all the bits and pieces ready for the final transfer.

the first transfer was right after the creation of the new dummy VM. it was transferring at around 20MiB/s relatively throughout all the procedure and took about 16 minutes to transfer the 20Gb disk.

for the second attempt, we first ran activate disks and did a zerofree on it before launching the transfer. that second attempt took only 6 minutes.

edit: in both cases we were running the move-instance script with the option --compress=lzop set

fwiw we booted the dummy instance on the dal cluster after the transfer and it reached the TTY, so the test transfer was successful.

note that the zerofree on the root (only) disk was done in this way, roughly:

on cluster master: gnt-instance activate-disks tests-transfer-01.tpo
on node:
- kpartx ... activate partitions
- fsck on the partition we wanted to run zerofree on
- zerofree -v /path/to/partition

marked the checklist item stop puppet on ganeti nodes on the dal cluster, finalize RAPI prep as completed

blah there's no "maintenance mode" available to use... it's a premium feature.

So we'll just proceed with the plan I laid out at first and rely on delaying the DNS change so that nothing will get written to the new location until we decide that the migration is over.

new IPv4: 204.8.99.149

alerts that fired:

HTTPSUnreachable: for gitlab.tpo, pages.tpo, containers.tpo and gitaly-02.tpo
PlaintextHTTPUnreachable: for the same four subdomains
jobdown for gitlab-02 with variants: job=gitlab (three alerts on three different ports), job=gitlab-workhorse, job=postgres, job=mtail, job=node, job=gitaly, job=nginx
SMTPUnreachable for gitlab-02.tpo
SSHUnreachable for gitlab-02.tpo
DRBDDegraded for dal-node-02 and dal-node-03 : they're the two destination nodes
systemd failed units on all fsn-node-* and dal-node-* ganeti nodes: we did disable puppet on all of them to avoid surprises in terms of firewall, in case puppet wanted to apply a change and reloaded the firewall. that would possibly have blocked the transfer
systemd failed unit, puppet-run.service where puppet tries to pull from a repository on gitlab.tpo: onionoo-backend-03, hetzner-nbg1-01, hetzner-nbg1-02.torproject.org, check-01.torproject.org, tb-build-03.torproject.org, collector-02.torproject.org, survey-01.torproject.org, colchicifolium.torproject.org, meronense.torproject.org, metricsdb-01.torproject.org, tb-build-02.torproject.org

I set silences for:

everything that has alias=gitlab-02.torproject.org or alias=gitaly-02.torproject.org
alias of dal-node-02.tpo or -03 and alertname=DRBDDegraded
alertname=SystedFailedUnits and all of the aliases identified above

edit: additionally to the above-mentioned alerts, there were also those that fired during the nigh:

PuppetAgentErrors and PuppetCatalogStale for all hosts mentioned in the systemd unit failed alert. we should setup an inhibition for PuppetAgentErrors when PuppetCatalogStale also fires.

systemd failed units on all fsn-node-* and dal-node-* ganeti nodes: we did disable puppet on all of them to avoid surprises in terms of firewall, in case puppet wanted to apply a change and reloaded the firewall. that would possibly have blocked the transfer

systemd failed unit, puppet-run.service where puppet tries to pull from a repository on gitlab.tpo: onionoo-backend-03, hetzner-nbg1-01, hetzner-nbg1-02.torproject.org, check-01.torproject.org, tb-build-03.torproject.org, collector-02.torproject.org, survey-01.torproject.org, colchicifolium.torproject.org, meronense.torproject.org, metricsdb-01.torproject.org, tb-build-02.torproject.org

those were particularly noisy, and kept repeating, rendering the notification channel pretty unusable. i think it's worth investigating this one separately, do you want me to file an issue, or how do you plan on following up on all of those? :)

I was thinking of adding a list of what needs to be silenced in our documentation. I don't know if that's enough to make this more calm for our future selves during other major maintenance, but it's a start.

This time around, I did not try to anticipate what alerts would be triggering and I let everything come out to IRC so that I could note everything down.

marked the checklist item enable maintenance mode on gitlab https://docs.gitlab.com/ee/administration/maintenance_mode/ and set message banner https://gitlab.torproject.org/admin/broadcast_messages if necessary to warn people of the read-only state as completed

marked the checklist item stop the instance and start the transfer as completed

marked the checklist item change IP in instance after the move as completed

marked the checklist item test that gitlab and all other websites are replying properly, reconfigure grub-pc, test reboot of instance as completed

marked the checklist item switch DNS entries to point to new IP as completed

marked the checklist item disable maintenance mode as completed

followup issue for old VM destruction created: #41904 (closed)

marked the checklist item schedule destruction of old instance as completed

marked the checklist item communicate to everyone that the move is over and that things are back to normal operation as completed

Unrelated to the migration I think but we might want to take a quick look at this. in the output of journalctl I saw this go by:

Nov 29 15:03:38 gitlab-02 prometheus-node-exporter[707]: ts=2024-11-29T15:03:38.131Z caller=textfile.go:227 level=error collector=textfile msg="failed to collect textfile data" file=tpa-gitlab-ci-wait.prom err="failed to parse textfile data from \"/var/lib/prometheus/node-exporter/tpa-gitlab-ci-wait.prom\": text format parsing error in line 3: expected float as value, got \"\""

added Doing label and removed Next label

marked this issue as related to #41905

mentioned in issue #41905

blah there's no "maintenance mode" available to use... it's a premium feature.

i made #41905 to followup on that.

marked this issue as related to #41906 (closed)

changed the description

marked this issue as related to #41909 (closed)

marked this issue as related to #41904 (closed)

@lavamind the password and certificate files were already present from previous instance migrations, so I'm wondering if we want to keep those around or if we should rather clean things up and re-create them for the next migrations.. do you have any thoughts on this?

marked the checklist item remove password files and cert files created by the prep before migration as completed

The TTL is back to the normal value on gitlab-02 (now visible on our ns servers). I've also cleared out the subdomain dip.torproject.org which was a CNAME to gitlab-02 since both anarcat and lavamind confirmed that it was not used anymore.

marked the checklist item bring the TTL back to the default value of 1h for gitlab-02 and *pages as completed

changed the description

I just saw an issue with graphs: we don't have any data on blackbox https probes since the migration, so we also won't have alerts if the sites go down.

I've tried restarting the prometheus-blackbox-exporter service but the metrics did not start coming in. restarting prometheus also did not bring the metrics back, so something else is at play. also the metrics for backbox probes stopped on the 27th

I just pushed a couple changes to the wiki section about cross-cluster migrations to make our documentation more up to date. I'll take a look at what needs to be updated in the disaster recovery section

marked the checklist item update disaster recover section of gitlab service docs with our findings as completed

marked the checklist item verify that we still have data for all elements in the gitlab omnibus dashboard on grafana as completed

closed

mentioned in commit wiki-replica@759023f9

migrate gitlab-02 to new gnt-dal cluster

Designs

Child items ...

Activity