Migrate tb-build-03 to new Hetzner Robot machine

The tb-build-03 machine has adequate build performance for monthly building and release signing, but where it lacks is in disk space. The applications team frequently has to ask other users to clear out old/unneeded data which ends up blocking builds/signing.

mentioned in issue #40736 (closed)

assigned to @lavamind

added Next label

tb-build-03 was deployed in an awkward manner on our new iSCSI SAN, so an opportunity to rebuild it would actually be welcome. I could take care of copying whatever data you want to carry over to the new machine, which would have up to 1.8TB of storage available according to the proposed iSCSI storage policy.

added lifecycle label

i was also thinking this would be an opportunity to create a new machine, because this one was kind of ... uh... experimental, in a sense.

@richard would you be okay with a new box? anything you'd need from the old?

A similar cheaper server (either the 8 or 12 core option) with 1 TB nvme storage would more than replace tb-build-03.

Expense has been approved by ED, so we'll spin up a new machine at Hetzner to replace tb-build-03.

i think there might be some confusion here. the new machine approved is a completely new machine, in #40736 (closed), not a rebuild of tb-build-03, right?

AFAICT approval given was concerning two new machines:

@isabela: tldr 300ish euro/month to replace tb-build-03 and spin up a new fast build server for Android to make devs happy(ier)

ack.

...

On 2022-05-03 13:59:42, Jérôme Charaoui (@lavamind) wrote:

AFAICT approval given was concerning two new machines:

@isabela: tldr 300ish euro/month to replace tb-build-03 and spin up a new fast build server for Android to make devs happy(ier)

-- Antoine Beaupré torproject.org system administration

added Doing label and removed Next label

The order has been placed at Hetzner:

  1 x Dedicated Root Server AX51-NVMe
     * Location: HEL1
     * For HEL1, support is only available in English.
     * Rescue system (English)
     * 1 x Primary IPv4
     * 2 x 1 TB NVMe SSD

changed title from Upgrade tb-build-03 storage capacity to Migrate tb-build-03 to new Hetzner Robot machine

added Next label and removed Doing label

added Doing label and removed Next label

@richard New machine tb-build-04 is ready. I can begin the process to move files the files in /home to the new server. Should I move all home directories, or only a subset? Are there other directories I should tranfer from tb-build-03 to tb-build-04? Thanks.

Another option of course is to not transfer any files... And also do you need a web server like on tb-build-03?

Please move all of the home directories except:

• aguestuser
• sysrqb

Also please setup the webserver as before (and set the webserver up on tb-build-05 as well if you haven't already).

I'll proceed to copy the directories over next Monday, June 6.

The transfer is in progress, it should take a couple of hours to complete as there's a significant amount of data to move around.

added Needs Review label and removed Doing label

added Next label and removed Needs Review label

added Doing label and removed Next label

mentioned in issue #40787 (closed)

@richard The transfer is completed, and both new boxes now have apache installed.

When should we schedule the retirement of tb-build-03?

@lavamind please wait until 11.0.14 is published and the blog post is merged. We should be good after that.

I understand this is now done, so I'll open a new ticket for the retirement of the old machine.

closed

mentioned in issue #40792 (closed)

marked this issue as related to #40792 (closed)