Rename chi-node-14 machine

When we moved out of Cymru (tpo/tpa/team#40929), we had physical machine chi-node-14 shipped to Quintex in Dallas and plugged it directly in the new network, without reinstalling the machine.

This ticket tracks the work to rename it to ci-runner-x86-03 to get rid of the legacy name and reflect its true usage.

added lifecycle label

assigned to @lavamind

Since we don't have a procedure for this, I'll be also documenting the steps here and eventually creating a wiki procedure.

Tentatively, all this needs to happens:

• remove the machine from puppet (from retirement procedure)
  • manifests (tor-puppet)
  • node clean + node deactivate
• disable and delete the backup configuration (from retirement procedure)
• change the hostname on the actual machine (unmanaged files)
  • /etc/hostname (via hostnamectl)
  • /etc/ssh/ssh_host_*_key.pub
  • /etc/hosts
• rename the machine in LDAP
• rename the machine in DNS
• rename the machine in mandos
• rename the machine in Nagios
• rename the machine on the Ganeti cluster N/A
• rename the machine in DNSWL N/A
• re-add it to Puppet
• rename the machine in tor-passwords
• rename the machine in docs (wiki)

@anarcat am I missing anything in the above?

the backups, at the very least.

if this would be a ganeti instance, there would be something to be done there as well, not sure if it's possible to rename an instance, but it should at least be mentioned at the top to avoid doing so if not, or checking...

we also have DNSWL in the new-machine/retire procedures.

Right, so fir this specific machine there's nothing to do for DNSWL nor Ganeti.

As for the backups, I already had it in the procedure above:

disable and delete the backup configuration (from retirement procedure)

Once the machine is readded to Puppet, the new backup configuration will be deployed.

oh oops, missed that, thanks

On chi-node-14:

# systemctl stop puppet.run.timer
# puppet agent --disable "node rename in progress"

On pauli (Puppet server):

# puppet node clean chi-node-14.torproject.org
# puppet node deactivate chi-node-14.torproject.org
# service restart apache2
# service restart puppetdb

On chi-node-14:

# hostnamectl set-hostname ci-runner-x86-03
# exec bash

In tsa-misc, with a slight modification:

$ ./retire -H chi-node-14.torproject.org remove-backups

I could have probably also used, for the puppet part:

$ retire -H chi-node-14.torproject.org revoke-puppet

These are also the only parts executed via retire-all but that one might get other, non rename-related bits added later on.

Renaming the SSH host key comments:

# sed -i 's/chi-node-14/ci-runner-x86-03/' /etc/ssh/ssh_host_*.pub

On mandos-01

# sed -i 's/chi-node-14/ci-runner-x86-03/' /etc/mandos/clients.conf

Re-adding the machine to Puppet:

# puppet agent --enable
~/tsa-misc# ./installer/puppet-bootstrap-client

do you still have the backlog of those commands?

it would be nice to still have the LDAP blob and the output of the retirement commands, in particular.

Some, yes.

The LDAP blob is identical, I just did a search/replace like this on the LDAP entry: s/chi-node-14/ci-runner-x86-03/. Nothing else was changed, including the SSH host keys.

The result was:

add: 0, rename: 1, modify: 1, delete: 0                                                                   
Action? [yYqQvVebB*rsf+?] y                                                                                                                                                                                          
Done.

The backups retirement command was:

$ ./retire -H chi-node-14.torproject.org remove-backups                                                                                              
starting tasks at 2023-09-20 15:48:54.620369+00:00                                                        
checking for path "/srv/backups/bacula/chi-node-14.torproject.org/" on bungei.torproject.org
scheduling rm -rf "/srv/backups/bacula/chi-node-14.torproject.org/" to run on bungei.torproject.org in 30 days
warning: commands will be executed using /bin/sh                                                          
job 110 at Fri Oct 20 15:48:00 2023              
checking for path "/srv/backups/pg/chi-node-14/" on bungei.torproject.org
path /srv/backups/pg/chi-node-14/ not found: [Errno 2] No such file                     
scheduling echo delete client=chi-node-14.torproject.org-fd yes | bconsole to run on bacula-director-01.torproject.org in 30 days
warning: commands will be executed using /bin/sh                                                          
job 61 at Fri Oct 20 15:49:00 2023                                                                        
completed tasks, elasped: 0:00:07.676209 (user 1.13 system 0.03 chlduser 0.0 chldsystem 0.0 RSS 52.2 MB)

The other commands, sed, hostnamectl, etc. didn't have any interesting output.

I'm going to upgrade chi-node-14 to bookworm, otherwise podman isn't going to work out for the runner.

ci-runner-x86-03 now upgraded to bookworm!

Last item remaining is to document this in the wiki.

marked this issue as related to #41295 (closed)

there are warnings from cron since this change, e.g.:

Date: Wed, 20 Sep 2023 22:45:01 +0000
From: root@ci-runner-x86-03.torproject.org (Cron Daemon)
To: root@ci-runner-x86-03.torproject.org
Subject: Cron <root@ci-runner-x86-03> docker image prune -f --filter until=24h --filter label=org.opencontainers.image.version=nightly > /dev/null

/bin/sh: 1: docker: not found

i suspect the cron jobs are not being cleaned up properly, sigh...

I cleaned them up manually from root's crontab.

added Doing label

mentioned in commit wiki-replica@04f62ddf

Created a draft rename-a-host procedure at https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/rename-a-host

I think we're all done here.

closed