Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T17:01:36Zhttps://gitlab.torproject.org/legacy/trac/-/issues/33834nevii IP address change planned for Ganeti migration2020-06-13T17:01:36Zanarcatnevii IP address change planned for Ganeti migrationI'm migrating nevii, our primary DNS server, to the Ganeti cluster. this implies an IP address change, and therefore all sorts of shenanigans.
after inspection, the changes are fairly "minimal": glue records should not change as the pri...I'm migrating nevii, our primary DNS server, to the Ganeti cluster. this implies an IP address change, and therefore all sorts of shenanigans.
after inspection, the changes are fairly "minimal": glue records should not change as the primary DNS server is not publicly exposed. we will need to change all secondary servers, but most of those are in Puppet.
we did have to request extra address space from Hetzner, but this was done in ticket 2020032503025825.anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/33731troodi IP address change planned for Ganeti migration2020-06-13T17:01:25Zanarcattroodi IP address change planned for Ganeti migrationI will soon migrate this virtual machine to the new ganeti cluster. this will involve an IP address change which might affect the service.
Please let me know if there are any problems you can think of. in particular, do let me know if a...I will soon migrate this virtual machine to the new ganeti cluster. this will involve an IP address change which might affect the service.
Please let me know if there are any problems you can think of. in particular, do let me know if any internal (inside the server) or external (outside the server) services hardcodes the IP address of the virtual machine.anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/33730vineale IP address change planned for Ganeti migration2020-06-13T17:01:25Zanarcatvineale IP address change planned for Ganeti migrationI will soon migrate this virtual machine to the new ganeti cluster. this will involve an IP address change which might affect the service.
Please let me know if there are any problems you can think of. in particular, do let me know if a...I will soon migrate this virtual machine to the new ganeti cluster. this will involve an IP address change which might affect the service.
Please let me know if there are any problems you can think of. in particular, do let me know if any internal (inside the server) or external (outside the server) services hardcodes the IP address of the virtual machine.anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/33729forestii IP address change planned for Ganeti migration2020-06-16T01:12:06Zanarcatforestii IP address change planned for Ganeti migrationI will soon migrate this virtual machine to the new ganeti cluster. this will involve an IP address change which might affect the service.
Please let me know if there are any problems you can think of. in particular, do let me know if a...I will soon migrate this virtual machine to the new ganeti cluster. this will involve an IP address change which might affect the service.
Please let me know if there are any problems you can think of. in particular, do let me know if any internal (inside the server) or external (outside the server) services hardcodes the IP address of the virtual machine.anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/33587puppet certificate revocation anomaly2020-06-13T17:01:14Zanarcatpuppet certificate revocation anomalytoday i revoked cupani's cert by mistake:
```
anarcat@curie:tsa-misc(master)$ ./retire -v -H cupani.torproject.org retire-all -p unifolium.torproject.org
checking for ganeti master on node unifolium.torproject.org
omeiense.torproject....today i revoked cupani's cert by mistake:
```
anarcat@curie:tsa-misc(master)$ ./retire -v -H cupani.torproject.org retire-all -p unifolium.torproject.org
checking for ganeti master on node unifolium.torproject.org
omeiense.torproject.org
polyanthum.torproject.org
instance cupani.torproject.org not running, no shutdown required
undefining instance cupani.torproject.org on host unifolium.torproject.org
error: failed to get domain 'cupani.torproject.org'
error: Domain not found: no domain with matching name 'cupani.torproject.org'
instance cupani.torproject.org not found on unifolium.torproject.org assuming retired: error: failed to get domain 'cupani.torproject.org'
error: Domain not found: no domain with matching name 'cupani.torproject.org'
scheduling cupani.torproject.org disk deletion on host unifolium.torproject.org
checking for path "/srv/vmstore/cupani.torproject.org/" on unifolium.torproject.org
scheduling rm -rf "/srv/vmstore/cupani.torproject.org/" to run on unifolium.torproject.org in 7 days
warning: commands will be executed using /bin/sh
job 4 at Tue Mar 17 17:45:00 2020
scheduling cupani.torproject.org backup disks removal on host bungei.torproject.org
checking for path "/srv/backups/bacula/cupani.torproject.org/" on bungei.torproject.org
scheduling rm -rf "/srv/backups/bacula/cupani.torproject.org/" to run on bungei.torproject.org in 30 days
warning: commands will be executed using /bin/sh
job 22 at Thu Apr 9 17:45:00 2020
Notice: Revoked certificate with serial 30
Notice: Removing file Puppet::SSL::Certificate cupani.torproject.org at '/var/lib/puppet/ssl/ca/signed/cupani.torproject.org.pem'
cupani.torproject.org
Submitted 'deactivate node' for cupani.torproject.org with UUID 7b5e6d74-cb31-4929-9082-4a2bcda08b88
```
i was following the migration procedure as part of #33446 and got over enthusiastic about the process. the cert shouldn't have been revoked, of course, as the machine is still up.
but when i tried to see the effect of this, it seemed the certificate still worked! cupani can do puppet runs without problems, even though the on-disk certificate is gone:
```
root@pauli:~# ls -al /var/lib/puppet/ssl/ca/signed/cupani.torproject.org.pem
ls: cannot access '/var/lib/puppet/ssl/ca/signed/cupani.torproject.org.pem': No such file or directory
```
so it seems our certificate revocation routine:
```
con.run('puppet node clean %s' % instance)
con.run('puppet node deactivate %s' % instance)
```
... does not work.anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/33448Migrate IP address of polyanthum.torproject.org (BridgeDB)2020-06-13T17:01:05ZCecylia BocovichMigrate IP address of polyanthum.torproject.org (BridgeDB)The IP address of polyanthum (bridgedb) will be migrated soon (see #33085). We should make sure that nothing breaks in this process.
In particular, moat uses domain fronting to connect to bridgedb. Do we need to worry about this? Anyth...The IP address of polyanthum (bridgedb) will be migrated soon (see #33085). We should make sure that nothing breaks in this process.
In particular, moat uses domain fronting to connect to bridgedb. Do we need to worry about this? Anything else we need to worry about?anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/33447migrate omeiense to the ganeti cluster, triggering an IP change2020-06-13T17:01:04Zanarcatmigrate omeiense to the ganeti cluster, triggering an IP changei will soon migrate this virtual machine to the new ganeti cluster. this will involve an IP address change which might affect the service.
please let me know if there are any problems you can think of. in particular, do let me know if a...i will soon migrate this virtual machine to the new ganeti cluster. this will involve an IP address change which might affect the service.
please let me know if there are any problems you can think of. in particular, do let me know if any internal (inside the server) or external (outside the server) services hardcodes the IP address of the virtual machine.
thanks!anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/33446migrate cupani/git-rw to the ganeti cluster, triggering an IP address change2020-06-13T17:01:03Zanarcatmigrate cupani/git-rw to the ganeti cluster, triggering an IP address changei will soon migrate cupani AKA git-rw.torproject.org to the new ganeti cluster. this will involve an IP address change which might affect the service.
please let me know if there are any problems you can think of. in particular, do let ...i will soon migrate cupani AKA git-rw.torproject.org to the new ganeti cluster. this will involve an IP address change which might affect the service.
please let me know if there are any problems you can think of. in particular, do let me know if any internal (inside the server) or external (outside the server) services hardcodes the IP address of cupani.
thanks!anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/33085decomission unifolium/kvm2, 6 VMs to migrate2020-06-13T17:00:42Zanarcatdecomission unifolium/kvm2, 6 VMs to migrate * [x] cupani.torproject.org (git-rw) migrated in #33446
* [x] polyanthum.torproject.org (bridges) #33448
* [x] omeiense.torproject.org (onionoo.torproject.org) (possibly to decom? see #32268) #33447
* [x] savii.torproject.org (static... * [x] cupani.torproject.org (git-rw) migrated in #33446
* [x] polyanthum.torproject.org (bridges) #33448
* [x] omeiense.torproject.org (onionoo.torproject.org) (possibly to decom? see #32268) #33447
* [x] savii.torproject.org (static content backend) retired in #33441
* [x] build-x86-07.torproject.org (buildbox) retired in #33442)
* [x] bracteata.torproject.org (sandstorm) retired in #32390
Requires a new gnt node (#33081).anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/33083new gnt-fsn node (fsn-node-05)2020-06-13T17:00:38Zanarcatnew gnt-fsn node (fsn-node-05)anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/32391Purge test accounts and data from riseup in February 4, 20202020-06-13T16:59:36ZGabagaba@torproject.orgPurge test accounts and data from riseup in February 4, 2020 We are migrating into nc.torproject.net. We are planning to remove any test account from nc.riseup.net in February 4, 2020. This is the ticket for us not to forget :) We are migrating into nc.torproject.net. We are planning to remove any test account from nc.riseup.net in February 4, 2020. This is the ticket for us not to forget :)micahmicah@torproject.orgmicahmicah@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/32198upgrade CRM* machines to buster2020-06-13T17:00:37Zanarcatupgrade CRM* machines to busterthose machines are downtime-sensitive enough to warrant a tracking ticket to ensure proper coordination among all teams.
we originally wanted to do this before november, but time is running out, so this is being pushed out to january.
...those machines are downtime-sensitive enough to warrant a tracking ticket to ensure proper coordination among all teams.
we originally wanted to do this before november, but time is running out, so this is being pushed out to january.
in the meantime, we could consider migrating the machines to the FSN cluster to ensure filesystem-level snapshot to give us rollback capabilities. we should do this in a near-zero downtime migration, that said.HiroHiro