Skip to content
Snippets Groups Projects

retire tpa-bootstrap-01

    • Closed Issue created by anarcat @anarcat

    now that dal-rescue-01 is online (#41135 (closed)), we can retire the tpa-bootstrap VM.

    • announcement
    • nagios
    • retire the host in fabric
    • remove from LDAP with ldapvi
    • power-grep
    • remove from tor-passwords
    • [ ] remove from DNSwl
    • remove from docs
    • remove from reverse DNS
    ✓ 8 of 8 checklist items completed · Edited

    Designs

      Child items 0

      2. No child items are currently assigned. Use child items to break down this issue into smaller parts.

      Activity

      • anarcat marked this issue as related to #41135 (closed)

        marked this issue as related to #41135 (closed)

      • anarcat changed milestone to %trusted high performance cluster (gnt-dal migration)

        changed milestone to %trusted high performance cluster (gnt-dal migration)

      • anarcat added Next lifecycle labels

        added Next lifecycle labels

      • anarcat assigned to @kez

        assigned to @kez

      • Kez added Doing label and removed Next label

        added Doing label and removed Next label

      • Kez
        Kez @kez ·

        removed form nagios in tor-nagios commit 44432fc

      • Kez
        Kez @kez ·

        retirement scheduled on dal-node-01 and bacula-director

         ./retire -v -H tpa-bootstrap-01.torproject.org retire-all --parent-host=dal-node-01.torproject.org
starting tasks at 2023-05-18 16:58:07.892270+00:00
checking for ganeti master on host dal-node-01.torproject.org
ganeti node detected with master dal-node-01.torproject.org
checking on dal-node-01.torproject.org if instance tpa-bootstrap-01.torproject.org is running
stopping instance tpa-bootstrap-01.torproject.org on dal-node-01.torproject.org
Waiting for job 41632 for tpa-bootstrap-01.torproject.org ...
scheduling tpa-bootstrap-01.torproject.org instance removal on host dal-node-01.torproject.org
scheduling gnt-instance remove --force tpa-bootstrap-01.torproject.org to run on dal-node-01.torproject.org in 7 days
warning: commands will be executed using /bin/sh
job 2 at Thu May 25 16:58:00 2023
scheduling tpa-bootstrap-01.torproject.org backup disks removal on host bungei.torproject.org and director bacula-director-01.torproject.org
checking for path "/srv/backups/bacula/tpa-bootstrap-01.torproject.org/" on bungei.torproject.org
scheduling rm -rf "/srv/backups/bacula/tpa-bootstrap-01.torproject.org/" to run on bungei.torproject.org in 30 days
warning: commands will be executed using /bin/sh
job 101 at Sat Jun 17 16:58:00 2023
checking for path "/srv/backups/pg/tpa-bootstrap-01/" on bungei.torproject.org
path /srv/backups/pg/tpa-bootstrap-01/ not found: [Errno 2] No such file
scheduling echo delete client=tpa-bootstrap-01.torproject.org-fd yes | bconsole to run on bacula-director-01.torproject.org in 30 days
warning: commands will be executed using /bin/sh
job 53 at Sat Jun 17 16:58:00 2023
Notice: Revoked certificate with serial 165
Notice: Removing file Puppet::SSL::Certificate tpa-bootstrap-01.torproject.org at '/var/lib/puppet/ssl/ca/signed/tpa-bootstrap-01.torproject.org.pem'
tpa-bootstrap-01.torproject.org
Submitted 'deactivate node' for tpa-bootstrap-01.torproject.org with UUID 38b3895c-de73-4b28-96c2-38dedf98e03a
completed tasks, elasped: 0:00:58.467702 (user 5.55 system 0.19 chlduser 0.0 chldsystem 0.0 RSS 54.6 MB)
        • Kez
          Kez @kez ·

          @anarcat a lot of the docs reference tpa-bootstrap, should i replace those with dal-rescue-01?

        • anarcat
          anarcat @anarcat ·
          Author Owner

          replace those with dal-rescue.torproject.org. in general, you can use a formulation like "the rescue host (currently dal-rescue)" or something to that effect.

          ...

          On 2023-05-18 17:10:11, kezzle wrote:

          @anarcat a lot of the docs reference tpa-bootstrap, should i replace those with dal-rescue-01?

        • Kez
          Kez @kez ·

          we still have the bootstrap puppet manifest (tor-puppet/modules/profile/manifests/bootstrap.pp). should that be removed entirely?

        • anarcat
          anarcat @anarcat ·
          Author Owner

          oh i don't think so!

          ...

          On 2023-05-18 17:52:47, kezzle wrote:

          we still have the bootstrap puppet manifest (tor-puppet/modules/profile/manifests/bootstrap.pp). should that be removed entirely?

        • Please register or sign in to reply
      • Kez
        Kez @kez ·

        removed from ldap:

        version: 1

dn: host=tpa-bootstrap-01,ou=hosts,dc=torproject,dc=org
changetype: delete
      • Kez
        Kez @kez · 
        grep -nH -r -e 204.8.99.136 -e 2620:7:6002::466:38ff:fe3d:791c -e tpa-bootstrap-01.torproject.org -e tpa-bootstrap-01 -e tpa-bootstrap.torproject.org -e tpa-bootstrap
dns/domains/0.0.0.0.2.0.0.6.7.0.0.0.0.2.6.2.ip6.arpa:61:; 2620:7:6002::466:38ff:fe3d:791c
dns/domains/0.0.0.0.2.0.0.6.7.0.0.0.0.2.6.2.ip6.arpa:62:c.1.9.7.d.3.e.f.f.f.8.3.6.6.4 IN PTR tpa-bootstrap-01.torproject.org
dns/domains/99.8.204.in-addr.arpa:153:136    IN PTR tpa-bootstrap-01.torproject.org.
dns/domains/torproject.org:228:tpa-bootstrap           IN      CNAME   tpa-bootstrap-01
grep: letsencrypt-domains/.git/index: binary file matches
letsencrypt-domains/domains:90:tpa-bootstrap.torproject.org
tor-nagios/.git/logs/refs/heads/master:25:10b36ef4b2afc970318a1449c5a1ff2efa2a39f1 fe1ca5364876a3f51ab1c1df6d92e19a09fb25a8 kez <kez@torproject.org> 1684428147 -0700	commit: Remove tpa-bootstrap-01 as part of retirement
tor-nagios/.git/logs/HEAD:31:10b36ef4b2afc970318a1449c5a1ff2efa2a39f1 fe1ca5364876a3f51ab1c1df6d92e19a09fb25a8 kez <kez@torproject.org> 1684428147 -0700	commit: Remove tpa-bootstrap-01 as part of retirement
tor-nagios/.git/logs/HEAD:33:7412bfe7cb0c66653dd2018e60111e71554c44ea 44432fcf80fe1c9cada6dca15e422c353a03c46c kez <kez@torproject.org> 1684428151 -0700	pull --rebase --autostash (pick): Remove tpa-bootstrap-01 as part of retirement
tor-nagios/.git/COMMIT_EDITMSG:1:Remove tpa-bootstrap-01 as part of retirement
grep: tor-puppet/.git/index: binary file matches
tor-puppet/modules/profile/manifests/bootstrap.pp:4:  String[1] $domain = 'tpa-bootstrap.torproject.org',
tor-puppet/modules/profile/manifests/bootstrap.pp:22:    content => epp('profile/tpa-bootstrap-vhost.epp', {
wiki/howto/ganeti.md:1959:    WARNING: Failed to run rename script for tpa-bootstrap-01.torproject.org on node dal-node-02.torproject.org: OS rename script failed (exited with exit code 1), last lines in the log file:\nCannot rename from tpa-bootstrap-01.torproject.org to tpa-bootstrap-01.torproject.org:\nInstance has a different hostname (tpa-bootstrap-01)
wiki/howto/quintex.md:178:override in `per-domain-config/tpa-bootstrap.torproject.org` with:
wiki/howto/quintex.md:186:<https://tpa-bootstrap.torproject.org/> site already.
wiki/howto/quintex.md:213:kernel https://tpa-bootstrap.torproject.org/vmlinuz
wiki/howto/quintex.md:214:initrd https://tpa-bootstrap.torproject.org/initrd.img
wiki/howto/quintex.md:215:initrd https://tpa-bootstrap.torproject.org/grml.iso /grml.iso
wiki/howto/quintex.md:361:chain https://tpa-bootstrap.torproject.org/grml.ipxe
wiki/policy/tpa-rfc-52-cymru-migration-timeline.md:29: - tpa-bootstrap-01
team.wiki/howto/ganeti.md:1976:    WARNING: Failed to run rename script for tpa-bootstrap-01.torproject.org on node dal-node-02.torproject.org: OS rename script failed (exited with exit code 1), last lines in the log file:\nCannot rename from tpa-bootstrap-01.torproject.org to tpa-bootstrap-01.torproject.org:\nInstance has a different hostname (tpa-bootstrap-01)
team.wiki/howto/quintex.md:165:override in `per-domain-config/tpa-bootstrap.torproject.org` with:
team.wiki/howto/quintex.md:173:<https://tpa-bootstrap.torproject.org/> site already.
team.wiki/howto/quintex.md:198:    kernel https://tpa-bootstrap.torproject.org/vmlinuz
team.wiki/howto/quintex.md:199:    initrd https://tpa-bootstrap.torproject.org/initrd.img
team.wiki/howto/quintex.md:200:    initrd https://tpa-bootstrap.torproject.org/grml.iso /grml.iso
team.wiki/howto/quintex.md:341:chain https://tpa-bootstrap.torproject.org/grml.ipxe
team.wiki/policy/tpa-rfc-52-cymru-migration-timeline.md:29: - tpa-bootstrap-01
        find -iname '*tpa-bootstrap*'
./letsencrypt-domains/per-domain-config/tpa-bootstrap.torproject.org
./tor-puppet/hiera-enc/nodes/tpa-bootstrap-01.torproject.org.yaml
./tor-puppet/modules/profile/templates/tpa-bootstrap-vhost.epp
        Edited by Kez
      • Kez
        Kez @kez ·

        removed from dns

        commit 6d4a2fdddef7ee74f0722da9a5c724d4fdb1082d (HEAD -> master)
Author: kez <kez@torproject.org>
Date:   Thu May 18 10:23:20 2023 -0700

    Remove tpa-bootstrap-01 from DNS
    
    Context: tpo/tpa/team#41174

diff --git a/0.0.0.0.2.0.0.6.7.0.0.0.0.2.6.2.ip6.arpa b/0.0.0.0.2.0.0.6.7.0.0.0.0.2.6.2.ip6.arpa
index c5d9e7e..7767152 100644
--- a/0.0.0.0.2.0.0.6.7.0.0.0.0.2.6.2.ip6.arpa
+++ b/0.0.0.0.2.0.0.6.7.0.0.0.0.2.6.2.ip6.arpa
@@ -58,8 +58,6 @@ a.b.a.b.b.a.e.f.f.f.8.3.6.6.4 IN PTR survey-01.torproject.org
 2.e.a.5.a.5.e.f.f.f.8.3.6.6.4 IN PTR rdsys-frontend-01.torproject.org
 ; 2620:7:6002::466:38ff:fec0:7bcb
 b.c.b.7.0.c.e.f.f.f.8.3.6.6.4 IN PTR ci-runner-x86-01.torproject.org
-; 2620:7:6002::466:38ff:fe3d:791c
-c.1.9.7.d.3.e.f.f.f.8.3.6.6.4 IN PTR tpa-bootstrap-01.torproject.org
 ; 2620:7:6002::466:39ff:fe7f:1826
 6.2.8.1.f.7.e.f.f.f.9.3.6.6.4.0 IN PTR web-dal-07.torproject.org.
 ; 2620:7:6002::466:39ff:fe32:e3dd
diff --git a/99.8.204.in-addr.arpa b/99.8.204.in-addr.arpa
index 03b55c0..4c59938 100644
--- a/99.8.204.in-addr.arpa
+++ b/99.8.204.in-addr.arpa
@@ -150,7 +150,7 @@
 133    IN PTR onionbalance-02.torproject.org.
 134    IN PTR rdsys-frontend-01.torproject.org.
 135    IN PTR metrics-psqlts-01.torproject.org.
-136    IN PTR tpa-bootstrap-01.torproject.org.
+;136    IN PTR .torproject.org.
 137    IN PTR telegram-bot-01.torproject.org.
 138    IN PTR btcpayserver-02.torproject.org.
 139    IN PTR gitlab-dev-01.torproject.org.
diff --git a/torproject.org b/torproject.org
index c314a38..88a4383 100644
--- a/torproject.org
+++ b/torproject.org
@@ -225,7 +225,6 @@ onionperf           IN      CNAME   static
 ; www A/AAAA records via services-auto
 ; for internal monitoring/testing only.  Don't publish.
 www-stellatum          IN      CNAME   stellatum
-tpa-bootstrap           IN      CNAME   tpa-bootstrap-01
 
 ; TBB testing - #13300
 test-reports.tbb       IN      A       93.95.228.164
      • Kez
        Kez @kez ·

        replaced wiki references to tpa-bootstrap with dal-rescue

      • Kez
        Kez @kez ·

        here's what's left from the grep and find commands:

        grep --exclude-dir=.git -nH -r -e 204.8.99.136 -e 2620:7:6002::466:38ff:fe3d:791c -e tpa-bootstrap-01.torproject.org -e tpa-bootstrap-01 -e tpa-bootstrap.torproject.org -e tpa-bootstrap
tor-puppet/modules/profile/manifests/bootstrap.pp:4:  String[1] $domain = 'tpa-bootstrap.torproject.org',
tor-puppet/modules/profile/manifests/bootstrap.pp:22:    content => epp('profile/tpa-bootstrap-vhost.epp', {
wiki/policy/tpa-rfc-52-cymru-migration-timeline.md:29: - tpa-bootstrap-01
        find -iname '*tpa-bootstrap*'
./tor-puppet/modules/profile/templates/tpa-bootstrap-vhost.epp

        we're leaving the boostrap module in puppet, so the power-grep step is finished

        • Kez
          Kez @kez ·

          actually, i haven't removed this from hiera yet. trying to push the commit results in

          Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 12 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 380 bytes | 380.00 KiB/s, done.
Total 4 (delta 3), reused 0 (delta 0), pack-reused 0
error: remote unpack failed: unable to create temporary object directory
To puppet.torproject.org:/srv/puppet.torproject.org/git/tor-puppet
 ! [remote rejected]   master -> master (unpacker error)
error: failed to push some refs to 'puppet.torproject.org:/srv/puppet.torproject.org/git/tor-puppet'

          so i guess i'll go find out how to fix that

        • anarcat
          anarcat @anarcat ·
          Author Owner

          are you sure you have the right URL there?

          here it is:

          git@puppet.torproject.org:/srv/puppet.torproject.org/git/tor-puppet
          ...

          On 2023-05-18 18:57:16, kezzle wrote:

          To puppet.torproject.org:/srv/puppet.torproject.org/git/tor-puppet
        • Kez
          Kez @kez ·

          ah, i was pushing as the wrong user!

        • Please register or sign in to reply
        • Kez
          Kez @kez ·

          trying to remove the host fomr tor passwords, i see Warning: the following recipients are invalid: 8C4CD511095E982EB0EFBFA21E8BF34923291265. Try again (or proceed)? [Y/n]. Trying again gives the same error, proceeding causes Error: hosts encrypted to an empty file. Edit again (or exit)? [Y/n]. The recipient in question is linus, who is listed in the tor-passwords .users file and is part of the @admins group in that file.

          the last commit that changed the .users file removed a lot of keys from the .keyring file, including linus' key. should linus also be removed from the .users file then? @anarcat

        • anarcat
          anarcat @anarcat ·
          Author Owner

          i don't think so. i don't remember removing linus from the file explicitly and i don't see a justification in the commit logs... maybe it was a mistake?

          i have a valid key for 8C4CD511095E982EB0EFBFA21E8BF34923291265 but maybe it's invalid in the keyring?

          ...

          On 2023-05-18 19:34:30, kezzle wrote:

          kezzle commented:

          trying to remove the host fomr tor passwords, i see Warning: the following recipients are invalid: 8C4CD511095E982EB0EFBFA21E8BF34923291265. Try again (or proceed)? [Y/n]. Trying again gives the same error, proceeding causes Error: hosts encrypted to an empty file. Edit again (or exit)? [Y/n]. The recipient in question is linus, who is listed in the tor-passwords .users file and is part of the @admins group in that file.

          the last commit that changed the .users file removed a lot of keys from the .keyring file, including linus' key. should linus also be removed from the .users file then? @anarcat

        • Kez
          Kez @kez ·

          i also have a key for 8C4CD511095E982EB0EFBFA21E8BF34923291265, but it looks like the keys expired a few years ago

          gpg -k 8C4CD511095E982EB0EFBFA21E8BF34923291265
pub   rsa4096 2010-05-07 [SC]
      8C4CD511095E982EB0EFBFA21E8BF34923291265
uid           [ unknown] Linus Nordberg <linus@nordberg.se>
uid           [ unknown] Linus Nordberg <linus@torproject.org>
uid           [ unknown] [jpeg image of size 2906]
uid           [ unknown] Linus Nordberg <linus@dfri.se>
uid           [ unknown] Linus Nordberg <linus@sunet.se>
uid           [ unknown] Linus Nordberg <linus@nordu.net>
sub   rsa4096 2018-02-17 [A]
        • anarcat
          anarcat @anarcat ·
          Author Owner

          that key is not rexpired, that's the creation date. That said, if you show that key in "verbose" mode, you'll see that linus's encryption key indeed did expire:

          anarcat@angela:tor-passwords$ gpg -v --list-keys 8C4CD511095E982EB0EF
BFA21E8BF34923291265
pub   rsa4096/1E8BF34923291265 2010-05-07 [SC]
      8C4CD511095E982EB0EFBFA21E8BF34923291265
uid                 [  full  ] Linus Nordberg <linus@nordberg.se>
uid                 [  full  ] Linus Nordberg <linus@dfri.se>
uid                 [marginal] Linus Nordberg <linus@sunet.se>
uid                 [  full  ] Linus Nordberg <linus@nordu.net>
uid                 [  full  ] Linus Nordberg <linus@torproject.org>
uid                 [  never ] [jpeg image of size 2906]
sub   rsa4096/95FA5E5B5681169D 2010-05-07 [E] [revoked: 2011-04-25]
sub   rsa4096/066FB4CAD564A6A9 2011-04-25 [E] [revoked: 2012-04-25]
sub   rsa4096/DEF09E0491BE3F46 2012-04-23 [E] [revoked: 2013-04-30]
sub   rsa4096/8318304C153E576C 2013-04-23 [E] [revoked: 2014-04-23]
sub   rsa4096/76BD0D3E2293E1E1 2014-04-23 [E] [revoked: 2016-04-14]
sub   rsa4096/39D965713B249393 2015-04-20 [E] [revoked: 2017-04-17]
sub   rsa4096/9CFD365BB5F7D1B1 2016-04-14 [E] [revoked: 2017-04-17]
sub   rsa4096/46B310F8DA945B54 2017-04-17 [E] [revoked: 2018-04-11]
sub   rsa4096/C1788C2B0B3120D3 2018-02-17 [A]
sub   rsa4096/9DC3E9C66DDABD48 2018-04-11 [E] [revoked: 2019-05-30]
sub   rsa4096/BFBF3660110EB862 2019-05-30 [E] [revoked: 2020-06-03]
sub   rsa4096/2D11AC8BFF2BC4A8 2020-06-03 [E] [expired: 2023-05-17]

          So yeah, either kick him out of the users file, or haggle @linus to update his darn key. :) I couldn't find an updated version on keys.openpgp.org myself.

          ...

          On 2023-05-22 16:38:34, kezzle (@kez) wrote:

          kezzle commented on a discussion: #41174 (comment 2904904)

          i also have a key for 8C4CD511095E982EB0EFBFA21E8BF34923291265, but it looks like the keys expired a few years ago

          gpg -k 8C4CD511095E982EB0EFBFA21E8BF34923291265
pub   rsa4096 2010-05-07 [SC]
      8C4CD511095E982EB0EFBFA21E8BF34923291265
uid           [ unknown] Linus Nordberg <linus@nordberg.se>
uid           [ unknown] Linus Nordberg <linus@torproject.org>
uid           [ unknown] [jpeg image of size 2906]
uid           [ unknown] Linus Nordberg <linus@dfri.se>
uid           [ unknown] Linus Nordberg <linus@sunet.se>
uid           [ unknown] Linus Nordberg <linus@nordu.net>
sub   rsa4096 2018-02-17 [A]
        • anarcat
          anarcat @anarcat ·
          Author Owner

          i pinged him on irc at least.

        • anarcat
          anarcat @anarcat ·
          Author Owner

          i removed linus' key from the .users file as i had to edit the pw manager in an unrelated ticket (#41161 (closed))

        • Kez
          Kez @kez ·

          every day i learn a new way i've been using gpg wrong :smile:

          thanks for taking care of the problematic key, i've removed the password from the repository

        • Please register or sign in to reply
      • Kez changed the description

        changed the description

      • Kez marked the checklist item remove from reverse DNS as completed

        marked the checklist item remove from reverse DNS as completed

      • Kez
        Kez @kez ·

        i don't believe this machine was handling its own mail and i can't find it in dnswl.org, so the checklist is complete, and the machine is retired

      • Kez closed

        closed

      • Kez changed the description

        changed the description

      • anarcat
        anarcat @anarcat ·
        Author Owner

        i properly destroyed the certificates here and will document how to do this next.

      • anarcat mentioned in commit wiki-replica@bbe2efe9

        mentioned in commit wiki-replica@bbe2efe9

      Please register or sign in to reply