Skip to content
Snippets Groups Projects
New look: Off

retire legacy onionoo infrastructure

    • Closed Issue created by anarcat @anarcat

    we rebuilt the onionoo service in a single machine in #41512 (closed), retire the old cluster of 4 machines.

    checklist:

    1. announcement
    2. retire the host in fabric
    3. remove from LDAP with ldapvi
    4. power-grep
    5. remove from tor-passwords
    6. remove from DNSwl
    7. remove from docs
    8. remove from racks (not a physical machine)
    9. remove from reverse DNS

    onionoo-frontend-01, -02, onionoo-backend-01, and -02, and NOT onionoo-backend-03, which remains.

    ✓ 8 of 8 checklist items completed · Edited

    Designs

    Child items ...

    • Activity

    • anarcat marked this issue as related to #41512 (closed)

      marked this issue as related to #41512 (closed)

    • anarcat added Metrics Next Technical Debt lifecycle labels

      added Metrics Next Technical Debt lifecycle labels

    • anarcat assigned to @anarcat

      assigned to @anarcat

    • anarcat mentioned in issue #41512 (closed)

      mentioned in issue #41512 (closed)

    • anarcat changed due date to October 29, 2024

      changed due date to October 29, 2024

    • anarcat assigned to @zen and unassigned @anarcat

      assigned to @zen and unassigned @anarcat

    • anarcat changed the description

      changed the description

    • zen changed due date to October 31, 2024

      changed due date to October 31, 2024

    • zen added Doing label and removed Next label

      added Doing label and removed Next label

    • zen changed due date to November 07, 2024

      changed due date to November 07, 2024

    • zen
      zen @zen ·
      Owner

      announcement

      No need for a specific announcement: this is a follow-up of an already-done issue (#41512 (closed)) and we checked with @hiro on IRC that it's fine to proceed.

    • zen marked the checklist item announcement as completed

      marked the checklist item announcement as completed

    • zen
      zen @zen ·
      Owner

      retire the host in fabric

      onionoo-frontend-01.torproject.org 
      starting tasks at 2024-11-07 19:23:58.058028+00:00
checking on fsn-node-01.torproject.org if instance onionoo-frontend-01.torproject.org is running
stopping instance onionoo-frontend-01.torproject.org on fsn-node-01.torproject.org
Waiting for job 1015576 for onionoo-frontend-01.torproject.org ...
scheduling onionoo-frontend-01.torproject.org instance removal on host fsn-node-01.torproject.org
scheduling gnt-instance remove --force onionoo-frontend-01.torproject.org to run on fsn-node-01.torproject.org in 7
syntax error. Last token seen: 7
Garbled time
scheduling onionoo-frontend-01.torproject.org backup disks removal on host bungei.torproject.org and director bacula-director-01.torproject.org
checking for path "/srv/backups/bacula/onionoo-frontend-01.torproject.org/" on bungei.torproject.org
scheduling rm -rf "/srv/backups/bacula/onionoo-frontend-01.torproject.org/" to run on bungei.torproject.org in 30 days
warning: commands will be executed using /bin/sh
job 130 at Sat Dec  7 19:24:00 2024
checking for path "/srv/backups/pg/onionoo-frontend-01/" on bungei.torproject.org
path /srv/backups/pg/onionoo-frontend-01/ not found: [Errno 2] No such file
scheduling echo delete client=onionoo-frontend-01.torproject.org-fd yes | bconsole to run on bacula-director-01.torproject.org in 30 days
warning: commands will be executed using /bin/sh
job 71 at Sat Dec  7 19:24:00 2024
Notice: Revoked certificate with serial 90
Notice: Removing file Puppet::SSL::Certificate onionoo-frontend-01.torproject.org at '/var/lib/puppet/ssl/ca/signed/onionoo-frontend-01.torproject.org.pem'
onionoo-frontend-01.torproject.org
Submitted 'deactivate node' for onionoo-frontend-01.torproject.org with UUID 153d551f-df7b-4904-99ee-70f58cc6a761
completed tasks, elasped: 0:00:44.453662 (user 4.06 system 0.16 chlduser 0.0 chldsystem 0.0 RSS 67.8 MB)
      onionoo-frontend-02.torproject.org 
      starting tasks at 2024-11-07 20:03:26.995846+00:00
checking on fsn-node-01.torproject.org if instance onionoo-frontend-02.torproject.org is running
stopping instance onionoo-frontend-02.torproject.org on fsn-node-01.torproject.org
Waiting for job 1015585 for onionoo-frontend-02.torproject.org ...
scheduling onionoo-frontend-02.torproject.org instance removal on host fsn-node-01.torproject.org
scheduling gnt-instance remove --force onionoo-frontend-02.torproject.org to run on fsn-node-01.torproject.org in 7 days
warning: commands will be executed using /bin/sh
job 36 at Thu Nov 14 20:03:00 2024
scheduling onionoo-frontend-02.torproject.org backup disks removal on host bungei.torproject.org and director bacula-director-01.torproject.org
checking for path "/srv/backups/bacula/onionoo-frontend-02.torproject.org/" on bungei.torproject.org
scheduling rm -rf "/srv/backups/bacula/onionoo-frontend-02.torproject.org/" to run on bungei.torproject.org in 30 days days
syntax error. Last token seen: days
Garbled time
checking for path "/srv/backups/pg/onionoo-frontend-02/" on bungei.torproject.org
path /srv/backups/pg/onionoo-frontend-02/ not found: [Errno 2] No such file
scheduling echo delete client=onionoo-frontend-02.torproject.org-fd yes | bconsole to run on bacula-director-01.torproject.org in 30 days days
syntax error. Last token seen: days
Garbled time
Notice: Revoked certificate with serial 110
Notice: Removing file Puppet::SSL::Certificate onionoo-frontend-02.torproject.org at '/var/lib/puppet/ssl/ca/signed/onionoo-frontend-02.torproject.org.pem'
onionoo-frontend-02.torproject.org
Submitted 'deactivate node' for onionoo-frontend-02.torproject.org with UUID fe4ab4a7-66da-4686-bcc1-0debd13a8d18
completed tasks, elasped: 0:00:41.268267 (user 3.94 system 0.18 chlduser 0.0 chldsystem 0.0 RSS 67.9 MB)
      onionoo-backend-01.torproject.org 
      starting tasks at 2024-11-07 20:10:30.916818+00:00
checking on fsn-node-01.torproject.org if instance onionoo-backend-01.torproject.org is running
stopping instance onionoo-backend-01.torproject.org on fsn-node-01.torproject.org
Waiting for job 1015588 for onionoo-backend-01.torproject.org ...
scheduling onionoo-backend-01.torproject.org instance removal on host fsn-node-01.torproject.org
scheduling gnt-instance remove --force onionoo-backend-01.torproject.org to run on fsn-node-01.torproject.org in 7 days
warning: commands will be executed using /bin/sh
job 39 at Thu Nov 14 20:10:00 2024
scheduling onionoo-backend-01.torproject.org backup disks removal on host bungei.torproject.org and director bacula-director-01.torproject.org
checking for path "/srv/backups/bacula/onionoo-backend-01.torproject.org/" on bungei.torproject.org
scheduling rm -rf "/srv/backups/bacula/onionoo-backend-01.torproject.org/" to run on bungei.torproject.org in 30 days
warning: commands will be executed using /bin/sh
job 131 at Sat Dec  7 20:11:00 2024
checking for path "/srv/backups/pg/onionoo-backend-01/" on bungei.torproject.org
path /srv/backups/pg/onionoo-backend-01/ not found: [Errno 2] No such file
scheduling echo delete client=onionoo-backend-01.torproject.org-fd yes | bconsole to run on bacula-director-01.torproject.org in 30 days
warning: commands will be executed using /bin/sh
job 72 at Sat Dec  7 20:11:00 2024
Notice: Revoked certificate with serial 83
Notice: Removing file Puppet::SSL::Certificate onionoo-backend-01.torproject.org at '/var/lib/puppet/ssl/ca/signed/onionoo-backend-01.torproject.org.pem'
onionoo-backend-01.torproject.org
Submitted 'deactivate node' for onionoo-backend-01.torproject.org with UUID 73995993-6acf-4948-bf73-2404936195a1
completed tasks, elasped: 0:00:50.830213 (user 4.19 system 0.21 chlduser 0.0 chldsystem 0.0 RSS 70.1 MB)
      onionoo-backend-02.torproject.org 
      starting tasks at 2024-11-07 20:48:51.520137+00:00
checking on fsn-node-01.torproject.org if instance onionoo-backend-02.torproject.org is running
stopping instance onionoo-backend-02.torproject.org on fsn-node-01.torproject.org
Waiting for job 1015597 for onionoo-backend-02.torproject.org ...
scheduling onionoo-backend-02.torproject.org instance removal on host fsn-node-01.torproject.org
scheduling gnt-instance remove --force onionoo-backend-02.torproject.org to run on fsn-node-01.torproject.org in 7 days
warning: commands will be executed using /bin/sh
job 41 at Thu Nov 14 20:49:00 2024
scheduling onionoo-backend-02.torproject.org backup disks removal on host bungei.torproject.org and director bacula-director-01.torproject.org
checking for path "/srv/backups/bacula/onionoo-backend-02.torproject.org/" on bungei.torproject.org
scheduling rm -rf "/srv/backups/bacula/onionoo-backend-02.torproject.org/" to run on bungei.torproject.org in 30 days
warning: commands will be executed using /bin/sh
job 134 at Sat Dec  7 20:49:00 2024
checking for path "/srv/backups/pg/onionoo-backend-02/" on bungei.torproject.org
path /srv/backups/pg/onionoo-backend-02/ not found: [Errno 2] No such file
scheduling echo delete client=onionoo-backend-02.torproject.org-fd yes | bconsole to run on bacula-director-01.torproject.org in 30 days
warning: commands will be executed using /bin/sh
job 74 at Sat Dec  7 20:49:00 2024
Notice: Revoked certificate with serial 97
Notice: Removing file Puppet::SSL::Certificate onionoo-backend-02.torproject.org at '/var/lib/puppet/ssl/ca/signed/onionoo-backend-02.torproject.org.pem'
onionoo-backend-02.torproject.org
Submitted 'deactivate node' for onionoo-backend-02.torproject.org with UUID 700f666c-8c33-443f-ab84-660dafa22b7f
completed tasks, elasped: 0:00:50.267270 (user 4.02 system 0.18 chlduser 0.0 chldsystem 0.0 RSS 70.0 MB)

      Notes:

      • During the run for onionoo-frontend-01, the scheduling gnt-instance remove task failed because of fabric-tasks@bc1021f4. So I tried to run it manually in the ganeti node, but unfortunately made a mistake in the command line and prematurely removed it.
      • During the run for onionoo-frontend-02, while I was trying to fix the scheduling issue above, the scheduling of removal of backup files and backup client from bacula failed, so I ran them manually (with oversight).
      • The other two nodes worked well.
    • zen marked the checklist item retire the host in fabric as completed

      marked the checklist item retire the host in fabric as completed

    • zen marked the checklist item remove from LDAP with ldapvi as completed

      marked the checklist item remove from LDAP with ldapvi as completed

    • anarcat mentioned in commit fabric-tasks@47fe6cc5

      mentioned in commit fabric-tasks@47fe6cc5

    • anarcat mentioned in commit wiki-replica@2a9d68ce

      mentioned in commit wiki-replica@2a9d68ce

    • anarcat
      anarcat @anarcat ·
      Author Owner

      we had an alert about those retirements in prometheus. it's not your fault: i think we overlooked the new monitoring system in our retirement procedures, as the puppet node retirement doesn't cleanup the reports generated by the current exporter. i've documented this in the pager playbook and, for now, patched the retirement procedure to add a silence instead.

      @lelutin i wonder if the retirement procedure should instead cleanup the old reports, so that we don't have an alert at all... do you know how we could do that? would also welcome your review of the docs i did on this in wiki-replica@2a9d68ce

      • zen
        zen @zen ·
        Owner
        Resolved by zen

        power-grep

        I see the following mentions to onionoo in our repos which I don't think need change (rationale: changing would be either removing historic info or updating command examples, which would be a difficult practice to maintain):

        • status-site: historic info
        • tor-nagios: deprecated repo
        • wiki: historic info and command output examples
        • fabric-tasks: command example
        • tor-puppet: command and output examples

        So at this point I don't see anything that needs cleanup. Please let me know if you find something that I overlooked.

    • zen marked the checklist item power-grep as completed

      marked the checklist item power-grep as completed

    • zen marked the checklist item remove from tor-passwords as completed

      marked the checklist item remove from tor-passwords as completed

    • zen marked the checklist item remove from DNSwl as completed

      marked the checklist item remove from DNSwl as completed

    • zen marked the checklist item remove from docs as completed

      marked the checklist item remove from docs as completed

    • zen changed the description

      changed the description

    • zen marked the checklist item remove from reverse DNS as completed

      marked the checklist item remove from reverse DNS as completed

    • zen
      zen @zen ·
      Owner
      • remove from DNSwl

      No matching entries, nothing to do there.

      • remove from docs

      Nothing relevant to be updated there.

      • remove from reverse DNS

      Removed all onionoo-* entries from Hetzner Robot.

    • zen
      zen @zen ·
      Owner

      Finished the retirement of legacy oninoo infrastructure.

    • zen closed

      closed

    • anarcat marked this issue as related to #41875 (closed)

      marked this issue as related to #41875 (closed)

    • anarcat mentioned in issue #41875 (closed)

      mentioned in issue #41875 (closed)

    Please register or sign in to reply