Skip to content
Snippets Groups Projects

decomission kvm3 AKA macrum, 7 VMs to migrate

    • Closed (moved) Issue created by anarcat @anarcat
    • crm-ext-01.torproject.org (part of #32198 (moved)) done at 116.202.120.186
    • crm-int-01.torproject.org (also part of #32198 (moved)) done at 116.202.120.190
    • forrestii.torproject.org (fpcentral, #33729 (moved)) server migrated to 116.202.120.185
    • nevii.torproject.org (DNS master) - #33834 (moved)
    • rude.torproject.org (RT) - migrated to 116.202.120.187
    • troodi.torproject.org (Trac, #33731 (moved)) - migrated to 116.202.120.188
    • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

    The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

    Will require a new Ganeti node (#33083 (moved)).

    ✓ 7 of 7 checklist items completed
    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    • Activity

    • Trac
      Trac @tracbot ·

      Trac:
      Child Ticket(s): #33729 (moved), #33730 (moved), #33731 (moved), #33834 (moved)

    • anarcat added component::internal services/tor sysadmin team owner::hiro priority::medium resolution::fixed severity::normal status::closed tpa-roadmap-april type::task labels

      added component::internal services/tor sysadmin team owner::hiro priority::medium resolution::fixed severity::normal status::closed tpa-roadmap-april type::task labels

    • anarcat
      anarcat @anarcat ·
      Author Owner

      Trac:
      Parent: N/A to N/A
      Actualpoints: N/A to N/A
      Points: N/A to N/A
      Sponsor: N/A to N/A
      Reviewer: N/A to N/A
      Description: * [ ] crm-ext-01.torproject.org

      • crm-int-01.torproject.org
      • forrestii.torproject.org (fpcentral)
      • nevii.torproject.org (DNS master)
      • rude.torproject.org (RT)
      • troodi.torproject.org (Trac)
      • vineale.torproject.org (gitweb)

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      to

      • crm-ext-01.torproject.org
      • crm-int-01.torproject.org
      • forrestii.torproject.org (fpcentral)
      • nevii.torproject.org (DNS master)
      • rude.torproject.org (RT)
      • troodi.torproject.org (Trac)
      • vineale.torproject.org (gitweb)

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

    • anarcat
      anarcat @anarcat ·
      Author Owner

      this is on! i'll start this next week, since fsn-node-05 is online (#33083 (moved))!

      Trac:
      Status: new to accepted

    • anarcat
      anarcat @anarcat ·
      Author Owner

      first sync performed with the automated tools, resync takes about 35 minutes.

      next step is to perform a sync with suspend, adopt and renumber the VMs for testing.

      Trac:
      Summary: decomission kvm3, 7 VMs to migrate to decomission kvm3 AKA macrum, 7 VMs to migrate

    • anarcat
      anarcat @anarcat ·
      Author Owner

      forgot to mention, this is the magic command that was ran and that takes 35 minutes on a good day:

      ./ganeti --verbose -H forrestii.torproject.org,nevii.torproject.org,rude.torproject.org,troodi.torproject.org,vineale.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org --verbose libvirt-import --ganeti-node=fsn-node-05.torproject.org --libvirt-host=macrum.torproject.org
    • anarcat
      anarcat @anarcat ·
      Author Owner

      created sub tickets for the notifications, remaining services are internal enough to not warrant coordination.

      Trac:
      Description: * [ ] crm-ext-01.torproject.org

      • crm-int-01.torproject.org
      • forrestii.torproject.org (fpcentral)
      • nevii.torproject.org (DNS master)
      • rude.torproject.org (RT)
      • troodi.torproject.org (Trac)
      • vineale.torproject.org (gitweb)

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

      to

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

    • anarcat
      anarcat @anarcat ·
      Author Owner

      i did an adoption of all the boxes, but it failed at crm-int-01 because we ran out of IPs. i removed nevii and reimported crm-int-01 because the latter seemed more important to migrate first.

      i tried to order a new ip block from hetzner, but it seems we can only order IPv6. i opened a ticket to clarify the next steps.

    • anarcat
      anarcat @anarcat ·
      Author Owner

      all machines renumbered (renumber-instances) internally and available for testing, except the CRM boxes which are on a different timeline.

      documented the IP addresses in the summary.

      Trac:
      Description: * [ ] crm-ext-01.torproject.org (part of #32198 (moved))

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

      to

      • crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186
      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) test server ready at 116.202.120.185
      • nevii.torproject.org (DNS master) - not migrated, ran out of IP, opened Ticket#2020032503025825 with hetzner to followup
      • rude.torproject.org (RT) - test server ready at 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - test server ready at 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - test server ready at 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

    • anarcat
      anarcat @anarcat ·
      Author Owner

      vineale done.

      Trac:
      Description: * [ ] crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186

      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) test server ready at 116.202.120.185
      • nevii.torproject.org (DNS master) - not migrated, ran out of IP, opened Ticket#2020032503025825 with hetzner to followup
      • rude.torproject.org (RT) - test server ready at 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - test server ready at 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - test server ready at 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

      to

      • crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186
      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) test server ready at 116.202.120.185
      • nevii.torproject.org (DNS master) - not migrated, ran out of IP, opened Ticket#2020032503025825 with hetzner to followup
      • rude.torproject.org (RT) - test server ready at 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - test server ready at 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

    • anarcat
      anarcat @anarcat ·
      Author Owner

      rude migration in progress.

      TTLs lowered at 22:51UTC.

      renumbering:

      --- /mnt/etc/network/interfaces.bak	2016-09-01 18:17:33.302001995 +0000
+++ /mnt/etc/network/interfaces	2020-03-31 23:59:06.952570913 +0000
@@ -1,14 +1,16 @@
 # This file describes the network interfaces available on your system
 # and how to activate them. For more information, see interfaces(5).
 
+# The loopback network interface
 auto lo
 iface lo inet loopback
 
-allow-hotplug eth0
+# The primary network interface
+auto eth0
 iface eth0 inet static
-    address 138.201.212.230/28
-    gateway 138.201.212.225
+    address 116.202.120.187/27
+    gateway 116.202.120.161
 iface eth0 inet6 static
     accept_ra 0
-    address 2a01:4f8:172:39ca:0:dad3:6:1/96
-    gateway 2a01:4f8:172:39ca:0:dad3:0:1
+    address 2a01:4f8:fff0:4f:266:37ff:fee0:8604/64
+    gateway 2a01:4f8:fff0:4f::1

      LDAP, nagios, and /etc on rude IP changes done.

      it should be done.

      Trac:
      Description: * [ ] crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186

      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) test server ready at 116.202.120.185
      • nevii.torproject.org (DNS master) - not migrated, ran out of IP, opened Ticket#2020032503025825 with hetzner to followup
      • rude.torproject.org (RT) - test server ready at 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - test server ready at 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

      to

      • crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186
      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) test server ready at 116.202.120.185
      • nevii.torproject.org (DNS master) - not migrated, ran out of IP, opened Ticket#2020032503025825 with hetzner to followup
      • rude.torproject.org (RT) - migrated to 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - test server ready at 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

    • anarcat
      anarcat @anarcat ·
      Author Owner

      a new netblock was allocated by hetzner, and was configured in Ganeti with:

      root@fsn-node-01:~# gnt-network add --network 49.12.57.128/27 --gateway 49.12.57.129 gnt-fsn13-02
root@fsn-node-01:~# gnt-network connect --nic-parameters=link=br0,vlan=4000,mode=openvswitch gnt-fsn13-02 default
root@fsn-node-01:~# gnt-network info gnt-fsn13-02
Network name: gnt-fsn13-02
UUID: f989bc71-7c0e-41c9-9bf1-5d6020726886
Serial number: 1
  Subnet: 49.12.57.128/27
  Gateway: 49.12.57.129
  IPv6 Subnet: None
  IPv6 Gateway: None
  Mac Prefix: None
  Size: 32
  Free: 29 (90.62%)
  Usage map:
        0 XX.............................X                                 63
         (X) used    (.) free
  externally reserved IPs:
    49.12.57.128, 49.12.57.129, 49.12.57.159
  connected to node groups:
    default (mode:openvswitch link:br0 vlan:4000)
  not used by any instances

      hopefully that will "just work"!

    • anarcat
      anarcat @anarcat ·
      Author Owner

      troodi/trac done!

      Trac:
      Description: * [ ] crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186

      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) test server ready at 116.202.120.185
      • nevii.torproject.org (DNS master) - not migrated, ran out of IP, opened Ticket#2020032503025825 with hetzner to followup
      • rude.torproject.org (RT) - migrated to 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - test server ready at 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

      to

      • crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186
      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) test server ready at 116.202.120.185
      • nevii.torproject.org (DNS master) - not migrated, ran out of IP, opened Ticket#2020032503025825 with hetzner to followup
      • rude.torproject.org (RT) - migrated to 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - migrated to 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

    • anarcat
      anarcat @anarcat ·
      Author Owner

      forrestii migrated!

      Trac:
      Description: * [ ] crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186

      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) test server ready at 116.202.120.185
      • nevii.torproject.org (DNS master) - not migrated, ran out of IP, opened Ticket#2020032503025825 with hetzner to followup
      • rude.torproject.org (RT) - migrated to 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - migrated to 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

      to

      • crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186
      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) server migrated to 116.202.120.185
      • nevii.torproject.org (DNS master) - not migrated, ran out of IP, opened Ticket#2020032503025825 with hetzner to followup
      • rude.torproject.org (RT) - migrated to 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - migrated to 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

    • anarcat
      anarcat @anarcat ·
      Author Owner

      create a ticket for nevii, as it's more complicated than the others

      Trac:
      Description: * [ ] crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186

      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) server migrated to 116.202.120.185
      • nevii.torproject.org (DNS master) - not migrated, ran out of IP, opened Ticket#2020032503025825 with hetzner to followup
      • rude.torproject.org (RT) - migrated to 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - migrated to 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

      to

      • crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186
      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) server migrated to 116.202.120.185
      • nevii.torproject.org (DNS master) - #33834 (moved)
      • rude.torproject.org (RT) - migrated to 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - migrated to 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

    • anarcat
      anarcat @anarcat ·
      Author Owner

      nevii done, only the CRM left, whoohoo!!!!!

      Trac:
      Description: * [ ] crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186

      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) server migrated to 116.202.120.185
      • nevii.torproject.org (DNS master) - #33834 (moved)
      • rude.torproject.org (RT) - migrated to 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - migrated to 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

      to

      • crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186
      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) server migrated to 116.202.120.185
      • nevii.torproject.org (DNS master) - #33834 (moved)
      • rude.torproject.org (RT) - migrated to 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - migrated to 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

    • anarcat
      anarcat @anarcat ·
      Author Owner

      this wasn't completed in march, so move to april

      Trac:
      Keywords: tpa-roadmap-march deleted, tpa-roadmap-april added

    • anarcat
      anarcat @anarcat ·
      Author Owner

      the old CRM boxes were migrated now, next step is to retire all of macrum, which i will start tomorrow.

      Trac:
      Description: * [ ] crm-ext-01.torproject.org (part of #32198 (moved)) test server ready (but shutdown) at 116.202.120.186

      • crm-int-01.torproject.org (also part of #32198 (moved)) test server ready (but shutdown) 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) server migrated to 116.202.120.185
      • nevii.torproject.org (DNS master) - #33834 (moved)
      • rude.torproject.org (RT) - migrated to 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - migrated to 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

      to

      • crm-ext-01.torproject.org (part of #32198 (moved)) done at 116.202.120.186
      • crm-int-01.torproject.org (also part of #32198 (moved)) done at 116.202.120.190
      • forrestii.torproject.org (fpcentral, #33729 (moved)) server migrated to 116.202.120.185
      • nevii.torproject.org (DNS master) - #33834 (moved)
      • rude.torproject.org (RT) - migrated to 116.202.120.187
      • troodi.torproject.org (Trac, #33731 (moved)) - migrated to 116.202.120.188
      • vineale.torproject.org (gitweb, #33730 (moved)) - done, new IP is 116.202.120.189

      The CRM machines might have already been migrated by the time we start this, see #32198 (moved).

      Will require a new Ganeti node (#33083 (moved)).

    • anarcat
      anarcat @anarcat ·
      Author Owner

      hiro wants to test this procedure. :)

    • anarcat
      anarcat @anarcat ·
      Author Owner

      Trac:
      Status: accepted to assigned
      Owner: anarcat to hiro

    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    Please register or sign in to reply