TPA-RFC-77-H: Move Tails nodes to the TPA Puppet Server

Some of the steps listed below come from howto/new-machine. Steps from the Main Procedure are tagged step-N.

Preparation

Fullfill all pre-requisites or workaround/document where needed:

  1. Minimal debian install
  2. Partitions correctly setup:
  3. A hostname:
    • has been set
    • was picked from the doc/naming-scheme → document that Tails nodes are special wrt. the doc/naming-scheme
    • resolves to a fully qualified domain name in the torproject.org domain
    • hostnames must not be longer than 21 chars (because it crashes bacula backups, which we don't use)
  4. A public IP adddress has been set and is available over SSH
  5. Reverse DNS is set for the machine
  6. DNS works on the machine
  7. A strong root password has been set in the password manager
  8. grub-pc/install_devices debconf parameter is correctly set

Prepare for migration:

  • Document all machines in the Nextcloud spreadsheet (step-1)
  • Add all mailing nodes to dnswl.org (step-6)
  • Add all hosts to LDAP (step-5)
  • Puppet: → puppet-control!119, puppet-hiera-enc!3
    • Add all hoster subnets to data/common/hosters.yaml (step-2) → puppet-control@140e8f0dc097827295fa00b0cdb9b58585784fec
    • Remove legacy Tails Puppet code
    • Install all 3rd-party modules that are used by Tails but not by Tor
    • Merge Hiera data:
      • Adapt hiera.yaml to support encrypted Hiera data (EYAML)
      • Import the tails::profile::puppet::eyaml profile into TPA's profile::puppet::server
      • Separate TPA and Tails hierarchies where needed
      • Add Tails roles to tor-puppet-hiera-enc
    • Copy code (legacy modules and profiles) from Tails to Tor
    • Conditionally include the corresponding base class (profile::tails or profile::common)
    • Handle the "masterless" node (stone.tails.net)
    • Handle SSH keys managed by Puppet (i.e. profile::tails::sshkeymaster)
    • Fix or workaround puppet-yapgp!3
  • Add root@stone.tails.net's SSH key as read-only for:
    • tpo/tpa/puppet-control>
    • tpo/tpa/puppet-hiera-enc>

Migration

  • Copy SSH keys managed by Puppet (i.e. profile::tails::sshkeymaster) to the TPA Puppet server
  • Run Puppet Agent everywhere: 
    # run for **all** nodes
cumin -b 10 -p 0 '*' patc
  • Merge the production branch of the TPA and Tails Puppet repos into puppet-control!119
  • Set Puppet config to noop = true in all nodes (Tails + TPA) 
    # run for **Tails and TPA** nodes
cumin -b 10 -p 0 '*' 'sed -i \'s/noop = false/noop = true/\' /etc/puppet/puppet.conf'
  • Stop the puppet-run.timer in all nodes 
    # run for **Tails and TPA** nodes
cumin -b 5 '*' 'systemctl stop puppet-run.timer'
  • Merge puppet-control!119 into the production branch of the Tails Puppet repo
  • Run Puppet (--dry-run) in all Tails nodes, only changes to Puppet should appear 
    # run for **Tails** nodes
cumin -b 10 -p 0 '*' 'patcn'
  • Bootstrap Puppet in Tails nodes (step-4)
    • Move the Tails Puppet CA to a backup folder in all Tails nodes 
      # run for **Tails** nodes    
cumin -b 10 -p 0 '*' 'mv /var/lib/puppet/ssl /var/lib/puppet/ssl.old; install -d -o puppet -g puppet /var/lib/puppet/ssl'
    • Deploy the TPA Puppet Server CA certificate in the Tails nodes: 
      ssh puppet.lizard puppetserver ca list --all | tail -n +2 | awk '{print $1}' | sort  > tails-hosts.txt
for node in $(cat tails-hosts.txt); do echo fab -H $node puppet.deploy-ca; done
    • Copy the EYAML keys from the Tails to the TPA Puppet Server 
      scp -3pr puppet.lizard:/etc/puppet/keys pauli.torproject.org:/etc/puppet/keys
# UIDs for the puppet user are different in the 2 servers, so fix it
ssh pauli.torproject.org chown -R puppet:puppet /etc/puppet/keys
    • Merge puppet-control!119 into the production branch of the TPA Puppet repo
    • Merge puppet-hiera-enc!3 into the main branch of the TPA Puppet Hiera ENC repo
    • Point Tails nodes to the TPA Puppet Server 
      # run for **Tails** nodes
cumin -b 10 -p 0 '*' 'patc --server puppet.torproject.org --waitforcert 0'
    • Confirm fingerprints and sign all Tails certificates in the TPA Puppet Server
    • Run the Puppet Agent in all machines to export resources 
      # Tails nodes still haven't applied the change of Puppet Server, so force it
cumin -b 10 -p 0 '*' 'patc --server puppet.torproject.org'
    • Run the Puppet Agent in all machines to ensure nothing changes 
      cumin -b 10 -p 0 '*' 'patc'
    • Set Puppet config to noop = false in all nodes (Tails + TPA) 
      cumin -b 10 -p 0 '*' 'sed -i \'s/noop = true/noop = false/\' /etc/puppet/puppet.conf'
    • Point the "masterless" node (stone.tails.net) to the TPA Puppet repo
  • Reboot all machines and run Puppet once again (step-8)

Follow-ups

  • Handle issue in stone.tails.net
  • Consider whether stone.tails.net should pull from puppet.torproject.org instead of GitLab
  • Remove the obsolete TLS data from all Tails nodes 
    # run for **Tails** nodes
cumin -b 10 -p 0 '*' 'rm -rf /var/lib/puppet/ssl.old'
  • Remove the now obsolete role definition using fact files (for Tails nodes)
  • Move EYAML keys in the password-store to reflect the migration to a different Puppet Server 
    git -C ~/.password-store/tor/ pull
pass mv tor/eyaml/puppet.lizard tor/eyaml/pauli.torproject.org
git -C ~/.password-store/tor push
  • Archive the Tails' puppet-code.git repo
  • Remove the custom LDAP facts from Tails nodes
  • Remove dragon.tails.net and *.dragon from Puppet and LDAP
  • Decommission puppet.lizard → #42102 (closed)
  • Cleanup unneeded certificate requests in pauli.torproject.org 
    ssh pauli.torproject.org puppetserver ca list
  • Handle dependency on the obsolete puppet-code repo in site/profile/manifests/tails/check_gpg_monitoring.pp → #42412

Reference

Initial plan: TPA-RFC-77

Unneeded steps from new machine main procedure (here for reference, only):

  • Deploy fabric-tasks> in all hosts (step-3)
  • Create /srv filesystem (step-7)
  • Silence backup alerts (step-9)
  • Configure bacula (step-10)
Edited by zen