Replace Tails machines with new ones

We recently had performance issues in Dragon (#18215 (closed), #18231 (closed)), which resulted in the need to lower the number of Jenkins workers, degrading CI performance. The machine is also somewhat old (as is Iguana) and we're worried it can deteriorate further.

Note that we also need to replace Lizard (#18218), so we may take this opportunity to review the whole Tails fleet.

Resources:

To do

  • spec 2 new machines → thread
  • decide where to host
  • write to accounting to get the budget approved
  • accounting approves the budget
  • inform other parties about our course of action
    • graber
    • riseup
  • order from Hetzner
  • burn-in
  • install base systems
  • add to Puppet
  • install Jenkins agents
  • decide what to do with dragon (see summary thread)
  • decide what to do with iguana (see summary thread)
  • workaround network speed limitations (see summary thread):
    • uploading artifacts from Jenkins agents in Hetzner to the Jenkins controller in Riseup is too slow
    • downloading the Tails git repository from Riseup to Hetzner is too slow
  • review amount of memory committed to huge pages
Edited by zen