import tails sysadmins doc

This wiki contains the public documentation of the Tails Sysadmin team that is

still valid. This documentation will be gradually superseded by the TPA doc

during [the merge

process](/tpo/tpa/team/-/wikis/policy/tpa-rfc-73-tails-infra-merge-roadmap).

These is the content that still lives here for now:

- [[how-tos]]

- [[processes]]

- [[providers]]

- [[role-description]]

- [[services]]

- [[scripts]]

Note: this wiki also contains non-markdown file, clone the corresponding repo

to see them.

- [[how-tos/decommission]]

- [[how-tos/grow-system-disks]]

- [[how-tos/install-an-isoworker]]

- [[how-tos/install-a-vm]]

- [[how-tos/install-base-systems]]

- [[how-tos/install-monitoring]]

- [[how-tos/rm-qa-for-sysadmins]]

- [[how-tos/spam]]

- [[how-tos/working-with-puppet-and-git-for-dummies]]

Initial setup

=============

Set some environment variables on your own system:

Set this variable to what "facter fqdn" would return on the system

which is being decommissioned; it's also the name of the Puppet node:

    FQDN=XXX

Set this variable to what "facter hostname" would return on the system

which is being decommissioned:

    HOSTNAME=XXX

Set this variable to what "facter ipaddress" would return on the system

which is being decommissioned:

    IP=XXX

Steps

=====

- Delete Puppet configuration in `puppet-code.git`:

   - the section dedicated to the decommissioned node in `nodes.pp`

   - `hieradata/node/$FQDN.yaml`

   - all references in `hierdata/common.yaml`

   - Look for other configuration:

       - `git grep --recurse-submodules -F "$FQDN"`

       - `git grep --recurse-submodules -F "$HOSTNAME"`

       - Look for files named `*"$HOSTNAME"*`

- Clean up certificates, cached facts, and reports on the Puppet master:

        ssh puppet.lizard sudo puppet node deactivate "${FQDN:?}"

        ssh puppet.lizard sudo puppet node clean "${FQDN:?}"

- Clean up Munin configuration

  If the decommissioned node was running Munin,

  ask micah and taggart to delete it from their configuration.

- Wipe sensitive data

  Depending on whether we control the hardware or not,

  on whether the underlying disks are SSDs or not,

  this may or not may not be feasible & worth it.

- Consider deleting backups

  If you decide to delete backups, see "Deleting backups of a decommissioned

  system" in `sysadmin.git:backups.mdwn`.

- Delete the VM definition in libvirt.

  For example, for a VM hosted on lizard, run:

        ssh lizard.tails.boum.org virsh undefine "${HOSTNAME:?}"

- Delete the storage volumes formerly used by this VM.

# Growing a VM's system disk

These are instructions for growing the size of a VM's system disk. For these

disks, there are 2 levels of LVM:

1. A logical volume is defined in lizard as /dev/lizard/[VM]-system and maps to

   /dev/vda inside the VM.

2. The /dev/vda is partitioned inside the VM and /dev/vda2 is made an LVM

   physical volume. That physical volume is a part of the "vg1" volume group

   and a "root" logical volume is created in that group, providing /dev/vg1/root.

**Attention:** these instructions do not apply to data disks, as their

partitioning scheme is different from system disks.

## Instructions

Please, double check these instructions before running them to make sure the

partitioning scheme makes sense for the case.

Resize the system disk in the host:

    VM=www

    AMOUNT=2G

    sudo virsh shutdown ${VM}

    # wait for VM to shutdown, then:

    sudo lvresize -L+${AMOUNT} /dev/lizard/${VM}-system

    sudo virsh start ${VM}

SSH into the VM:

    ssh ${VM}.lizard

Resize the block device and LVM volumes from inside the VM:

    sudo parted /dev/vda resizepart "2 -1s"

    sudo pvresize /dev/vda2

    sudo lvresize -l+100%FREE /dev/vg1/root

    sudo resize2fs /dev/vg1/root

This should be enough!

# Installing a VM

1. Copy the install-vm.sh script to the hypervisor. 

2. Run ./install-vm.sh [-d disksize] [-v vcpu] [-r ram] -n hostname -i ip. This script starts by outputting the root password, be sure to copy that.

3. In puppet-hiera-node, create a file called <fqdn>.yaml and add an entry for tails::profile::network::interfaces.

4. In puppet-code, update the hieradata/node submodule and add a node definition in manifest/nodes.pp

5. Once the install is done, log in on the console as root and run puppet agent -t.

6. Log in to the puppetmaster and run ``puppet ca sign <fqdn>``.

7. Go back to the node you're installing and run ``puppet agent -t`` several times. Then, reboot the machine.

8. Add the SSH onion address (``cat /var/lib/tor/ssh-hidden-v3/hostname``) to onions.mdwn in this repo, as well as the appropriate file under Machines/Servers in summit.wiki.

9. Add the SSH fingerprints (``cd /etc/ssh;for i in `ls *pub`;do ssh-keygen -l -f $i;done``) to the appropriate file under Machines/Servers in summit.wiki.

10. Add the root password to our pass repository.

11. Wait for all the other nodes to collect the exported resources from the new node (this should be done within half an hour) and you're done!