Onboarding new Tails sysadmins

This document describes the process to include a new person in the Tails sysadmin team.

⚠️ This process should become obsolete at some point during the Tails/Tor merge process.

• Onboarding new Tails sysadmins
• Documentation
  • Security policy
  • Accesses
    • OpenPGP
    • Git repositories
    • GitLab
    • Mailing lists
• Monitoring
  • Misc
  • SSH and sudo

Documentation

Our documentation is stored in this wiki. See our ../role-description as it gives insight on the way we currently organize. Check the pages linked from there for info about services and some important pages in GitLab which we need to keep an eye on.

Security policy

Ensure the new sysadmin complies with our team's security policy (Level B):

• https://gitlab.tails.boum.org/tails/summit/-/wikis/Security_policies/

Also, see the integration of Tails and TPA security policies in:

• #41727
• https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-18-security-policy

Accesses

Once we have the necessary information, there are some steps to do to get the new sysadmin in the team.

OpenPGP

• Have the new sysadmin to generate an authentication-capable subkey for their OpenPGP key.
• Have the new sysadmin upload their OpenPGP key, including the authentication subkey, to hkps://hkps.pool.sks-keyservers.net and hkps://keys.openpgp.org; the latter requires an email-based confirmation.

Git repositories

We have a meta-repository that documents all important repositories. During the onboarding process, you should receive a signed copy of the known_hosts file in that repository to bootstrap trust on those SSH servers.

Onboarding steps:

• Add the new sysadmin to .sysadmins in gitlab-config.git.
• Add the new sysadmin's SSH public key in the keys directory in gitolite@git.tails.net:gitolite-admin, commit and push.
• Add the new sysadmin to the @sysadmins variable in conf/gitolite.conf in gitolite@git.tails.net:gitolite-admin, commit and push.
• Add her OpenPGP key to the list of git-remote-gcrypt recipients for sysadmin.git and update README accordingly.
• Password store: credentials are stored in TPA's password-store, see service/password-manager#on-boarding-new-staff.
• Send the new sysadmin a signed copy of the known_hosts file that contains the hashes for the SSHd host key for git.tails.net and also share the onboarding info with them.

GitLab

Sysadmin issues are tracked in Torproject's Gitlab

Onboarding steps:

• Create an account for the new sysadmin in our GitLab at: https://gitlab.tails.boum.org
• Make sure they know that the GitLab admin credentials live in our Password Store repository.
• Have them subscribe to the relevant labels in GitLab in the "tails" group level (see https://gitlab.tails.boum.org/groups/tails/-/labels):
  • C:Server
  • C:Infrastructure
  • Core Work:Sysadmin They might also want to subscribe to priority labels, at least in the project level for example for the "tails-sysadmin" and "tails/puppet-tails" projects (see https://gitlab.torproject.org/tpo/tpa/tails-sysadmin/-/labels?subscribed=&search=P%3A and the corresponding URL for the "tails/puppet-tails" project):
  • P:Urgent
  • P:High
  • P:Elevated At the time of reading there might be others and this doc might be outdated, please check!

Mailing lists

We currently use the following mailing lists:

• sysadmins at tails.net, a Schleuder list used for:
  • accounts in external services
  • communication with upstream providers
  • general requests (eg. GitLab accounts, occasional bug reports)
  • cron reports which eventually need acting upon
• tails-notifications at lists.puscii.nl, used for Icinga2 notifications

Onboarding steps:

• Add the new sysadmin's public OpenPGP key to the keyring of the sysadmins@tails.net list.
• Subscribe the new sysadmin to the sysadmins@tails.net list.
• Add the new sysadmin to the list of administrators of the sysadmins@puscii.nl list.
• Add the new sysadmin the the tails-notifications@lists.puscii.nl list and set her as an owner of that list.

Monitoring

We use Icinga2 with the Icingaweb2 web interface. The shared passphrase can be found in the Password Store (see the Git repositories section).

pass tor/services/icingaweb2.tails.boum.org/icingaadmin

Misc

• Send an email on assembly@tails.net to announce this new sysadmin if not already advertised.
• Point the new sysadmin to the admin account in pass for https://icingaweb2.tails.net/icingaweb2/
• Inform the new sysadmin about the public XMPP channels in chat.disroot.org, mainly tails-dev where most of the work happens.
• Give access to and inform new sysadmin about private XMPP channels in the Tails XMPP server, mainly tails-sysadmins for private sysadmins conversations, tails-summit for private work conversations, and tails-bar for fun.
• Give access to the tails-sysadmins@puscii.nl calendar that we use for self-organizing and publishing info like who's on shift, meetings, sprint dates, etc.
• Monitoring can be configured in Android using the aNag app with the following configuration:
  • Instance type: Icinga 2 API
  • URL: https://w4whlrdxqh26l4frpyngcb36g66t7nbj2onspizlbcgk6z32c3kdhayd.onion:5665/
  • Username: icingaweb2
  • Password: See the value of monitoring::master::apipasswd by executing the following command in the puppet-code Git repo:

    eyaml decrypt -e hieradata/node/ecours.tails.net.eyaml

  • Check "Allow insecure certificate", because the cert doesn't include the onion address. (This can be further improved in the future)
  • Check "Enabled".
• Grant the new sysadmin "owner" affiliation on our public XMPP channels at chat.disroot.org: tails and tails-dev.

SSH and sudo

Once you have confirmed the known_hosts file (see the Git repositories section), you can fetch a list of all hosts from the Puppet Server:

ssh -p 3005 lizard.tails.net sudo puppetserver ca list --all

You can also fetch SSH fingerprints for know hosts:

mkdir -p ~/.ssh/tails
scp -P 3005 lizard.tails.net:/etc/ssh/ssh_known_hosts ~/.ssh/tails/known_hosts

An example SSH config file can be seen here.

All public systems are reachable via the tails.net namespace and, once inside, all private VMs are accessible via their hostnames and FQDNs. TCP forwarding works so you can use any public system as a jumphost.

Physical servers and VMs hosted by third-parties have OOB access, and such instructions can be found in sysadmin-private.git:systems/.

Onboarding steps:

• Send the new sysadmin the SSH connection information (onion service, port, SSHd host key hashes) for all our systems.
• For hieradata/common.eyaml and hieradata/node/stone.tails.net.eyaml:
  • Add the user name to the sysadmins entry of rbac::roles.
  • Add the user data to the rbac::users hash, including the new sysadmin's SSH and OpenPGP public keys.
• Commit these changes to our Puppet manifests repository and push.
• Check that the new sysadmin can SSH from the lizard.tails.net virtualization host to VMs hosted there and elsewhere, e.g. misc.lizard and isoworker1.dragon.
• Ensure the new sysadmin uses a UTF-8 locale when logged into our systems. Otherwise, some Puppet facts (e.g. jenkins_plugins) will return different values, and Puppet will do weird things.
• Ask micah and taggart to add the new sysadmin's SSH public key to ~tails/.ssh/authorized_keys on magpie.riseup.net so she has access to lizard's IPMI interface.
• Ask tachanka-collective@lists.tachanka.org to add the new sysadmin's OpenPGP key to the access list for ecours' OOB interface.
• Ask noc@lists.paulla.asso.fr to add the new sysadmin to their OOB interface.
• Login to service.coloclue.net and add the new sysadmin's SSH key to the OOB interface.