From 2e6a7caf66f803b6cecd4f11c2a14406b55b4704 Mon Sep 17 00:00:00 2001 From: Linus Nordberg <linus@torproject.org> Date: Sun, 3 Mar 2019 20:11:38 +0100 Subject: [PATCH] add three pages --- tsa/doc/bits-and-pieces.mdwn | 108 +++++++++++++++++++++++++++++++++++ tsa/howto/letsencrypt.mdwn | 39 +++++++++++++ tsa/howto/nagios.mdwn | 18 ++++++ 3 files changed, 165 insertions(+) create mode 100644 tsa/doc/bits-and-pieces.mdwn create mode 100644 tsa/howto/letsencrypt.mdwn create mode 100644 tsa/howto/nagios.mdwn diff --git a/tsa/doc/bits-and-pieces.mdwn b/tsa/doc/bits-and-pieces.mdwn new file mode 100644 index 00000000..c0df55df --- /dev/null +++ b/tsa/doc/bits-and-pieces.mdwn @@ -0,0 +1,108 @@ +# Bits and pieces of Tor Project infrastructure + +A collection of information looking for a better place, perhaps after +being expanded a bit to deserve their own page. + +## Backups + +* We use [[Bacula|https://en.wikipedia.org/wiki/Bacula]] to make + backups, with one host running a director (currently dictyotum.tpo) + and another host for storage (currently brulloi.tpo). +* There are `BASE` files and `WAL` files, the latter for incremental + backups. +* The logs found in `/var/log/bacula-main.log` and `/var/log/bacula/` + seem mostly empty, just like the systemd journals. + +### Servers + +* There's one `director` and one `storage node`. + +* The director runs `/usr/local/sbin/dsa-bacula-scheduler` which reads + `/etc/bacula/dsa-clients` for a list of clients to back up. This + file is populated by puppet (puppetdb + `$bacula::tag_bacula_dsa_client_list`) and will list clients until + they're being deactivated in puppet. + +### Clients +* `tor-puppet/modules/bacula/manifests/client.pp` gives an idea of + where things are at on backup clients. +* Clients run the Bacula File Daemon, `bacula-fd(8)`. + +## Onion sites + +- Example from a vhost template + + <% if scope.function_onion_global_service_hostname(['crm-2018.torproject.org']) -%> + <Virtualhost *:80> + ServerName <%= scope.function_onion_global_service_hostname(['crm-2018.torproject.org']) %> + Use vhost-inner-crm-2018.torproject.org + </VirtualHost> + <% end -%> + +- Function defined in + `tor-puppet/modules/puppetmaster/lib/puppet/parser/functions/onion_global_service_hostname.rb` + parses + `/srv/puppet.torproject.org/puppet-facts/onionbalance-services.yaml`. +- `onionbalance-services.yaml` is populated through + `onion::balance` (`tor-puppet/modules/onion/manifests/balance.pp`) +- `onion:balance` uses the `onion_balance_service_hostname` fact from `tor-puppet/modules/torproject_org/lib/facter/onion-services.rb` + +## Puppet + +### Before it all starts + +- `puppet.tpo` is currently being run on `pauli.tpo` +- This is where the tor-puppet git repo lives +- The repo has hooks to populate `/etc/puppet` with its contents, most + notably the modules directory. +- All paths in this document are relative to the root of this + repository. + +### File layout + +- The root of definitions and execution is found in + `tor-puppet/manifests/site.pp`. + +- `local.yaml` (modules/torproject_org/misc/local.yaml) defines + services and list which host(s) supply each service. local.yaml is + read by [site.pp](manifests/site.pp) for setting up the $localinfo + and $nodeinfo variables. + +- `roles init.pp` (modules/roles/manifests/init.pp) maps services to + roles, using the `$nodeinfo` variable. + +- `torproject.org init.pp` (modules/torproject_org/manifests/init.pp) + performs basic host initialisation, like configuring Debian mirrors + and APT sources, installing a base set of packages, configuring + puppet and timezone, setting up a bunch of rc-files and running + ud-replicate. + +- `hoster.yaml` (modules/torproject_org/misc/hoster.yaml) defines + hosting providers and specifies things like which net blocks they + use, if they have a DNS resolver or a debian mirror. hoster.yaml is + read by + - the `nodeinfo()` function + (modules/puppetmaster/lib/puppet/parser/functions/nodeinfo.rb), + used for setting up the `$nodeinfo` variable + - `ferm's def.conf template` (modules/ferm/templates/defs.conf.erb) + - the `entropy provider` + (modules/puppetmaster/lib/puppet/parser/functions/entropy_provider.rb) + TODO + +### Generating secrets, an example + +- `bacula::director` inherits `bacula` which defines +- `$bacula_director_secret` using +- `hkdf()` and generates +- `/etc/bacula/bacula-dir.conf` using that + +### Custom facts + +`modules/torproject_org/lib/facter/software.rb` defines our custom +facts, making it possible to get answer to questions like "Is this +host running apache2?" byt simply looking at a puppet variable. + +### Misc + +- `puppet-lint` is a thing +- TODO: how to debug things diff --git a/tsa/howto/letsencrypt.mdwn b/tsa/howto/letsencrypt.mdwn new file mode 100644 index 00000000..b344d6b0 --- /dev/null +++ b/tsa/howto/letsencrypt.mdwn @@ -0,0 +1,39 @@ +# How to get an X.509 certificate for your new name + +## The letnsencrypt-domains git repository + +If not already done, clone git repos letsencrypt-domains and +backup-keys. + + git clone ssh://git@git-rw.torproject.org/admin/letsencrypt-domains + cd letsencrypt-domains + git clone pauli.torproject.org:/srv/puppet.torproject.org/git/tor-backup-keys.git backup-keys + +## Add your new name and generate a private key + + $EDIT domains # add your domain name and optional SAN(s) + ./bin/manage-backup-keys create # see tor-passwords/000-backup-keys for the passphrase + +## Push they new key to the backup-keys repo + + cd backup-keys + git status + git add $yourfiles + git commit + git push + cd .. + +## Push the updated domain list to the letsencrypt-domains repo + + git diff domains + git add domains + git commit + git push + +- dehydrated is now being run on DNS master (nevii.tpo), see the + `letsencrypt` user and `/srv/letsencrypt`. +- Resulting keys and certs are being copied to the LDAP host + (currently pauli.tpo) under + `/srv/puppet.torproject.org/from-letsencrypt/`, from where they're + being picked up by the host running the service somehow. +- FIXME: and then what? diff --git a/tsa/howto/nagios.mdwn b/tsa/howto/nagios.mdwn new file mode 100644 index 00000000..6ee73a77 --- /dev/null +++ b/tsa/howto/nagios.mdwn @@ -0,0 +1,18 @@ +# Nagios/Icinga service for Tor Project infrastructure + +## Getting status updates + +- Using a web browser: https://nagios.torproject.org/cgi-bin/icinga/status.cgi?allunhandledproblems&sortobject=services&sorttype=1&sortoption=2 +- On IRC: /j #tor-nagios +- Over email: Add your email address to `tor-nagios/config/static/objects/contacts.cfg` + +## How to run a nagios check manually on a host (TARGET.tpo) + + NCHECKFILE=$(egrep -A 4 THE-SERVICE-TEXT-FROM-WEB | egrep '^ *nrpe:' | cut -d : -f 2 | tr -d ' |"') + NCMD=$(ssh -t TARGET.tpo grep "$NCHECKFILE" /etc/nagios -r) + : NCMD is the command that's being run. If it looks sane, run it. With --verbose if you like more output. + ssh -t TARGET.tpo "$NCMD" --verbose + +## How to add a host + +- Add new hosts to `tor-nagios/config/nagios-master.cfg`. -- GitLab