Merge branch 'master' of ssh://torgit/project/help/wiki

tsa/doc/accounts.creole

@@ -7,6 +7,11 @@ their email forwards.
 It also stores group memberships which in turn affects which users can log into
 which [[hosts|https://db.torproject.org/machines.cgi]].
 
+This document should be consistent with the [[Tor membership
+policy|https://gitweb.torproject.org/community/policies.git/plain/membership.txt]],
+in case of discrepancy between the two documents, the membership
+policy overrules this document.
+
 == <a id="ldap-or-alias">Decision tree: LDAP account or email alias?</a> ==
 
 Here is a simple decision tree to help you decide if a new contributor

tsa/doc/debian-archive.mdwn

+Internal Debian archive operations
+----------------------------------
+
+Packages should be uploaded to: 
+
+    alberti.torproject.org:/srv/db.torproject.org/ftp-archive/archive/pool/tpo-all/
+
+Then to install the packages:
+
+    ssh alberti.tpo make -C /srv/db.torproject.org/ftp-archive
+
+This might require fixing some permissions. Do a `chmod g+w` on the
+broken directories if this happens and open a ticket if it does, to
+create a service account instead of that hack.

tsa/doc/naming-scheme.mdwn

+There are multiple naming schemes in use:
+
+ * onion species
+ * role-based
+ * location-based
+
+Onion species
+=============
+
+[Wikipedia list of onion species][], preferably picking a first letter
+matching purpose (e.g. "m" for monitoring, "b" for backups, "p" for
+puppet) and ideally not overlapping with [existing machines at
+debian.org][] in the first three letters or at least the short
+hostname part
+
+[Wikipedia list of onion species]: https://en.wikipedia.org/wiki/List_of_Allium_species
+[existing machines at debian.org]: https://db.debian.org/machines.cgi
+
+> Example: monticola.torproject.org was picked as a "monitoring"
+> ("mon") server to run the experimental Prometheus server. no
+> machine is named "monticola" at Debian.org and no machine has
+> "mon" or smaller as its first three letters there either.
+
+Role
+====
+
+Another naming scheme is `role-ID`, where:
+
+ * `role` is what the server is for, for example `gitlab`, `mon` for
+   monitoring, `crm`, etc. try to keep it short and abbreviate to
+   at most three letters if role is longer than five. `role` might
+   have a dash (`-`) in it to describe the service better (`crm-ext`
+   vs `crm-int`)
+ * `ID` is a two-character number, padded with zero, starting from
+   one, to distinguish between multiple instances of the same server
+   (e.g. `mon-01`, `mon-02`)
+
+Location
+========
+
+Another naming scheme used for virtual machines is `hoster-locN-ID`
+(example `hetzner-hel1-01`), where:
+
+ * `hoster`: is the hosting provider (example `hetzner`)
+ * `locN`: is the three-letter code of the city where the machine is
+   located, followed by a digit in case there are multiple locations
+   in the same city (e.g. `hel1`)
+ * `ID`: is an two-character number, padded with zero, starting from
+   one, to distinguish multiple instances at the same location
+
+This is used for virtual machines at Hetzner that are bound to a
+specific location.

tsa/howto/grafana.mdwn

+Grafana
+=======
+
+[Grafana](https://grafana.com/) is a graphing engine and dashboard management tool that
+processes data from multiple data sources. We use it to trend various
+metrics collected from servers by [[Prometheus]].
+
+<https://grafana.torproject.org/>
+
+Puppet implementation
+---------------------
+
+Grafana was installed with Puppet using the upstream Debian
+package, following a debate regarding the merits of Debian packages
+versus Docker containers when neither are trusted, see [this
+comment](https://trac.torproject.org/projects/tor/ticket/29684#comment:3) for a summary.
+
+Manual configuration was performed after the installed:
+
+ 1. admin password reset on first install, stored in
+    `torr-passwords.git`, in `hosts-extra-info`.
+
+ 2. Prometheus datasource configured by hand
+
+ 3. 5 dashboards imported and configured by hand
+
+Those should be integrated in the Puppet configuration and
+authentication needs to be figured out.

tsa/howto/ldap.mdwn

+LDAP primer
+===========
+
+LDAP is not accessible to the outside world, so you need to be behind
+the firewall. Once that's resolved, you can use [ldapvi(1)](https://manpages.debian.org/ldapvi.1.en.html) or
+[ldapsearch(1)](https://manpages.debian.org/ldapsearch.1.en.html) to inspect the database. User documentation on that
+process is in [[doc/accounts]].
+
+Basic design
+------------
+
+The LDAP setup at Tor is based on the one from
+Debian.org. `/etc/password` and `groups` files are synchronized from
+the central LDAP server using the `sshdist` account, which means
+things keep working when LDAP is down. Most operations can be
+performed on the [db.torproject.org](https://db.torproject.org/) site or by [email](https://db.torproject.org/doc-mail.html).
+
+DNS zone files are also managed (at least partly) in LDAP. This is
+automated through cron jobs, but if you're in a hurry, the zones get
+generated by `ud-generate` on `alberti` (as `sshdist`?) and replicate
+(?) on `nevii` with `ud-replicate` (as `root`?).
+
+Handy commands
+--------------
+
+This will load a text editor with a dump of all the users (useful to
+modify an existing user or add a new one):
+
+    ldapvi -ZZ --encoding=ASCII --ldap-conf -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org"
+
+This will list all known hosts in LDAP:
+
+    ldapsearch -ZZ -vLxW -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org" -b "ou=hosts,dc=torproject,dc=org" '(objectclass=*)' | grep ^dn:
+

tsa/howto/new-machine-hetzner-cloud.mdwn

+Go to the [Heztner console][] and clikety on the web interface to get
+a new instance. Credentials are in `tor-passwords.git` in
+`hosts-extra-info` under `hetzner`.
+
+ [Heztner console]: https://console.hetzner.cloud/
+
+Pick the following settings:
+
+ 1. Location: depends on the project, a monitoring server might be
+    better in a different location than the other VMs
+ 2. Image: Debian 9
+ 3. Type: depends on the project
+ 4. Volume: only if extra space is required
+ 5. Additional features: nothing (no user data or backups)
+ 6. SSH key: enable all configured keys
+ 7. Name: FQDN picked from the [[doc/naming-scheme]]
+ 8. Create the server
+ 9. Login to the server as root with SSH
+
+Then follow the rest of [[new-machine]].

tsa/howto/new-machine.mdwn

+
+(all commands to be run as root unless otherwise noted)
+
+* set a hostname - pick an unused one from the [[doc/naming-scheme]]
+
+* sanitize DNS configuration:
+
+        grep torproject.org /etc/resolv.conf || echo 'domain torproject.org' >>  /etc/resolv.conf
+        vi /etc/hosts # make sure the local host is there with both FQDN and just hostname
+
+* add to ldap on `alberti` using:
+
+        ldapvi -ZZ --encoding=ASCII --ldap-conf -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org"
+
+  Copy-paste the host block of another similar host, and change the
+  leading number to an `add`. See also the [ldapvi manual](http://www.lichteblau.com/ldapvi/manual/) for more
+  help. To generate the ssh host key block, use:
+
+        for i in /etc/ssh/ssh_host_rsa_key.pub /etc/ssh/ssh_host_ed25519_key.pub; do
+            echo "sshRSAHostKey: `cat $i`"
+        done
+
+* then:
+  * on alberti:
+  
+        sudo -u sshdist ud-generate && sudo -H ud-replicate && sudo puppet agent -t
+
+  * on nevii:
+
+        sudo -H ud-replicate
+
+* set up puppet:
+  * on pauli:
+
+        ( puppet agent -t --environment=production || true ) && \
+        ud-replicate && \
+        sudo -H -u puppet make -C /srv/puppet.torproject.org/auto-ca install
+
+  * on new machine:
+
+        me=$(hostname -f) && \
+        [ "$me" != "${me%torproject.org}" ] && \
+        apt-get update && \
+        apt-get install -y --no-install-recommends puppet lsb-release && \
+        service puppet stop && \
+        systemctl mask puppet.service && \
+        (puppet agent -t || true ) && \
+        cd /var/lib/puppet/ssl/certificate_requests && \
+        echo sha256sum output: && \
+        echo && \
+        sha256sum $me.pem && \
+        echo && \
+        echo && \
+        cd /
+
+  * on pauli:
+
+        cd /var/lib/puppet/ssl/ca/requests && \
+        ( [ -e $filename ] || (echo "$filename does not exist."; exit 1) ) && \
+        echo "paste sha256sum output followed by control-d: " && \
+        sha256sum -c && \
+        puppet ca sign $(basename "$filename" .pem) && \
+        echo && \
+        echo "--- 8< --- CUT HERE --- 8< ---" && \
+        echo && \
+        echo 'cat > /var/lib/puppet/ssl/certs/ca.pem << EOF ' && \
+        cat /var/lib/puppet/ssl/certs/ca.pem && \
+        echo 'EOF' && \
+        echo "cat > /var/lib/puppet/ssl/certs/$filename << EOF " && \
+        cat /var/lib/puppet/ssl/ca/signed/$filename && \
+        echo 'EOF' && \
+        echo 'puppet agent --enable' && \
+        echo 'puppet agent -t --pluginsync' && \
+        echo 'puppet agent -t --pluginsync' && \
+        echo
+        echo "--- 8< --- CUT HERE --- 8< ---" && \
+        echo
+        cd /
+
+* copy paste the thing between `CUT HERE` from `pauli` into a root
+  shell on the new host
+
+* do more puppet runs, and run a ud-replicate to get ldap users, then
+  more puppet runs since we now have more users:
+
+        puppet agent -t
+        ud-replicate
+        puppet agent -t
+        puppet agent -t
+
+* get rid of unneeded network services (only needed if we did not install this
+  from a minbase but from somebody else's install/image):
+
+        dpkg --purge portmap nfs-common exim4 exim4-base exim4-config exim4-daemon-light
+        userdel -r Debian-exim
+
+* fix `/etc/aliases`:
+
+        ( ! grep '^root:' /etc/aliases && echo 'root: torproject-admin@torproject.org' >> /etc/aliases ) &&
+        sed -i -e 's/^root:.*/root: torproject-admin@torproject.org/' /etc/aliases && newaliases
+
+* set new root password, and document it
+
+* swap and tmp
+
+        grep tmpfs /etc/fstab || echo 'tmpfs /tmp tmpfs defaults 0 0' >> /etc/fstab
+
+        ! grep swap /etc/fstab && \
+        dd if=/dev/zero bs=1024k count=1024 of=/swapfile && \
+        chmod 0 /swapfile && \
+        mkswap /swapfile && \
+        echo '/swapfile none swap sw 0 0' >> /etc/fstab && \
+        swapon -a
+
+* add to [[nagios]]
+
+* document the machine in the evil Google spreadsheet (!)

tsa/howto/prometheus.mdwn

+Prometheus
+==========
+
+[Prometheus][] is a monitoring system that is designed to process a
+large number of metrics, centralize them on one (or multiple) servers
+and serve them with a well-defined API. That API is queried through a
+domain-specific language (DSL) called "PromQL" or "Prometheus Query
+Language". Prometheus also supports basic graphing capabilities
+although those are limited enough that we use a separate graphing
+layer on top (see [[Grafana]]).
+
+[Prometheus]: https://prometheus.io/
+
+Basic design
+------------
+
+The Prometheus web interface is available at:
+
+<https://prometheus.torproject.org>
+
+A simple query you can try is to pick any metric in the list and click
+`Execute`. For example, [this link][] will show the 5-minute load
+over the last two weeks for the known servers.
+
+[this link]: https://prometheus1.torproject.org/graph?g0.range_input=2w&g0.expr=node_load5&g0.tab=0
+
+Here you can see, from the [Prometheus overview documentation][] the
+basic architecture of a Prometheus site:
+
+[Prometheus overview documentation]: https://prometheus.io/docs/introduction/overview/
+
+<img src="https://prometheus.io/assets/architecture.png" alt="A
+drawing of Prometheus' architecture, showing the push gateway and
+exporters adding metrics, service discovery through file_sd and
+Kubernetes, alerts pushed to the Alertmanager and the various UIs
+pulling from Prometheus" />
+
+As you can see, Prometheus is somewhat tailored towards
+[Kubernetes][] but it can be used without it. We're deploying it with
+the `file_sd` discovery mechanism, where Puppet collects all exporters
+into the central server, which then scrapes those exporters every
+`scrape_interval` (by default 15 seconds). The architecture graph also
+shows the Alertmanager which could be used to (eventually) replace our
+Nagios deployment.
+
+[Kubernetes]: https://kubernetes.io/
+
+It does not show that Prometheus can federate to multiple instances
+and the Alertmanager can be configured with High availability.
+
+Munin expatriates
+-----------------
+
+Here's a quick cheat sheet from people used to Munin and switching to
+Prometheus:
+
+| What              | Munin           | Prometheus                          |
+| ---               | -----           | ----------                          |
+| Scraper           | munin-update    | prometheus                          |
+| Agent             | munin-node      | prometheus node-exporter and others |
+| Graphing          | munin-graph     | prometheus or grafana               |
+| Alerting          | munin-limits    | prometheus alertmanager             |
+| Network port      | 4949            | 9100 and others                     |
+| Protocol          | TCP, text-based | HTTP, [text-based][]                |
+| Storage format    | RRD             | custom TSDB                         |
+| Downsampling      | yes             | no                                  |
+| Default interval  | 5 minutes       | 15 seconds                          |
+| Authentication    | no              | no                                  |
+| Federation        | no              | yes (can fetch from other servers)  |
+| High availability | no              | yes (alert-manager gossip protocol) |
+
+[text-based]: https://prometheus.io/docs/instrumenting/exposition_formats/
+
+Basically, Prometheus is similar to Munin in many ways:
+
+ * it "pulls" metrics from the nodes, although it does it over HTTP
+   (to <http://host:9100/metrics>) instead of a custom TCP protocol
+   like Munin
+
+ * the agent running on the nodes is called `prometheus-node-exporter`
+   instead of `munin-node`. it scrapes only a set of built-in
+   parameters like CPU, disk space and so on, different exporters are
+   necessary for different applications (like
+   `prometheus-apache-exporter`) and any application can easily
+   implement an exporter by exposing a Prometheus-compatible
+   `/metrics` endpoint
+
+ * like Munin, the node exporter doesn't have any form of
+   authentication built-in. we rely on IP-level firewalls to avoid
+   leakage
+
+ * the central server is simply called `prometheus` and runs as a
+   daemon that wakes up on its own, instead of `munin-update` which is
+   called from `munin-cron` and before that `cron`
+
+ * graphics are generated on the fly through the crude Prometheus web
+   interface or by frontends like Grafana, instead of being constantly
+   regenerated by `munin-graph`
+
+ * samples are stored in a custom "time series database" (TSDB) in
+   Prometheus instead of the (ad-hoc) RRD standard
+   
+ * Prometheus performs *no* downsampling like RRD and Prom relies on
+   smart compression to spare disk space, but it uses more than Munin
+
+ * Prometheus scrapes samples much more aggressively than Munin by
+   default, but that interval is configurable
+
+ * Prometheus can scale horizontally (by sharding different services
+   to different servers) and vertically (by aggregating different
+   servers to a central one with a different sampling frequency)
+   natively - `munin-update` and `munin-graph` can only run on a
+   single (and same) server
+
+ * Prometheus can act as an high availability alerting system thanks
+   to its `alertmanager` that can run multiple copies in parallel
+   without sending duplicate alerts - `munin-limits` can only run on a
+   single server
+
+Puppet implementation
+---------------------
+
+Every node is configured as a `node-exporter` through the
+`roles::monitored` that is included everywhere. The role might
+eventually be expanded to cover alerting and other monitoring
+resources as well. This role, in turn, includes the
+`profile::prometheus::client` which configures each client correctly
+with the right firewall rules.
+
+The firewall rules are exported from the server, defined in
+`profile::prometheus::server`. We hacked around limitations of the
+upstream Puppet module to install Prometheus using backported Debian
+packages. The monitoring server itself is defined in
+`roles::monitoring`.
+
+The [Prometheus Puppet module][] was patched to [allow scrape job
+collection][] and [use of Debian packages for installation][]. Much of
+the initial Prometheus configuration was also documented in
+[ticket #29681][] and especially [ticket #29388][] which investigates
+storage requirements and possible alternatives for data retention
+policies.
+
+[ticket #29388]: https://trac.torproject.org/projects/tor/ticket/29388
+[ticket #29681]: https://trac.torproject.org/projects/tor/ticket/29681
+[use of Debian packages for installation]: https://github.com/voxpupuli/puppet-prometheus/pull/303
+[allow scrape job collection]: https://github.com/voxpupuli/puppet-prometheus/pull/304
+[Prometheus Puppet module]: https://github.com/voxpupuli/puppet-prometheus/

tsa/howto/puppet.mdwn

+Listing all hosts under puppet
+==============================
+
+This will list all active hosts known to the Puppet master:
+
+    ssh -t pauli.torproject.org 'sudo -u postgres psql puppetdb -P pager=off -A -t -c "SELECT c.certname FROM certnames c WHERE c.deactivated IS NULL"'
+
+The following will list all hosts under Puppet and their `virtual`
+value:
+
+    ssh -t pauli.torproject.org "sudo -u postgres psql puppetdb -P pager=off -F',' -A -t -c \"SELECT c.certname, value_string FROM factsets fs INNER JOIN facts f ON f.factset_id = fs.id INNER JOIN fact_values fv ON fv.id = f.fact_value_id INNER JOIN fact_paths fp ON fp.id = f.fact_path_id INNER JOIN certnames c ON c.certname = fs.certname WHERE fp.name = 'virtual' AND c.deactivated IS NULL\""  | tee hosts.csv
+
+The resulting file is a Comma-Seperated Value (CSV) file which can be
+used for other purposes later.
+
+Possible values of the `virtual` field can be obtain with a similar
+query:
+    
+    ssh -t pauli.torproject.org "sudo -u postgres psql puppetdb -P pager=off -A -t -c \"SELECT DISTINCT value_string FROM factsets fs INNER JOIN facts f ON f.factset_id = fs.id INNER JOIN fact_values fv ON fv.id = f.fact_value_id INNER JOIN fact_paths fp ON fp.id = f.fact_path_id WHERE fp.name = 'virtual';\""
+
+The currently known values are: `kvm`, `physical`, and `xenu`.
+
+## Other ways of extracting a host list
+
+ * Using the [PuppetDB API][]:
+
+        curl -s -G http://localhost:8080/pdb/query/v4/facts  | jq -r ".[].certname"
+
+   The [fact API][] is quite extensive and allows for very complex
+   queries. For example, this shows all hosts with the `apache2` fact
+   set to `true`:
+
+        curl -s -G http://localhost:8080/pdb/query/v4/facts --data-urlencode 'query=["and", ["=", "name", "apache2"], ["=", "value", true]]' | jq -r ".[].certname"
+
+ * Using [Cumin][], see below
+
+[Cumin]: https://doc.wikimedia.org/cumin/master/introduction.html
+
+ * Using LDAP:
+ 
+        HOSTS=$(ssh alberti.torproject.org 'ldapsearch -h db.torproject.org -x -ZZ -b dc=torproject,dc=org -LLL "hostname=*.torproject.org" hostname | awk "\$1 == \"hostname:\" {print \$2}" | sort')
+        for i in `echo $HOSTS`; do mkdir hosts/x-$i 2>/dev/null || continue; echo $i; ssh $i ' ...'; done
+
+    the mkdir is so that I can run the same command in many terminal
+    windows and each host gets only one once
+
+ [PuppetDB API]: https://puppet.com/docs/puppetdb/4.3/api/index.html
+ [fact API]: https://puppet.com/docs/puppetdb/4.3/api/query/v4/facts.html
+
+Batch jobs on all hosts
+=======================
+
+With that trick, a job can be ran on all hosts with
+[parallel-ssh](https://code.google.com/archive/p/parallel-ssh/), for example, check the uptime:
+
+    cut -d, -f1 hosts.hsv | parallel-ssh -i -h /dev/stdin uptime
+
+This would do the same, but only on physical servers:
+
+    grep 'physical$' hosts.hsv | cut -d -f1 | parallel-ssh -i -h /dev/stdin uptime
+
+This would fetch the `/etc/motd` on all machines:
+
+    cut -d -f1 hosts.csv | parallel-slurp -h /dev/stdin -L motd /etc/motd motd
+
+To run batch commands through sudo that requires a password, you will need to fool both sudo and ssh a little more:
+
+    cut -d -f1 hosts.csv | parallel-ssh -P -I -i -x -tt -h /dev/stdin -o pvs sudo pvs
+
+You should then type your password then Control-d. Warning: this will
+show your password on your terminal and probably in the logs as well. 
+
+## Using Cumin
+
+You can also use [Cumin][] to operate arbitrary shell commands on
+Puppet hosts or a subset of hosts. First, install Cumin and setup a
+tunnel to connect to the Puppet locally:
+
+    virtualenv --python=python3 ~/.venv/cumin
+    ~/.venv/cumin/bin/pip3 install cumin
+    ssh -L8080:localhost:8080 pauli.torproject.org
+
+Notice how Cumin is installed in a [Python virtualenv](https://virtualenv.pypa.io/): it is not
+yet [in Debian](https://bugs.debian.org/924685). You'll also need a patch to enable plain HTTP
+access, see [this bug report](https://phabricator.wikimedia.org/T218441). You might also want to disable the
+[root check](https://phabricator.wikimedia.org/T218440) as well. Then drop the following configuration in
+`~/.config/cumin/config.yaml`:
+
+    transport: clustershell
+    puppetdb:
+        host: localhost
+        scheme: http
+        port: 8080
+        api_version: 4  # Supported versions are v3 and v4. If not specified, v4 will be used.
+    log_file: cumin.log
+    default_backend: puppetdb
+
+From here on we'll assume you use the following alias:
+
+    alias cumin="~/.venv/cumin/bin/cumin --config ~/.config/cumin/config.yaml"
+
+This will run the `uptime` command on all hosts:
+
+    cumin '*' uptime
+
+To run against only a subset, you need to use the Cumin grammar, which
+is [briefly described in the Wikimedia docs](https://wikitech.wikimedia.org/wiki/Cumin#PuppetDB_host_selection). For example, this
+will run the same command only on physical hosts:
+
+    cumin 'F:virtual=physical' uptime
+
+Just check the monitoring server:
+
+    cumin 'R:class=roles::monitoring' uptime
+
+Any Puppet fact or class can be queried that way. This also serves as
+a ad-hoc interface to query PuppetDB for certain facts, as you don't
+have to provide a command. In that case, `cumin` runs in "dry mode"
+and will simply show which hosts match the request:
+
+    $ cumin 'F:virtual=physical'
+    16 hosts will be targeted:
+    [...]

tsa/howto/raid.mdwn

+Hardware RAID
+=============
+
+Some TPO machines have hardware RAID with `megaraid`
+controllers. Those are controlled with the `MegaCLI` command that is
+... rather hard to use.
+
+First, alias the megacli command because the package (derived from the
+upstream RPM by Alien) installs it in a strange location:
+
+    alias megacli=/opt/MegaRAID/MegaCli/MegaCli
+
+This will confirm you are using hardware raid:
+
+    root@moly:/home/anarcat# lspci | grep -i raid
+    05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05)
+
+This will show the RAID levels of each enclosure, for example this is
+RAID-10:
+
+    root@moly:/home/anarcat# megacli -LdPdInfo -aALL | grep "RAID Level"
+    RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
+
+This lists a summary of all the disks, for example the first disk has
+failed here:
+
+    root@moly:/home/anarcat# megacli -PDList -aALL | grep -e '^Enclosure' -e '^Slot' -e '^PD' -e '^Firmware' -e '^Raw' -e '^Inquiry'
+    Enclosure Device ID: 252
+    Slot Number: 0
+    Enclosure position: 0
+    PD Type: SAS
+    Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
+    Firmware state: Failed
+    Inquiry Data: SEAGATE ST3600057SS     [REDACTED]
+    Enclosure Device ID: 252
+    Slot Number: 1
+    Enclosure position: 0
+    PD Type: SAS
+    Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
+    Firmware state: Online, Spun Up
+    Inquiry Data: SEAGATE ST3600057SS     [REDACTED]
+    Enclosure Device ID: 252
+    Slot Number: 2
+    Enclosure position: 0
+    PD Type: SAS
+    Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
+    Firmware state: Online, Spun Up
+    Inquiry Data: SEAGATE ST3600057SS     [REDACTED]
+    Enclosure Device ID: 252
+    Slot Number: 3
+    Enclosure position: 0
+    PD Type: SAS
+    Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
+    Firmware state: Online, Spun Up
+    Inquiry Data: SEAGATE ST3600057SS     [REDACTED]
+
+This will make the drive blink (slot number 0 in enclosure 252):
+
+    megacli -PdLocate -start -physdrv[252:0] -aALL
+
+SMART monitoring
+----------------
+
+Some servers will fail to properly detect disk drives in their SMART
+configuration. In particular, `smartd` does not support:
+
+ * virtual disks (e.g. `/dev/nbd0`)
+ * MMC block devices (e.g. `/dev/mmcblk0`, commonly found on ARM
+   devices)
+ * out of the box, CCISS raid devices (e.g. `/dev/cciss/c0d0`)
+
+The latter can be configured with the following snippet in
+`/etc/smartd.conf`:
+
+    #DEVICESCAN -d removable -n standby -m root -M exec /usr/share/smartmontools/smartd-runner
+    DEFAULT -n standby -m root -M exec /usr/share/smartmontools/smartd-runner
+    /dev/cciss/c0d0 -d cciss,0
+    /dev/cciss/c0d0 -d cciss,1
+    /dev/cciss/c0d0 -d cciss,2
+    /dev/cciss/c0d0 -d cciss,3
+    /dev/cciss/c0d0 -d cciss,4
+    /dev/cciss/c0d0 -d cciss,5
+
+Notice how the `DEVICESCAN` is commented out to be replaced by the
+CCISS configuration. One line for each drive should be added (and no,
+it does not autodetect all drives unfortunately). This hack was
+deployed on `listera` which uses that hardware RAID.
+
+Other hardware RAID controllers are better supported. For example, the
+`megaraid` controller on `moly` was correctly detected by `smartd`
+which accurately found a broken hard drive.
+
+References
+----------
+
+Here are some external documentation links:
+
+ * <https://cs.uwaterloo.ca/twiki/view/CF/MegaRaid>
+ * <https://raid.wiki.kernel.org/index.php/Hardware_Raid_Setup_using_MegaCli>
+ * <https://sysadmin.compxtreme.ro/how-to-replace-an-lsi-raid-disk-with-megacli/>

tsa/howto/retire-a-host.creole

 = Decommissioning a host =
 
 * remove the host from tor-nagios/config/nagios-master.cfg
-* stop the VM: virsh destroy $host
-* undefine it: virsh undefine $host
-* if applicable, possibly only on moly.tpo: remove the LVM logical volumes
-* remove it from ud-ldap
-* on pauli:
-** puppet node clean $host.torproject.org
-** puppet node deactive $host.torproject.org
+* if applicable, stop the VM: virsh destroy $host
+* if applicable, undefine it: virsh undefine $host
+* if applicable, remove the LVM logical volumes or virtual disk files
+* remove it from ud-ldap: the host entry and any @<host> group memberships there might be as well as any sudo passwords users might have configured for that host
+* if it has any associated records in tor-dns/domains or auto-dns, remove it from there too
+* on pauli:   read x ; puppet node clean $x.torproject.org && puppet node deactivate $x.torproject.org
+* grep the tor-puppet repo for the host (and maybe its IP addresses) and clean up

tsa/old.mdwn

+This section is all old documentation imported from alberti.
+
+[[!map pages="tsa/old/*"]]

tsa/old/backuppc.mdwn

+
+on serra:
+
+in /etc/backuppc:
+
+- edit hosts, add the host in question
+- create a file called $host.pl,
+  . include the common file,
+  . list shares to be backed up
+  (use an existing file as template)

tsa/old/nagios-client.mdwn

+
+sudo apt-get update &&
+sudo apt-get install tor-nagios-checks &&
+sudo sed -i -e 's/^#DAEMON/DAEMON/' /etc/default/nagios-nrpe-server &&
+sudo env -i /etc/init.d/nagios-nrpe-server restart &&
+if [ -e '/etc/firehol/firehol.conf' ] ; then
+        echo 'You might want to add these to /etc/firehol/firehol.conf:'
+        echo
+        cat << EOF
+server custom monit tcp/4949 default accept src 38.229.70.32
+server custom nrpe tcp/5666 default accept src 38.229.70.32
+server ntp accept src 38.229.70.32
+server icmp accept src 38.229.70.32
+EOF
+fi
+

tsa/old/new-kvm-virsh-vm.mdwn

+#!/bin/bash
+
+# RUN ON moly
+#############
+
+set -e
+set -u
+
+VG=vg0
+SUITE=wheezy
+MIRROR=http://ftp.us.debian.org/debian
+#SPONSOR="xx"
+
+check_installed() {
+    local p
+    for p in "$@"; do
+        if ! dpkg -l "$p" 2>/dev/null | grep -q '^ii'; then
+            echo >&2 "Error: package $p not installed:"
+            echo >&2 "  apt-get install $*"
+            exit 1
+        fi
+    done
+}
+get_sshkey_fprs() {
+    local f
+    for f in etc/ssh/ssh_host*_key.pub; do
+        echo -n "  "
+        ssh-keygen -l -f "$f"
+    done
+}
+cleanup() {
+    set +x
+    if [ -n "$disk" ]; then
+        echo "Maybe:"
+        echo "lvremove $disk"
+    fi
+    echo
+    if [ -n "${target:-}" ] && [ -e "$target" ]; then
+        cd /
+        if [ "$(stat -f / -c %i)" != "$(stat -f "$target" -c %i)" ]; then
+            umount "$target"
+        fi
+        rmdir "$target"
+    fi
+
+    if [ -n "${part1:-}" ]; then
+        kpartx -d -p -p -v "$disk"
+    fi
+}
+check_installed debootstrap debian-archive-keyring kpartx ipv6calc
+
+echo -n "New VM's name: "
+if [ -n "${1:-}" ]; then echo "$1"; guest="$1"; shift; else read guest; fi
+echo
+echo -n "Disk size: [8g]"
+if [ -n "${1:-}" ]; then echo "$1"; disksize="$1"; shift; else read disksize; fi
+disksize=${disksize:-8g}
+
+trap cleanup EXIT
+
+LV="$guest"
+lvcreate -L "$disksize" -n "$LV" "$VG"
+
+disk="/dev/mapper/$VG-$(echo $guest | sed -e 's/-/--/')"
+target="/mnt/target-$guest"
+
+if ! [ -e "$disk" ]; then
+    echo >&2 "Error: Disk $disk does not exist."
+    exit 1
+fi
+if [ -e "$target" ]; then
+    echo >&2 "Error: Directory $target already exists."
+    exit 1
+fi
+
+if [ "$(head -c 65536 "$disk" | sha1sum | awk '{print $1}')" != "1adc95bebe9eea8c112d40cd04ab7a8d75c4f961" ]; then
+    echo -n "Warning: Disk appears to be not be empty.  Continue anyway? [y/N] "
+    read ans
+    [ "$ans" = "y" ] || exit 0
+fi
+
+echo -n "ipaddr: "
+if [ -n "${1:-}" ]; then echo "$1"; ipaddr="$1"; shift; else read ipaddr; fi
+netmask=255.255.255.192
+gateway=38.229.72.1
+ip6addr=2620:0:6b0:b:1a1a::${ipaddr}
+ip6gateway=2620:0:6b0:b::1
+
+set -x
+
+if [ -e "$disk-part1" ]; then
+    # we already had a partition table and udev/kpartx got it at boot time
+    kpartx -v -d -p -part $disk
+fi
+echo '2048,,L,*' | sfdisk -u S --Linux "$disk"
+kpartx -v -p -p -a "$disk"
+part1="${disk}-p1"
+mkfs.ext4 "$part1"
+
+mkdir "$target"
+mount "$part1" "$target"
+cd "$target"
+
+debootstrap --variant=minbase --keyring=/usr/share/keyrings/debian-archive-keyring.gpg "$SUITE" . "$MIRROR"
+
+### Set up swap and fstab
+dd if=/dev/zero of=swapfile bs=1024k count=512
+chmod 0 swapfile
+mkswap ./swapfile
+
+uuidroot=$(blkid -s UUID -o value ${part1}) &&
+cat > etc/fstab << EOF
+UUID=$uuidroot    /               ext4   errors=remount-ro 0       1
+/swapfile none swap sw 0 0
+EOF
+echo 'RAMTMP=yes' >> etc/default/tmpfs
+
+### Set up basic networking stuff
+echo "$guest" > etc/hostname
+cat > etc/hosts << EOF
+127.0.0.1       localhost
+
+# The following lines are desirable for IPv6 capable hosts
+::1     localhost ip6-localhost ip6-loopback
+fe00::0 ip6-localnet
+ff00::0 ip6-mcastprefix
+ff02::1 ip6-allnodes
+ff02::2 ip6-allrouters
+ff02::3 ip6-allhosts
+EOF
+rm -fv etc/udev/rules.d/70-persistent-*
+mkdir -p etc/udev/rules.d/
+touch etc/udev/rules.d/75-persistent-net-generator.rules
+
+cat > etc/network/interfaces << EOF
+auto lo
+iface lo inet loopback
+
+auto eth0
+iface eth0 inet static
+    address $ipaddr
+    netmask $netmask
+    up   ip route add $gateway/32 dev \$IFACE
+    up   ip route add default via $gateway dev \$IFACE
+    down ip route del default via $gateway dev \$IFACE
+    down ip route del $gateway/32 dev \$IFACE
+iface eth0 inet6 static
+    address $ip6addr
+    gateway $ip6gateway
+    netmask 64
+    accept_ra 0
+EOF
+
+cat > etc/resolv.conf << EOF
+nameserver 8.8.8.8
+search torproject.org
+EOF
+
+### A couple packages
+mv etc/apt/sources.list etc/apt/sources.list.d/debian.list
+chroot . apt-get update
+echo "Apt::Install-Recommends 0;" > etc/apt/apt.conf.d/local-recommends
+chroot . apt-get install -y locales-all net-tools iproute ifupdown dialog vim netbase udev psmisc usbutils pciutils
+
+### Set up kernel and bootloader
+chroot . apt-get install -y linux-image-amd64
+DEBIAN_FRONTEND=noninteractive chroot . apt-get install -y grub2
+
+! [ -e dev/vda ]
+! [ -e dev/vda1 ]
+cp -av `readlink -f "$disk"` dev/new-root
+cp -av `readlink -f "$part1"` dev/new-root1
+chroot . grub-install --modules=part_msdos /dev/new-root
+rm -v dev/new-root*
+
+cp -av `readlink -f "$disk"` dev/vda
+cp -av `readlink -f "$part1"` dev/vda1
+rm boot/grub/device.map
+chroot . update-grub
+rm -v dev/vda*
+
+rootpw="$(head -c 12 /dev/urandom | base64)"
+echo "root:$rootpw" | chroot . chpasswd
+
+### install ssh
+chroot . apt-get install -y ssh
+sed -i -e "s/`hostname`\$/$guest/" etc/ssh/ssh_host*_key.pub
+sshkeys="$(get_sshkey_fprs)"
+rsahostkey="$(cat etc/ssh/ssh_host_rsa_key.pub)"
+
+### clean up
+trap - EXIT
+cleanup
+
+
+echo "$guest's root password is $rootpw"
+echo "SSH host key fingerprints are:"
+echo "$sshkeys"
+echo "IP addresses:"
+echo "  $ipaddr"
+echo "  $ip6addr"
+
+echo
+echo "ud-ldap ldpavi snippet:"
+cat << EOF
+add host=$guest,ou=hosts,dc=torproject,dc=org
+host: $guest
+hostname: $guest.torproject.org
+objectClass: top
+objectClass: debianServer
+l: Germany
+distribution: Debian GNU/Linux
+access: restricted
+admin: torproject-admin@torproject.org
+architecture: amd64
+sshRSAHostKey: $rsahostkey
+ipHostNumber: $ipaddr
+ipHostNumber: `ipv6calc --in ipv6addr --out ipv6addr $ip6addr`
+mXRecord: 0 INCOMING-MX
+physicalHost: `hostname -f`
+description: XXX
+purpose: XXX
+
+EOF
+#sponsor: $SPONSOR
+
+echo "Maybe run this now: "
+echo " cd /etc/tor-virt && git pull && virsh define `hostname`/$guest.xml && virsh autostart $guest"
+echo "And then:"
+echo " virsh start $guest"

tsa/old/new-machine-cymru.mdwn

+After installing from our tor iso:
+
+- set up dedicated /srv disk, if we have one
+
+- make a swapfile (either on / or on /srv if that's a dedicated FS)
+  cd /srv && [ -e lost+found ] &&
+  dd if=/dev/zero of=swapfile bs=1024k count=4096 &&
+  chmod 0 swapfile &&
+  mkswap swapfile &&
+  echo "`readlink -f swapfile` none swap sw 0 0" >> /etc/fstab &&
+  swapon -a
+
+- follow normal new-machine docs

tsa/old/new-machine-nondebian.mdwn

+for fc14:
+
+11-03-08 23:11:34 <weasel> I cloned the git repo into usr/local/, added two symlinks to local/sbin,
+11-03-08 23:11:41 <weasel> copied the config file to /etc/userdir-ldap
+11-03-08 23:11:47 <weasel> modified nsswitch.conf,
+11-03-08 23:11:53 <weasel> modified the pam stuff (added mkhomedir)
+11-03-08 23:12:12 <weasel> created a symlink from /var/db/.... to ../lib/misc/foo
+11-03-08 23:12:32 <weasel> disabled selinux, so ssh can read the key files
+11-03-08 23:12:38 <weasel> update ssh config
+11-03-08 23:12:40 <weasel> fix timezone
+11-03-08 23:12:52 <weasel> and created the zsh symlink
+11-03-08 23:13:03 <weasel> I think that's it.  maybe I forgot something
+
+
+
+* fix up hostname.  It needs to have the fully qualified hostname in 'hostname -f'
+
+cd /usr/local && git clone https://git.torproject.org/admin/userdir-ldap.git
+
+cd /usr/local/sbin && ln -s ../ln -s ../userdir-ldap/ud-config ../userdir-ldap/ud-replicate .
+
+mkdir /etc/userdir-ldap && cd /etc/userdir-ldap && cp /usr/local/userdir-ldap/userdir-ldap.conf .
+
+echo alberti.torproject.org,alberti,db.torproject.org,db,38.229.70.7 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqFvZsXVYuzrgDO7IbBjeBO5WKk+sXmb0rRzPcSwIRTaMS4h3QdLDG1VjeNA5CUeAjTOYC0hAWZiXzfsl4u0KwNJUWRGZCclbIt6V7Tk74mM0405A+y0JP3NwUnTevcRcVxiDo8mrI85y5MXvipaWnPdGYayL09h9EeNDzBVKNZooCeKQBqkejhH69gyy4gdN9HgfMep3uOInyjr86W49pZ4n7CXoVt8QkTWtoBX/qPHK8igqX/dcYkOgCclVYRrQ1G4FbxEOGD+QzwPnCGDWCUgapFXoqh7HpG0Xfg5iDXGFcIu1JgDdr/SFJkr6hmYjW0gmkge0ihGj7GZ6onWhzQ== root@alberti > /etc/ssh/ssh_known_hosts &&
+ud-replicate
+
+maybe change chown in ud-replicate
+
+cd /var/db && ln -s ../lib/misc/group.db ../lib/misc/passwd.db ../lib/misc/shadow.db .
+
+edit /etc/nsswitch.conf to read:
+ | passwd:     files db
+ | shadow:     files db
+ | group:      db files
+
+check if 'id weasel' works
+
+disable selinux
+
+include pam mkhomedir for ssh and su.
+
+add to sshd_config:
+AuthorizedKeysFile /etc/ssh/userkeys/%u
+AuthorizedKeysFile2 /var/lib/misc/userkeys/%u
+
+crontab:
+cat > /etc/cron.d/ud-replicate << EOF
+PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+10,25,40,55 * * * * root ud-replicate
+EOF
+
+sudoers

tsa/old/new-machine.orig.mdwn

+This is probably not relevant anymore. When the next host is created,
+review these docs and add what's missing to [[new-machine]].
+
+* set a hostname - pick an unused one from https://svn.torproject.org/svn/projects/misc-sysadmin/onion-names.txt
+* sane base setup
+    echo "Apt::Install-Recommends 0;" > /etc/apt/apt.conf.d/local-recommends &&
+    apt-get install locales-all rsync sudo zsh subversion git-core mtr-tiny ntp &&
+    cat /dev/null > /etc/default/locale
+
+* fix TZ
+    echo 'Etc/UTC' > /etc/timezone &&
+    dpkg-reconfigure tzdata -pcritical -fnoninteractive
+
+* ssh setup
+    cd /etc/ssh/ && rm -f ssh_host_dsa_key ssh_host_dsa_key.pub ssh_host_key ssh_host_key.pub &&
+    mkdir -p /etc/ssh/userkeys && ln -s /root/.ssh/authorized_keys /etc/ssh/userkeys/root &&
+    sed -i -e 's/^HostKey.*_dsa_key/# &/;
+               s/^X11Forwarding yes/X11Forwarding no/;
+               $ a AuthorizedKeysFile /etc/ssh/userkeys/%u
+               $ a AuthorizedKeysFile2 /var/lib/misc/userkeys/%u' sshd_config &&
+    (cd / && env -i /etc/init.d/ssh restart)
+* re-key ssh
+    cd /etc/ssh/ && rm -f ssh_host_rsa_key ssh_host_rsa_key.pub &&
+    dpkg-reconfigure openssh-server
+
+* torproject sources list entry:
+
+sudo apt-key add - << EOF &&
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+Version: GnuPG v1.4.9 (GNU/Linux)
+
+mQENBEvgZPYBCADQeFoNmFWv156s+JPiUv6vFZb1sm3sx5g61Wel38MYgEuYEdan
+mFnULzdRc5ScCqXD9iC7vJtAFWv9xobQkpffy8uQNAL6Dom/4A4z8Ywhdt8qwZWt
+qeJQ5HSv/ollXW1jd5B+VCFaLh70PMbooitq8F5uBbVhFzvd4XxbBIWw2PzFzBbI
+0daBpEdwjrtNH/E+M+ZQLMtaYyTZ1vMx+KmP2hrWtKyK4ZLmr+/2rxmoJrFGQwmp
+uBohXRHMrekrdbHPfJHPXqj4SgpP9DRj2MPemQLRByHX6Hll6xy0GKkBhg1Em5Qr
+GCCFXIiSS/kP16f7hpyBxke859m/RXLzCHHDABEBAAG0I2RiLnRvcnByb2plY3Qu
+b3JnIGFyY2hpdmUga2V5IDIwMTAtiQE8BBMBAgAmBQJL4GT2AhsDBQkFo5qABgsJ
+CAcDAgQVAggDBBYCAwECHgECF4AACgkQwsdoQg4eEkBqFAf8DtnZo0flz0IkmKDU
+D1FBAl6SHE5HN7f57mW/0CLMSvWohSKIouSBJH4dUTM8484Z15ikSRW9urzv9dsW
+w24+9EEaxBBVJqoJIMZmvqaM452kZ/zwQR4NBIGxhSJ8UblpQ0gttMB90oVoAx9a
+2erJUD8sRwCxcwPTE3fQMJZEu6oB5jIPnQQAPOznMO19CJmnZIlzWPALFC3NPRSX
+QFEZPO9CGHzpB4UDzpoBctTpTfHot33ep1c5qaLfRkmTIdImqNe2gRykglHXHCa5
+BLU4M6In3gMIoeUFeRzbE7eTm1j7NDUG3EbQf5aguRSWMWbIGWAnZdTH5ZhzSb72
+fVoq6g==
+=dBbT
+-----END PGP PUBLIC KEY BLOCK-----
+EOF
+if ! [ -e /etc/apt/sources.list.d/db.torproject.org.list ] ; then
+        echo 'deb     http://db.torproject.org/torproject-admin          lenny            main' | sudo tee /etc/apt/sources.list.d/db.torproject.org.list
+fi
+
+* install userdir-ldap
+    apt-get update &&  apt-get install userdir-ldap
+
+* fix nsswitch for ud fu.  (you might have to restart sshd here)
+    sed -i -e 's/^passwd:[[:space:]]\+compat$/passwd:         compat db/;
+              s/^group:[[:space:]]\+compat$/group:          db compat/;
+              s/^shadow:[[:space:]]\+compat$/shadow:         compat db/' \
+        /etc/nsswitch.conf
+    (cd / && env -i /etc/init.d/ssh restart)
+
+* add pam_mkhomedir to common-session:
+    grep pam_mkhomedir /etc/pam.d/common-session || \
+    echo "session optional        pam_mkhomedir.so skel=/etc/skel umask=0022" >> /etc/pam.d/common-session
+
+* setup sudo
+    grep '^%adm' /etc/sudoers || echo '%adm    ALL=(ALL) ALL' >> /etc/sudoers
+    grep '^%adm.*apt-get' /etc/sudoers || echo '%adm    ALL=(ALL) NOPASSWD: /usr/bin/apt-get update, /usr/bin/apt-get dist-upgrade, /usr/bin/apt-get clean' >> /etc/sudoers
+
+* add host to ud-ldap
+  : on alberti : && sudo -u sshdist ud-generate && sudo -H ud-replicate
+
+* fix resolver
+  sed -i -e 's/search localdomain/search torproject.org/' /etc/resolv.conf
+
+* do one ud-replicate:
+  echo alberti.torproject.org,alberti,db.torproject.org,db,38.229.70.7 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqFvZsXVYuzrgDO7IbBjeBO5WKk+sXmb0rRzPcSwIRTaMS4h3QdLDG1VjeNA5CUeAjTOYC0hAWZiXzfsl4u0KwNJUWRGZCclbIt6V7Tk74mM0405A+y0JP3NwUnTevcRcVxiDo8mrI85y5MXvipaWnPdGYayL09h9EeNDzBVKNZooCeKQBqkejhH69gyy4gdN9HgfMep3uOInyjr86W49pZ4n7CXoVt8QkTWtoBX/qPHK8igqX/dcYkOgCclVYRrQ1G4FbxEOGD+QzwPnCGDWCUgapFXoqh7HpG0Xfg5iDXGFcIu1JgDdr/SFJkr6hmYjW0gmkge0ihGj7GZ6onWhzQ== root@alberti > /etc/ssh/ssh_known_hosts &&
+  ud-replicate
+
+* apply phobos' sudo defaults
+  sed -i -e '
+          /^Defaults/ a Defaults mail_badpass\
+Defaults mail_no_host\
+Defaults mail_no_perms\
+Defaults tty_tickets\
+Defaults insults\
+Defaults !lecture
+      ' /etc/sudoers
+* try to become root using sudo.
+
+* disable password auth with ssh (again: once you verified you can log in and become root using keys.)
+    #vi /etc/ssh/sshd_config
+    #  | PasswordAuthentication no
+
+    if grep '^PasswordAuthentication' /etc/ssh/sshd_config; then
+      sed -i -e 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config;
+    else
+      sed -i -e '$ a PasswordAuthentication no' /etc/ssh/sshd_config
+    fi &&
+    (cd / && env -i /etc/init.d/ssh restart)
+
+* get rid of unneeded network services:
+
+  dpkg --purge portmap nfs-common
+
+  dpkg --purge exim4 exim4-base exim4-config exim4-daemon-light  at bsd-mailx
+  userdel -r Debian-exim
+
+* install postfix
+  apt-get install postfix postfix-cdb bsd-mailx
+
+  rm /etc/mailname
+
+  cat > /etc/postfix/main.cf << 'EOF'
+# See /usr/share/postfix/main.cf.dist for a commented, more complete version
+
+mydomain = torproject.org
+myorigin = $myhostname
+smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
+biff = no
+
+# appending .domain is the MUA's job.
+append_dot_mydomain = no
+
+# Uncomment the next line to generate "delayed mail" warnings
+#delay_warning_time = 4h
+
+readme_directory = no
+
+# TLS parameters
+smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
+smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
+smtpd_use_tls=yes
+smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
+smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
+smtp_use_tls=yes
+
+# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
+# information on enabling SSL in the smtp client.
+
+alias_maps =
+        hash:/etc/aliases
+        cdb:/var/lib/misc/thishost/mail-forward
+alias_database = hash:/etc/aliases
+mydestination = $myhostname
+                localhost.$mydomain
+                localhost
+relayhost =
+mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
+mailbox_command = procmail -a "$EXTENSION"
+mailbox_size_limit = 0
+recipient_delimiter = +
+inet_interfaces = all
+inet_protocols = all
+EOF
+
+   env -i /etc/init.d/postfix restart
+
+   sed -i -e 's/^root:.*/root: torproject-admin@torproject.org/' /etc/aliases && newaliases
+
+
+* install root admin key
+  echo 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEA2ToYPcCcP+/+E8dVlorOeeQpAauQN2rHF8LBy0BPzY9W4x0eMkPI5r5vDSN48+w9W/SELTd1ZFy7d2n6bjt06W0+78YwLeRN4wOoyiryh3yd2Kiy74Plb81RRx+Px9erh0mpMF9krnWfGyrr+q8be2ACWNHLiSUyx5PC5QmetBH+Ug7O+NpAL/grmLgdzkOEzaatrI+re1RyfwvJQ6ZC3Wc+KKUOVvB6zoRk9qegUydvx1NX5SrfDQrBzFBv6vcL0N1TESxDx8Pu6uFFsuWWBlhb3AKZKJEtLV8AqGQ007A3oK5PP/fdUvgKOcbw3bWXZKy/dgcH+httQPPt8brmXtRvNgXozsH6soegnoWA/opNVHcWwLjfk9mFMGLvY5zzX26xg9sVnBK1dI6geqGhv20oWIcIqEhmFr0HQnCIE9tMRb6jDmzymXoX2gnR2nfgpg81jLFSYm/BJYjISVrLAFy50N+4HcJ/VYTO/KBVoERfkz88LK5p2zOQNdMNSN4WIGizl6G7lQrFAZAkdk2dfaIA/PkxVXG0/Rfh+R2z94UCiAfc06FzkYRfFreThpgPWI4B/QFcBQ0MsxuQzl+ea3jRVKRGyKHpwRtjh2Ebt0YfCpyn1ZfU+enLIYJ0TmpC6HXpWiSc8qSaP6gMF0ULqZUtip1NfJiaEkUyhR5XbIU= Peter Palfrader - torproject adm key (2010-01-09)' >> /root/.ssh/authorized_keys &&
+  wc -l /root/.ssh/authorized_keys
+
+* clean away broken firewall
+  rm -f /etc/network/if-pre-up.d/iptables /etc/iptables.rules /etc//iptables.up.rules
+  for j in INPUT FORWARD OUTPUT; do iptables -P $j ACCEPT; done; iptables -F
+
+* set new root password
+
+* sane editor
+  sudo apt-get install vim && sudo  update-alternatives --set editor /usr/bin/vim.basic
+
+
+* add more software
+apt-get install ferm git-core logwatch rkhunter munin-node sudo fail2ban htop etckeeper wget
+
+* configure the firewall
+* take ferm defaults, but need something for ferm here.
+
+* rkhunter
+rkhunter --update --propupd
+
+* fail2ban:
+/etc/init.d/fail2ban start
+
+* copy munin-node.conf from schmitzi to /etc/munin/
+* on new host:
+sudo /etc/init.d/munin-node restart
+* on schmitzi, add the host to /etc/munin/munin.conf

tsa/old/new-vm-rethem.mdwn

+= HOW I CREATED A NEW VM AT RETHEMHOSTING (CYMRU) FOR MUMBLE
+
+A quick guide for how to make VMs.
+
+18 Sep 2015
+
+== WHAT YOU WILL NEED
+
+ * Access to many git repositories
+ * Sudo access all over the place
+ * GPG key that can decrypt files in those git repositories
+ * Your ssh configuration file set up to use the appropriate
+   ProxyCommand magic to ssh into various .torproject.org hosts.
+ * Enough Unix shell skillz to be dangerous to yourself and others
+
+== THE STEPS
+
+ * Go to the approach for rethemhosting.  You can find it in
+   hosts-extra-info in the tor-passwords repository.
+
+   If your browser hates you when you do this, it's possible that
+   rethemhosting still hasn't upgraded their DH parameters to
+   something non-embarrassing. Disable the DHE ciphersuites and try
+   again.  Very sorry.  I emailed them about it.
+
+ * Pick an onion species name:
+
+   https://en.wikipedia.org/wiki/List_of_Allium_species
+
+   (I went with "tuburosum", since I have some of that in my
+   fridge. Try not to pick something that has the first 3 characters
+   in common with some other host.)
+
+ * In the rethemhosting interface, go to the list of instances, then
+   select "add instance" (currently towards the top right.)
+
+   It will walk you through a wizard.
+
+   1: Select ISO
+
+   2: Select "my ISOs", then tor-debian-7.8-20150221.
+
+      (If there is something more recent than that, ask weasel if it
+      is safe to use!)
+
+      You will have only one option for hypervisor.
+
+   3: Configure the CPU/RAM requirements as needed
+
+   4: Select 20 GB disk, always. More disks can get added later.
+
+   5: Is easy; nothing to do here.
+
+   6: Select 'Tor External Dual', then make it the default, then
+      deselect the original network.
+
+      Weasel explains: "one is I think an internal network that isn't
+      routed anywhere, and the two external things are two networks
+      that are in the internet but with different address space.  I
+      think the first tor external one is full."
+
+   7: Give it a name.  No group.
+
+  * Launch the VM!  It will boot from a custom install CD.
+
+  * Did you think you were done? Lolololollol11ol!
+
+  * Once the VM is running, click on it in the rethem interface, and
+    go to the NICs tab.  Learn the IP addresses (v4 and v6) and
+    gateways for the new VM.
+
+  * Edit the domains git repository.  (It is at
+    git@git-rw.torproject.org:admin/dns/domains.)
+
+    DO NOT EDIT torproject.org!!!!!!
+
+    Instead, add the reverse-dns records as appropriate to the *.arpa files.
+
+    ipv6cal can help with ipv6 reverse fu.  e.g.:
+
+      weasel@defiant:~$ ipv6calc --in ipv6addr --out revnibbles.arpa 2607:8500:5:1::5a2c
+      c.2.a.5.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.5.0.0.0.0.0.5.8.7.0.6.2.ip6.arpa.
+
+    Commit and push when you are done; magic will happen.
+
+  * Now you need to get console access to the new VM: Click the rightmost
+    icon on the details screen at the rethem cloudstack thing.
+
+    The debian installer is about to ask you for some advice:
+
+      - Tell it the IP address to use.
+      - Tell it a netmask
+      - Tell it a gateway.
+      - Tell it the hostname.
+
+    It might ask you if you want to create a swap partition.  You don't.
+
+    It will ask you if you want to write the new partition map to the
+    the disk.  You do.
+
+    Now it will do a debian installation!  This will take a while.
+
+  * You need to add the new host to nagios.
+
+    git clone git@git-rw.torproject.org:admin/tor-nagios
+
+    cd config
+
+    edit nagios-master.cfg
+
+    Add the new host right below the most recently added host.  By
+    default, you should put it in these hostgroups:
+           computers, syslog-ng-hosts, no-ntp-peer.
+
+    (Other hostgroups are documented at XXXXXXXXXXXXX.)
+
+    Run make.  Did that work?
+
+    If so, run make install.
+
+    Commit and push.
+
+  * ssh into your shiny new host!  (as root.)
+
+    (The root password is the one marked as "rethemhosting
+    torproject.org preseed debian-install image default root password"
+    in hosts-extra-info in tor-passwords.git.  Don't worry, we'll
+    change it.)
+
+    (Use -o HostKeyAlgorithms=ssh-rsa to avoid a warning later on.)
+
+  * If you got in, detach the ISO.
+
+    It's one of the icons in the cloudstack thing.  It looks like a paperclip.
+
+
+  * Get a shell on alberti, then run:
+
+
+    ldapvi -ZZ --encoding=ASCII --ldap-conf -h db.torproject.org -D uid=nickm,ou=users,dc=torproject,dc=org
+
+    You will need to use your ldap password.
+
+    Now you are editing the LDAP database!
+
+    Look for the last host= entry for a rethemhosting host in the file.
+
+    Copy that whole block to the end of the file, and replace the
+    number with the word "add".  Then set the hostname correctly, and
+    update the other stuff.  (like memory, ssh host key, ip addresses,
+    purpose.) drop or replace allowedGroups.
+
+    Use dpkg --print-architecture if you don't know what the architecture is.
+    (It's amd64.)
+
+    Save and quit, then say yes.
+
+
+  * Open an editor on the new host as root, and edit
+    /etc/network/interfaces.  Edit it to be a copy of the one on the
+    most recently created host, except fill in the correct values for
+    this host.  I got:
+
+========
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+allow-hotplug eth0
+iface eth0 inet static
+    address 154.35.132.152/28
+    gateway 154.35.132.145
+iface eth0 inet6 static
+    address 2607:8500:5:1::5a2c/64
+    gateway 2607:8500:5:1::1
+    accept_ra 0
+========
+
+    Your IP will vary.
+
+  * Edit /etc/resolv.conf and /etc/hosts so that they include the
+    proper domain.  (It is probably torproject.org, not
+    rethemhosting.)
+
+  * Reboot the new host (with shutdown -r now), and make sure it comes
+    up with the right IP.
+
+  * Now you are going to read new-machine-cymru on alberti!  It lives
+    in /src/db.torproject.org.
+
+    Follow its instructions _on the new host_ as root.
+
+  * You will be told to recursively follow the instructions in
+    new-machine on alberti, which lives in the same repository.  Do
+    so.
+
+    Some notes:
+         - You will need to be root for nearly all of this.
+         - If something just won't work, try puppet agent -t on the
+           new host again, and give it another try.
+         - Use weasel's magical pws scripts to manage the
+           tor-passwords repository.
+
+  * On the new host, if there is an /etc/apt/sources.list.d/debian.list,
+    remove /etc/apt/sources.list, then apt-get update.
+
+  * Run one last  "apt-get update && apt-get dist-upgrade && apt-get clean"
+
+  * Reboot again for good measure!
+
+  * (Unless you used -o HostKeyAlgorithms=ssh-rsa before:)
+    When you next ssh into the new host, the key will probably have
+    changed.  That's because it switched from ECDSA to RSA.  Don't
+    worry, but make sure you got the right one.
+
+NOW YOU HAVE A HOST!
+
+But, nobody can log into it.  That's sad.
+
+I wanted to set it up so that ioerror could log in and become a new
+"tormumble" role and administer the stuff.
+
+= So here's how I made the tormumble role and stuff!
+
+  * Do the magic thing on alberti again to edit the ldap database.
+
+    Create a role account and associated group in ldap - ldapvi and
+    copy one from before, picking a free uid/gid number pair.
+
+    Add the group to the allowedGroups for the host.
+
+    Add the group to the supplemenataryGids for the users who should
+    be in it.
+
+    Then, in puppet:
+
+       /usr/sbin/visudo -f ./modules/sudo/files/sudoers
+
+    And add a line of the form:
+
+       %rolename   hostname=(groupname)  ALL
+
+  * Then on the host, you need to log in as the role user to create
+    their homedir and stuff.  Do su - <role account> to do this, then
+    log out again.
+
+    If you couldn't log in, then the LDAP info hasn't propagated
+    yet. Run this on alberti:
+         sudo -u sshdist ud-generate
+    And this on the new host as root:
+         ud-replicate
+
+  * Move the home directory to the right place:
+
+      mkdir /srv/<roledirectoryservicewhatever>.torproject.org
+      mv /home/<roleuser> to /srv/$foo/home
+
+      , then replace the original location with a symlink
+
+      Also, chown /srv/<roledirectoryservicewhatever>.torproject.org
+      to the new <roleuser>:<rolegroup>