... | ... | @@ -80,7 +80,7 @@ from a sysadmin perspective. User documentation lives in [doc/static-sites](doc/ |
|
|
|
|
|
%torwww,%metrics STATICMASTER=(mirroradm) NOPASSWD: /usr/local/bin/static-master-update-component onionperf.torproject.org, /usr/local/bin/static-update-component onionperf.torproject.org
|
|
|
|
|
|
10. add to nagios monitoring, in `tor-nagios/config/nagios-master.cfg`:
|
|
|
10. add to Nagios monitoring, in `tor-nagios/config/nagios-master.cfg`:
|
|
|
|
|
|
-
|
|
|
name: mirror static sync - atlas
|
... | ... | @@ -117,7 +117,7 @@ from a sysadmin perspective. User documentation lives in [doc/static-sites](doc/ |
|
|
[...]
|
|
|
}
|
|
|
|
|
|
7. remove the sudo rules for the role user
|
|
|
7. remove the `sudo` rules for the role user
|
|
|
|
|
|
8. remove the home directory specified on the server (often
|
|
|
`staticiforme`, but can be elsewhere) and mirrors, for example:
|
... | ... | @@ -129,7 +129,7 @@ from a sysadmin perspective. User documentation lives in [doc/static-sites](doc/ |
|
|
9. consider removing the role user and group in LDAP, if there are no
|
|
|
files left owned by that user
|
|
|
|
|
|
10. remove from nagios, e.g.:
|
|
|
10. remove from Nagios, e.g.:
|
|
|
|
|
|
-
|
|
|
name: mirror static sync - atlas
|
... | ... | @@ -227,7 +227,7 @@ files and directories in the `tor-puppet.git` repository: |
|
|
* `roles::static_mirror` - a generic mirror, see
|
|
|
`staticsync::static_mirror` below
|
|
|
* `roles::static_mirror_web` - a web mirror, including most (but
|
|
|
not necessarily all) components defined in the YAMl
|
|
|
not necessarily all) components defined in the YAML
|
|
|
configuration. configures Apache (which the above
|
|
|
doesn't). includes `roles::static_mirror` (and therefore
|
|
|
`staticsync::static_mirror`)
|
... | ... | @@ -262,7 +262,7 @@ files and directories in the `tor-puppet.git` repository: |
|
|
* exports the SSH key to the mirrors and sources
|
|
|
* `staticsync::base`, included by all of the above, deploys:
|
|
|
* `/etc/static-components.conf`: a file derived from the
|
|
|
`static-components.yaml` config file
|
|
|
`static-components.yaml` configuration file
|
|
|
* `/etc/staticsync.conf`: polyglot (bash and Python)
|
|
|
configuration file propagating the `base` (currently
|
|
|
`/srv/static.torproject.org`, `masterbase` (currently
|
... | ... | @@ -275,22 +275,22 @@ not directly the `YAML` file shipped to hosts, in |
|
|
`staticsync::base`. See the `static-components.conf.erb` Puppet
|
|
|
template.
|
|
|
|
|
|
### Scripts walkthrough
|
|
|
### Scripts walk through
|
|
|
|
|
|
<!-- this is a reformatted copy of the `OVERVIEW` in the staticsync
|
|
|
puppet module -->
|
|
|
|
|
|
- `static-update-component` is run by the user on the **source** host.
|
|
|
|
|
|
If not run under sudo as the `staticuser` already, it sudos to the
|
|
|
`staticuser`, re-execing itself. It then SSH to the `static-master`
|
|
|
If not run under sudo as the `staticuser` already, it `sudo`'s to the
|
|
|
`staticuser`, re-executing itself. It then SSH to the `static-master`
|
|
|
for that component to run `static-master-update-component`.
|
|
|
|
|
|
LOCKING: none, but see `static-master-update-component`
|
|
|
|
|
|
- `static-master-update-component` is run on the **master** host
|
|
|
|
|
|
It rsyncs the contents from the **source** host to the static
|
|
|
It `rsync`'s the contents from the **source** host to the static
|
|
|
**master**, and then triggers `static-master-run` to push the
|
|
|
content to the mirrors.
|
|
|
|
... | ... | @@ -326,11 +326,11 @@ puppet module --> |
|
|
When instructed by `static-master-run`, we update the symlink and
|
|
|
remove the old tree.
|
|
|
|
|
|
`static-mirror-run` rsyncs either `-current-push` or `-current-live`
|
|
|
`static-mirror-run` `rsync`'s either `-current-push` or `-current-live`
|
|
|
for a component.
|
|
|
|
|
|
LOCKING: during all of `static-mirror-run`, we keep an exclusive
|
|
|
lock on the `<component>` dir, i.e., the directory that holds
|
|
|
lock on the `<component>` directory, i.e., the directory that holds
|
|
|
`tree-[ab]` and `cur`.
|
|
|
|
|
|
- `static-mirror-run-all`
|
... | ... | @@ -354,7 +354,7 @@ Python 2. |
|
|
|
|
|
### Authentication
|
|
|
|
|
|
Authentication between the static site hosts is entirely done through
|
|
|
The authentication between the static site hosts is entirely done through
|
|
|
SSH. The source hosts are accessible by normal users, which can `sudo`
|
|
|
to a "role" user which has privileges to run the static sync scripts
|
|
|
as sync user. That user then has privileges to contact the master
|
... | ... | @@ -391,7 +391,7 @@ file (`.serial`) to make sure everyone has the same copy of the site. |
|
|
## Logs and metrics
|
|
|
|
|
|
All tor webservers keep a minimal amount of logs. The IP address and
|
|
|
time (but not the date) are zero'd (`00:00:00`). The referer is
|
|
|
time (but not the date) are clear (`00:00:00`). The referrer is
|
|
|
disabled on the client side by sending the `Referrer-Policy
|
|
|
"no-referrer"` header.
|
|
|
|
... | ... | @@ -403,14 +403,14 @@ The IP addresses are replaced with: |
|
|
|
|
|
Logs are kept for two weeks.
|
|
|
|
|
|
Errrors may be sent by email.
|
|
|
Errors may be sent by email.
|
|
|
|
|
|
Metrics are scraped by [Prometheus](prometheus) using the "apache"
|
|
|
Metrics are scraped by [Prometheus](prometheus) using the "Apache"
|
|
|
exporter.
|
|
|
|
|
|
## Backups
|
|
|
|
|
|
The `source` hosts are backed up with [bacula](backups) without any special
|
|
|
The `source` hosts are backed up with [Bacula](backups) without any special
|
|
|
provision.
|
|
|
|
|
|
TODO: check if master / mirror nodes need to be backup. Probably not?
|
... | ... | @@ -443,7 +443,7 @@ difficult because the dsa-puppet and tor-puppet have disconnected |
|
|
histories. Even if they would have a common ancestor, the code is
|
|
|
spread in multiple directories, which makes it hard to track. There
|
|
|
has been some refactoring to move most of the code in a `staticsync`
|
|
|
module, but we still have files strewn over otehr modules.
|
|
|
module, but we still have files strewn over other modules.
|
|
|
|
|
|
The static mirror system was written for Debian.org by Peter
|
|
|
Palfrader. It has also been patches by other DSA members (Stephen
|
... | ... | @@ -509,4 +509,17 @@ of copies of the sites we have to keep around. |
|
|
|
|
|
* [GitLab pages](https://docs.gitlab.com/ee/administration/pages/) could be used as a source?
|
|
|
* the [cache system](cache) could be used as a replacement in the
|
|
|
frontend |
|
|
front-end
|
|
|
|
|
|
<!-- LocalWords: atomicity DDOS YAML Hiera webserver NFS CephFS TLS
|
|
|
-->
|
|
|
<!-- LocalWords: filesystem GitLab scalable frontend CDN HTTPS DNS
|
|
|
-->
|
|
|
<!-- LocalWords: howto Nagios SSL TOC dns letsencrypt sudo LDAP SLA
|
|
|
-->
|
|
|
<!-- LocalWords: rsync cron hostname symlink webservers Bacula DSA
|
|
|
-->
|
|
|
<!-- LocalWords: torproject debian TPO Palfrader Julien Cristau TPA
|
|
|
-->
|
|
|
<!-- LocalWords: LocalWords
|
|
|
--> |