Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
Wiki Replica
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container Registry
Model registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
The Tor Project
TPA
Wiki Replica
Commits
73d8679d
Unverified
Commit
73d8679d
authored
4 years ago
by
anarcat
Browse files
Options
Downloads
Patches
Plain Diff
spell-check
parent
ffa527c9
No related branches found
Branches containing commit
No related tags found
No related merge requests found
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
howto/static-component.md
+31
-18
31 additions, 18 deletions
howto/static-component.md
with
31 additions
and
18 deletions
howto/static-component.md
+
31
−
18
View file @
73d8679d
...
...
@@ -80,7 +80,7 @@ from a sysadmin perspective. User documentation lives in [doc/static-sites](doc/
%torwww,%metrics STATICMASTER=(mirroradm) NOPASSWD: /usr/local/bin/static-master-update-component onionperf.torproject.org, /usr/local/bin/static-update-component onionperf.torproject.org
10.
add to
n
agios monitoring, in
`tor-nagios/config/nagios-master.cfg`
:
10.
add to
N
agios monitoring, in
`tor-nagios/config/nagios-master.cfg`
:
-
name: mirror static sync - atlas
...
...
@@ -117,7 +117,7 @@ from a sysadmin perspective. User documentation lives in [doc/static-sites](doc/
[...]
}
7.
remove the sudo rules for the role user
7.
remove the
`
sudo
`
rules for the role user
8.
remove the home directory specified on the server (often
`staticiforme`
, but can be elsewhere) and mirrors, for example:
...
...
@@ -129,7 +129,7 @@ from a sysadmin perspective. User documentation lives in [doc/static-sites](doc/
9.
consider removing the role user and group in LDAP, if there are no
files left owned by that user
10.
remove from
n
agios, e.g.:
10.
remove from
N
agios, e.g.:
-
name: mirror static sync - atlas
...
...
@@ -227,7 +227,7 @@ files and directories in the `tor-puppet.git` repository:
*
`roles::static_mirror`
- a generic mirror, see
`staticsync::static_mirror`
below
*
`roles::static_mirror_web`
- a web mirror, including most (but
not necessarily all) components defined in the YAM
l
not necessarily all) components defined in the YAM
L
configuration. configures Apache (which the above
doesn't). includes
`roles::static_mirror`
(and therefore
`staticsync::static_mirror`
)
...
...
@@ -262,7 +262,7 @@ files and directories in the `tor-puppet.git` repository:
*
exports the SSH key to the mirrors and sources
*
`staticsync::base`
, included by all of the above, deploys:
*
`/etc/static-components.conf`
: a file derived from the
`static-components.yaml`
config file
`static-components.yaml`
config
uration
file
*
`/etc/staticsync.conf`
: polyglot (bash and Python)
configuration file propagating the
`base`
(currently
`/srv/static.torproject.org`
,
`masterbase`
(currently
...
...
@@ -275,22 +275,22 @@ not directly the `YAML` file shipped to hosts, in
`staticsync::base`
. See the
`static-components.conf.erb`
Puppet
template.
### Scripts walkthrough
### Scripts walk
through
<!-- this is a reformatted copy of the
`OVERVIEW`
in the staticsync
puppet module -->
-
`static-update-component`
is run by the user on the
**source**
host.
If not run under sudo as the
`staticuser`
already, it sudos to the
`staticuser`
, re-execing itself. It then SSH to the
`static-master`
If not run under sudo as the
`staticuser`
already, it
`
sudo
`
'
s to the
`staticuser`
, re-exec
ut
ing itself. It then SSH to the
`static-master`
for that component to run
`static-master-update-component`
.
LOCKING: none, but see
`static-master-update-component`
-
`static-master-update-component`
is run on the
**master**
host
It rsyncs the contents from the
**source**
host to the static
It
`
rsync
`
'
s the contents from the
**source**
host to the static
**master**
, and then triggers
`static-master-run`
to push the
content to the mirrors.
...
...
@@ -326,11 +326,11 @@ puppet module -->
When instructed by
`static-master-run`
, we update the symlink and
remove the old tree.
`static-mirror-run`
rsyncs either
`-current-push`
or
`-current-live`
`static-mirror-run`
`
rsync
`
'
s either
`-current-push`
or
`-current-live`
for a component.
LOCKING: during all of
`static-mirror-run`
, we keep an exclusive
lock on the
`<component>`
dir, i.e., the directory that holds
lock on the
`<component>`
dir
ectory
, i.e., the directory that holds
`tree-[ab]`
and
`cur`
.
-
`static-mirror-run-all`
...
...
@@ -354,7 +354,7 @@ Python 2.
### Authentication
A
uthentication between the static site hosts is entirely done through
The a
uthentication between the static site hosts is entirely done through
SSH. The source hosts are accessible by normal users, which can
`sudo`
to a "role" user which has privileges to run the static sync scripts
as sync user. That user then has privileges to contact the master
...
...
@@ -391,7 +391,7 @@ file (`.serial`) to make sure everyone has the same copy of the site.
## Logs and metrics
All tor webservers keep a minimal amount of logs. The IP address and
time (but not the date) are
zero'd
(
`00:00:00`
). The referer is
time (but not the date) are
clear
(
`00:00:00`
). The refer
r
er is
disabled on the client side by sending the
`Referrer-Policy
"no-referrer"`
header.
...
...
@@ -403,14 +403,14 @@ The IP addresses are replaced with:
Logs are kept for two weeks.
Err
r
ors may be sent by email.
Errors may be sent by email.
Metrics are scraped by
[
Prometheus
](
prometheus
)
using the "
a
pache"
Metrics are scraped by
[
Prometheus
](
prometheus
)
using the "
A
pache"
exporter.
## Backups
The
`source`
hosts are backed up with
[
b
acula
](
backups
)
without any special
The
`source`
hosts are backed up with
[
B
acula
](
backups
)
without any special
provision.
TODO: check if master / mirror nodes need to be backup. Probably not?
...
...
@@ -443,7 +443,7 @@ difficult because the dsa-puppet and tor-puppet have disconnected
histories. Even if they would have a common ancestor, the code is
spread in multiple directories, which makes it hard to track. There
has been some refactoring to move most of the code in a
`staticsync`
module, but we still have files strewn over ot
e
hr modules.
module, but we still have files strewn over oth
e
r modules.
The static mirror system was written for Debian.org by Peter
Palfrader. It has also been patches by other DSA members (Stephen
...
...
@@ -509,4 +509,17 @@ of copies of the sites we have to keep around.
*
[
GitLab pages
](
https://docs.gitlab.com/ee/administration/pages/
)
could be used as a source?
*
the
[
cache system
](
cache
)
could be used as a replacement in the
frontend
front-end
<!-- LocalWords: atomicity DDOS YAML Hiera webserver NFS CephFS TLS
-->
<!-- LocalWords: filesystem GitLab scalable frontend CDN HTTPS DNS
-->
<!-- LocalWords: howto Nagios SSL TOC dns letsencrypt sudo LDAP SLA
-->
<!-- LocalWords: rsync cron hostname symlink webservers Bacula DSA
-->
<!-- LocalWords: torproject debian TPO Palfrader Julien Cristau TPA
-->
<!-- LocalWords: LocalWords
-->
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment