Verified Commit e884a50b authored by anarcat's avatar anarcat
Browse files

upload status-site design docs after gitlab migration

parent 97170d91

service/status.md

+51 −59
Original line number Diff line number Diff line
@@ -15,7 +15,7 @@ have a copy of the [Hugo](https://gohugo.io/) static site generator 
and the git repository:



    sudo apt install hugo

    git clone --recursive -b main git@git-rw.torproject.org:project/web/status-site

    git clone --recursive -b main https://gitlab.torproject.org/tpo/tpa/status-site.git

    cd status-site



WARNING: the URL of the Git repository changed! It used to be hosted
@@ -60,39 +60,21 @@ before it is ready. 


## Uploading site to the static mirror system



The `status.torproject.org` site currently lives in the [static mirror

system](howto/static-component) and uses the [git-rw repository](https://gitweb.torproject.org/project/web/status-site.git/) which get built via

Jenkins once you push to the repository.



In other words, uploading the site is automated by continuous

integration. So you simply need to commit and push:

Uploading the site is automated by continuous integration. So you

simply need to commit and push:



    git commit -a -myolo

    git push



Note that only the `webwml` group has access to the repository for

now.



You will see progress of the Jenkins jobs:

Note that only the `TPA` group has access to the repository for now,

but other users can request access as needed.



 * [hugo-website-status](https://jenkins.torproject.org/job/hugo-website-status/) (build job)

 * [hugo-website-status-install](https://jenkins.torproject.org/job/hugo-website-status-install/) (install job)

You can see the progress of build jobs in the [GitLab CI

pipelines](https://gitlab.torproject.org/tpo/tpa/status-site/-/pipelines). If all goes well, successful webhook deliveries should

show up in [this control panel](https://gitlab.torproject.org/tpo/tpa/status-site/-/hooks/1198/edit) as well.



If all goes well, the changes should propagate to the mirrors within

about 5 to 10 minutes, depending on how busy Jenkins is.



If the jobs did not trigger, make sure you are pushing to the Gitolite

server (`git-rw.torproject.org`) and *NOT* the GitLab server, which is

just a mirror and cannot currently trigger Jenkins jobs.



Merge requests may also be issued from the mirror of the repository on

GitLab:



<https://gitlab.torproject.org/tpo/tpa/status-site>



... but will need to be merged into the `git-rw` server by someone in

the above group to take effect. More people have access to the GitLab

repository and should therefore be able to collaborate there.

a few seconds to a minute.



See also the [disaster recovery](#disaster-recovery) options below.
@@ -137,26 +119,23 @@ Those procedures were not tested. 


### Manual deployment to the static mirror system



If `git-rw` is down, you can upload the `public/` folder content under

`/srv/status.torproject.org/htdocs`.

If GitLab is down, you can upload the `public/` folder content under

`/srv/static-gitlab-shim/status.torproject.org/`.



The canonical source for the static websites rotation is defined in

Puppet (in `modules/roles/misc/static-components.yaml`) and is

currently set to `staticiforme.torproject.org`. This `rsync` command

currently set to `static-gitlab-shim.torproject.org`. This `rsync` command

should be enough:



    rsync -rtP public/ anarcat@staticiforme.torproject.org:/srv/status.torproject.org/htdocs



NOTE: there is a copy of the git repository in `/status-site` as

well. Ignore it: it's out of date but could be used to build the

website in a pinch.

    rsync -rtP public/ anarcat@staticiforme.torproject.org:/srv/static-gitlab-shim/status.torproject.org



Then the new source material needs to be synchronized to the mirrors,

with:



    sudo -u torwww static-update-component status.torproject.org

    sudo -u mirroradm static-update-component status.torproject.org



This requires membership to the `torwww` group.

This requires access to the `mirroradm` group, although typically the

machine is only accessible to TPA anyways.



Don't forget to push the changes to the git repository, once that is

available. It's important so that the next people can start from your
@@ -190,9 +169,6 @@ Then DNS needs to be tweaked to point there as well. 


# Reference



The `status.tpo` dashboard is built with [cstate](https://github.com/cstate/cstate/), which is a theme

for the [Hugo](https://gohugo.io/) static site generator.



## Installation



See the instructions on how to [setup a local development
@@ -208,25 +184,14 @@ deploy it to a third-party provider. 
## Design



The status site is part of the [static mirror system](howto/static-component) and is built

with Jenkins jobs, from a git repository on the [git](howto/git) server. This

was setup this way because that is how every other static website is

currently built.

with [cstate](https://github.com/cstate/cstate/), which is a theme for the [Hugo](https://gohugo.io/) static site

generator. The site is managed in a git repository on the [GitLab](howto/gitlab)

server and uses [GitLab CI](service/ci) to get built. The

[static-gitlab-shim service](service/static-gitlab-shim) propagates the builds to the [static

mirror system](howto/static-component) for high availability.



This involved:



 * a new [static component](howto/static-component) owned by `torwww` (in the

   `tor-puppet.git` repository)

 * a new [build script](https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/linux/hugo-website-status?id=7fff67041e41206fef7d128ac60488db375652a1) in the [jenkins/tools.git](https://gitweb.torproject.org/project/jenkins/tools.git/) repository

 * a new [build job](https://gitweb.torproject.org/project/jenkins/jobs.git/commit/?id=170fd9879ff0da65670aaa36c20e63c0db1ed039) in the [jenkins/jobs.git](https://gitweb.torproject.org/project/jenkins/jobs.git/) repository

 * a [new entry](https://gitweb.torproject.org/admin/static-builds.git/commit/?id=b2344aa1d68f4f065764c6f23d14494020b81f86) in the [ssh wrapper](https://gitweb.torproject.org/admin/static-builds.git/tree/ssh-wrap?id=b2344aa1d68f4f065764c6f23d14494020b81f86) in the

   [admin/static-builds.git](https://gitweb.torproject.org/admin/static-builds.git/) repository

 * a new [gitolite](howto/git) repository with hooks to ping the Jenkins server and

   mirror to GitLab



We also considered using GitLab CI for deployment but (a) GitLab pages

is not yet setup and (b) it doesn't integrate well with the static

mirror system for now. See [the broader discussion of the static site

system improvements](howto/static-component#proposed-solution).

See the [static-gitlab-shim service design document](service/static-gitlab-shim#design) for more

information.



## Issues
@@ -413,10 +378,37 @@ experience: 
  * [Wikimedia](http://status.wikimedia.org/) - based on [proprietary nimsoft software](http://www.nimsoft.com/solutions/nimsoft-cloud-user-experience/key-features/public-status-page.html?m=41159&c=pspfoot),

    deprecated in favor of Grafana



## Previous implementation

## Previous implementations



### IRC bot



A similar service was ran by @weasel around 2014. It would bridge the

`status` comments on IRC into a website, see [this archived

version](https://web.archive.org/web/20141017061031/https://gs.torproject.org/)

and [the source code](https://github.com/weaselp/groupstatus), which

is still available.



### Jenkins jobs



The site used to be built with Jenkins jobs, from a git repository on

the [git](howto/git) server. This was setup this way because that is how every

other static website was built back then.



This involved:



 * a new [static component](howto/static-component) owned by `torwww` (in the

   `tor-puppet.git` repository)

 * a new [build script](https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/linux/hugo-website-status?id=7fff67041e41206fef7d128ac60488db375652a1) in the [jenkins/tools.git](https://gitweb.torproject.org/project/jenkins/tools.git/) repository

 * a new [build job](https://gitweb.torproject.org/project/jenkins/jobs.git/commit/?id=170fd9879ff0da65670aaa36c20e63c0db1ed039) in the [jenkins/jobs.git](https://gitweb.torproject.org/project/jenkins/jobs.git/) repository

 * a [new entry](https://gitweb.torproject.org/admin/static-builds.git/commit/?id=b2344aa1d68f4f065764c6f23d14494020b81f86) in the [ssh wrapper](https://gitweb.torproject.org/admin/static-builds.git/tree/ssh-wrap?id=b2344aa1d68f4f065764c6f23d14494020b81f86) in the

   [admin/static-builds.git](https://gitweb.torproject.org/admin/static-builds.git/) repository

 * a new [gitolite](howto/git) repository with hooks to ping the Jenkins server and

   mirror to GitLab



We also considered using GitLab CI for deployment but (a) GitLab pages

was not yet setup and (b) it didn't integrate well with the static

mirror system for now. See [the broader discussion of the static site

system improvements](howto/static-component#proposed-solution).



Both issues have now been fixed thanks to the [static-gitlab-shim

service](service/static-gitlab-shim).