A "status" dashboard is a simple website that allows service admins to
clearly and simply announce down times and recovery.

Note that this be considered part of the [documentation](service/documentation) system,
but is documented separately.

[[_TOC_]]

# Tutorial

## Local development environment

To install the development environment for the status site, you should
have a copy of the [Hugo](https://gohugo.io/) static site generator
and the git repository:

    sudo apt install hugo
    git clone --recursive -b main git@git-rw.torproject.org:project/web/status-site
    cd status-site

WARNING: the URL of the Git repository changed! It used to be hosted
at GitLab, but is now hosted at Gitolite. The repository is *mirrored*
to GitLab, but pushing there will not trigger build jobs.

Then you can start a local development server to preview the site
with:

    hugo serve --baseUrl=http://localhost/
    firefox https://localhost:1313/

The content can also be built in the `public/` directory with, simply:

    hugo

## Creating new issues

Issues are stored in `content/issues/`. You can create a new issue
with `hugo new`, for example:

    hugo new issues/2021-02-03-testing-cstate-again.md

This create the file from a pre-filled template (called an
[archetype in Hugo](https://gohugo.io/content-management/archetypes/))
and put it in `content/issues/2021-02-03-testing-cstate-again.md`.

If you do not have hugo installed locally, you can also copy the
template directly (from `themes/cstate/archetypes/default.md`), or
copy an existing issue and use it as a template.

Otherwise the [upstream guide on how to create issues](https://github.com/cstate/cstate/wiki/Usage#creating-incidents-method-1) is fairly
thorough and should be followed.

In general, keep in mind that the `date` field is when the issue
*started*, not when you posted the issue, see [this feature
request](https://github.com/cstate/cstate/issues/168) asking for an explicit "update" field.

Also note that you can add `draft: true` to the front-matter (the
block on top) to keep the post from being published on the front page
before it is ready.

## Uploading site to the static mirror system

The `status.torproject.org` site currently lives in the [static mirror
system](howto/static-component) and uses the [git-rw repository](https://gitweb.torproject.org/project/web/status-site.git/) which get built via
Jenkins once you push to the repository.

In other words, uploading the site is automated by continuous
integration. So you simply need to commit and push:

    git commit -a -myolo
    git push

Note that only the `webwml` group has access to the repository for
now.

You will see progress of the Jenkins jobs:

 * [hugo-website-status](https://jenkins.torproject.org/job/hugo-website-status/) (build job)
 * [hugo-website-status-install](https://jenkins.torproject.org/job/hugo-website-status-install/) (install job)

If all goes well, the changes should propagate to the mirrors within
about 5 to 10 minutes, depending on how busy Jenkins is.

If the jobs did not trigger, make sure you are pushing to the Gitolite
server (`git-rw.torproject.org`) and *NOT* the GitLab server, which is
just a mirror and cannot currently trigger Jenkins jobs.

Merge requests may also be issued from the mirror of the repository on
GitLab:

<https://gitlab.torproject.org/tpo/tpa/status-site>

... but will need to be merged into the `git-rw` server by someone in
the above group to take effect. More people have access to the GitLab
repository and should therefore be able to collaborate there.

See also the [disaster recovery](#disaster-recovery) options below.

Keep in mind that this is a public website. You might want to talk
with the `comms@` people before publishing big or sensitive
announcements.

# How-to

## Changing categories

cState relies on "systems" which live inside a "category" For example,
the "v3 onion services" are in the "Tor network" category. Those are
defined in the `config.yml` file, and each issue (in `content/issues`)
refers to one or more "system" that is affected by it.

## Theming

The logo lives in `static/logo.png`. Some colors are defined in
`config.yml`, search for `Colors throughout cState`.

## Pager playbook

The only Nagios warning that can come out of this service is if the
static synchronisation fails. See the [static site system](howto/static-component) for more
information on diagnosing those.

## Disaster recovery

It should be possible to deploy the static website anywhere that
supports plain HTML, assuming you have a copy of the git repository. 

The instructions below assume you have a copy of the git
repository. Make sure you follow the [installation instructions](#local-development-environment) to
also clone the submodules! If the git repository is not available, you
could start from scratch using [the example repository](https://github.com/cstate/example) as well.

From here on, it is assumed you have a copy of the git repository (or
the example one).

Those procedures were not tested.

### Manual deployment to the static mirror system

If `git-rw` is down, you can upload the `public/` folder content under
`/srv/status.torproject.org/htdocs`.

The canonical source for the static websites rotation is defined in
Puppet (in `modules/roles/misc/static-components.yaml`) and is
currently set to `staticiforme.torproject.org`. This `rsync` command
should be enough:

    rsync -rtP public/ anarcat@staticiforme.torproject.org:/srv/status.torproject.org/htdocs

NOTE: there is a copy of the git repository in `/status-site` as
well. Ignore it: it's out of date but could be used to build the
website in a pinch.

Then the new source material needs to be synchronized to the mirrors,
with:

    sudo -u torwww static-update-component status.torproject.org

This requires membership to the `torwww` group.

Don't forget to push the changes to the git repository, once that is
available. It's important so that the next people can start from your
changes:

    git commit -a -myolo
    git push

### Netlify deployment

Upstream has instructions to [deploy to Netlify](https://github.com/cstate/cstate#-netlify-and-netlify-cms), which, in our
case, might be as simple as [following this link](https://app.netlify.com/start/deploy?repository=https://gitlab.torproject.org/tpo/tpa/status-site.git) and filling in
those settings:

 * Build command: `hugo`
 * Publish directory: `public`
 * Add one build environment variable
   * Key: `HUGO_VERSION`
   * Value: `0.48` (or later)

Then, of course, DNS needs to be updated to point there.

### GitLab pages deployment

A site could also be deployed on *another* GitLab server with "GitLab
pages" enabled. For example, if the repository is pushed to
<https://gitlab.com/>, the GitLab CI/CD system there will
automatically pick it up and publish it.

Then DNS needs to be tweaked to point there as well.

# Reference

## Installation

See the instructions on how to [setup a local development
environment](#local-development-environment) and the [design section](#design) for more information on
how this is setup.

## SLA

This service should be highly available. It should support failure
from one or all point of presence: if all fail, it should be easy to
deploy it to a third-party provider.

## Design

The status site is part of the [static mirror system](howto/static-component) and is built
with Jenkins jobs, from a git repository on the [git](howto/git) server. This
was setup this way because that is how every other static website is
currently built.

This involved:

 * a new [static component](howto/static-component) owned by `torwww` (in the
   `tor-puppet.git` repository)
 * a new [build script](https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/linux/hugo-website-status?id=7fff67041e41206fef7d128ac60488db375652a1) in the [jenkins/tools.git](https://gitweb.torproject.org/project/jenkins/tools.git/) repository
 * a new [build job](https://gitweb.torproject.org/project/jenkins/jobs.git/commit/?id=170fd9879ff0da65670aaa36c20e63c0db1ed039) in the [jenkins/jobs.git](https://gitweb.torproject.org/project/jenkins/jobs.git/) repository
 * a [new entry](https://gitweb.torproject.org/admin/static-builds.git/commit/?id=b2344aa1d68f4f065764c6f23d14494020b81f86) in the [ssh wrapper](https://gitweb.torproject.org/admin/static-builds.git/tree/ssh-wrap?id=b2344aa1d68f4f065764c6f23d14494020b81f86) in the
   [admin/static-builds.git](https://gitweb.torproject.org/admin/static-builds.git/) repository
 * a new [gitolite](howto/git) repository with hooks to ping the Jenkins server and
   mirror to GitLab

We also considered using GitLab CI for deployment but (a) GitLab pages
is not yet setup and (b) it doesn't integrate well with the static
mirror system for now. See [the broader discussion of the static site
system improvements](howto/static-component#proposed-solution).

## Issues

There is no issue tracker specifically for this project, [File][] or
[search][] for issues in the [team issue tracker][search].

 [File]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/new
 [search]: https://gitlab.torproject.org/tpo/tpa/team/-/issues

Upstream issues can be found and filed in the [GitHub issue tracker](https://github.com/cstate/cstate/issues).

## Monitoring and testing

The site, like other static mirrors, is monitored by [Nagios](howto/nagios) with
the `dsa_check_staticsync` check, which ensures all mirrors are up to
date.

## Logs and metrics

There are no logs or metrics specific to this service, see the [static
site service](howto/static-component) for details.

## Backups

Does not need special backups: backed up as part of the regular [static
site](howto/static-component) and [git](howto/git) services.

## Other documentation

 * [cState home page](https://github.com/cstate/cstate/)
 * [demo site](https://cstate.mnts.lt/)
 * [cState wiki](https://github.com/cstate/cstate/wiki), see in particular the [usage](https://github.com/cstate/cstate/wiki/Usage) and
   [configuration](https://github.com/cstate/cstate/wiki/Customization) guides

# Discussion

## Overview

This project comes from two places:

 1. during the [2020 TPA user survey](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40061), some respondents suggested
    to document "down times of 1h or longer" and better communicate
    about service statuses

 2. separately, following a major outage in the Tor network due to a
    DDOS, the network team and network health teams [asked for a
    dashboard to inform tor users about such problems in the
    future](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40138)

This is therefore a project spanning multiple teams, with different
stakeholders. The general idea is to have a site (say
status.torproject.org) that simply shows users how things are going,
in an easy to understand form.

## Goals

In general, the goal is to provide a simple interface to provide users
with status updates.

### Must have

 * **user-friendly**: the public website must be easy to understand by
   the Tor wider community of users (not just TPI/TPA)
 * **status updates and progress**: "post status problem we know about
   so the world can learn if problems are known to the Tor team."
   * example: "[recent] v3 outage where we could have put out a small
     FAQ right away (go static HTML!) and then update the world as we
     figure out the problem but also expected return to normal."
 * **multi-stakeholder**: "easily editable by many of us namely likely
   the network health team and we could also have the network team to
   help out"
 * **simple to deploy and use**: pushing an update shouldn't require
   complex software or procedures. editing a text file, committing and
   pushing, or building with a single command and pushing the HTML,
   for example, is simple enough. installing a MySQL database and PHP
   server, for example, is not simple enough.
 * keep it simple
 * free-software based

### Nice to have

 * deployment through GitLab (pages?), with contingency plans
 * separate TLD to thwart DNS-based attacks against torproject.org
 * same tool for multiple teams
 * per-team filtering
 * RSS feeds
 * integration with social media?
 * responsive design

### Non-Goals

 * automation: updating the site is a manual process. no automatic
   reports of sensors/metrics or Nagios, as this tends to complicate
   the implementation and cause false positives

## Approvals required

TPA, network team, network health team.

## Proposed Solution

We're experimenting with [cstate](https://github.com/cstate/cstate) because it's the only static
website generator with such a nice template out of the box that we
could find.

## Cost

Just research and development time. Hosting costs are negligible.

## Alternatives considered

Those are the status dashboards we know about and that are still
somewhat in active development:

  * [Cachet](https://cachethq.io/)
    * PHP
    * MySQL database
    * [demo site](https://demo.cachethq.io/) (test@test.com, test123)
    * responsive
    * [not decentralized](https://twitter.com/theanarcat/status/575061666532102144)
    * [no Nagios support](https://github.com/cachethq/Cachet/issues/225)
    * user-friendly
    * publicly accessible
    * fairly easy to use
    * [aims for LDAP support](https://github.com/CachetHQ/Cachet/issues/2108)
    * no Twitter, Identica, IRC or XMPP support for now
    * [dropped RSS support](https://github.com/CachetHQ/Cachet/issues/3313)
    * future of the project uncertain ([4037](https://github.com/CachetHQ/Cachet/issues/4037), [3968](https://github.com/CachetHQ/Cachet/issues/3968))
  * [cstate](https://github.com/cstate/cstate), Hugo-based static site generator, tag-based RSS
    feeds, easy setup on Netlify, GitLab CI integration, badges,
    read only API
  * [Staytus](http://staytus.co/)
    * ruby
    * MySQL database
    * responsive
    * email notifications
    * mobile-friendly
    * not distributed
    * no Nagios integration
    * [no Twitter notifications](https://github.com/adamcooke/staytus/issues/2)
    * user-friendly - seems to be even nicer than Cachet, as there are links to individual announcements and notifications
    * no LDAP support
    * MIT-licensed
    * [similar performance problems than Cachet](https://github.com/adamcooke/staytus/issues/4)

### Abandonware

Those were previously evaluated in a previous life but ended up being
abandoned upstream:

  * [Overseer](https://github.com/disqus/overseer) - used at [Disqus.com](http://disqus.com/), Python/Django, user-friendly/simple, [administrator non-friendly](https://overseer.readthedocs.org/en/latest/admin.html), twitter integration, Apache2 license, development stopped, Disqus replaced it with [Statuspage.io](https://www.statuspage.io/)
  * [Stashboard](http://www.stashboard.org/) - used at [Twilio](http://www.twilio.com/), MIT license, [demo](http://stashboard.appspot.com/), Twitter integration,
    REST API, abandon-ware, no authentication, no Unicode support,
    depends on Google App engine, requires daily updates
  * [Baobab](https://github.com/Gandi/baobab) - previously used at [Gandi](https://gandi.net/), replaced with `statuspage.io`, Django based

### Hacks

Those were discarded because they do not provide an "out of the box"
experience:

  * use Jenkins to run jobs that check a bunch of things and report a
    user-friendly status?
  * just use a social network account (e.g. Twitter)
  * "just use the wiki"
  * use Drupal ("there's a module for that")
  * roll our own with [Lektor](https://www.getlektor.com/), e.g. using [this template](https://www.hamma.dev/hamma1/)

### example sites

  * [Amazon Service Health Dashboard](http://status.aws.amazon.com/)
  * [Disqus](http://status.disqus.com) - based on [statuspage.io](https://www.statuspage.io/)
  * [GitLab](https://status.gitlab.com) - based on [status.io](https://status.io/)
  * [Github](https://status.github.com/) - "Battle station fully operational", auto-refresh,
    twitter-connected, simple color coded (see [this blog post for
    more details](https://github.com/blog/1240-new-status-site)), not open-source (confirmed in personal email
    between GitHub support and anarcat on 2013-05-02)
  * [Potager.org](https://meteo.potager.org/) - ikiwiki based
  * [Riseup.net](https://status.riseup.net/) - RSS feeds
  * [Signal](https://status.signal.org/) - simple, plain HTML page
  * [sr.ht](https://status.sr.ht/) - cState
  * [Twilio](https://status.twilio.com/) - email, slack, RSS subscriptions, lots of services
    shown
  * [Wikimedia](http://status.wikimedia.org/) - based on [proprietary nimsoft software](http://www.nimsoft.com/solutions/nimsoft-cloud-user-experience/key-features/public-status-page.html?m=41159&c=pspfoot),
    deprecated in favor of Grafana

## Previous implementation

A similar service was ran by @weasel around 2014. It would bridge the
`status` comments on IRC into a website, see [this archived
version](https://web.archive.org/web/20141017061031/https://gs.torproject.org/)
and [the source code](https://github.com/weaselp/groupstatus), which
is still available.
