|
|
How we manage documentation inside TPA, but also touch on other wikis
|
|
|
and possibly other documentation systems inside TPO.
|
|
|
|
|
|
[[_TOC_]]
|
|
|
|
|
|
# Tutorial
|
|
|
|
|
|
## Editing the wiki
|
|
|
|
|
|
If you have the right privileges (currently: being part of TPA, but we
|
|
|
hope to improve this), you should have an `Edit` button at the
|
|
|
top-right of pages in the wiki here:
|
|
|
|
|
|
<https://gitlab.torproject.org/tpo/tpa/team/-/wikis/>
|
|
|
|
|
|
If not, you need to issue a merge request in the [wiki replica](https://gitlab.torproject.org/tpo/tpa/wiki-replica).
|
|
|
|
|
|
# How-to
|
|
|
|
|
|
<!-- more in-depth procedure that may require interpretation -->
|
|
|
|
|
|
## Pager playbook
|
|
|
|
|
|
### Wiki unavailable
|
|
|
|
|
|
If the GitLab server is down, the wiki will be unavailable. For that
|
|
|
reason, it is highly preferable to keep a copy of the git repository
|
|
|
backing the wiki on your local computer.
|
|
|
|
|
|
If for some reason you do not have such a copy, it is extremely
|
|
|
unlikely you will be able to read this page in the first place. But,
|
|
|
if for some reason you are able to, you should find the [gitlab](gitlab)
|
|
|
documentation to restore *that* service and then immediately clone a
|
|
|
copy of this repository:
|
|
|
|
|
|
git@gitlab.torproject.org:tpo/tpa/team.wiki.git
|
|
|
|
|
|
or:
|
|
|
|
|
|
https://gitlab.torproject.org/tpo/tpa/team.wiki.git
|
|
|
|
|
|
If you can't find the GitLab documentation in the wiki, you can try to
|
|
|
read the [latest copy in the wayback machine](https://web.archive.org/web/2/https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab/).
|
|
|
|
|
|
## Disaster recovery
|
|
|
|
|
|
If GitLab disappears in a flaming ball of fire, it should be possible
|
|
|
to build a static copy of this website somehow. Originally, GitLab's
|
|
|
wiki was based on [Gollum](https://github.com/gollum/gollum/), a simple Git-based wiki. In practice,
|
|
|
GitLab's design has diverged wildly and is now a [separate
|
|
|
implementation](https://docs.gitlab.com/ee/development/wikis.html).
|
|
|
|
|
|
The GitLab instructions still say you can run `gollum` to start a
|
|
|
server rendering the source git repository to HTML. Unfortunately,
|
|
|
that is done dynamically and cannot be done as a one-time job, or as a
|
|
|
post-update git hook, so you would have to setup [gollum as a
|
|
|
service](https://github.com/gollum/gollum/wiki/Gollum-as-a-service) in the short term.
|
|
|
|
|
|
In the long term, it might be possible to migrate back to [ikiwiki][]
|
|
|
or another static site generator.
|
|
|
|
|
|
[ikiwiki]: https://ikiwiki.info/
|
|
|
|
|
|
# Reference
|
|
|
|
|
|
## Installation
|
|
|
|
|
|
"Installation" was trivial insofar as we consider the [GitLab](gitlab)
|
|
|
step to be abstracted away: just create a wiki inside the `team` and
|
|
|
start editing/pushing content.
|
|
|
|
|
|
In practice, the wiki was migrated from [ikiwiki][] (see [issue
|
|
|
34437](https://gitlab.torproject.org/tpo/tpa/team/-/issues/34437)) using anarcat's [ikiwiki2hugo](https://gitlab.com/anarcat/scripts/raw/master/ikiwiki2hugo.py) converter, which happened
|
|
|
to be somewhat compatible with GitLab's wiki syntax.
|
|
|
|
|
|
The ikiwiki repository was archived inside GitLab in the
|
|
|
[wiki-archive](https://gitlab.torproject.org/tpo/tpa/wiki-archive) and [wiki-infra-archive](https://gitlab.torproject.org/tpo/tpa/wiki-infra-archive) repositories. History of
|
|
|
those repositories is, naturally, also available in the history of the
|
|
|
current wiki.
|
|
|
|
|
|
## SLA
|
|
|
|
|
|
This service should be as available as GitLab or better, assuming TPA
|
|
|
members keep a copy of the documentation cloned on their computers.
|
|
|
|
|
|
## Design
|
|
|
|
|
|
Documentation for TPA is hosted inside a git repository, which is
|
|
|
[hosted inside a GitLab wiki](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/). It is replicated inside a [git
|
|
|
repository at GitLab](https://gitlab.torproject.org/tpo/tpa/wiki-replica) to allow external users to contribute by
|
|
|
issuing pull requests.
|
|
|
|
|
|
GitLab wikis support Markdown, RDoc, AsciiDoc, and Org formats.
|
|
|
|
|
|
### Scope
|
|
|
|
|
|
This documentation mainly concerns the TPA wiki, but there are other
|
|
|
wikis on GitLab which are not directly covered by this documentation
|
|
|
and may have a different policy.
|
|
|
|
|
|
### Structure
|
|
|
|
|
|
The wiki has a *minimalist* structure: we try to avoid deeply nested
|
|
|
pages. Any page inside the wiki should be reachable within 2 or 3
|
|
|
clicks from the main page. Flat is better than tree.
|
|
|
|
|
|
All *services* running at torproject.org MUST have a documentation
|
|
|
page in the [service directory](service) which SHOULD at least include
|
|
|
a "disaster recovery" and "pager playbook" section. It is strongly
|
|
|
encouraged to follow the [documentation template](howto/template) for
|
|
|
new services.
|
|
|
|
|
|
This documentation is based on the [Grand Unified Theory of
|
|
|
Documentation](https://www.divio.com/blog/documentation/), by Daniele Procida. To quote that excellent guide
|
|
|
(which should, obviously, be self-documenting):
|
|
|
|
|
|
> There is a secret that needs to be understood in order to write good
|
|
|
> software documentation: there isn’t one thing called documentation,
|
|
|
> there are four.
|
|
|
>
|
|
|
> They are: tutorials, how-to guides, technical reference and
|
|
|
> explanation. They represent four different purposes or functions,
|
|
|
> and require four different approaches to their
|
|
|
> creation. Understanding the implications of this will help improve
|
|
|
> most documentation - often immensely.
|
|
|
|
|
|
We express this structure in a rather odd way: each service page has
|
|
|
that structure embedded. This is partly due to limitations in the
|
|
|
tools we use to manage the documentation -- GitLab wikis do not offer
|
|
|
much in terms of structure -- but also because we have a *large*
|
|
|
variety of services being documented. To give a concrete example, it
|
|
|
would not make much sense to have a top-level "Tutorials" section with
|
|
|
tutorials for GitLab, caching, emails, followed by "How to guides"
|
|
|
with guides for... exactly the same list! So instead we flip that
|
|
|
structure around and the top-level structure is by service: within
|
|
|
those pages we follow the suggested structure.
|
|
|
|
|
|
### Authentication
|
|
|
|
|
|
The entire wiki is public and no private or sensitive information
|
|
|
should be committed to it.
|
|
|
|
|
|
### People
|
|
|
|
|
|
Most of the documentation has been written by anarcat, which may be
|
|
|
considered the editor of the wiki, but any other contributors is
|
|
|
strongly encouraged to contribute to the knowledge accumulating in the
|
|
|
wiki.
|
|
|
|
|
|
## Issues
|
|
|
|
|
|
There is no issue tracker specifically for this project, [File][] or
|
|
|
[search][] for issues in the [team issue tracker][search].
|
|
|
|
|
|
[File]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/new
|
|
|
[search]: https://gitlab.torproject.org/tpo/tpa/team/-/issues
|
|
|
|
|
|
Notable issues:
|
|
|
|
|
|
* [wikis are not editable publicly](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/76)
|
|
|
* [wikis are hard to find](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/66)
|
|
|
* [wiki migration is incomplete](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/5)
|
|
|
|
|
|
See also the [limitations](#Limitations) section below.
|
|
|
|
|
|
## Monitoring and testing
|
|
|
|
|
|
There is not monitoring of this service, outside of the main GitLab
|
|
|
monitoring systems.
|
|
|
|
|
|
There are no continuous tests of the documentation.
|
|
|
|
|
|
See the "alternatives considered" section for ideas on tests that
|
|
|
*could* be ran.
|
|
|
|
|
|
## Logs and metrics
|
|
|
|
|
|
No logs or metrics specific to the wiki are kept, other than what
|
|
|
GitLab already does.
|
|
|
|
|
|
## Backups
|
|
|
|
|
|
Backed up alongside GitLab, and hopefully in git clones on all TPA
|
|
|
members machines.
|
|
|
|
|
|
## Other documentation
|
|
|
|
|
|
* [GitLab wiki documentation](https://docs.gitlab.com/ee/user/project/wiki/#wiki)
|
|
|
* [GitLab markdown format](https://docs.gitlab.com/ee/user/markdown.html)
|
|
|
|
|
|
# Discussion
|
|
|
|
|
|
Documentation is a critical part of any project. Without
|
|
|
documentation, things lose their meaning, training is impossible, and
|
|
|
memories are lost. Updating documentation is also hard: things change
|
|
|
after documentation is written and keeping documentation in sync with
|
|
|
reality is a constant challenge.
|
|
|
|
|
|
This section talks about the known problems with the current
|
|
|
documentation (systems) and possible solutions.
|
|
|
|
|
|
## Limitations
|
|
|
|
|
|
### Redundancy
|
|
|
|
|
|
The current TPA documentation system is a GitLab wiki, but used to be
|
|
|
a fairly old [ikiwiki][] site, part of the [static site system](howto/static-component).
|
|
|
|
|
|
As part of the ikiwiki migration, that level of redundancy was lost:
|
|
|
if GitLab goes down, the wiki goes down, along with the
|
|
|
documentation. This is mitigated by the fact that the wiki is backed
|
|
|
by a Git repository. So TPA members are strongly encouraged to keep a
|
|
|
copy of the Git repository locally to not only edit the content (which
|
|
|
makes sure the copy is up to date) but also consult it in case of an
|
|
|
infrastructure failure.
|
|
|
|
|
|
### Unity
|
|
|
|
|
|
We have lots of documentation spaces. There's this wiki for TPA, but
|
|
|
there are also different wikis for different teams. There's a proposal
|
|
|
to create a [community hub](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/66) which could help. But that idea assumes
|
|
|
people will know about the hub, which adds an extra layer of indirection.
|
|
|
|
|
|
It would be better if we could have group wikis, which were [published
|
|
|
as part of the 13.5 release](https://about.gitlab.com/releases/2020/10/22/gitlab-13-5-released/) but, unfortunately, only in the
|
|
|
commercial version. So we're stuck with our current approach of having
|
|
|
the "team" projects inside each group to hold the wiki.
|
|
|
|
|
|
It should also be noted that we have documentation scattered outside
|
|
|
the wiki as well: some teams have documentation in text files, others
|
|
|
are entire static websites. The above community hub could benefit from
|
|
|
linking to those other resources as well.
|
|
|
|
|
|
### Testing
|
|
|
|
|
|
There is no continuous testing/integration of the documentation. Typos
|
|
|
frequently show up in documentation, and probably tons of broken links
|
|
|
as well. Style is incoherent at best, possibly unreadable at
|
|
|
worst. This is a tough challenge in any documentation system, due to
|
|
|
the complexity and ambiguity of language, but it shouldn't deter us
|
|
|
from running basic tests on the documentation.
|
|
|
|
|
|
This would require hooking up the wiki in GitLab CI, which is not
|
|
|
currently possible within GitLab wikis. We'd need to switch the wiki
|
|
|
to a full Git repository, possibly pushing to the wiki using a deploy
|
|
|
key on succesful runs. But then why would we keep the wiki?
|
|
|
|
|
|
### Structure
|
|
|
|
|
|
Wikis are notorious for being hard to structure. They can quickly
|
|
|
become a tangled mess with oral tradition the only memory to find your
|
|
|
way inside of the forest. The GitLab wikis are especially vulnerable
|
|
|
to this as they do not offer many tools to structure content: no
|
|
|
includes, limited macros and so on.
|
|
|
|
|
|
The *is* a mechanism to [add a sidebar](https://docs.gitlab.com/ee/user/project/wiki/#customizing-sidebar) in certain sections, that
|
|
|
said, which can help quite a bit in giving a rough structure. But
|
|
|
restructuring the wiki is hard: renaming pages breaks all links
|
|
|
pointing to it and [there is no way to do redirects](https://gitlab.com/gitlab-org/gitlab/-/issues/257892) which is a
|
|
|
major regression from ikiwiki.
|
|
|
|
|
|
Using a static site generator (SSG) could help here: many of them
|
|
|
support redirections (and so [does GitLab Pages](https://docs.gitlab.com/ee/user/project/pages/redirects.html), although in a
|
|
|
very limited way). Many SSGs also support more "structure" features
|
|
|
like indexes, hierarchical (and automatic) sidebars (based on
|
|
|
structure, e.g. [Sphinx](https://www.sphinx-doc.org/) or [mkdocs](https://www.mkdocs.org/)), paging, per-section RSS
|
|
|
feeds (for "blog" or "news" type functionality) and so on.
|
|
|
|
|
|
### Syntax
|
|
|
|
|
|
Markdown is great for jotting down notes, filing issues and so on, but
|
|
|
it has been [heavily criticised](https://www.ericholscher.com/blog/2016/mar/15/dont-use-markdown-for-technical-docs/) for use in formal
|
|
|
documentation. One of the problem with Markdown is its lack of
|
|
|
standardized syntax: there is [CommonMark](http://commonmark.org/) but it has yet to see
|
|
|
wider adoption.
|
|
|
|
|
|
This makes Markdown not portable across different platforms supposedly
|
|
|
supporting markdown.
|
|
|
|
|
|
It also lacks special mechanisms for more elaborate markups like
|
|
|
[admonitions](http://docutils.sourceforge.net/0.7/docs/ref/rst/directives.html#admonitions) (or generally: "semantic meanings") or "quick links"
|
|
|
(say: bug#1234 pointing directly to the bug tracker).
|
|
|
|
|
|
It has to be said, however, that Markdown is widely used, much more
|
|
|
than the alternatives (e.g. asciidoc or rst), for better or for
|
|
|
worse. So it might be better to stick with it than to force users to
|
|
|
learn a new markup language, however good it is supposed to be.
|
|
|
|
|
|
## Goals
|
|
|
|
|
|
Note: considering we *just* migrated from ikiwiki to GitLab wikis, it
|
|
|
is unlikely we will make any major change on the documentation system
|
|
|
in the short term, unless one of the above issues becomes so critical
|
|
|
it needs to immediately be fixed.
|
|
|
|
|
|
That said, improvements or replacements to the current sytem should
|
|
|
include...
|
|
|
|
|
|
### Must have
|
|
|
|
|
|
* highly available: it should be possible to have readonly access to
|
|
|
the documentation even in case of a total catastrophe (global
|
|
|
[EMP](https://en.wikipedia.org/wiki/Electromagnetic_pulse) catastrophe excluded)
|
|
|
|
|
|
* testing: the documentation should be "testable" for typos, broken
|
|
|
links and other quality issues
|
|
|
|
|
|
* structure: it should be possible to structure the documentation in
|
|
|
a way that makes things easy to find and new users easily orient
|
|
|
themselves
|
|
|
|
|
|
* discoverability: our documentation should be easy to find and
|
|
|
navigate for new users
|
|
|
|
|
|
* minimal friction: it should be easy to contribute to the
|
|
|
documentation (e.g. the "Edit" button on a wiki is easier than
|
|
|
"make a merge request", as a workflow)
|
|
|
|
|
|
### Nice to have
|
|
|
|
|
|
* offline write: it should be possible to write documentation offline
|
|
|
and push the changes when back online. a git repository is a good
|
|
|
example of such functionality
|
|
|
|
|
|
* nice-looking, easily themable
|
|
|
|
|
|
* coherence: documentation systems should be easy to cross-reference
|
|
|
between each other
|
|
|
|
|
|
* familiarity: users shouldn't have to learn a new markup language or
|
|
|
tool to work on documentation
|
|
|
|
|
|
### Non-Goals
|
|
|
|
|
|
* repeat after me: we should not write our own documentation system
|
|
|
|
|
|
## Approvals required
|
|
|
|
|
|
TPA, although it might be worthwhile to synchronize this technology
|
|
|
with other teams so we have coherence across the organisation.
|
|
|
|
|
|
## Proposed Solution
|
|
|
|
|
|
We currently use GitLab wikis.
|
|
|
|
|
|
## Cost
|
|
|
|
|
|
Staff hours, hosting costs shadowed by GitLab.
|
|
|
|
|
|
## Alternatives considered
|
|
|
|
|
|
### Static site generators
|
|
|
|
|
|
* [11ty](https://www.11ty.dev/): [picked by mozilla](https://hacks.mozilla.org/2020/10/to-eleventy-and-beyond/), javascript
|
|
|
* [hugo](https://gohugo.io/): golang
|
|
|
* [ikiwiki][]: previously used, old, Perl, hard to setup, occult templating system, slow
|
|
|
* [lektor](https://www.getlektor.com/): used at Tor for other public websites
|
|
|
* [Nanoc][]: [used by GitLab](https://docs.gitlab.com/ee/development/documentation/site_architecture/index.html)
|
|
|
* [pelican](https://getpelican.com/): watch out for pelican, another user reports that,
|
|
|
with caching, generating a 500 page site takes 30 seconds, 2
|
|
|
minutes without caching
|
|
|
* [zola](https://www.getzola.org/): rust
|
|
|
|
|
|
### Testing
|
|
|
|
|
|
To use those tests, we'd need to switch the wiki into a Git
|
|
|
repository, as it is not (currently) possible to run CI on changes in
|
|
|
GitLab wikis.
|
|
|
|
|
|
* GitLab has a [test suite](https://docs.gitlab.com/ee/development/documentation/testing.html) for their documentation which:
|
|
|
* runs [markdownlint][]: checks that Markdown syntax
|
|
|
* runs [vale][]: grammar, style, and word usage linter for the
|
|
|
English language
|
|
|
* checks the internal anchors and links using Nanoc
|
|
|
* [Danger systems](https://danger.systems/ruby/) has a bunch of plugins which could be used to
|
|
|
check documentation
|
|
|
* [textlint](https://textlint.github.io/): pluggable text linting approach recognizing markdown
|
|
|
* [proselint](http://proselint.com/): grammar and style checkign
|
|
|
* [languagetool](https://www.languagetool.org/): Grammar, Style and Spell Checker
|
|
|
* [anorack](https://github.com/jwilk/anorack): spots errors based on phonemes
|
|
|
* [redpen](https://redpen.cc/): huge JAR, can be noisy
|
|
|
|
|
|
See also [this LWN article](https://lwn.net/Articles/822969/).
|
|
|
|
|
|
[vale]: https://errata-ai.gitbook.io/vale/
|
|
|
[markdownlint]: https://github.com/DavidAnson/markdownlint |