[GitLab](https://gitlab.com/) is *a web-based DevOps lifecycle tool that provides a
Git-repository manager providing wiki, issue-tracking and continuous
integration/continuous deployment pipeline features, using an
open-source license, developed by GitLab Inc* ([Wikipedia](https://en.wikipedia.org/wiki/GitLab)). Tor
uses GitLab mainly for issue tracking, wiki hosting and code review
for now, at <https://gitlab.torproject.org>, after migrating from
[howto/trac](howto/trac).
Note that continuous integration is documented separately, in [the CI page](service/ci).
[[_TOC_]]
# Tutorial
<!-- simple, brainless step-by-step instructions requiring little or -->
<!-- no technical background -->
## How to get an account?
You might already *have* an account! If you were active on Trac, your
account was migrated with the same username and email address as Trac,
unless you have an LDAP account, in which case that was used. So head
over to the [password reset page](https://gitlab.torproject.org/users/password/new) to get access to your account.
If your account was *not* migrated, send a mail to
<gitlab-admin@torproject.org> to request a new one.
If you did not have an account in Trac and want a new account, you
should request a new one at <https://gitlab.onionize.space/>.
## How to report an issue in Tor software?
You first need to figure out which project the issue resides in. The
[project list][] is a good place to get started. Here are a few quick
links for popular projects:
[project list]: https://gitlab.torproject.org/tpo
* [core tor](https://gitlab.torproject.org/tpo/core/tor): [issues](https://gitlab.torproject.org/tpo/core/tor/-/issues), [new issue](https://gitlab.torproject.org/tpo/core/tor/-/issues/new)
* [Tor Browser](https://gitlab.torproject.org/tpo/applications/tor-browser): [issues](https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues), [new issue](https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/new)
* [gitlab](https://gitlab.torproject.org/tpo/tpa/gitlab): [issues](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues), [new issue](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/new)
If you do not have a GitLab account or can't figure it out for any
reason, you can also use the mailing lists. The
<tor-dev@lists.torproject.org> mailing list is the best for now.
## How to report an issue in the bugtracker itself?
If you have access to GitLab, you can [file a new issue][File] after
you have [searched the GitLab project for similar bugs][search].
If you do *not* have access to GitLab, you can email
<gitlab-admin@torproject.org>.
### Note about confidential issues
Note that you can mark issues as "confidentials" which will make them
private to the members of the project the issue is reported on (the
"developers" group and above, specifically).
Keep in mind, however, that it is still possible issue information
gets leaked in cleartext, however. For example, GitLab [sends email
notifications in cleartext for private issue](https://gitlab.com/gitlab-org/gitlab/-/issues/5816), an known upstream
issue. (We have [decided we cannot fix this ourselves in GitLab for
now](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/23).) Some repositories might also have "web hooks" that notify
IRC bots in clear text as well, although at the time of writing all
projects are correctly configured.
## How to contribute code?
As reporting an issue, you first need to figure out which project you
are working on in the GitLab [project list][]. Then, if you are not
familiar with merge requests, you should read the [merge requests
introduction](https://gitlab.torproject.org/help/user/project/merge_requests/getting_started.md) in the GitLab documentation. If you are unfamiliar
with merge requests but familiar with GitHub's pull requests, those
are similar.
Note that we do not necessarily use merge requests in all teams yet,
and Gitolite still has the canonical version of the code. See [issue
36][] for a followup on this.
[issue 36]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/36
Also note that different teams might have different workflows. If a
team has a special workflow that diverges from the one here, it should
be documented here. Those are the workflows we know about:
* [Network Team](https://gitlab.torproject.org/tpo/core/tor/-/wikis/NetworkTeam/GitlabReviews)
* [Web Team](https://gitlab.torproject.org/tpo/web/community/-/wikis/Git-flow-and-merge-requests)
* Bridge DB: merge requests
If you do not have access to GitLab, please use one of the mailing
lists: <tor-dev@lists.torproject.org> would be best.
## How to quote a comment in a reply?
The "Reply" button only creates a new comment without any quoted text
by default. It seems the solution to that is currently highlighting
the text to quote and then pressing the `r`-key. See also the [other
keyboard shortcuts](https://docs.gitlab.com/ee/user/shortcuts.html).
Alternatively, you can copy-paste the text in question in the comment
form, select the pasted text, and hit the `Insert a quote` button
which look like a styled, curly, and closing quotation mark `”`.
# How-to
## Continuous Integration (CI)
All CI documentation resides in a different document see
[service/ci](service/ci).
## Email interactions
You can interact with GitLab by email too.
### Creating a new issue
Clicking on the project issues gives a link at the bottom of the page,
which says say "Email a new issue to this project".
That link should go into the "To" field of your email. The email
subject becomes the title of the issue and the body the
description. You can use shortcuts in the body, like `/assign @foo`,
`/estimate 1d`, etc.
See [the upstream docs for more details](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#new-issue-via-url-with-prefilled-fields).
### Commenting on an issue
If you just reply to the particular comment notification you received
by email, as you would reply to an email in a thread, that comment
will show up in the issue.
You need to have email notifications enabled for this to work,
naturally.
You can also add a new comment to any issue by copy-pasting the
issue-specific email address in the right sidebar (labeled "Issue
email", [introduced in GitLab 13.8](https://gitlab.com/gitlab-org/gitlab/-/issues/18816)).
This also works with shortcuts like `/estimate 1d` or `/spend
-1h`. Note: for those you won't get notification emails back, though,
while for others like `/assign @foo` you would.
See [the upstream docs for more details](https://docs.gitlab.com/ee/administration/reply_by_email.html).
### Quick status updates by email
There are a bunch of [quick actions](https://gitlab.torproject.org/help/user/project/quick_actions.md) available which are handy to
update an issue. As mentioned above they can be sent by email as well,
both within a comment (be it as a reply to a previous one or in a new
one) or just instead of it. So, for example, if you want to update the
amount of time spent on ticket $foo by one hour, find any notification
email for that issue and reply to it by replacing any quoted text with
`/spend 1h`.
## How to migrate a Git repository from legacy to GitLab?
See the [git documentation for this procedure](howto/git#how-to-migrate-a-git-repository-from-legacy-to-gitlab).
## How to mirror a Git repository from legacy to GitLab?
See the [git documentation for this procedure](howto/git#how-to-migrate-a-git-repository-from-legacy-to-gitlab).
## How to mirror a Git repository from GitLab to GitHub
Some repositories are mirrored to THE [torproject organization on
GitHub](https://github.com/torproject). This section explains how that works and how to create a
new mirror from GitLab. In this example, we're going to mirror the
[tor browser manual](https://gitlab.torproject.org/tpo/web/manual).
1. head to the "Mirroring repositories" section of the
[settings/repository](https://gitlab.torproject.org/tpo/web/manual/-/settings/repository) part of the project
2. as a Git repository URL, enter:
ssh://git@github.com/torproject/tb-manual.git
3. click "detect host keys"
4. choose "SSH" as the "Authentication method"
5. don't check any of the boxes, click "Mirror repository"
6. the page will reload and show the mirror in the list of "Mirrored
repositories". click the little "paperclip" icon which says "Copy
SSH public key"
7. head over to https://github.com and authenticate with
`torproject-pusher`, password is in `external-services-git`, in
the password manager
8. go to [settings/keys](https://github.com/settings/keys) and hit [new SSH key](https://github.com/settings/ssh/new)
9. paste the public key in the bigger field, as a title, use the URL
of the repository, for example:
Title: https://gitlab.torproject.org/tpo/web/manual mirror key
Key: ssh-rsa AAAA[...]
10. click "Add SSH key"
11. to speed up the process, you can [import the repository in
GitHub](https://github.com/new/import), otherwise create a [new repository](https://github.com/new). in *both*
cases make sure you change the namespace from the default
(`torproject-pusher`, which is incorrect) to the `torproject`
namespace (which is correct)
12. then hit the "reload" button in the repositories mirror list
If there is an error, it will show up as a little red "Error"
button. Hovering your mouse over the button will show you the error.
## How to find the right emoji?
It's possible to add "reaction emojis" to comments and issues and
merge requests in GitLab. Just hit the little smiley face and a dialog
will pop up. You can then browse through the list and pick the right
emoji for how you feel about the comment, but remember to be nice!
It's possible you get lost in the list. You can type the name of the
emoji to restrict your search, but be warned that some emojis have
[particular, non-standard names](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29057) that might not be immediately
obvious. For example, `🎉`, `U+1F389 PARTY POPPER`, is found as
`tada` in the list! See [this upstream issue for more details](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29057).
<a id="hooking-up-a-project-with-the-bots" />
## Publishing notifications on IRC
By default, new projects do *not* have notifications setup in
`#tor-bots` like all the others. To do this, you need to configure a
"Webhook", in the `Settings -> Webhooks` section of the project. The
URL should be:
https://kgb-bot.torproject.org/webhook/
... and you should select the notifications you wish to see in
`#tor-bots`. You can also enable notifications to other channels by
adding more parameters to the URL, like (say)
`?channel=tor-foo`. Important note: do not try to put the `#` in
the channel name, or if you do, URL-encode it (e.g. like `%23tor-foo`),
otherwise this will silently fail to change the target channel. Other
parameters are documented the [KGB documentation](https://salsa.debian.org/kgb-team/kgb/-/wikis/usage).
Note that GitLab admins might be able to configure [system-wide
hooks](https://gitlab.torproject.org/help/system_hooks/system_hooks) in [the admin section](https://gitlab.torproject.org/admin/hooks), although it's not entirely clear
how does relate to the per-project hooks so those have not been
enabled. Furthermore, it is possible for GitLab admins with root
access to enable webhooks on *all* projects, with the [webhook rake
task](https://docs.gitlab.com/ee/raketasks/web_hooks.html#webhooks). For example, running this on the GitLab server (currently
`gitlab-02`) will enable the above hook on all repositories:
sudo gitlab-rake gitlab:web_hook:add URL='https://kgb-bot.torproject.org/webhook/'
Note that by default, the rake task only enables `Push` events. You
need the following patch to enable others:
modified lib/tasks/gitlab/web_hook.rake
@@ -10,7 +10,19 @@ namespace :gitlab do
puts "Adding webhook '#{web_hook_url}' to:"
projects.find_each(batch_size: 1000) do |project|
print "- #{project.name} ... "
- web_hook = project.hooks.new(url: web_hook_url)
+ web_hook = project.hooks.new(
+ url: web_hook_url,
+ push_events: true,
+ issues_events: true,
+ confidential_issues_events: false,
+ merge_requests_events: true,
+ tag_push_events: true,
+ note_events: true,
+ confidential_note_events: false,
+ job_events: true,
+ pipeline_events: true,
+ wiki_page_events: true,
+ )
if web_hook.save
puts "added".color(:green)
else
See also the [upstream issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17966) and [our GitLab issue 7](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/7) for
details.
You can also remove a given hook from all repos with:
sudo gitlab-rake gitlab:web_hook:rm URL='https://kgb-bot.torproject.org/webhook/'
And, finally, list all hooks with:
sudo gitlab-rake gitlab:web_hook:list
## Setting up two-factor authentication (2FA)
We strongly recommend you enable two-factor authentication on
GitLab. This is [well documented in the GitLab manual](https://gitlab.torproject.org/help/user/profile/account/two_factor_authentication.md#two-factor-authentication), but basically:
1. first, pick a 2FA "app" (and optionally a hardware token) if you
don't have one already
2. head to your [account settings](https://gitlab.torproject.org/profile/account)
3. register your 2FA app and save the recovery codes somewhere. if
you need to enter a URL by hand, you can scan the qrcode with your
phone or create one by following this format:
otpauth://totp/$ACCOUNT?secret=$KEY&issuer=gitlab.torproject.org
where...
* `$ACCOUNT` is the `Account` field in the 2FA form
* `$KEY` is the `Key` field in the 2FA form, without spaces
4. register the 2FA hardware token if available
GitLab requires a 2FA "app" even if you intend to use a hardware
token. The 2FA "app" must implement the TOTP protocol, for example the
[Google Authenticator](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2) or a free alternative (for example [free OTP
plus](https://github.com/helloworld1/FreeOTPPlus/), see also this [list from the Nextcloud project](https://github.com/nextcloud/twofactor_totp#readme)). The
hardware token must implement the U2F protocol, which is supported by
security tokens like the [YubiKey](https://en.wikipedia.org/wiki/YubiKey), [Nitrokey](https://www.nitrokey.com/), or similar.
## Deleting sensitive attachments
If a user uploaded a secret attachment by mistake, just deleting the
issue is not sufficient: it turns out that doesn't remove the
attachments from disk!
To fix this, ask a sysadmin to find the file in the
`/var/opt/gitlab/gitlab-rails/uploads/` directory. Assuming the
attachment URL is:
<https://gitlab.torproject.org/anarcat/test/uploads/7dca7746b5576f6c6ec34bb62200ba3a/openvpn_5.png>
There should be a "hashed" directory and a hashed filename in there,
which looks something like:
./@hashed/08/5b/085b2a38876eeddc33e3fbf612912d3d52a45c37cee95cf42cd3099d0a3fd8cb/7dca7746b5576f6c6ec34bb62200ba3a/openvpn_5.png
The second directory (`7dca7746b5576f6c6ec34bb62200ba3a` above) is the
one visible in the attachment URL. The last part is the actual
attachment filename, but since those can overlap between issues, it's
safer to look for the hash. So to find the above attachement, you
should use:
find /var/opt/gitlab/gitlab-rails/uploads/ -name 7dca7746b5576f6c6ec34bb62200ba3a
And delete the file in there. The following should do the trick:
find /var/opt/gitlab/gitlab-rails/uploads/ -name 7dca7746b5576f6c6ec34bb62200ba3a | sed 's/^/rm /' > delete.sh
Verify `delete.sh` and run it if happy.
Note that GitLab is working on an [attachment manager](https://gitlab.com/gitlab-org/gitlab/-/issues/16229) that should
allow web operators to delete old files, but it's unclear how or when
this will be implemented, if ever.
## Publishing GitLab pages
GitLab features a way to publish websites directly from the continuous
integration pipelines, called [GitLab pages](https://docs.gitlab.com/ee/user/project/pages/). Complete
documentation on how to publish such pages is better served by the
official documentation, but creating a `.gitlab-ci.yml` should get you
rolling. For example, this will publish a `hugo` site:
image: registry.gitlab.com/pages/hugo/hugo_extended:0.65.3
pages:
script:
- hugo
artifacts:
paths:
- public
only:
- main
If `.gitlab-ci.yml` already contains a job in the `build` stage that
generates the required artifacts in the `public` directory, then
including the `pages-deploy.yml` CI template should be sufficient:
include:
- project: tpo/tpa/ci-templates
file: pages-deploy.yml
GitLab pages are published under the `*.pages.torproject.org` wildcard
domain. There are two types of projects hosted at the TPO GitLab:
sub-group projects, usually under the `tpo/` super-group, and user
projects, for example `anarcat/myproject`. You can also publish a page
specifically for a user. The URLs will look something like this:
| Type of GitLab page | Name of the project created in GitLab | Website URL |
|---------------------|---------------------------------------|------------------------------------------------------|
| User pages | `username.pages.torproject.net` | `https://username.pages.torproject.net` |
| User projects | `user/projectname` | `https://username.pages.torproject.net/projectname` |
| Group projects | `tpo/group/projectname` | `https://tpo.pages.torproject.net/group/projectname` |
## Accepting merge requests on wikis
Wiki permissions are not great, but there's a workaround: accept merge
requests for a git replica of the wiki.
This documentation was [moved to the documentation section](service/documentation#accepting-merge-requests-on-wikis).
## Renaming a branch globally
While `git` supports renaming branches locally with the `git branch
--move $to_name` command, this doesn't actually rename the remote
branch. That process is more involved.
Changing the name of a default branch both locally and on remotes can
be partially automated with the use of [anarcat's branch rename
script](https://gitlab.com/anarcat/scripts/-/blob/main/git-branch-rename-remote). The script basically renames the branch locally, pushes
the new branch and deletes the old one, with special handling of
GitLab remotes, where it "un-protects" and "re-protects" the branch.
You should run the script with an account that has "Maintainer" or
"Owner" access to GitLab, so that it can do the above GitLab API
changes. You will then need to provide an [access token](https://gitlab.torproject.org/-/profile/personal_access_tokens) through
the `GITLAB_PRIVATE_TOKEN` environment variable, which should have the
scope `api`.
So, for example, this will rename the `master` branch to `main` on the
local and remote repositories:
GITLAB_PRIVATE_TOKEN=REDACTED git-branch-rename-remote
If you want to rename another branch or remote, you can specify those
on the commandline as well. For example, this will rename the
`develop` branch to `dev` on the `gitlab` remote:
GITLAB_PRIVATE_TOKEN=REDACTED git-branch-rename-remote --remote gitlab --from-branch develop --to-branch dev
The command can also be used to fix *other* repositories so that they
correctly rename their local branch too. In that case, the GitLab
repository is already up to date, so there is no need for an access
token.
Other users, then can just run this command will rename `master` to
`main` on the local repository, including remote tracking branches:
git-branch-rename-remote
Obviously, users without any extra data in their local repository can
just destroy their local repository and clone a new one to get the
correct configuration.
Keep in mind that there may be a few extra steps and considerations to
make when changing the name of a heavily used branch, detailed below.
### Modifying open Merge Requests
A merge request that is open against the modified branch may be
bricked as a result of deleting the old branch name from the Gitlab
remote. To avoid this, after creating and pushing the new branch name,
edit each merge request to target the new branch name **before**
deleting the old branch.
### Updating gitolite
Many GitLab repositories are mirrored or maintained manually on
Gitolite (`git-rw.torproject.org`) and [Gitweb](https://gitweb.torproject.org). The `ssh` step for
the above automation script will fail for Gitolite and these steps
need to be done manually by a sysadmin. [Open a TPA ticket](https://gitlab.torproject.org/tpo/tpa/team/-/issues/new) with a
list of the Gitolite repositories you would like to update and a
sysadmin will perform the following magic:
cd /srv/git.torproject.org/repositories/
for repo in $list; do
git -C "$repo" symbolic-ref HEAD refs/heads/$to_branch
done
This will update Gitolite, but it won't update Gitweb until the
repositories have been pushed to. To update Gitweb immediately, ask
your friendly sysadmin to run the above command on the Gitweb server
as well.
### Updating Transifex
If your repository relies on Transifex for translations, make sure to
update the Transifex config to pull from the new branch. To do so,
[open a l10n ticket](https://gitlab.torproject.org/tpo/community/l10n/-/issues/new?issue%5Bassignee_id%5D=&issue%5Bmilestone_id%5D=) with the new branch name changes.
## Find the project associated with a project ID
Sometimes you'll find a numeric project ID instead of a human-readable
one. For example, you can see on the [arti project](https://gitlab.torproject.org/tpo/core/arti) that it says:
Project ID: 647
So you can easily find the project ID of a project right on the
project's front page. But what if you only have the ID and need to
find what project it represents? You can talk with the API, with a URL
like:
https://gitlab.torproject.org/api/v4/projects/<PROJECT_ID>
For example, this is how I found the above arti project from the
`Project ID 647`:
```
$ curl -s 'https://gitlab.torproject.org/api/v4/projects/647' | jq .web_url
"https://gitlab.torproject.org/tpo/core/arti"
```
## Connect to the PostgreSQL server
The GitLab Omnibus setup is special: it ships its own embedded
PostgreSQL server (!), which means the regular `sudo -u postgres psql`
command doesn't work.
To get access to the PostgreSQL server, you need to [follow the
upstream instructions](https://docs.gitlab.com/omnibus/maintenance/#starting-a-postgresql-superuser-psql-session) which are, at the time of writing:
sudo gitlab-psql -d gitlabhq_production
This actually expands to the following command:
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -p 5432 -h /var/opt/gitlab/postgresql -d gitlabhq_production -d gitlabhq_production
An emergency dump, therefore, could be taken with:
cd /tmp ; sudo -u gitlab-psql /opt/gitlab/embedded/bin/pg_dump -p 5432 -h /var/opt/gitlab/postgresql -d gitlabhq_production -d gitlabhq_production | pv -s 2G > /srv/gitlab-backup/db/db-$(date '+%Y-%m-%d').sql
Yes, that is silly. See also [issue 20][].
## Pager playbook
<!-- information about common errors from the monitoring system and -->
<!-- how to deal with them. this should be easy to follow: think of -->
<!-- your future self, in a stressful situation, tired and hungry. -->
* Grafana Dashboards:
* [GitLab overview](https://grafana.torproject.org/d/QrDJktiMz/gitlab-omnibus)
* [Gitaly](https://grafana.torproject.org/d/x6Z50y-iz/gitlab-gitaly)
TODO: document how to handle common problems in GitLab
### Troubleshooting
Upstream recommends running this command to self-test a GitLab
instance:
sudo gitlab-rake gitlab:check SANITIZE=true
This command also shows general info about the GitLab instance:
sudo gitlab-rake gitlab:check SANITIZE=true
it is especially useful to find on-disk files and package versions.
### GitLab pages not found
If you're looking for a way to track GitLab pages error, know that the
webserver logs are in `/var/log/nginx/gitlab_pages_access`, but that
only proxies requests for the GitLab Pages engine, which (JSON!) logs
live in `/var/log/gitlab/gitlab-pages/current`.
If you get a `"error":"domain does not exist"` problem, make sure the
entire *pipeline* actually succeeds. Typically, the "pages:deploy" job
can fail with:
Artifacts for pages are too large
In that case, you need to go into the Admin Area -> Settings ->
Preferences -> Pages and bump the size limit. It defaults to 100MB and
we bumped it to 1024MB at the time of writing. Note that GitLab CI/CD
also have a similar setting which might (or might not?) affect such
problems.
### PostgreSQL debugging
The PostgreSQL configuration in GitLab is [particular][issue 20]. See the
[connect to the PostgreSQL server](#connect-to-the-postgresql-server) section above on how to connect
to it.
### Disk full on GitLab server
If the main GitLab server is running out of space (as opposed to
runners, see [Runner disk fills up](service/ci#runner-disk-fills-up) for that scenario), then it's
projects that are taking up space. We've typically had trouble with
artifacts taking up space, for example (tpo/tpa/team#40615,
tpo/tpa/team#40517).
You can see the largest disk users in the GitLab admin area in
[Overview -> Projects -> Sort by: Largest repository](https://gitlab.torproject.org/admin/projects?sort=storage_size_desc).
Note that, although it's unlikely, it's technically possible that an
archived project takes up space, so make sure you check the "Show
archived projects" option in the "Sort by" drop down.
In the past, we have worked around that problem by reducing the
default artifact retention period from 4 to 2 weeks
(tpo/tpa/team#40516) but obviously does not take effect
immediately. More recently, we have tried to tweak individual
project's retention policies and scheduling strategies (details in
tpo/tpa/team#40615).
Please be aware of the [known upstream issues](#issues) that affect those
diagnostics as well.
To obtain a list of project sorted by space usage, log on to GitLab using an
account with administrative privileges and open the [Projects page](https://gitlab.torproject.org/admin/projects?sort=storage_size_desc)
sorted by `Largest repository`. The total space consumed by each project is
displayed and clicking on a specific project shows a breakdown of how this space
is consumed by different components of the project (repository, LFS, CI
artifacts, etc.).
If a project is consuming an unexpected amount of space for artifacts, the
scripts from the [tpo/tpa/gitlab-tools](https://gitlab.torproject.org/tpo/tpa/gitlab-tools)
project can by utilized to obtain a breakdown of the space used by job logs and
artifacts, per job or per pipeline. These scripts can also be used to manually
remove such data, see the [gitlab-tools README](https://gitlab.torproject.org/tpo/tpa/gitlab-tools/README.md).
It's also possible to compile some CI artifact usage statistics directly on the
GitLab server. To see if expiration policies work (or if "kept" artifacts or
old `job.log` are a problem), use this command (which takes a while to
run):
find -mtime +14 -print0 | du --files0-from=- -c -h | tee find-mtime+14-du.log
To limit this to `job.log`, of course, you can do:
find -name "job.log" -mtime +14 -print0 | du --files0-from=- -c -h | tee find-mtime+14-joblog-du.log
### Incoming email routing
Incoming email get routed through either eugeni or the submission
service, then end up on the Postfix server on `gitlab-02`, and from
there, to a dovecot mailbox. You can use `postfix-trace` to confirm
the message correctly ended up there.
Normally, GitLab should be picking mails from the mailbox
(`/srv/mail/git@gitlab.torproject.org/Maildir/`) regularly, and
deleting them when done. If that is not happening, look at the
mailroom logs:
tail -f /var/log/gitlab/mailroom/mail_room_json.log | jq -c
A working run will look something like this:
```
{"severity":"INFO","time":"2022-08-29T20:15:57.734+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Processing started"}
{"severity":"INFO","time":"2022-08-29T20:15:57.734+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"uid":7788,"action":"asking arbiter to deliver","arbitrator":"MailRoom::Arbitration::Redis"}.734+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Getting new messages","unread":{"count":1,"ids":[7788]},"to_be_delivered":{"count":1,"ids":[7788]}}ext":{"email":"git@gitlab.torproject.org","name":"inbox"},"uid":7788,"action":"sending to deliverer","deliverer":"MailRoom::Delivery::Sidekiq","byte_size":4162}","delivery_method":"Sidekiq","action":"message pushed"}
{"severity":"INFO","time":"2022-08-29T20:15:57.744+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Processing started"}
{"severity":"INFO","time":"2022-08-29T20:15:57.744+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Getting new messages","unread":{"count":0,"ids":[]},"to_be_delivered":{"count":0,"ids":[]}}0","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Idling"}
```
Emails should be processed every minute or so.
### Outgoing email
Follow the [email not sent](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/smtp.md#email-not-sent) procedure. TL;DR:
sudo gitlab-rails console
(Yes it takes forever.) Then check if the settings are sane:
```
--------------------------------------------------------------------------------
Ruby: ruby 3.0.5p211 (2022-11-24 revision ba5cf0f7c5) [x86_64-linux]
GitLab: 15.10.0 (496a1d765be) FOSS
GitLab Shell: 14.18.0
PostgreSQL: 12.12
------------------------------------------------------------[ booted in 28.31s ]
Loading production environment (Rails 6.1.7.2)
irb(main):003:0> ActionMailer::Base.delivery_method
=> :smtp
irb(main):004:0> ActionMailer::Base.smtp_settings
=>
{:user_name=>nil,
:password=>nil,
:address=>"localhost",
:port=>25,
:domain=>"localhost",
:enable_starttls_auto=>false,
:tls=>false,
:ssl=>false,
:openssl_verify_mode=>"none",
:ca_file=>"/opt/gitlab/embedded/ssl/certs/cacert.pem"}
```
Then test an email delivery:
Notify.test_email('noreply@torproject.org', 'Hello World', 'This is a test message').deliver_now
A working delivery will look something like this, with the last line
in *green*:
```
irb(main):001:0> Notify.test_email('noreply@torproject.org', 'Hello World', 'This is a test message').deliver_now
Delivered mail 64219bdb6e919_10e66548d042948@gitlab-02.mail (20.1ms)
=> #<Mail::Message:296420, Multipart: false, Headers: <Date: Mon, 27 Mar 2023 13:36:27 +0000>, <From: GitLab <git@gitlab.torproject.org>>, <Reply-To: GitLab <noreply@torproject.org>>, <To: noreply@torproject.org>, <Message-ID: <64219bdb6e919_10e66548d042948@gitlab-02.mail>>, <Subject: Hello World>, <Mime-Version: 1.0>, <Content-Type: text/html; charset=UTF-8>, <Content-Transfer-Encoding: 7bit>, <Auto-Submitted: auto-generated>, <X-Auto-Response-Suppress: All>>
```
A *failed* delivery will *also* say `Delivered mail` *but* will
include an error message as well. For example, in [issue 139][] we had
this error:
```
irb(main):006:0> Notify.test_email('noreply@torproject.org', 'Hello World', 'This is a test message').deliver_now
Delivered mail 641c797273ba1_86be948d03829@gitlab-02.mail (7.2ms)
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:46:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)
```
[issue 139]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/139
## Disaster recovery
In case the entire GitLab machine is destroyed, a new server should be
provisionned in the [howto/ganeti](howto/ganeti) cluster (or elsewhere) and backups
should be restored using the below procedure.
### Running an emergency backup
A full backup can be ran as root with:
/usr/bin/gitlab-rake gitlab:backup:create
Backups are stored as a tar file in `/srv/gitlab-backup` and do *not*
include secrets, which are backed up separately, for example with:
umask 0077 && tar -C /var/opt/gitlab -czf /srv/gitlab-backup/config_backup$(date +"\%Y\%m\%dT\%H\%M").tar.gz
See `/etc/cron.d/gitlab-config-backup`, and the `gitlab::backup` and
`profile::gitlab::app` classes for the actual jobs that runs nightly.
### Recovering this wiki from backups
If you need to immediately restore the wiki from backups, you can head
to the backup server and restore the directory:
/var/opt/gitlab/git-data/repositories/@hashed/11/f8/11f8e31ccbdbb7d91589ecf40713d3a8a5d17a7ec0cebf641f975af50a1eba8d.git
The hash above is the SHA256 checksum of the [wiki-replica](https://gitlab.torproject.org/tpo/tpa/wiki-replica/)
project id (695):
$ printf 695 | sha256sum
11f8e31ccbdbb7d91589ecf40713d3a8a5d17a7ec0cebf641f975af50a1eba8d -
On the backup server, that would be something like:
bconsole
restore
5
46
cd /var/opt/gitlab/git-data/repositories/@hashed/11/f8
mark 11f8e31ccbdbb7d91589ecf40713d3a8a5d17a7ec0cebf641f975af50a1eba8d.git
done
yes
The files will end up in `/var/tmp/bacula-restore` on
`gitlab-02`. Note that the number `46`, above, will vary according to
other servers backed up on the backup server, of course.
This should give you a copy of the git repository, which you can then
use, presumably, to read this procedure and restore the rest of
GitLab.
(Although then, how did you read *this* part of the procedure?
Anyways, I thought this could save your future self one day. You'll
thank me later.)
### Restoring from backups
The [upstream documentation](https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations) has a fairly good restore procedure,
but because our backup procedure is non-standard -- we exclude
repositories and artifacts, for example -- you should follow this
procedure instead.
Note that the procedure assumes some familiarity with the general
[backup and restore procedures](howto/backup), particularly how to restore a
bunch of files from the backup server (see the [restore files
section](howto/backup#restore-files).
This entire procedure will take many hours to complete. In our tests,
it took:
1. an hour or two to setup a VM
2. less than an hour to do a basic GitLab install
3. 20 minutes to restore the basic system (database, tickets are
visible at this point)
4. an hour to restore repositories
5. another hour to restore artifacts
This gives a time to recovery of about 5 to 6 hours. Most of that time
is spent waiting for files to be copied, interspersed with a few
manual commands.
So here's the procedure that was followed to deploy a development
server, from backups, in [tpo/tpa/team#40820](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40820) (run everything as
root):
1. [install GitLab using Puppet](#installation): basically create a server large
enough for everything, apply the Puppet `role::gitlab`
That includes creating new certificates and DNS records, if not
already present (those may be different if you are created a dev
server from backups, for example, which was the case for the the
above ticket).
Also note that you need to install the *same* GitLab version as
the one from the backup. If you are unsure of the GitLab version
that's in the backup (bad day uh?), try to restore the
`/var/opt/gitlab/gitlab-rails/VERSION` file from the backup server
first.
2. at this point, a blank GitLab installation should be
running. verify that you can reach the login page, possibly trying
to login with the root account, because a working GitLab
installation is a pre-requisite for the rest of the restore
procedure.
(it might be technically possible to restore the entire server
from scratch using only the backup server, but that procedure has
not been established or tested.)
3. on the backup server (currently `bacula-director-01`), restore the
latest GitLab backup job from the `/srv/gitlab-backup` and the
secrets from `/etc/gitlab`:
# bconsole
*restore
To select the JobIds, you have the following choices:
[...]
5: Select the most recent backup for a client
[...]
Select item: (1-13): 5
Defined Clients:
[...]
47: gitlab-02.torproject.org-fd
[...]
Select the Client (1-98): 47
Automatically selected FileSet: Standard Set
[...]
Building directory tree for JobId(s) 199535,199637,199738,199847,199951 ... ++++++++++++++++++++++++++++++++
596,949 files inserted into the tree.
[...]
cwd is: /
$ cd /etc
cwd is: /etc/
$ mark gitlab
84 files marked.
$ cd /srv
cwd is: /srv/
$ mark gitlab-backup
12 files marked.
$ done
This took about 20 minutes in a simulation done in June 2022,
including 5 minutes to load the file list.
4. move the files in place and fix ownership, possibly moving
pre-existing backups out of place, if the new server has been
running for more than 24 hours:
mkdir /srv/gitlab-backup.blank
mv /srv/gitlab-backup/* /srv/gitlab-backup.blank
cd /var/tmp/bacula-restores/srv/gitlab-backup
mv *.tar.gz backup_information.yml db /srv/gitlab-backup/
cd /srv/gitlab-backup/
chown git:git *.tar.gz backup_information.yml db/ db/database.sql.gz
5. stop GitLab services that talk with the database (those might have
changed since the time of writing, [review upstream documentation
just in case](https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations):
gitlab-ctl stop puma
gitlab-ctl stop sidekiq
6. restore the secrets files (note: this wasn't actually tested, but
should work):
chown root:root /var/tmp/bacula-restores/etc/gitlab/*
mv /var/tmp/bacula-restores/etc/gitlab/{gitlab-secrets.json,gitlab.rb} /etc/gitlab/
Note that if you're setting up a development environment, you do
*not* want to perform that step, which means that CI/CD variables
and 2FA tokens will be lost, which means people will need to reset
those and login with their recovery codes. This is what you want
for a dev server, because you do not want a possible dev server
compromise to escalate to the production server, or the dev server
to have access to the prod deployments.
Also note that this step was *not* performed on the dev server
test and this lead to problems during login: while it was possible
to use a recovery code to bypass 2FA, it wasn't possible to reset
the 2FA configuration afterwards.
6. restore the files:
gitlab-backup restore
This last step will ask you to confirm the restore, because it
actually destroys the existing install. It will also ask you to
confirm the rewrite of the `authorized_keys` file, which you want
to accept (unless you specifically want to restore that from
backup as well).
7. restart the services and check everything:
gitlab-ctl reconfigure
gitlab-ctl restart
gitlab-rake gitlab:check SANITIZE=true
gitlab-rake gitlab:doctor:secrets
gitlab-rake gitlab:lfs:check
gitlab-rake gitlab:uploads:check
gitlab-rake gitlab:artifacts:check
Note: in the simulation, GitLab was started like this instead,
which just worked as well:
gitlab-ctl start puma
gitlab-ctl start sidekiq
We did try the "verification" tasks above, but many of them
failed, especially in the `gitlab:doctor:secrets` job, possibly
because we didn't restore the secrets (deliberately).
At this point, basic functionality like logging-in and issues should
be working again, but not wikis (because they are not restored
yet). Note that it's **normal** to see a 502 error message ("Whoops,
GitLab is taking too much time to respond.") when GitLab restarts: it
takes a *long* time to start (think minutes)... You can follow its
progress in `/var/log/gitlab/gitlab-rails/*.log`.
Be warned that the new server *will* start sending email
notifications, for example for issues with an due date, which might be
confusing for users if this is a development server. If this is a
production server, that's a good thing. If it's a development server,
you may want to disable email altogether in the GitLab server, with
this line in Hiera data (eg. `hiera/roles/gitlab_dev.yml`) in the
`tor-puppet.git` repository:
profile::gitlab::app::email_enabled: false
So the above procedure only restores a *part* of the system, namely
what is covered by the nightly backup job. To restore the rest (at the
time of writing: artifacts and repositories, which includes wikis!),
you also need to specifically restore those files from the backup
server.
For example, this procedure will restore the repositories from the
backup server:
$ cd /var/opt/gitlab/git-data
cwd is: /var/opt/gitlab
$ mark repositories
113,766 files marked.
$ done
The files will then end up in
`/var/tmp/bacula-restores/var/opt/gitlab/git-data`. They will need to
be given to the right users and moved into place:
chown -R git:root /var/tmp/bacula-restores/var/opt/gitlab/git-data/repositories
mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.orig
mv /var/tmp/bacula-restores/var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories/
During the last simulation, restoring repositories took an hour.
Restoring artifacts is similar:
$ cd /srv/gitlab-shared
cwd is: /srv/gitlab-shared/
$ mark artifacts
434,788 files marked.
$ done
Then the files need to be given and moved as well, notice the
`git:git` instead of `git:root`:
chown -R git:git /var/tmp/bacula-restores/srv/gitlab-shared/artifacts
mv /var/opt/gitlab/gitlab-rails/shared/artifacts/ /var/opt/gitlab/gitlab-rails/shared/artifacts.orig
mv /var/tmp/bacula-restores/srv/gitlab-shared/artifacts /var/opt/gitlab/gitlab-rails/shared/artifacts/
Restoring the artifacts took another hour of copying.
And that's it! Note that this procedure may vary if the subset of
files backed up by the GitLab backup job changes.
# Reference
## Installation
### Main GitLab installation
The current GitLab server was setup in the [howto/ganeti](howto/ganeti) cluster in a
regular virtual machine. It was configured with [howto/puppet](howto/puppet) with the
`roles::gitlab`. That, in turn, includes a series of `profile`
classes which configure:
* `profile::gitlab::web`: nginx vhost and TLS cert, which depends on
`profile::nginx` built for the [howto/cache](howto/cache) service and relying on the
[puppet/nginx](https://forge.puppet.com/puppet/nginx) module from the Forge
* `profile::gitlab::app`: the core of the configuration of gitlab
itself, uses the [puppet/gitlab](https://forge.puppet.com/puppet/gitlab) module from the Forge, with
Prometheus, Grafana, and Nginx support disabled, but Redis,
PostgreSQL, and other exporters enabled
* `profile::dovecot::private`: a simple IMAP server to receive mails
destined to GitLab
[issue 20]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20
This installs the [GitLab Omnibus](https://docs.gitlab.com/omnibus/) distribution which duplicates a
lot of resources we would otherwise manage elsewhere in Puppet,
including (but possibly not limited to):
* [howto/prometheus](howto/prometheus) exporters (see [issue 40077](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40077) for example)
* [howto/postgresql](howto/postgresql)
* redis
This therefore leads to a "particular" situation regarding monitoring
and PostgreSQL backups, in particular. See [issue 20](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20) for details.
The install takes a *long* time to complete. It's going to take a few
minutes to download, unpack, and configure GitLab. There's no precise
timing of this procedure yet, but assume each of those steps takes
about 2 to 5 minutes.
After the install, the administrator account details are stored in
`/etc/gitlab/initial_root_password`. After logging in, you most likely
want to [disable new signups](https://docs.gitlab.com/ee/user/admin_area/settings/sign_up_restrictions.html#disable-new-sign-ups) as recommended, or possibly restore
from backups.
Note that the first gitlab server (gitlab-01) was setup using the
Ansible recipes used by the Debian.org project. That install was not
working so well (e.g. [503 errors on merge requests](https://gitlab.torproject.org/tpo/tpa/team/-/issues/32197)) so we
[migrated to the omnibus package](https://gitlab.torproject.org/tpo/tpa/team/-/issues/32949) in March 2020, which seems to
work better. There might still be some leftovers of that configuration
here and there, but some effort was done during the 2022 hackweek
(2022-06-28) to clean that up in Puppet at least. See
[tpo/tpa/gitlab#127](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/127) for some of that cleanup work.
### GitLab CI installation
See [the CI documentation](service/ci) for documentation specific to GitLab CI.
### GitLab pages installation
To setup GitLab pages, we [followed the GitLab Pages administration
manual](https://docs.gitlab.com/ee/administration/pages). The steps taken were as follows:
1. add `pages.torproject.net` to the [public suffix list](https://publicsuffix.org/) ([issue
40121](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40121) and [upstream PR](https://github.com/publicsuffix/list/pull/1196)) (although that takes months or
*years* to propagate everywhere)
1. add `*.pages.torproject.net` and `pages.torproject.net` to DNS
(`dns/domains.git` repository), as A records so that LE DNS-01
challenges still work, along with a CAA record to allow the
wildcard on `pages.torproject.net`
2. get the wildcard cert from Let's Encrypt (in
`letsencrypt-domains.git`)
3. deploy the TLS certificate, some GitLab config and a nginx vhost to gitlab-02
with Puppet
4. run the [status-site pipeline](https://gitlab.torproject.org/tpo/tpa/status-site/-/pipelines) to regenerate the pages
The GitLab pages configuration lives in the `profile::gitlab::app`
Puppet class. The following GitLab settings were added:
gitlab_pages => {
ssl_certificate => '/etc/ssl/torproject/certs/pages.torproject.net.crt-chained',
ssl_certificate_key => '/etc/ssl/private/pages.torproject.net.key',
},
pages_external_url => 'https://pages.torproject.net',
The virtual host for the `pages.torproject.net` domain was configured
through the `profile::gitlab::web` class.
## SLA
<!-- this describes an acceptable level of service for this service -->
## Design
<!-- how this is built -->
<!-- should reuse and expand on the "proposed solution", it's a -->
<!-- "as-built" documented, whereas the "Proposed solution" is an -->
<!-- "architectural" document, which the final result might differ -->
<!-- from, sometimes significantly -->
<!-- a good guide to "audit" an existing project's design: -->
<!-- https://bluesock.org/~willkg/blog/dev/auditing_projects.html -->
GitLab is a fairly large program with multiple components. The
[upstream documentation](https://docs.gitlab.com/ee/development/architecture.html) has a good details of the architecture but
this section aims at providing a shorter summary. Here's an overview
diagram, first:
Note: the above image may be broken as upstream frequently changes the
URL. It should be visible in the [Simplified component overview](https://docs.gitlab.com/ee/development/architecture.html#simplified-component-overview) of
the [architecture documentation](https://docs.gitlab.com/ee/development/architecture.html).
The web frontend is Nginx (which we incidentally also use in our
[howto/cache](howto/cache) system) but GitLab wrote their own reverse proxy called
[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse/) which in turn talks to the underlying GitLab
Rails application, served by the [Unicorn](https://yhbt.net/unicorn/) application
server. The Rails app stores its data in a [howto/postgresql](howto/postgresql) database
(although not our own deployment, for now, [should be fixed](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20)). GitLab also offloads
long-term background tasks to a tool called [sidekiq](https://github.com/mperham/sidekiq).
Those all server HTTP(S) requests but GitLab is of course also
accessible over SSH to push/pull git repositories. This is handled by
a separate component called [gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell) which acts as a shell
for the `git` user.
Workhorse, Rails, sidekiq and gitlab-shell all talk with Redis to
store temporary information, caches and session information. They can
also communicate with the [Gitaly](https://gitlab.com/gitlab-org/gitaly) server which handles all
communication with the git repositories themselves.
### Continuous integration
GitLab also features Continuous Integration (CI). CI is handled by
[GitLab runners](https://gitlab.com/gitlab-org/gitlab-runner/) which can be deployed by anyone and registered in
the Rails app to pull CI jobs. This is documented in the [service/ci
page](service/ci).
### Spam control
TODO: document lobby.
Discuss alternatives, e.g. [this hackernews discussion about mediawiki
moving to gitlab](https://news.ycombinator.com/item?id=24919569). Their [gitlab migration](https://www.mediawiki.org/wiki/GitLab_consultation) documentation might
give us hints on how to improve the spam situation on our end.
A few ideas on tools:
* [Tornevall blocklist](https://www.tornevall.net/about/)
* [Mediawiki spam control tricks](https://m.mediawiki.org/wiki/Manual:Combating_spam)
* [Friendly CAPTCHA](https://friendlycaptcha.com/), [considered for inclusion in GitLab](https://gitlab.com/gitlab-org/gitlab/-/issues/273480)
### Scalability
We have not looked a lot into GitLab scalability. Upstream has
[reference architectures](https://docs.gitlab.com/ee/administration/reference_architectures/) which explain how to scale for various
user sizes. We have not yet looked into this, and so far have just
thrown hardware at GitLab when performance issues come up.
### GitLab pages
[GitLab pages](https://gitlab.com/gitlab-org/gitlab-pages) is "a simple HTTP server written in Go, made to
serve GitLab Pages with CNAMEs and SNI using HTTP/HTTP2". In practice,
the way this works is that artifacts from GitLab CI jobs get sent back
to the central server.
GitLab pages is designed to scale horizontally: multiple pages servers
can be deployed and fetch their content and configuration through NFS.
They are [rearchitecturing this with Object storage](https://docs.gitlab.com/ee/architecture/blueprints/cloud_native_gitlab_pages/) (ie. S3
through minio by default, or external existing providers) which might
simplify running this but this actually adds complexity to a
previously fairly simple design. Note that they have tried using
CephFS instead of NFS but that did not work for some reason.
The [new pages architecture](https://docs.gitlab.com/ee/architecture/blueprints/cloud_native_gitlab_pages/) also relies on the GitLab rails API
for configuration (it was a set of JSON files before), which makes it
dependent on the Rails API for availability, although [that part of
the design](https://gitlab.com/groups/gitlab-org/-/epics/4242) has [exponential back-off time](https://gitlab.com/groups/gitlab-org/-/epics/4242) for unavailability
of the rails API, so maybe it would survive a downtime of the rails
API.
GitLab pages is not currently in use in our setup, but could be used
as an alternative to the [static mirroring system](howto/static-component). See the
[discussion there](howto/static-component#alternatives-considered) for more information about how that compares
with the static mirror system.
Update: [some tests of GitLab pages](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/91) were performed in January
2021, with moderate success. There are still concerns about the
reliability and scalability of the service, but the service could be
used for small sites at this stage. See the [GitLab pages installation
instructions](#gitlab-pages-installation) for details on how this was setup.
Note that the pages are actually on disk, in
`/var/opt/gitlab/gitlab-rails/shared/pages/GROUP/.../PROJECT`, for
example the status site pipeline publishes to:
/var/opt/gitlab/gitlab-rails/shared/pages/tpo/tpa/status-site/
Maybe this could be abused to act as a static source in the static
mirror system?
Update: see [service/static-shim](service/static-shim) for the chosen
solution to deploy websites built in GitLab CI to the static mirror
system.
## Issues
[File][] or [search][] for issues in the [gitlab project][search].
[File]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/new
[search]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues
### Known
* Wikis:
* [Wikis are not publicly editable](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/76)
* [non-existent wiki pages are not 404 errors](https://gitlab.com/gitlab-org/gitlab/-/issues/361471), also [List all
dead links in wiki](https://gitlab.com/gitlab-org/gitlab/-/issues/39518)
* [support changing the default wiki branch name](https://gitlab.com/gitlab-org/gitlab/-/issues/364109)
* Issues:
* [Issues warn about LFS](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/59)
* [Confidential issues leak cleartext by email](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/23) (see the [Note
about confidential issues](#note-about-confidential-issues) above)
* [Cannot move issues to projects I do not maintain](https://gitlab.com/gitlab-org/gitlab/-/issues/331610)
* General:
* Does not [allow users to select 12 vs 24-hour format](https://gitlab.com/gitlab-org/gitlab/-/issues/15206), and
[fails to detect fr-CA locale](https://gitlab.com/gitlab-org/gitlab/-/issues/391747); workaround: use en-GB
See also [issues YOU have voted on](https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=updated_desc&state=opened&my_reaction_emoji=Any).
### Resolved
* [keep gitlab artifacts disk space usage under control](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/119), resolved
through a home-made script ([gitlab-pipeline-vacuum](https://gitlab.torproject.org/tpo/tpa/gitlab-tools/-/blob/main/gitlab-pipeline-vacuum)) but also
upstream, partially: [Clean up old expired artifacts for self-managed
instances](https://gitlab.com/gitlab-org/gitlab/-/issues/353128) is done, but not:
* [Unlock latest artifacts on stale branches](https://gitlab.com/gitlab-org/gitlab/-/issues/296779)
* [After deleting build artifacts, empty logfile and empty
artifacts folder still persists on the filesystem](https://gitlab.com/gitlab-org/gitlab/-/issues/17465)
* [How to "unkeep" artifacts?](https://gitlab.com/gitlab-org/gitlab/-/issues/289954)
* [Add BuildLog expiration options](https://gitlab.com/gitlab-org/gitlab/-/issues/17245)
* [wiki anchors don't work](https://gitlab.com/gitlab-org/gitlab/-/issues/362163)
## Monitoring and testing
Monitoring right now is minimal: normal host-level metrics like disk
space, CPU usage, web port and TLS certificates are monitored by
Nagios with our normal infrastructure, as a black box.
Prometheus monitoring is built into the GitLab Omnibus package, so it
is *not* configured through our Puppet like other Prometheus
servers. It has still been (manually) integrated in our Prometheus
setup and Grafana dashboards (see [pager playbook](#pager-playbook)) have been deployed.
One problem with the current monitoring is that the GitLab exporters
are [currently hardcoded](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40077).
We could also use the following tools to integrate alerting into
GitLab better:
* [moosh3/gitlab-alerts](https://github.com/moosh3/gitlab-alerts): autogenerate issues based from Prometheus
Alert Manager (with the webhook)
* [FUSAKLA/prometheus-gitlab-notifier](https://github.com/FUSAKLA/prometheus-gitlab-notifier): similar
* 11.5 shipped [a bunch of alerts](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45740) which we might want to use
directly
* the "Incident management" support has various [integrations](https://docs.gitlab.com/ee/operations/incident_management/integrations.html)
including Prometheus (starting from 13.1) and Pagerduty (which is
supported by Prometheus)
We also lack visibility on certain key aspects of GitLab. For example,
it would be nice to [monitor issue counts in Prometheus](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40591) or have
better monitoring of GitLab pipelines like wait time, success/failure
rates and so on. There was an issue open about [monitoring individual
runners](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41042) but the runners do not expose (nor do they have access to)
that information, so that was scrapped.
There used to be a development server called `gitlab-dev-01` that
could be used to test dangerous things if there is a concern a change
could break the production server, but it was retired, see
[tpo/tpa/team#41151][] for details.
[tpo/tpa/team#41151]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41151
## Logs and metrics
GitLab keeps an extensive (excessive?) amount of logs, in
`/var/log/gitlab`, which includes PII, including IP addresses.
To see live logs, you can type the handy command:
gitlab-ctl tail
... but that is sort of like drinking from a fire hose. You can
inspect the logs of a specific component by passing it as an argument,
for example to inspect the mail importer:
gitlab-ctl tail mailroom
Each component is in his own directory, so the equivalent to the above
is:
tail -f /var/log/gitlab/mailroom/{current,mail_room_json.log}
Notice how both regular and JSON logs are kept.
Logs seem to be kept for a month.
## Backups
There is a backup job ( `tpo-gitlab-backup`, in the `root` user
crontab) that is a simple wrapper script which calls `gitlab-backup`
to dump some components of the GitLab installation in the backup
directory (`/srv/gitlab-backup`).
The backup system is deployed by Puppet and (*at the time of
writing*!) **skips** *repositories** and **artifacts**. It contains:
* GitLab CI build logs (`builds.tar.gz`)
* a compressed database dump (`db/database.sql.gz`)
* Git Large Files (Git LFS, `lfs.tar.gz`)
* packages (`packages.tar.gz`)
* GitLab pages (`pages.tar.gz`)
* some terraform thing (`terraform_state.tar.gz`)
* uploaded files (`uploads.tar.gz`)
The backup job is ran nightly. GitLab also creates a backup on
upgrade. Those are purged after two weeks by the wrapper script.
The backup job does **NOT** contain those components because they take
up a tremendous amount of disk space, and are already backed up by
Bacula. Those need to be restored from the regular backup server,
separately:
* Git repositories (found in
`/var/opt/gitlab/git-data/repositories/`)
* GitLab CI artifacts (normally found in
`/var/opt/gitlab/gitlab-rails/shared/artifacts/`, in our case
bind-mounted over `/srv/gitlab-shared/artifacts`)
It is assumed that the existing [howto/backup](howto/backup) system
will pick up those files, but also the actual backup files in
`/srv/gitlab-backup` and store them for our normal rotation periods.
This implies that the files covered by the `gitlab-backup` job are
*also* already backed up by Bacula and are therefore duplicated on the
backup storage server. See [issue 40518][] to review that strategy.
Ideally, this rather exotic backup system would be harmonized with our
existing backup system, but this would require (for example) using our
existing PostgreSQL infrastructure ([issue 20][]). Other ideas
(including filesystem snapshots) are also in [issue 40518][].
[issue 40518]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40518
## Other documentation
* GitLab has a [built-in help system](https://gitlab.torproject.org/help)
* [Support forum](https://forum.gitlab.com/)
# Discussion
## Meetings
Some meetings about tools discussed GitLab explicitly. Those are the minutes:
- [2020 September 15th](http://meetbot.debian.net/tor-meeting/2020/tor-meeting.2020-09-15-15.00.html)
- [2020 July 7th](http://meetbot.debian.net/tor-meeting/2020/tor-meeting.2020-07-07-15.08.log.html)
## Overview
<!-- describe the overall project. should include a link to a ticket -->
<!-- that has a launch checklist -->
The GitLab project at Tor has been a long time coming. If you look at
the [history](#history) section above, you'll see it has been worked on since
at least 2016, at which point an external server was setup for the
"network team" to do code review. This server was ultimately retired.
The current server has been worked on since 2019, with the master
ticket, [issue 29400](https://gitlab.torproject.org/tpo/tpa/services/-/issues/29400), created in the footsteps of the [2019
Brussels meeting](https://trac.torproject.org/projects/tor/wiki/org/meetings/2019BrusselsAdminTeamMinutes). The service launched some time in June 2020,
with a full migration of Trac tickets.
## Goals
### Must have
* replacement of the Trac issue tracking server
* rough equivalent of Trac features in GitLab
### Nice to have
* identical representation of Trac issues in GitLab, including proper
issue numbering
### Non-Goals
* replacement of Gitolite (git hosting)
* replacement of Gitweb (git hosting)
* replacement of Jenkins (CI) -- although that was eventually done
* replacement of the static site hosting system
Those are not part of the first phase of the project, but it is
understood that if one of those features gets used more heavily in
GitLab, the original service MUST be eventually migrated into GitLab
and turned off. We do *not* want to run multiple similar services at
the same time (for example run both gitolite and gitaly on all git
repositories, or run Jenkins and GitLab runners).
## Approvals required
The GitLab migration was approved at the 2019 Brussels dev meeting.
## Proposed Solution
The solution to the "code review" and "project management" problems
are to deploy a GitLab instance which does *not* aim at managing all
source code, in the first stage.
## Cost
Staff not evaluated.
In terms of hardware, we start with a single virtual machine and agree
that, in the worst case, we can throw a full Hetzner PX62-NVMe node at
the problem (~70EUR/mth).
## Alternatives considered
GitLab is such a broad project that multiple alternatives exist for
different components:
* GitHub
* Pros:
* widely used in the open source community
* Good integration between ticketing system and code
* Cons
* It is hosted by a third party (Microsoft!)
* Closed source
* GitLab:
* Pros:
* Mostly free software
* Feature-rich
* Cons:
* Complex software, high maintenance
* "Opencore" - some interesting features are closed-source
### GitLab command line clients
If you want to do batch operations or integrations with GitLab, you
might want to use one of those tools, depending on your environment or
prefered programming language:
* [bugwarrior](https://github.com/ralphbean/bugwarrior) ([Debian](https://tracker.debian.org/pkg/bugwarrior)) - support for GitLab, GitHub and
other bugtrackers for the [taskwarrior](http://taskwarrior.org/) database
* [git-lab](https://invent.kde.org/sdk/git-lab) - python commandline client, lists, pulls MR; creates
snippets
* [GitLab-API-v4](https://metacpan.org/release/GitLab-API-v4) ([Debian](https://tracker.debian.org/pkg/libgitlab-api-v4-perl)) - perl library and [commandline
client](https://manpages.debian.org/buster/libgitlab-api-v4-perl/gitlab-api-v4.1p.en.html)
* [GitLabracadabra](https://gitlab.com/gitlabracadabra/gitlabracadabra) ([Debian](https://tracker.debian.org/pkg/gitlabracadabra)) - *configure a GitLab instance
from a YAML configuration, using the API*: project settings like
labels, admins, etc
* [glab](https://github.com/profclems/glab) ([not in Debian](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969482)) - inspired by GitHub's official `gh`
client
* [python-gitlab](https://github.com/python-gitlab/python-gitlab) (also known as `gitlab-cli` in [Debian](https://tracker.debian.org/pkg/python-gitlab))
* [ruby-gitlab](https://github.com/narkoz/gitlab) ([Debian](https://tracker.debian.org/pkg/ruby-gitlab)), also includes a [commandline
client](https://narkoz.github.io/gitlab/)
* [salsa](https://manpages.debian.org/buster/devscripts/salsa.1.en.html) (in [Debian devscripts](https://tracker.debian.org/pkg/devscripts)) is specifically built for
salsa but might be coerced into talking to other GitLab servers
GitLab upstream has a [list of third-party commandline tools](https://about.gitlab.com/partners/#cli-clients) that
is interesting as well.
### Migration tools
ahf implemente the gitlab using his own home-made tools that talk to
the GitLab and Trac API. but there's also [tracboat](https://github.com/tracboat/tracboat) which is
designed to migrate from trac to GitLab.
We did not use Tracboat because it uses gitlab's DB directly and thus
only works with some very specific version. Each time the database
schema changes at GitLab, Tracboat needs to port to it. We prefered to
use something that talked with the GitLab API.
We also didn't like the output entirely, so we modified it but still
used some of its regular expressions and parser.
We also needed to implement the "ticket movement" hack (with the legacy
project) which wasn't implemented in Tracboat.
Finally, we didn't want to do complete user migration, but lazily
transfer only some users.
## Git repository integrity solutions
> This section is a summary of the discussion in [ticket
> tpo/tpa/gitlab#81](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81). A broader discussion of the security issues
> with GitLab vs Gitolite and the choices made during that migration
> are available in [Gitolite: security concerns](howto/git#security-concerns).
Some developers expressed concerns about using GitLab as a canonical
location for source code repositories, mainly because of the much
broader attack surface GitLab provides, compared to the legacy,
[gitolite-based infrastructure](howto/git), especially considering
that the web application basically has write access to everything.
One solution to this problem is to use cryptographic signatures. We
already use OpenPGP extensively in the Tor infrastructure, and it's
well integrated in git, so it's an obvious candidate. But it's not
necessarily obvious how OpenPGP would be used to sign code inside Tor,
so this section provides a short review of existing solutions in that
space.
### Guix: sign all commits
[Guix uses OpenPGP to sign commits](https://guix.gnu.org/blog/2020/securing-updates/), using an approach that is
basically:
> 1. The repository contains a .guix-authorizations file that lists
> the OpenPGP key fingerprints of authorized committers.
> 2. A commit is considered authentic if and only if it is signed by
> one of the keys listed in the .guix-authorizations file of each
> of its parents. This is the authorization invariant.
>
> \[...\] Since .guix-authorizations is a regular file under version
> control, granting or revoking commit authorization does not require
> special support.
Note the big caveat:
> It has one downside: it prevents pull-request-style
> workflows. Indeed, merging the branch of a contributor not listed in
> .guix-authorizations would break the authorization invariant. It’s a
> good tradeoff for Guix because our workflow relies on patches carved
> into stone tablets (patch tracker), but it’s not suitable for every
> project out there.
Also note there's a bootstrapping problem in their design:
> Which commit do we pick as the first one where we can start
> verifying the authorization invariant?
They solve this with an out of band "*channel introduction*" mechanism
which declares a good hash and a signing key.
This also requires a custom client. But it serves as a good example of
an extreme approach (validate everything) one could take.
Note that GitLab Premium (non-free) has support for [push rules](https://docs.gitlab.com/ee/user/project/repository/push_rules.html)
and in particular a "Reject unsigned commits" rule.
Another implementation is SourceWare's [gitsigur](https://sourceware.org/git/gitsigur.git) which verifies
all commits (200 lines Python script), see also [this discussion](https://inbox.sourceware.org/overseers/ZJ3Tihvu6GbOb8%2FR@elastic.org/T/)
for a comparison. A similar project is Gentoo's [update-02-gpg](https://gitweb.gentoo.org/infra/githooks.git/tree/local/update-02-gpg)
bash script.
### Arista: sign all commits in Gerrit
Arista wrote a blog post called [Commit Signing with Git at Enterprise
Scale](https://eos.arista.com/commit-signing-with-git-at-enterprise-scale/) ([archive](https://web.archive.org/web/20201020180304/https://eos.arista.com/commit-signing-with-git-at-enterprise-scale/)) which takes a radically different approach.
* all OpenPGP keys are centrally managed (which solves the "web of
trust" mess) in a [Vault](https://www.vaultproject.io/)
* Gerrit is the gatekeeper: for patches to be merged, they must be
signed by a trusted key
It is a rather obtuse system: because the final patches are rebased on
top of the history, the git signatures are actually lost so they have
a system to keep a reference to the Gerrit change id in the git
history, which does have a copy of the OpenPGP signature.
### Gerwitz: sign all commits or at least merge commits
Mike Gerwitz wrote an [article in 2012](https://mikegerwitz.com/2012/05/a-git-horror-story-repository-integrity-with-signed-commits) (which he warns is out of
date) but which already correctly identified the issues with merge and
rebase workflows. He argues there is a way to implement the desired
workflow by signing *merges*: because maintainers are the one
committing merge requests to the tree, they are in a position to
actually sign the code provided by third-parties. Therefore it can be
assume that if a merge commit is signed, then the code it imported is
also signed.
The article also provides a crude checking script for such a scenario.
Obviously, in the case of GitLab, it would make the "merge" button
less useful, as it would break the trust chain. But it's possible to
merge "out of band" (in a local checkout) and push the result, which
GitLab generally correctly detect as closing the merge request.
### Torvalds: signed tags
Linus Torvalds, the original author and maintainer of the Linux
kernel, simply signs the release tags. In an article called "[what
does a pgp signature on a git commit prove?](https://people.kernel.org/monsieuricon/what-does-a-pgp-signature-on-a-git-commit-prove)", Konstantin Ryabitsev
(the `kernel.org` sysadmin), provides a good primer on OpenPGP signing
in git. It also shows how to validate Linux releases by checking the
tag and argues this is sufficient to ensure trust.
### Vick: git signatures AKA git notes
The [git-signatures](https://github.com/hashbang/git-signatures/) project, authored by [Lance R. Vick](https://github.com/lrvick), makes
it possible to "*attach an arbitrary number of GPG signatures to a
given commit or tag.*":
> Git already supports commit signing. These tools are intended to
> compliment that support by allowing a code reviewer and/or release
> engineer attach their signatures as well.
Downside: third-party tool not distributed with git and not packaged
in Debian.
The idea of using git-notes was also [proposed by Owen Jacobsen](https://grimoire.ca/git/detached-sigs/).
### Walters: extended validation tags
The [git-evtag](https://github.com/cgwalters/git-evtag) projects from Colin Walters tries to address the
perceived vulnerability of the SHA-1 hash by implementing a new
signing procedure for tags, based on SHA-512 and OpenPGP.
### Ryabitsev: b4 and patch attestations
Konstantin Ryabitsev (the kernel.org sysadmin, again) proposed a new
cryptographic scheme to sign patches in Linux, he called "[patch
attestation](https://people.kernel.org/monsieuricon/introducing-b4-and-patch-attestation)". The protocol is designed to survive mailing list
transports, rebases and all sorts of mangling. It does not use GnuPG
and is based on a Trust On First Use (TOFU) model.
The model is not [without critics](https://lwn.net/Articles/813646/#Comments).
Update, 2021-06-04: there was [another iteration](https://people.kernel.org/monsieuricon/end-to-end-patch-attestation-with-patatt-and-b4) of that concept,
this time based on DKIM-like headers, with support for OpenPGP
signatures but also "native" ed25519.
One key takeaway from this approach, which we could reuse, is the way
public keys are stored. In [patatt](https://pypi.org/project/patatt/), the git repository itself
holds the public keys:
> On the other hand, within the context of git repositories, we
> already have a suitable mechanism for distributing developer public
> keys, which is the repository itself. Consider this:
>
> * git is already decentralized and can be mirrored to multiple
> locations, avoiding any single points of failure
> * all contents are already versioned and key additions/removals can
> be audited and “git blame’d”
> * git commits themselves can be cryptographically signed, which
> allows a small subset of developers to act as “trusted
> introducers” to many other contributors (mimicking the
> “keysigning” process)
>
> The idea of using git itself for keyring management was originally
> suggested by the did:git project, though we do not currently
> implement the proposed standard itself.
>
> <https://github.com/dhuseby/did-git-spec/blob/master/did-git-spec.md>
It's unclear, however, why the latter spec wasn't reused. To be
investigated.
Update, 2022-04-20: someone actually went through the trouble of
[auditing the transparency log](https://tlog.linderud.dev/), which is an interesting exercise
in itself. The [verifier source code](https://github.com/Foxboron/kernel.org-git-verifier) is available, but probably
too specific to Linux for our use case. [Their notes are also
interesting](https://linderud.dev/blog/monitoring-the-kernel.org-transparency-log-for-a-year/).
### Ryabitsev: Secure Scuttlebutt
A more exotic proposal is to [use the Secure Scuttlebutt (SSB)
protocol](https://people.kernel.org/monsieuricon/patches-carved-into-developer-sigchains) instead of emails to exchange (and also, implicitly) sign
git commits. There is even a [git-ssb](https://github.com/clehner/git-ssb) implementation, although
it's hard to see because it's been migrated to .... SSB!
Obviously, this is not quite practical and is shown only as a more
radical example, as a stand-in for the other end of the
decentralization spectrum.
### Stelzer: ssh signatures
Fabian Stelzer made a [pull request for git](https://github.com/git/git/pull/1041) which was actually
[merged](https://github.com/git/git/commit/18c6653da0be924f83415f987d76f6813b81f086) in October 2021 and therefore might make it to 2.34. The
PR adds support for SSH signatures on top of the already existing
OpenPGP and X.509 systems that git already supports.
It does not address the above issues of "which commits to sign" or
"where to store keys", but it does allow users to drop the
OpenPGP/GnuPG dependency if they so desire. Note that there may be
compatibility issues with different OpenSSH releases, as the PR
explicitly says:
> I will add this feature in a follow up patch afterwards since the
> released 8.7 version has a broken ssh-keygen implementation which
> will break ssh signing completely.
We do not currently have plans to get rid of OpenPGP internally, but
it's still nice to have options.
### Lorenc: sigstore
[Dan Lorenc][], an engineer at Google, designed a tool that allows
users to sign "artifacts". Typically, those are container images
(e.g. [cosign][] is named so because it signs "containers"), but
anything can be signed.
[Dan Lorenc]: https://github.com/dlorenc
[cosign]: https://github.com/sigstore/cosign
It also works with a transparency log server called [rekor][]. They
run a public instance, but we could also run our own. It is currently
unclear if we could have both, but it's apparently possible to run a
"monitor" that would check the log for consistency.
[rekor]: https://github.com/sigstore/rekor
There's also a system for [signing binaries with ephemeral keys][]
which seems counter-intuitive but actually works nicely for CI jobs.
[signing binaries with ephemeral keys]: https://shibumi.dev/posts/first-look-into-cosign/
Seems very promising, maintained by Google, RedHat, and supported by
the Linux foundation. Complementary to [in-toto][] and [TUF][]. TUF is
actually used to create the [root keys][] which are controlled, at
the time of writing, by:
[in-toto]: https://github.com/in-toto/in-toto
[TUF]: https://theupdateframework.io/
[root keys]: https://github.com/sigstore/root-signing
* [Bob Callaway][] (Google)
* [Dan Lorenc][] (Google)
* [Luke Hinds][] (RedHat)
* [Marina Moore][] (NYU)
* [Santiago Torres][] (Purdue)
[Luke Hinds]: https://github.com/lukehinds
[Marina Moore]: https://github.com/mnm678
[Santiago Torres]: https://github.com/SantiagoTorres
[Bob Callaway]: https://github.com/bobcallaway
Update: [gitsign](https://github.com/sigstore/gitsign) is specifically built to use this infrastructure
for Git. GitHub and GitLab are currently lacking support for verifying
those signatures. See [tutorial](https://dev.to/erikaheidi/enable-gitsign-today-and-start-signing-your-commits-2gda).
Similar projects:
* [SLSA](https://slsa.dev/), which has a well documented [threat model](https://slsa.dev/spec/v0.1/threats)
* [Trillian](https://github.com/google/trillian-examples) (Google)
* [sigsum](https://www.sigsum.org/), similar to sigstore, but [more minimal](https://git.sigsum.org/sigsum/tree/archive/2022-03-15-notes-on-sigsum-and-rekor.md)
### Other caveats
Also note that git has limited security guarantees regarding
checksums, since it uses SHA-1, but [that is about to change](https://lwn.net/Articles/823352/). Most
Git implementations also have protections against collisions, see for
example [this article from GitHub](https://github.com/blog/2338-sha-1-collision-detection-on-github-com).
There are, of course, a large number of usability (and some would say
security) issues with OpenPGP (or, more specifically, the main
implementation, GnuPG). There has even been [security issues with
signed Git commits](https://mgorny.pl/articles/attack-on-git-signature-verification.html), specifically.
So I would also be open to [alternative signature verification
schemes](https://blog.gtank.cc/modern-alternatives-to-pgp/). Unfortunately, none of those are implemented in git, as
far as I can tell.
There are, however, alternatives to GnuPG itself. [This article from
Saoirse Shipwreckt](https://boats.gitlab.io/blog/post/signing-commits-without-gpg/) shows how to verify commits without GnuPG, for
example. That still relies on OpenPGP keys of course...
... which brings us to the web of trust and key distribution
problems. The OpenPGP community is in this problematic situation right
now where the traditional key distribution mechanisms (the old
keyserver network) has been under attack and is not as reliable as it
should be. This brings the question of keyring management, but that is
already being discussed in [tpo/tpa/team#29671](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29671).
Finally, note that OpenPGP keys are not permanent: they can be
revoked, or expired. Dealing with this problem has its specific [set
of solutions](https://karl.kornel.us/2017/10/welp-there-go-my-git-signatures/) as well. GitHub [marks signatures as verified](https://github.blog/changelog/2022-05-31-improved-verification-of-historic-git-commit-signatures/) for
expired or revoked (but not compromised) keys, but has a special
mouse-over showing exactly what's going on with that key, which seems
like a good compromise.
### Related
* [gitid](https://github.com/Luiserebii/gitid): easier identity management for git
* [signed git pushes](https://people.kernel.org/monsieuricon/signed-git-pushes)
* [TUF](https://theupdateframework.io/): generic verification mechanism, used by Docker, no known
Git implementation just yet
* [SLSA](https://slsa.dev/): "security framework, a check-list of
standards and controls to prevent tampering, improve integrity, and
secure packages and infrastructure", built on top of [in-toto][]
* [jcat](https://github.com/hughsie/libjcat): used by fwupd
* [git-signify](https://leahneukirchen.org/dotfiles/bin/git-signify): using [signify](https://github.com/aperezdc/signify), a non-OpenPGP alternative
* [crev](https://github.com/crev-dev/): Code REView system, used by Rust (and Cargo) to vet
dependencies, delegates sharing signatures to git, but
cryptographically signs them so should be resilient against a
server compromise
* [arch linux upstream tag verifications](https://vulns.xyz/2022/05/auth-tarball-from-git/)
* [Linux kernel OpenPGP keys distribution repository](https://git.kernel.org/pub/scm/docs/kernel/pgpkeys.git/tree/README.rst)
## Migration from Trac
GitLab was put online as part of a migration from [Trac](howto/trac),
see the [Trac documentation for details on the migration](howto/trac#gitlab-migration).