gitlab.md

[GitLab](https://gitlab.com/) is *a web-based DevOps lifecycle tool that provides a
Git-repository manager providing wiki, issue-tracking and continuous
integration/continuous deployment pipeline features, using an
open-source license, developed by GitLab Inc* ([Wikipedia](https://en.wikipedia.org/wiki/GitLab)). Tor
uses GitLab mainly for issue tracking, wiki hosting and code review
for now, at <https://gitlab.torproject.org>, after migrating from
[howto/trac](howto/trac).

Note that continuous integration is documented separately, in [the CI page](service/ci).

[[_TOC_]]

# Tutorial

<!-- simple, brainless step-by-step instructions requiring little or -->
<!-- no technical background -->

## How to get an account?

You might already *have* an account! If you were active on Trac, your
account was migrated with the same username and email address as Trac,
unless you have an LDAP account, in which case that was used. So head
over to the [password reset page](https://gitlab.torproject.org/users/password/new) to get access to your account.

If your account was *not* migrated, send a mail to
<gitlab-admin@torproject.org> to request a new one.

If you did not have an account in Trac and want a new account, you
should request a new one at <https://gitlab.onionize.space/>.

## How to report an issue in Tor software?

You first need to figure out which project the issue resides in. The
[project list][] is a good place to get started. Here are a few quick
links for popular projects:

[project list]: https://gitlab.torproject.org/tpo

 * [core tor](https://gitlab.torproject.org/tpo/core/tor): [issues](https://gitlab.torproject.org/tpo/core/tor/-/issues), [new issue](https://gitlab.torproject.org/tpo/core/tor/-/issues/new)
 * [Tor Browser](https://gitlab.torproject.org/tpo/applications/tor-browser): [issues](https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues), [new issue](https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/new)
 * [gitlab](https://gitlab.torproject.org/tpo/tpa/gitlab): [issues](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues), [new issue](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/new)

If you do not have a GitLab account or can't figure it out for any
reason, you can also use the mailing lists. The
<tor-dev@lists.torproject.org> mailing list is the best for now.

## How to report an issue in the bugtracker itself?

If you have access to GitLab, you can [file a new issue][File] after
you have [searched the GitLab project for similar bugs][search]. 

If you do *not* have access to GitLab, you can email
<gitlab-admin@torproject.org>.

### Note about confidential issues

Note that you can mark issues as "confidentials" which will make them
private to the members of the project the issue is reported on (the
"developers" group and above, specifically).

Keep in mind, however, that it is still possible issue information
gets leaked in cleartext, however. For example, GitLab [sends email
notifications in cleartext for private issue](https://gitlab.com/gitlab-org/gitlab/-/issues/5816), an known upstream
issue. (We have [decided we cannot fix this ourselves in GitLab for
now](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/23).) Some repositories might also have "web hooks" that notify
IRC bots in clear text as well, although at the time of writing all
projects are correctly configured.

## How to contribute code?

As reporting an issue, you first need to figure out which project you
are working on in the GitLab [project list][]. Then, if you are not
familiar with merge requests, you should read the [merge requests
introduction](https://gitlab.torproject.org/help/user/project/merge_requests/getting_started.md) in the GitLab documentation. If you are unfamiliar
with merge requests but familiar with GitHub's pull requests, those
are similar.

Note that we do not necessarily use merge requests in all teams yet,
and Gitolite still has the canonical version of the code. See [issue
36][] for a followup on this.

[issue 36]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/36

Also note that different teams might have different workflows. If a
team has a special workflow that diverges from the one here, it should
be documented here. Those are the workflows we know about:

 * [Network Team](https://gitlab.torproject.org/tpo/core/tor/-/wikis/NetworkTeam/GitlabReviews)
 * [Web Team](https://gitlab.torproject.org/tpo/web/community/-/wikis/Git-flow-and-merge-requests)
 * Bridge DB: merge requests

If you do not have access to GitLab, please use one of the mailing
lists: <tor-dev@lists.torproject.org> would be best.

## How to quote a comment in a reply?

The "Reply" button only creates a new comment without any quoted text
by default.  It seems the solution to that is currently highlighting
the text to quote and then pressing the `r`-key. See also the [other
keyboard shortcuts](https://docs.gitlab.com/ee/user/shortcuts.html).

Alternatively, you can copy-paste the text in question in the comment
form, select the pasted text, and hit the `Insert a quote` button
which look like a styled, curly, and closing quotation mark `”`.

# How-to

## Continuous Integration (CI)

All CI documentation resides in a different document see
[service/ci](service/ci).

## Container registry operations

### Logging in

To upload content to the registry, you first need to login. This can
be done with the `login` command:

    podman login

This will ask you for your GitLab username and a password, for which
you should use a [personal access token](https://gitlab.torproject.org/-/profile/personal_access_tokens).

### Uploading an image

Assuming you already have an image built (below we have it labeled
with `containers.torproject.org/anarcat/test/airsonic-test`), you can
upload it with:

    podman push containers.torproject.org/anarcat/test/airsonic-test containers.torproject.org/anarcat/test

Notice the two arguments: the first is the label of the image to
upload and the second is *where* to upload it, or "destination". The
destination is made of two parts, the first component is the host name
of the container registry (in our case `containers.torproject.org`)
and the second part is the path to the project to upload into (in our
case [`anarcat/test`](https://gitlab.torproject.org/anarcat/test).

The uploaded container image should appear under Deploy -> Container
Registry in your project. In the above case, it is in:

<https://gitlab.torproject.org/anarcat/test/container_registry/4>

## Email interactions

You can interact with GitLab by email too. 

### Creating a new issue

Clicking on the project issues gives a link at the bottom of the page,
which says say "Email a new issue to this project".

That link should go into the "To" field of your email. The email
subject becomes the title of the issue and the body the
description. You can use shortcuts in the body, like `/assign @foo`,
`/estimate 1d`, etc.

See [the upstream docs for more details](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#new-issue-via-url-with-prefilled-fields).

### Commenting on an issue

If you just reply to the particular comment notification you received
by email, as you would reply to an email in a thread, that comment
will show up in the issue.

You need to have email notifications enabled for this to work,
naturally.

You can also add a new comment to any issue by copy-pasting the
issue-specific email address in the right sidebar (labeled "Issue
email", [introduced in GitLab 13.8](https://gitlab.com/gitlab-org/gitlab/-/issues/18816)).

This also works with shortcuts like `/estimate 1d` or `/spend
-1h`. Note: for those you won't get notification emails back, though,
while for others like `/assign @foo` you would.

See [the upstream docs for more details](https://docs.gitlab.com/ee/administration/reply_by_email.html).

### Quick status updates by email

There are a bunch of [quick actions](https://gitlab.torproject.org/help/user/project/quick_actions.md) available which are handy to
update an issue. As mentioned above they can be sent by email as well,
both within a comment (be it as a reply to a previous one or in a new
one) or just instead of it. So, for example, if you want to update the
amount of time spent on ticket $foo by one hour, find any notification
email for that issue and reply to it by replacing any quoted text with
`/spend 1h`.

## How to migrate a Git repository from legacy to GitLab?

See the [git documentation for this procedure](howto/git#how-to-migrate-a-git-repository-from-legacy-to-gitlab).

## How to mirror a Git repository from legacy to GitLab?

See the [git documentation for this procedure](howto/git#how-to-migrate-a-git-repository-from-legacy-to-gitlab).

## How to mirror a Git repository from GitLab to GitHub

Some repositories are mirrored to THE [torproject organization on
GitHub](https://github.com/torproject). This section explains how that works and how to create a
new mirror from GitLab. In this example, we're going to mirror the
[tor browser manual](https://gitlab.torproject.org/tpo/web/manual).

 1. head to the "Mirroring repositories" section of the
    [settings/repository](https://gitlab.torproject.org/tpo/web/manual/-/settings/repository) part of the project

 2. as a Git repository URL, enter:

        ssh://git@github.com/torproject/manual.git

 3. click "detect host keys"

 4. choose "SSH" as the "Authentication method"

 5. don't check any of the boxes, click "Mirror repository"

 6. the page will reload and show the mirror in the list of "Mirrored
    repositories". click the little "paperclip" icon which says "Copy
    SSH public key"

 7. head over to the [settings/keys](https://github.com/torproject/manual/settings/keys)
    section of the target GitHub project and click "Add deploy key"

        Title: https://gitlab.torproject.org/tpo/web/manual mirror key
        Key: <paste public key here>

 8. check the "Allow write access" checkbox and click "Add key"

 9. back in the "Mirroring repositories" section of the GitLab project, click
    the "Update now" button represented by circling arrows

If there is an error, it will show up as a little red "Error"
button. Hovering your mouse over the button will show you the error.

If you want retry the "Update now" button, you need to let the update interval
pass (1 minute for protected branch mirroring, 5 minutes for all branches)
otherwise it will have no effect.

## How to find the right emoji?

It's possible to add "reaction emojis" to comments and issues and
merge requests in GitLab. Just hit the little smiley face and a dialog
will pop up. You can then browse through the list and pick the right
emoji for how you feel about the comment, but remember to be nice!

It's possible you get lost in the list. You can type the name of the
emoji to restrict your search, but be warned that some emojis have
[particular, non-standard names](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29057) that might not be immediately
obvious. For example, `🎉`, `U+1F389 PARTY POPPER`, is found as
`tada` in the list! See [this upstream issue for more details](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29057).

<a name="hooking-up-a-project-with-the-bots" />

## Publishing notifications on IRC

By default, new projects do *not* have notifications setup in
`#tor-bots` like all the others. To do this, you need to configure a
"Webhook", in the `Settings -> Webhooks` section of the project. The
URL should be:

    https://kgb-bot.torproject.org/webhook/

... and you should select the notifications you wish to see in
`#tor-bots`. You can also enable notifications to other channels by
adding more parameters to the URL, like (say)
`?channel=tor-foo`. Important note: do not try to put the `#` in
the channel name, or if you do, URL-encode it (e.g. like `%23tor-foo`),
otherwise this will silently fail to change the target channel. Other
parameters are documented the [KGB documentation](https://salsa.debian.org/kgb-team/kgb/-/wikis/usage).

Note that GitLab admins might be able to configure [system-wide
hooks](https://gitlab.torproject.org/help/system_hooks/system_hooks) in [the admin section](https://gitlab.torproject.org/admin/hooks), although it's not entirely clear
how does relate to the per-project hooks so those have not been
enabled. Furthermore, it is possible for GitLab admins with root
access to enable webhooks on *all* projects, with the [webhook rake
task](https://docs.gitlab.com/ee/raketasks/web_hooks.html#webhooks). For example, running this on the GitLab server (currently
`gitlab-02`) will enable the above hook on all repositories:

    sudo gitlab-rake gitlab:web_hook:add URL='https://kgb-bot.torproject.org/webhook/'

Note that by default, the rake task only enables `Push` events. You
need the following patch to enable others:

    modified   lib/tasks/gitlab/web_hook.rake
    @@ -10,7 +10,19 @@ namespace :gitlab do
           puts "Adding webhook '#{web_hook_url}' to:"
           projects.find_each(batch_size: 1000) do |project|
             print "- #{project.name} ... "
    -        web_hook = project.hooks.new(url: web_hook_url)
    +        web_hook = project.hooks.new(
    +          url: web_hook_url,
    +          push_events: true,
    +          issues_events: true,
    +          confidential_issues_events: false,
    +          merge_requests_events: true,
    +          tag_push_events: true,
    +          note_events: true,
    +          confidential_note_events: false,
    +          job_events: true,
    +          pipeline_events: true,
    +          wiki_page_events: true,
    +        )
             if web_hook.save
               puts "added".color(:green)
             else

See also the [upstream issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17966) and [our GitLab issue 7](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/7) for
details.

You can also remove a given hook from all repos with:

    sudo gitlab-rake gitlab:web_hook:rm URL='https://kgb-bot.torproject.org/webhook/'

And, finally, list all hooks with:

    sudo gitlab-rake gitlab:web_hook:list

## Setting up two-factor authentication (2FA)

We strongly recommend you enable two-factor authentication on
GitLab. This is [well documented in the GitLab manual](https://gitlab.torproject.org/help/user/profile/account/two_factor_authentication.md#two-factor-authentication), but basically:

 1. first, pick a 2FA "app" (and optionally a hardware token) if you
    don't have one already

 2. head to your [account settings](https://gitlab.torproject.org/profile/account)

 3. register your 2FA app and save the recovery codes somewhere. if
    you need to enter a URL by hand, you can scan the qrcode with your
    phone or create one by following this format:

        otpauth://totp/$ACCOUNT?secret=$KEY&issuer=gitlab.torproject.org

    where...

      * `$ACCOUNT` is the `Account` field in the 2FA form
      * `$KEY` is the `Key` field in the 2FA form, without spaces

 4. register the 2FA hardware token if available

GitLab requires a 2FA "app" even if you intend to use a hardware
token. The 2FA "app" must implement the TOTP protocol, for example the
[Google Authenticator](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2) or a free alternative (for example [free OTP
plus](https://github.com/helloworld1/FreeOTPPlus/), see also this [list from the Nextcloud project](https://github.com/nextcloud/twofactor_totp#readme)). The
hardware token must implement the U2F protocol, which is supported by
security tokens like the [YubiKey](https://en.wikipedia.org/wiki/YubiKey), [Nitrokey](https://www.nitrokey.com/), or similar.

## Deleting sensitive attachments

If a user uploaded a secret attachment by mistake, just deleting the
issue is not sufficient: it turns out that doesn't remove the
attachments from disk!

To fix this, ask a sysadmin to find the file in the
`/var/opt/gitlab/gitlab-rails/uploads/` directory. Assuming the
attachment URL is:

<https://gitlab.torproject.org/anarcat/test/uploads/7dca7746b5576f6c6ec34bb62200ba3a/openvpn_5.png>

There should be a "hashed" directory and a hashed filename in there,
which looks something like:

    ./@hashed/08/5b/085b2a38876eeddc33e3fbf612912d3d52a45c37cee95cf42cd3099d0a3fd8cb/7dca7746b5576f6c6ec34bb62200ba3a/openvpn_5.png

The second directory (`7dca7746b5576f6c6ec34bb62200ba3a` above) is the
one visible in the attachment URL. The last part is the actual
attachment filename, but since those can overlap between issues, it's
safer to look for the hash. So to find the above attachment, you
should use:

    find /var/opt/gitlab/gitlab-rails/uploads/ -name 7dca7746b5576f6c6ec34bb62200ba3a

And delete the file in there. The following should do the trick:

    find /var/opt/gitlab/gitlab-rails/uploads/ -name 7dca7746b5576f6c6ec34bb62200ba3a | sed 's/^/rm /' > delete.sh

Verify `delete.sh` and run it if happy.

Note that GitLab is working on an [attachment manager](https://gitlab.com/gitlab-org/gitlab/-/issues/16229) that should
allow web operators to delete old files, but it's unclear how or when
this will be implemented, if ever.

## Publishing GitLab pages

GitLab features a way to publish websites directly from the continuous
integration pipelines, called [GitLab pages](https://docs.gitlab.com/ee/user/project/pages/). Complete
documentation on how to publish such pages is better served by the
official documentation, but creating a `.gitlab-ci.yml` should get you
rolling. For example, this will publish a `hugo` site:

    image: registry.gitlab.com/pages/hugo/hugo_extended:0.65.3
    pages:
      script:
        - hugo
      artifacts:
        paths:
          - public
      only:
        - main

If `.gitlab-ci.yml` already contains a job in the `build` stage that
generates the required artifacts in the `public` directory, then
including the `pages-deploy.yml` CI template should be sufficient:

    include:
      - project: tpo/tpa/ci-templates
        file: pages-deploy.yml

GitLab pages are published under the `*.pages.torproject.org` wildcard
domain. There are two types of projects hosted at the TPO GitLab:
sub-group projects, usually under the `tpo/` super-group, and user
projects, for example `anarcat/myproject`. You can also publish a page
specifically for a user. The URLs will look something like this:

| Type of GitLab page | Name of the project created in GitLab | Website URL                                          |
|---------------------|---------------------------------------|------------------------------------------------------|
| User pages          | `username.pages.torproject.net`       | `https://username.pages.torproject.net`              |
| User projects       | `user/projectname`                    | `https://username.pages.torproject.net/projectname`  |
| Group projects      | `tpo/group/projectname`               | `https://tpo.pages.torproject.net/group/projectname` |

## Accepting merge requests on wikis

Wiki permissions are not great, but there's a workaround: accept merge
requests for a git replica of the wiki.

This documentation was [moved to the documentation section](service/documentation#accepting-merge-requests-on-wikis).

## Renaming a branch globally

While `git` supports renaming branches locally with the `git branch
--move $to_name` command, this doesn't actually rename the remote
branch. That process is more involved.

Changing the name of a default branch both locally and on remotes can
be partially automated with the use of [anarcat's branch rename
script](https://gitlab.com/anarcat/scripts/-/blob/main/git-branch-rename-remote). The script basically renames the branch locally, pushes
the new branch and deletes the old one, with special handling of
GitLab remotes, where it "un-protects" and "re-protects" the branch.

You should run the script with an account that has "Maintainer" or
"Owner" access to GitLab, so that it can do the above GitLab API
changes. You will then need to provide an [access token](https://gitlab.torproject.org/-/profile/personal_access_tokens) through
the `GITLAB_PRIVATE_TOKEN` environment variable, which should have the
scope `api`.

So, for example, this will rename the `master` branch to `main` on the
local and remote repositories:

    GITLAB_PRIVATE_TOKEN=REDACTED git-branch-rename-remote

If you want to rename another branch or remote, you can specify those
on the commandline as well. For example, this will rename the
`develop` branch to `dev` on the `gitlab` remote:

    GITLAB_PRIVATE_TOKEN=REDACTED git-branch-rename-remote --remote gitlab --from-branch develop --to-branch dev

The command can also be used to fix *other* repositories so that they
correctly rename their local branch too. In that case, the GitLab
repository is already up to date, so there is no need for an access
token.

Other users, then can just run this command will rename `master` to
`main` on the local repository, including remote tracking branches:

    git-branch-rename-remote

Obviously, users without any extra data in their local repository can
just destroy their local repository and clone a new one to get the
correct configuration.

Keep in mind that there may be a few extra steps and considerations to
make when changing the name of a heavily used branch, detailed below.

### Modifying open Merge Requests

A merge request that is open against the modified branch may be
bricked as a result of deleting the old branch name from the Gitlab
remote. To avoid this, after creating and pushing the new branch name,
edit each merge request to target the new branch name **before**
deleting the old branch.

### Updating gitolite

Many GitLab repositories are mirrored or maintained manually on
Gitolite (`git-rw.torproject.org`) and [Gitweb](https://gitweb.torproject.org). The `ssh` step for
the above automation script will fail for Gitolite and these steps
need to be done manually by a sysadmin. [Open a TPA ticket](https://gitlab.torproject.org/tpo/tpa/team/-/issues/new) with a
list of the Gitolite repositories you would like to update and a
sysadmin will perform the following magic:

    cd /srv/git.torproject.org/repositories/
    for repo in $list; do
        git -C "$repo" symbolic-ref HEAD refs/heads/$to_branch
    done

This will update Gitolite, but it won't update Gitweb until the
repositories have been pushed to. To update Gitweb immediately, ask
your friendly sysadmin to run the above command on the Gitweb server
as well.

### Updating Transifex

If your repository relies on Transifex for translations, make sure to
update the Transifex config to pull from the new branch. To do so,
[open a l10n ticket](https://gitlab.torproject.org/tpo/community/l10n/-/issues/new?issue%5Bassignee_id%5D=&issue%5Bmilestone_id%5D=) with the new branch name changes.

## Find the project associated with a project ID

Sometimes you'll find a numeric project ID instead of a human-readable
one. For example, you can see on the [arti project](https://gitlab.torproject.org/tpo/core/arti) that it says:

    Project ID: 647

So you can easily find the project ID of a project right on the
project's front page. But what if you only have the ID and need to
find what project it represents? You can talk with the API, with a URL
like:

    https://gitlab.torproject.org/api/v4/projects/<PROJECT_ID>

For example, this is how I found the above arti project from the
`Project ID 647`:

```
$ curl -s 'https://gitlab.torproject.org/api/v4/projects/647' | jq .web_url
"https://gitlab.torproject.org/tpo/core/arti"
```

## Find the project associated with a hashed repository name

Git repositories are not stored under the project name in GitLab
anymore, but under a hash of the project ID. The easiest way to get to
the project URL from a hash is [through the rails console](https://docs.gitlab.com/ee/administration/repository_storage_paths.html#from-hashed-path-to-project-name), for
example:

    sudo gitlab-rails console

then:

    ProjectRepository.find_by(disk_path: '@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9').project

The project's full path is also available in the `config` file in the
git repository, under the `fullpath` config entry.

Finally, you can also generate a rainbow table of all possible hashes
to get the project ID, and from there, find the project using the API
above. Here's a Python blob that will generate a hash for every
project ID up to 2000:

    import hashlib

    for i in range(2000):
        h = hashlib.sha256()
        h.update(str(i).encode('ascii'))
        print(i, h.hexdigest())

## Connect to the PostgreSQL server

The GitLab Omnibus setup is special: it ships its own embedded
PostgreSQL server (!), which means the regular `sudo -u postgres psql`
command doesn't work.

To get access to the PostgreSQL server, you need to [follow the
upstream instructions](https://docs.gitlab.com/omnibus/maintenance/#starting-a-postgresql-superuser-psql-session) which are, at the time of writing:

    sudo gitlab-psql -d gitlabhq_production

This actually expands to the following command:

    sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -p 5432 -h /var/opt/gitlab/postgresql -d gitlabhq_production -d gitlabhq_production

An emergency dump, therefore, could be taken with:

    cd /tmp ; sudo -u gitlab-psql /opt/gitlab/embedded/bin/pg_dump -p 5432 -h /var/opt/gitlab/postgresql -d gitlabhq_production -d gitlabhq_production | pv -s 2G > /srv/gitlab-backup/db/db-$(date '+%Y-%m-%d').sql

Yes, that is silly. See also [issue 20][].

## Pager playbook

<!-- information about common errors from the monitoring system and -->
<!-- how to deal with them. this should be easy to follow: think of -->
<!-- your future self, in a stressful situation, tired and hungry. -->

 * Grafana Dashboards:
   * [GitLab overview](https://grafana.torproject.org/d/QrDJktiMz/gitlab-omnibus)
   * [Gitaly](https://grafana.torproject.org/d/x6Z50y-iz/gitlab-gitaly)

TODO: document how to handle common problems in GitLab

### Troubleshooting

Upstream recommends running this command to self-test a GitLab
instance:

    sudo gitlab-rake gitlab:check SANITIZE=true

This command also shows general info about the GitLab instance:

    sudo gitlab-rake gitlab:check SANITIZE=true

it is especially useful to find on-disk files and package versions.

### GitLab pages not found

If you're looking for a way to track GitLab pages error, know that the
webserver logs are in `/var/log/nginx/gitlab_pages_access`, but that
only proxies requests for the GitLab Pages engine, which (JSON!) logs
live in `/var/log/gitlab/gitlab-pages/current`.

If you get a `"error":"domain does not exist"` problem, make sure the
entire *pipeline* actually succeeds. Typically, the "pages:deploy" job
can fail with:

    Artifacts for pages are too large

In that case, you need to go into the Admin Area -> Settings ->
Preferences -> Pages and bump the size limit. It defaults to 100MB and
we bumped it to 1024MB at the time of writing. Note that GitLab CI/CD
also have a similar setting which might (or might not?) affect such
problems.

### PostgreSQL debugging

The PostgreSQL configuration in GitLab is [particular][issue 20]. See the
[connect to the PostgreSQL server](#connect-to-the-postgresql-server) section above on how to connect
to it.

### Disk full on GitLab server

If the main GitLab server is running out of space (as opposed to
runners, see [Runner disk fills up](service/ci#runner-disk-fills-up) for that scenario), then it's
projects that are taking up space. We've typically had trouble with
artifacts taking up space, for example (tpo/tpa/team#40615,
tpo/tpa/team#40517).

You can see the largest disk users in the GitLab admin area in
[Overview -> Projects -> Sort by: Largest repository](https://gitlab.torproject.org/admin/projects?sort=storage_size_desc). 

Note that, although it's unlikely, it's technically possible that an
archived project takes up space, so make sure you check the "Show
archived projects" option in the "Sort by" drop down.

In the past, we have worked around that problem by reducing the
default artifact retention period from 4 to 2 weeks
(tpo/tpa/team#40516) but obviously does not take effect
immediately. More recently, we have tried to tweak individual
project's retention policies and scheduling strategies (details in
tpo/tpa/team#40615).

Please be aware of the [known upstream issues](#issues) that affect those
diagnostics as well.

To obtain a list of project sorted by space usage, log on to GitLab using an
account with administrative privileges and open the [Projects page](https://gitlab.torproject.org/admin/projects?sort=storage_size_desc)
sorted by `Largest repository`. The total space consumed by each project is
displayed and clicking on a specific project shows a breakdown of how this space
is consumed by different components of the project (repository, LFS, CI
artifacts, etc.).

If a project is consuming an unexpected amount of space for artifacts, the
scripts from the [tpo/tpa/gitlab-tools](https://gitlab.torproject.org/tpo/tpa/gitlab-tools)
project can by utilized to obtain a breakdown of the space used by job logs and
artifacts, per job or per pipeline. These scripts can also be used to manually
remove such data, see the [gitlab-tools README](https://gitlab.torproject.org/tpo/tpa/gitlab-tools/README.md).

It's also possible to compile some CI artifact usage statistics directly on the
GitLab server. To see if expiration policies work (or if "kept" artifacts or
old `job.log` are a problem), use this command (which takes a while to
run):

    find -mtime +14 -print0 | du --files0-from=- -c -h | tee find-mtime+14-du.log

To limit this to `job.log`, of course, you can do:

    find -name "job.log" -mtime +14 -print0 | du --files0-from=- -c -h | tee find-mtime+14-joblog-du.log

If we ran out of space on the object storage because of the GitLab
registry, consider [purging untagged manifests](https://docs.gitlab.com/ee/administration/packages/container_registry.html#removing-untagged-manifests-and-unreferenced-layers) by tweaking the
cron job defined in `profile::gitlab::app` in Puppet.

### Incoming email routing

Incoming email get routed through either eugeni or the submission
service, then end up on the Postfix server on `gitlab-02`, and from
there, to a dovecot mailbox. You can use `postfix-trace` to confirm
the message correctly ended up there.

Normally, GitLab should be picking mails from the mailbox
(`/srv/mail/git@gitlab.torproject.org/Maildir/`) regularly, and
deleting them when done. If that is not happening, look at the
mailroom logs:

    tail -f /var/log/gitlab/mailroom/mail_room_json.log | jq -c

A working run will look something like this:

```
{"severity":"INFO","time":"2022-08-29T20:15:57.734+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Processing started"}
{"severity":"INFO","time":"2022-08-29T20:15:57.734+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"uid":7788,"action":"asking arbiter to deliver","arbitrator":"MailRoom::Arbitration::Redis"}.734+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Getting new messages","unread":{"count":1,"ids":[7788]},"to_be_delivered":{"count":1,"ids":[7788]}}ext":{"email":"git@gitlab.torproject.org","name":"inbox"},"uid":7788,"action":"sending to deliverer","deliverer":"MailRoom::Delivery::Sidekiq","byte_size":4162}","delivery_method":"Sidekiq","action":"message pushed"}
{"severity":"INFO","time":"2022-08-29T20:15:57.744+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Processing started"}
{"severity":"INFO","time":"2022-08-29T20:15:57.744+00:00","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Getting new messages","unread":{"count":0,"ids":[]},"to_be_delivered":{"count":0,"ids":[]}}0","context":{"email":"git@gitlab.torproject.org","name":"inbox"},"action":"Idling"}
```

Emails should be processed every minute or so.

### Outgoing email

Follow the [email not sent](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/smtp.md#email-not-sent) procedure. TL;DR:

    sudo gitlab-rails console

(Yes it takes forever.) Then check if the settings are sane:

```
--------------------------------------------------------------------------------
 Ruby:         ruby 3.0.5p211 (2022-11-24 revision ba5cf0f7c5) [x86_64-linux]
 GitLab:       15.10.0 (496a1d765be) FOSS
 GitLab Shell: 14.18.0
 PostgreSQL:   12.12
------------------------------------------------------------[ booted in 28.31s ]
Loading production environment (Rails 6.1.7.2)
irb(main):003:0> ActionMailer::Base.delivery_method
=> :smtp
irb(main):004:0> ActionMailer::Base.smtp_settings
=> 
{:user_name=>nil,
 :password=>nil,
 :address=>"localhost",
 :port=>25,
 :domain=>"localhost",
 :enable_starttls_auto=>false,
 :tls=>false,
 :ssl=>false,
 :openssl_verify_mode=>"none",
 :ca_file=>"/opt/gitlab/embedded/ssl/certs/cacert.pem"}
```

Then test an email delivery:

    Notify.test_email('noreply@torproject.org', 'Hello World', 'This is a test message').deliver_now

A working delivery will look something like this, with the last line
in *green*:

```
irb(main):001:0> Notify.test_email('noreply@torproject.org', 'Hello World', 'This is a test message').deliver_now
Delivered mail 64219bdb6e919_10e66548d042948@gitlab-02.mail (20.1ms)
=> #<Mail::Message:296420, Multipart: false, Headers: <Date: Mon, 27 Mar 2023 13:36:27 +0000>, <From: GitLab <git@gitlab.torproject.org>>, <Reply-To: GitLab <noreply@torproject.org>>, <To: noreply@torproject.org>, <Message-ID: <64219bdb6e919_10e66548d042948@gitlab-02.mail>>, <Subject: Hello World>, <Mime-Version: 1.0>, <Content-Type: text/html; charset=UTF-8>, <Content-Transfer-Encoding: 7bit>, <Auto-Submitted: auto-generated>, <X-Auto-Response-Suppress: All>>
```

A *failed* delivery will *also* say `Delivered mail` *but* will
include an error message as well. For example, in [issue 139][] we had
this error:

```
irb(main):006:0> Notify.test_email('noreply@torproject.org', 'Hello World', 'This is a test message').deliver_now
Delivered mail 641c797273ba1_86be948d03829@gitlab-02.mail (7.2ms)
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:46:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)
```

[issue 139]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/139

### Gitlab registry troubleshooting

If something goes with the GitLab Registry feature, you should first
look at the logs in:

    tail -f /var/log/gitlab/registry/current /var/log/gitlab/nginx/gitlab_registry_*.log /var/log/gitlab/gitlab-rails/production.log

The first one might be the one with more relevant information, but is
the hardest to parse, as it's this weird "date {JSONBLOB}" format that
no human or machine can parse.

You can restart *just* the registry with:

    gitlab-ctl restart registry

A misconfiguration of the object storage backend will look like this
when uploading a container:

    Error: trying to reuse blob sha256:61581d479298c795fa3cfe95419a5cec510085ec0d040306f69e491a598e7707 at destination: pinging container registry containers.torproject.org: invalid status code from registry 503 (Service Unavailable)

The registry logs might have something like this:

```
2023-07-18_21:45:26.21751 time="2023-07-18T21:45:26.217Z" level=info msg="router info" config_http_addr="127.0.0.1:5000" config_http_host= config_http_net= config_http_prefix= config_http_relative_urls=true correlation_id=01H5NFE6E94A566P4EZG2ZMFMT go_version=go1.19.8 method=HEAD path="/v2/anarcat/test/blobs/sha256:61581d479298c795fa3cfe95419a5cec510085ec0d040306f69e491a598e7707" root_repo=anarcat router=gorilla/mux vars_digest="sha256:61581d479298c795fa3cfe95419a5cec510085ec0d040306f69e491a598e7707" vars_name=anarcat/test version=v3.76.0-gitlab
2023-07-18_21:45:26.21774 time="2023-07-18T21:45:26.217Z" level=info msg="authorized request" auth_project_paths="[anarcat/test]" auth_user_name=anarcat auth_user_type=personal_access_token correlation_id=01H5NFE6E94A566P4EZG2ZMFMT go_version=go1.19.8 root_repo=anarcat vars_digest="sha256:61581d479298c795fa3cfe95419a5cec510085ec0d040306f69e491a598e7707" vars_name=anarcat/test version=v3.76.0-gitlab
2023-07-18_21:45:26.30401 time="2023-07-18T21:45:26.303Z" level=error msg="unknown error" auth_project_paths="[anarcat/test]" auth_user_name=anarcat auth_user_type=personal_access_token code=UNKNOWN correlation_id=01H5NFE6CZBE49BZ6KBK4EHSJ1 detail="SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.\n\tstatus code: 403, request id: 17731468F69A0F79, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8" error="unknown: unknown error" go_version=go1.19.8 host=containers.torproject.org method=HEAD remote_addr=64.18.183.94 root_repo=anarcat uri="/v2/anarcat/test/blobs/sha256:a55f9a4279c12800590169f7782b956e5c06ec88ec99c020dd111a7a1dcc7eac" user_agent="containers/5.23.1 (github.com/containers/image)" vars_digest="sha256:a55f9
```

If you suspect the object storage backend to be the problem, you
should try to communicate with the MinIO server by configuring the
`rclone` client on the GitLab server and trying to manipulate the
server. Look for the access token in `/etc/gitlab/gitlab.rb` and use
it to configure `rclone` like this:

    rclone config create minio s3 provider Minio endpoint https://minio.torproject.org:9000/  region dallas access_key_id gitlab-registry secret_access_key REDACTED

Then you can list the registry bucket:

    rclone ls minio:gitlab-registry/

See how to [Use rclone as an object storage client](service/minio#use-rclone-as-an-object-storage-client) for more ideas.

The above may reproduce the above error from the registry:

    SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.

That is either due to an incorrect access key or bucket. An error that
was made during the original setup was to treat `gitlab/registry` as a
bucket, while it's a subdirectory... This was fixed by switching to
`gitlab-registry` as a bucket name. Another error we had was to use
`endpoint` instead of `regionendpoint`.

Another tweak that was done was to set a region in MinIO. Before the
right region was set and matching in the configuration, we had this
error in the registry logs:

    2023-07-18_21:04:57.46099 time="2023-07-18T21:04:57.460Z" level=fatal msg="configuring application: 1 error occurred:\n\t* validating region provided: dallas\n\n"

As a last resort, you can revert back to the [filesystem storage](https://docs.gitlab.com/ee/administration/packages/container_registry.html#use-file-system)
by commenting out the `storage => { ... 's3' ... }` block in
`profile::gitlab::app` and adding a line in the `gitlab_rails` blob
like:

    registry_path                  => '/var/opt/gitlab/gitlab-rails/shared/registry',

Note that this is a risky operation, as you might end up with a "split
brain" where some images are on the filesystem, and some on object
storage. Warning users with maintenance announcement on the GitLab
site might be wise.

In the same section, you can [disable the registry by default](https://docs.gitlab.com/ee/administration/packages/container_registry.html#disable-container-registry-for-new-projects-site-wide) on
all projects with:

    gitlab_default_projects_features_container_registry => false,

... or [disable it site-wide](https://docs.gitlab.com/ee/administration/packages/container_registry.html#disable-container-registry-site-wide) with:

    registry => {
      enable => false
      # [...]
    }

Note that the `registry` configuration is stored inside the Docker
Registry `config.yaml` file as a single line that looks like JSON. You
*may* think it's garbled and the reason why things don't work, but it
isn't, that is valid YAML, just harder to parse. Blame `gitlab-ctl`'s
Chef cookbook on that... A non-mangled version of the working config
would look like:

```
storage:
  s3:
    accesskey: gitlab-registry
    secretkey: REDACTED
    region: dallas
    regionendpoint: https://minio.torproject.org:9000/
    bucket: gitlab-registry
```

Another option that was explored while setting up the registry is
enabling the [debug server](https://docs.gitlab.com/ee/administration/packages/container_registry.html#enable-the-registry-debug-server).

## Disaster recovery

In case the entire GitLab machine is destroyed, a new server should be
provisionned in the [howto/ganeti](howto/ganeti) cluster (or elsewhere) and backups
should be restored using the below procedure.

### Running an emergency backup

A full backup can be ran as root with:

    /usr/bin/gitlab-rake gitlab:backup:create

Backups are stored as a tar file in `/srv/gitlab-backup` and do *not*
include secrets, which are backed up separately, for example with:

    umask 0077 && tar -C /var/opt/gitlab -czf /srv/gitlab-backup/config_backup$(date +"\%Y\%m\%dT\%H\%M").tar.gz

See `/etc/cron.d/gitlab-config-backup`, and the `gitlab::backup` and
`profile::gitlab::app` classes for the actual jobs that runs nightly.

### Recovering this wiki from backups

If you need to immediately restore the wiki from backups, you can head
to the backup server and restore the directory:

    /var/opt/gitlab/git-data/repositories/@hashed/11/f8/11f8e31ccbdbb7d91589ecf40713d3a8a5d17a7ec0cebf641f975af50a1eba8d.git

The hash above is the SHA256 checksum of the [wiki-replica](https://gitlab.torproject.org/tpo/tpa/wiki-replica/)
project id (695):

    $ printf 695 | sha256sum 
    11f8e31ccbdbb7d91589ecf40713d3a8a5d17a7ec0cebf641f975af50a1eba8d  -

On the backup server, that would be something like:

    bconsole
    restore
    5
    46
    cd /var/opt/gitlab/git-data/repositories/@hashed/11/f8
    mark 11f8e31ccbdbb7d91589ecf40713d3a8a5d17a7ec0cebf641f975af50a1eba8d.git
    done
    yes

The files will end up in `/var/tmp/bacula-restore` on
`gitlab-02`. Note that the number `46`, above, will vary according to
other servers backed up on the backup server, of course.

This should give you a copy of the git repository, which you can then
use, presumably, to read this procedure and restore the rest of
GitLab. 

(Although then, how did you read *this* part of the procedure?
Anyways, I thought this could save your future self one day. You'll
thank me later.)

### Restoring from backups

The [upstream documentation](https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations) has a fairly good restore procedure,
but because our backup procedure is non-standard -- we exclude
repositories and artifacts, for example -- you should follow this
procedure instead.

Note that the procedure assumes some familiarity with the general
[backup and restore procedures](howto/backup), particularly how to restore a
bunch of files from the backup server (see the [restore files
section](howto/backup#restore-files).

This entire procedure will take many hours to complete. In our tests,
it took:

 1. an hour or two to setup a VM
 2. less than an hour to do a basic GitLab install
 3. 20 minutes to restore the basic system (database, tickets are
    visible at this point)
 4. an hour to restore repositories
 5. another hour to restore artifacts

This gives a time to recovery of about 5 to 6 hours. Most of that time
is spent waiting for files to be copied, interspersed with a few
manual commands.

So here's the procedure that was followed to deploy a development
server, from backups, in [tpo/tpa/team#40820](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40820) (run everything as
root):

 1. [install GitLab using Puppet](#installation): basically create a server large
    enough for everything, apply the Puppet `role::gitlab`
    
    That includes creating new certificates and DNS records, if not
    already present (those may be different if you are created a dev
    server from backups, for example, which was the case for the the
    above ticket).
    
    Also note that you need to install the *same* GitLab version as
    the one from the backup. If you are unsure of the GitLab version
    that's in the backup (bad day uh?), try to restore the
    `/var/opt/gitlab/gitlab-rails/VERSION` file from the backup server
    first.

 2. at this point, a blank GitLab installation should be
    running. verify that you can reach the login page, possibly trying
    to login with the root account, because a working GitLab
    installation is a pre-requisite for the rest of the restore
    procedure.
    
    (it might be technically possible to restore the entire server
    from scratch using only the backup server, but that procedure has
    not been established or tested.)

 3. on the backup server (currently `bacula-director-01`), restore the
    latest GitLab backup job from the `/srv/gitlab-backup` and the
    secrets from `/etc/gitlab`:
    
        # bconsole
        *restore
        To select the JobIds, you have the following choices:
        [...]
         5: Select the most recent backup for a client
        [...]
        Select item:  (1-13): 5
        Defined Clients:
        [...]
            47: gitlab-02.torproject.org-fd
        [...]
        Select the Client (1-98): 47
        Automatically selected FileSet: Standard Set
        [...]
        Building directory tree for JobId(s) 199535,199637,199738,199847,199951 ...  ++++++++++++++++++++++++++++++++
        596,949 files inserted into the tree.
        [...]
        cwd is: /
        $ cd /etc
        cwd is: /etc/
        $ mark gitlab