Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
  • tpa-rfc-80-merge-tor-and-tails-support-policies
  • puppet-merge-costs
  • rfc-77-improv
  • tails-merge-admin-mailing-lists
  • mkdocs
  • new-blog-doc
7 results
Blame Permalink
gitlab.md 112.76 KiB

GitLab is a web-based DevOps lifecycle tool that provides a Git-repository manager providing wiki, issue-tracking and continuous integration/continuous deployment pipeline features, using an open-source license, developed by GitLab Inc (Wikipedia). Tor uses GitLab for issue tracking, source code and wiki hosting, at https://gitlab.torproject.org, after migrating from Trac and gitolite.

Note that continuous integration is documented separately, in the CI page.

Tutorial

How to get an account?

You might already have an account! If you were active on Trac, your account was migrated with the same username and email address as Trac, unless you have an LDAP account, in which case that was used. So head over to the password reset page to get access to your account.

If your account was not migrated, send a mail to gitlab-admin@torproject.org to request a new one.

If you did not have an account in Trac and want a new account, you should request a new one at https://gitlab.onionize.space/.

How to report an issue in Tor software?

You first need to figure out which project the issue resides in. The project list is a good place to get started. Here are a few quick links for popular projects:

If you do not have a GitLab account or can't figure it out for any reason, you can also use the mailing lists. The tor-dev@lists.torproject.org mailing list is the best for now.

How to report an issue in the bugtracker itself?

If you have access to GitLab, you can file a new issue after you have searched the GitLab project for similar bugs.

If you do not have access to GitLab, you can email gitlab-admin@torproject.org.

Note about confidential issues

Note that you can mark issues as "confidentials" which will make them private to the members of the project the issue is reported on (the "developers" group and above, specifically).

Keep in mind, however, that it is still possible issue information gets leaked in cleartext, however. For example, GitLab sends email notifications in cleartext for private issue, an known upstream issue.

We have deployed a workaround for this which redacts outgoing mail, but there's still some metadata leaking there:

  • the issue number
  • the reporter
  • the project name
  • the reply token (allowing someone to impersonate a reply)

Some repositories might also have "web hooks" that notify IRC bots in clear text as well, although at the time of writing all projects are correctly configured. The IRC side of things, of course, might also leak information.

Note that internal notes are currently not being redacted, unless they are added to confidential issues, see issue 145.

How to contribute code?

As reporting an issue, you first need to figure out which project you are working on in the GitLab project list. Then, if you are not familiar with merge requests, you should read the merge requests introduction in the GitLab documentation. If you are unfamiliar with merge requests but familiar with GitHub's pull requests, those are similar.

Note that we do not necessarily use merge requests in all teams yet, and Gitolite still has the canonical version of the code. See issue 36 for a followup on this.

Also note that different teams might have different workflows. If a team has a special workflow that diverges from the one here, it should be documented here. Those are the workflows we know about:

If you do not have access to GitLab, please use one of the mailing lists: tor-dev@lists.torproject.org would be best.

How to quote a comment in a reply?

The "Reply" button only creates a new comment without any quoted text by default. It seems the solution to that is currently highlighting the text to quote and then pressing the r-key. See also the other keyboard shortcuts.

Alternatively, you can copy-paste the text in question in the comment form, select the pasted text, and hit the Insert a quote button which look like a styled, curly, and closing quotation mark ”.

How-to

Continuous Integration (CI)

All CI documentation resides in a different document see service/ci.

Container registry operations

Logging in

To upload content to the registry, you first need to login. This can be done with the login command:

podman login

This will ask you for your GitLab username and a password, for which you should use a personal access token.

Uploading an image

Assuming you already have an image built (below we have it labeled with containers.torproject.org/anarcat/test/airsonic-test), you can upload it with:

podman push containers.torproject.org/anarcat/test/airsonic-test containers.torproject.org/anarcat/test

Notice the two arguments: the first is the label of the image to upload and the second is where to upload it, or "destination". The destination is made of two parts, the first component is the host name of the container registry (in our case containers.torproject.org) and the second part is the path to the project to upload into (in our case anarcat/test.

The uploaded container image should appear under Deploy -> Container Registry in your project. In the above case, it is in:

https://gitlab.torproject.org/anarcat/test/container_registry/4

Email interactions

You can interact with GitLab by email too.

Creating a new issue

Clicking on the project issues gives a link at the bottom of the page, which says say "Email a new issue to this project".

That link should go into the "To" field of your email. The email subject becomes the title of the issue and the body the description. You can use shortcuts in the body, like /assign @foo, /estimate 1d, etc.

See the upstream docs for more details.

Commenting on an issue

If you just reply to the particular comment notification you received by email, as you would reply to an email in a thread, that comment will show up in the issue.

You need to have email notifications enabled for this to work, naturally.

You can also add a new comment to any issue by copy-pasting the issue-specific email address in the right sidebar (labeled "Issue email", introduced in GitLab 13.8).

This also works with shortcuts like /estimate 1d or /spend -1h. Note: for those you won't get notification emails back, though, while for others like /assign @foo you would.

See the upstream docs for more details.

Quick status updates by email

There are a bunch of quick actions available which are handy to update an issue. As mentioned above they can be sent by email as well, both within a comment (be it as a reply to a previous one or in a new one) or just instead of it. So, for example, if you want to update the amount of time spent on ticket $foo by one hour, find any notification email for that issue and reply to it by replacing any quoted text with /spend 1h.

How to migrate a Git repository from legacy to GitLab?

See the git documentation for this procedure.

How to mirror a Git repository from legacy to GitLab?

See the git documentation for this procedure.

How to mirror a Git repository from GitLab to GitHub

Some repositories are mirrored to the torproject organization on GitHub. This section explains how that works and how to create a new mirror from GitLab. In this example, we're going to mirror the tor browser manual.

  1. head to the "Mirroring repositories" section of the settings/repository part of the project

  2. as a Git repository URL, enter:

    ssh://git@github.com/torproject/manual.git

  3. click "detect host keys"

  4. choose "SSH" as the "Authentication method"

  5. don't check any of the boxes, click "Mirror repository"

  6. the page will reload and show the mirror in the list of "Mirrored repositories". click the little "paperclip" icon which says "Copy SSH public key"

  7. head over to the settings/keys section of the target GitHub project and click "Add deploy key"

    Title: https://gitlab.torproject.org/tpo/web/manual mirror key
Key: <paste public key here>

  8. check the "Allow write access" checkbox and click "Add key"

  9. back in the "Mirroring repositories" section of the GitLab project, click the "Update now" button represented by circling arrows

If there is an error, it will show up as a little red "Error" button. Hovering your mouse over the button will show you the error.

If you want retry the "Update now" button, you need to let the update interval pass (1 minute for protected branch mirroring, 5 minutes for all branches) otherwise it will have no effect.

How to find the right emoji?

It's possible to add "reaction emojis" to comments and issues and merge requests in GitLab. Just hit the little smiley face and a dialog will pop up. You can then browse through the list and pick the right emoji for how you feel about the comment, but remember to be nice!

It's possible you get lost in the list. You can type the name of the emoji to restrict your search, but be warned that some emojis have particular, non-standard names that might not be immediately obvious. For example, 🎉, U+1F389 PARTY POPPER, is found as tada in the list! See this upstream issue for more details.

Publishing notifications on IRC

By default, new projects do not have notifications setup in #tor-bots like all the others. To do this, you need to configure a "Webhook", in the Settings -> Webhooks section of the project. The URL should be:

https://kgb-bot.torproject.org/webhook/

... and you should select the notifications you wish to see in #tor-bots. You can also enable notifications to other channels by adding more parameters to the URL, like (say) ?channel=tor-foo.

Important note: do not try to put the # in the channel name, or if you do, URL-encode it (e.g. like %23tor-foo), otherwise this will silently fail to change the target channel.

Other parameters are documented the KGB documentation. In particular, you might want to use private=yes;channel=tor-foo if you do not want to have the bot send notifications in #tor-bots, which is also does by default.

IMPORTANT: Again, even if you tell the bot to send a notification to the channel #tor-foo, the bot still defaults to also sending to #tor-bots, unless you use that private flag above. Be careful to not accidentally leak sensitive information to a public channel, and test with a dummy repository if you are unsure.

The KGB bot can also send notifications to channels that require a password. In the /etc/kgb.conf configuration file, add a secret to a channel so the bot can access a password-protected channel. For example:

channels:
    -
        name: '#super-secret-channel
        network: 'MyNetwork'
        secret: 'ThePasswordIsPassw0rd'
        repos:
            - SecretRepo

Note: support for channel passwords is not implemented in the upstream KGB bot. There's an open merge request for it and the patch has been applied to TPA's KGB install, but new installs will need to manually apply that patch.

Note that GitLab admins might be able to configure system-wide hooks in the admin section, although it's not entirely clear how does relate to the per-project hooks so those have not been enabled. Furthermore, it is possible for GitLab admins with root access to enable webhooks on all projects, with the webhook rake task. For example, running this on the GitLab server (currently gitlab-02) will enable the above hook on all repositories:

sudo gitlab-rake gitlab:web_hook:add URL='https://kgb-bot.torproject.org/webhook/'

Note that by default, the rake task only enables Push events. You need the following patch to enable others:

modified   lib/tasks/gitlab/web_hook.rake
@@ -10,7 +10,19 @@ namespace :gitlab do
       puts "Adding webhook '#{web_hook_url}' to:"
       projects.find_each(batch_size: 1000) do |project|
         print "- #{project.name} ... "
-        web_hook = project.hooks.new(url: web_hook_url)
+        web_hook = project.hooks.new(
+          url: web_hook_url,
+          push_events: true,
+          issues_events: true,
+          confidential_issues_events: false,
+          merge_requests_events: true,
+          tag_push_events: true,
+          note_events: true,
+          confidential_note_events: false,
+          job_events: true,
+          pipeline_events: true,
+          wiki_page_events: true,
+        )
         if web_hook.save
           puts "added".color(:green)
         else

See also the upstream issue and our GitLab issue 7 for details.

You can also remove a given hook from all repos with:

sudo gitlab-rake gitlab:web_hook:rm URL='https://kgb-bot.torproject.org/webhook/'

And, finally, list all hooks with:

sudo gitlab-rake gitlab:web_hook:list

The hook needs a secret token to be operational. This secret is stored in Puppet's Trocla database as profile::kgb_bot::gitlab_token:

trocla get profile::kgb_bot::gitlab_token plain

That is configured in profile::kgb_bot in case that is not working.

Note that if you have a valid personal access token, you can manage the hooks with the gitlab-hooks.py script in gitlab-tools script. For example, this created a webhook for the tor-nagios project:

export HTTP_KGB_TOKEN=$(ssh root@puppet.torproject.org trocla get profile::kgb_bot::gitlab_token plain)
./gitlab-hooks.py -p tpo/tpa/debian/deb.torproject.org-keyring create --no-releases-events --merge-requests-events --issues-events --push-events --url https://kgb-bot.torproject.org/webhook/?channel=tor-admin

Setting up two-factor authentication (2FA)

We strongly recommend you enable two-factor authentication on GitLab. This is well documented in the GitLab manual, but basically:

  1. first, pick a 2FA "app" (and optionally a hardware token) if you don't have one already

  2. head to your account settings

  3. register your 2FA app and save the recovery codes somewhere. if you need to enter a URL by hand, you can scan the qrcode with your phone or create one by following this format:

    otpauth://totp/$ACCOUNT?secret=$KEY&issuer=gitlab.torproject.org

    where...

    • $ACCOUNT is the Account field in the 2FA form
    • $KEY is the Key field in the 2FA form, without spaces

  4. register the 2FA hardware token if available

GitLab requires a 2FA "app" even if you intend to use a hardware token. The 2FA "app" must implement the TOTP protocol, for example the Google Authenticator or a free alternative (for example free OTP plus, see also this list from the Nextcloud project). The hardware token must implement the U2F protocol, which is supported by security tokens like the YubiKey, Nitrokey, or similar.

Deleting sensitive attachments

If a user uploaded a secret attachment by mistake, just deleting the issue is not sufficient: it turns out that doesn't remove the attachments from disk!

To fix this, ask a sysadmin to find the file in the /var/opt/gitlab/gitlab-rails/uploads/ directory. Assuming the attachment URL is:

https://gitlab.torproject.org/anarcat/test/uploads/7dca7746b5576f6c6ec34bb62200ba3a/openvpn_5.png

There should be a "hashed" directory and a hashed filename in there, which looks something like:

./@hashed/08/5b/085b2a38876eeddc33e3fbf612912d3d52a45c37cee95cf42cd3099d0a3fd8cb/7dca7746b5576f6c6ec34bb62200ba3a/openvpn_5.png

The second directory (7dca7746b5576f6c6ec34bb62200ba3a above) is the one visible in the attachment URL. The last part is the actual attachment filename, but since those can overlap between issues, it's safer to look for the hash. So to find the above attachment, you should use:

find /var/opt/gitlab/gitlab-rails/uploads/ -name 7dca7746b5576f6c6ec34bb62200ba3a

And delete the file in there. The following should do the trick:

find /var/opt/gitlab/gitlab-rails/uploads/ -name 7dca7746b5576f6c6ec34bb62200ba3a | sed 's/^/rm /' > delete.sh

Verify delete.sh and run it if happy.

Note that GitLab is working on an attachment manager that should allow web operators to delete old files, but it's unclear how or when this will be implemented, if ever.

Publishing GitLab pages

GitLab features a way to publish websites directly from the continuous integration pipelines, called GitLab pages. Complete documentation on how to publish such pages is better served by the official documentation, but creating a .gitlab-ci.yml should get you rolling. For example, this will publish a hugo site:

image: registry.gitlab.com/pages/hugo/hugo_extended:0.65.3
pages:
  script:
    - hugo
  artifacts:
    paths:
      - public
  only:
    - main

If .gitlab-ci.yml already contains a job in the build stage that generates the required artifacts in the public directory, then including the pages-deploy.yml CI template should be sufficient:

include:
  - project: tpo/tpa/ci-templates
    file: pages-deploy.yml

GitLab pages are published under the *.pages.torproject.org wildcard domain. There are two types of projects hosted at the TPO GitLab: sub-group projects, usually under the tpo/ super-group, and user projects, for example anarcat/myproject. You can also publish a page specifically for a user. The URLs will look something like this:

Type of GitLab page Name of the project created in GitLab Website URL
User pages username.pages.torproject.net https://username.pages.torproject.net
User projects user/projectname https://username.pages.torproject.net/projectname
Group projects tpo/group/projectname https://tpo.pages.torproject.net/group/projectname

Accepting merge requests on wikis