gitlab.md

GitLab is a web-based DevOps lifecycle tool that provides a Git-repository manager providing wiki, issue-tracking and continuous integration/continuous deployment pipeline features, using an open-source license, developed by GitLab Inc (Wikipedia). Tor uses GitLab mainly for issue tracking, wiki hosting and code review for now, at https://gitlab.torproject.org, after migrating from trac.

!toc levels=3

Tutorial

How to get an account?

You might already have an account! If you were active on Trac, your account was migrated with the same username and email address as Trac, unless you have an LDAP account, in which case that was used. So head over to the password reset page to get access to your account.

If your account was not migrated, send a mail to gitlab-admin@torproject.org to request a new one.

We hope to have something else in place for user registration once the migration is successful.

How to report an issue in Tor software?

You first need to figure out which project the issue resides in. The project list is a good place to get started. Here are a few quick links for popular projects:

• core tor: issues, new issue
• Tor Browser: issues, new issue
• gitlab: issues, new issue

If you do not have a GitLab account or can't figure it out for any reason, you can also use the mailing lists. The tor-dev@lists.torproject.org mailing list is the best for now.

How to report an issue in the bugtracker itself?

If you have access to GitLab, you can file a new issue after you have searched the GitLab project for similar bugs.

If you do not have access to GitLab, you can email gitlab-admin@torproject.org.

How to contribute code?

As reporting an issue, you first need to figure out which project you are working on in the GitLab project list. Then, if you are not familiar with merge requests, you should read the merge requests introduction in the GitLab documentation. If you are unfamiliar with merge requests but familiar with GitHub's pull requests, those are similar.

If you do not have access to GitLab, please use one of the mailing lists: tor-dev@lists.torproject.org would be best.

How-to

Hooking up a project with the bots

By default, new projects do not have notifications setup in #tor-bots like all the others. To do this, you need to configure a "Webhook", in the Settings -> Webhooks section of the project. The URL should be:

https://kgb-bot.torproject.org/webhook/

... and you should select the notifications you wish to see in #tor-bots. You can also enable notifications to other channels by adding more parameters to the URL, like (say) ?channel=%23tor-foo. The parameters are documented the KGB documentation.

Note that GitLab admins might be able to configure system-wide hooks in the admin section, although it's not entirely clear how does relate to the per-project hooks so those have not been enabled. Furthermore, it is possible for GitLab admins with root access to enable webhooks on all projects, with the webhook rake task. For example, running this on the GitLab server (currently gitlab-02) will enable the above hook on all repositories:

sudo gitlab-rake gitlab:web_hook:add URL='https://kgb-bot.torproject.org/webhook/?channel=%23tor-bots'

Note that by default, the rake task only enables Push events. You need the following patch to enable others:

modified   lib/tasks/gitlab/web_hook.rake
@@ -10,7 +10,19 @@ namespace :gitlab do
       puts "Adding webhook '#{web_hook_url}' to:"
       projects.find_each(batch_size: 1000) do |project|
         print "- #{project.name} ... "
-        web_hook = project.hooks.new(url: web_hook_url)
+        web_hook = project.hooks.new(
+          url: web_hook_url,
+          push_events: true,
+          issues_events: true,
+          confidential_issues_events: false,
+          merge_requests_events: true,
+          tag_push_events: true,
+          note_events: true,
+          confidential_note_events: false,
+          job_events: true,
+          pipeline_events: true,
+          wiki_page_events: true,
+        )
         if web_hook.save
           puts "added".color(:green)
         else

See also the upstream issue and our GitLab issue 7 for details.

Setting up two-factor authentication (2FA)

We strongly recommend you enable two-factor authentication on GitLab. This is well documented in the GitLab manual, but basically:

1. first, pick a 2FA "app" (and optionally a hardware token) if you don't have one already

2. head to your account settings

3. register your 2FA app and save the recovery codes somewhere. if you need to enter a URL by hand, you can scan the qrcode with your phone or create one by following this format:

   otpauth://totp/$ACCOUNT?secret=$KEY&issuer=gitlab.torproject.org

   where...

   • $ACCOUNT is the Account field in the 2FA form
   • $KEY is the Key field in the 2FA form, without spaces

4. register the 2FA hardware token if available

GitLab requires a 2FA "app" even if you intend to use a hardware token. The 2FA "app" must implement the TOTP protocol, for example the Google Authenticator or a free alternative (for example free OTP plus, see also this list from the Nextcloud project). The hardware token must implement the U2F protocol, which is supported by security tokens like the YubiKey, Nitrokey, or similar.

Pager playbook

• Grafana Dashboards:
  • GitLab overview
  • Gitaly

Disaster recovery

In case the entire GitLab machine is destroyed, a new server should be provisionned in the ganeti cluster (or elsewhere) and backups should be restored using the below procedure.

Running an emergency backup

TBD

baremetal recovery

TBD

Reference

Installation

The current GitLab server was setup in the ganeti cluster in a regular virtual machine. It was configured with puppet with the roles::gitlab.

This installs the GitLab Omnibus distribution which duplicates a lot of resources we would otherwise manage elsewhere in Puppet, including (but possibly not limited to):

• prometheus
• postgresql

This therefore leads to a "particular" situation regarding monitoring and PostgreSQL backups, in particular.

SLA

Migration

GitLab was migrated from Trac in June 2020, after a few months of testing. Tests were done first on a server called dip.torproject.org, a reference to salsa.debian.org, the GitLab server ran by the Debian project. We identified some problems with merge requests during the test so the server was reinstalled with the "GitLab Omnibus" package on the current server, gitlab-02 which will enter production in the week of June 15th 2020.

Why migrate?

We're hoping gitlab will be a good fit because:

• Gitlab will allow us to collect our different engineering tools into a single application: Git repository handling, Wiki, Issue tracking, Code reviews, and project management tooling.
• Gitlab is well-maintained, while Trac plugins are not well maintained and Trac itself hasn't seen a release for over a year (since 2019)
• Gitlab will allow us to build a more modern approach to handling CI for our different projects. This is going to happen after the ticket and wiki migration.

(Note that we're only planning to install and use the freely licensed version of gitlab. There is an "enterprise" version with additional features, but we prefer to use free software whenever possible.)

Migrated content

The issues and wiki of the "Tor" project are migrated. There are no other projects in Trac.

• Trac wiki: https://gitlab.torproject.org/legacy/trac
• Trac issues: https://gitlab.torproject.org/legacy/trac/-/issues

Trac issues that remain are really legacy issues, others issues have been "moved" to the respective projects. @ahf, who did the migration, created a copy of the mapping for those looking for their old stuff.

Not migrated

We are not migrating away from Gitolite and Jenkins just yet. This means those services are still fully operational and their equivalent features in GitLab are not supported (namely Git hosting and CI). Those services might eventually be migrated to GitLab, but that's not part of the current migration plan.

Again, the canonical copy for source code hosted by git is:

• git-rw.torproject - writable git repositories over SSH
• https://git.torproject.org/ - readonly clones
• https://gitweb.torproject.org/ - web interface

We also do not host "GitLab pages", the static site hosting provided by GitLab.

The priority of those features would be:

1. gitolite replacement and migration
2. CI deployment, with people migrating their own job from Jenkinks and TPA shutting down Jenkins on a flag date
3. GitLab pages replacement and migration from the current static site hosting system

Those are each large projects and will be undertaken at a later stage, progressively.

Feature equivalence

Feature	Trac	GitLab	Comments
Ticket relations	parent/child	checklists	checklists show up as "X of Y tasks completed"¹
Milestones	yes	yes
Estimates	points/actual	estimation/spending	requires conversion from days to hours
Private issues	no	yes
Issue subscription	RSS, email, ML	email	Trac sends email to trac-bugs
User projects	no	yes	if users can create projects
User registration	optional	disabled	²
Search	advanced	basic	no support for custom queries in GitLab³
Markup	WikiCreole	Markdown, GitHub-like	⁴
IRC bot	yes	yes	zwiebelbot has to be patched, other bots to be deployed for notifications⁵
Git hosting	no, gitolite	yes, builtin	concerns about trusting GitLab with our code
CI	no, Jenkins	yes, builtin	maybe in the future
Upstream maintenance	slow	fast	Trac does not seem well maintained
Wikis	one big wiki	per-project	⁶
API	XML-RPC	REST, multiple clients
Javascript	optional	required	Drag-and-drop boards seem not to work but the list of issues still can be used.

Notes:

1. Trac parent/child issue relationships have been converted into a simple comment at the beginnning of the ticket linking to the child/parent tickets. It was originally hoped to use the "checklists" features but this was not implemented for lack of time.

2. User registration is perfectly possible in GitLab but since GitLab instances are frequently attacked by spammers, it is disabled until we find an alternative. See missing features below for details).

3. GitLab, in particular, does not support inline searches, see Missing features below for details.

4. The wiki and issue formatting markup is different. Whereas Trac uses wiki formatting inspired by old wikis like MoinMoin, a subset of the somewhat standard Wikicreole markup, GitLab uses Markdown, specifically their own GitLab version of markdown inspired by GitHub's markdown extensions. The wiki and issues were automatically converted to Markdown, but when you file new issues, you will need to use Markdown, not Creole.

5. specifically, zwiebelbot now knows about foo#N pointing to issue N in project foo in GitLab. We need to update (or replace) the nsa bot in #tor-bots to broadcast announcements to projects. This could be done with the KGB bot for which we now have a Puppet module so it could easily be deployed here

6. because Trac does not allow users to create projects, we have historically used one gigantic project for everything, which means we had only one wiki. technically, Trac also supports one wiki per project, but because project creation requires an admin intervention, this never concretized.

Ticket fields equivalence

Trac	GitLab	Comments
id	id	keep the ticket id in legacy project, starts at 40000 in GitLab
Summary	?	unused?
Reporter	Reporter
Description	Body
Type	Label	use templates to make sure those are filled
Milestone	Milestone, Label
Version	Label
Keywords	Label
Points, in days	/estimate, in hours	requires conversion
Actual points	/spending
Sponsor	Label
Priority	Board, Label	boards can sort issues instead of assigning arbitrary keywords
Component	Subproject, Label
Severity	Label	mark only blocker issues to resolve
Cc	@people	paid plans also have multiple assignees
Parent issue	#reference	issue mentions and checklists
Reviewer	Label
Attachements	Attachements, per comment
Status	Label	Kanban boards panels

Notice how the Label field is used as a fallback when no equivalent field exists.

Missing features

GitLab does not provide one-to-one feature parity with Trac, but it comes pretty close. It has issue tracking, wikis, milestones, keywords, time estimates, and much more.

But one feature it is missing is the advanced ticket query features of Trac. It's not possible to create "reports" in GitLab to have pre-cooked issue listings. And it's especially not possible to embed special searches in wiki pages the same way it is done in Trac.

We suggest people use the "dashboard" feature of GitLab instead. This featuers follows the Kanban development strategy which is implemented in GitLab as issue boards. It is also, of course, possible to link so specific searches from the wiki, but not embed those tickets in the output.

We do not have a anonymous account (AKA cypherpunks) for now. GitLab will be in closed registration for now, with users needing to request approval on a per-person basis for now. Eventually, we're going to consider other options to work around this (human) bottleneck.

Interesting new features

1. Using pull requests to your project repositories, and assigning reviewers on pull requests, rather than using reviewer and needs_review labels on issues. Issues can refer to pull requests and vice versa.

2. Your team can work on using Gitlab boards for handling the different stages of issue handling. All the way from selection to finalization with code in a PR. You can have as many boards as you like: per subproject, per sponsor, per week, all of this is something we can experiment with.

3. You can now use time estimation in Gitlab simply by adding a specially formatted comment in your issues/pull requests instead of using points and actual_points. See the time tracking documentation for details

4. Familiarize yourself with new interfaces such as the "to do" dashboard where you can see what needs your input since last visit

5. Create email filters for tickets: Gitlab adds a lot more email headers to each notification you receive (if you want it via email), which for example allows you split notifications in your mail program into different directories.

   Bonus info: You will be able to reply via email to the notifications you receive from Gitlab, and Gitlab will put your responses into the system as notes on issues :-)

bugs.torproject.org redirections

The https://bugs.torproject.org redirection now points at GitLab. The following rules apply:

1. legacy tickets: bugs.torproject.org/N redirects to gitlab.torproject.org/legacy/trac/-/issues/N
2. new issues: bugs.tpo/PROJECT/N redirects to gitlab.tpo/PROJECT/-/issues/N
3. merge requests: bugs.tpo/PROJECT!N redirects to gitlab.tpo/PROJECT/-/merge_requests/N
4. catch all: bugs.tpo/FOO redirects to gitlab.tpo/FOO
5. ticket list: a bare bugs.tpo redirects to https://gitlab.torproject.org/tpo/-/issues

It used to be that bugs.tpo/N would redirect to issue N the Trac "tor" project. But unfortunately, there's no global "number space" for issues in GitLab (or at least not a user-visible one), so N is not distinct across projects. We therefore need the prefix to disambiguate.

We considered enforcing the tpo prefix there to shorten links, but we decided against it because it would forbid pointers to user-specific projects and would make it extremely hard to switch away from the global tpo group if we ever decide to do that.

Content organisation

Projects are all stored under the over-arching tpo group. This is done this way to allow project managers to have an overview of all projects going on at TPO. It also allows us to host other organisations on our GitLab in a different namespace.

Under the tpo group, each team has its own subgroup and they have autonomy under that group to manage accesses and projects.

Permissions

Given the above Team/Group organization, users will be members in gitlab for the groups/teams they belong to.

Any projects that need to be shared between multiple groups should be shared using the “Share Project” functionality.

There should be a limited number of members in the Tor Project group, as these will have access to all subgroups and their projects. Currently this is limited to Project Managers and Services and Sysadmins.

A reminder of the GitLab permission system and types of users:

• Guests: anybody that may need to report issues on a project and/or make comments on an issue.
• Reporter: they can also manage labels
• Developer: they can create branches, manage merge requests, force push to non-protected branches
• Maintainer: edit projects, manage runners, edit comments, delete wiki pages.
• Owner: we are setting this role for every member in the TPO team. They can also transfer projects to other name spaces, switch visilbity level, delete issues.

Labels

At group level we have sponsor labels and state labels. The ones that are used by the whole organization are in the tpo group. Each team can decide which other labels they add for their projects.

• Kanban columns
  • Icebox
  • Backlog
  • Next
  • Doing
  • Needs Review
• Types of Issue
  • Defect
  • Enhancement
  • Task
• Related to a project
  • Scalability
  • UX
• Sponsors
  • Sponsor X
• Keywords
  • Other possible keywords needed at group level.

Note that those labels are being worked on ticket 4.

Remaining work

As of June 17th:

3. Make sure TPO project issue change notifications are emailed to the tor-bugs mailing list.

4. Begin handing out some of the accounts to community members that we have created in their name(s).

5. Begin figuring out how to solve the account creation and anonymous users in the longer run.

See also the list of known issues.

What will break, and when will you fix it?

Most notably, we're going to have an interruption in the ability to open new accounts and new tickets. We did not want to migrate without a solution here; we'll try to have at least a stop-gap solution in place soon, and something better in the future. For now, we're planning for people that want to get a new account please send a mail to gitlab-admin@torproject.org. We hope to have something else in place once the migration is succesful.

We're not going to migrate long-unused accounts.

Some wiki pages that contained automated listings of tickets will stop containing those lists: that's a trac feature that gitlab doesn't have. We'll have to adjust our workflows to work around this. In some cases, we can use gitlab milestone pages or projects that do not need a wiki page as a work around.

History

• lost in the mists of time: migration from Bugzilla to Flyspray (40 tickets)
• 2010-04-23: migration from Flyspray to Trac completed (last Flyspray ticket is 1393, first Trac ticket is 2000)
• 2016-11-29: first request to setup a GitLab server
• ~2017: oniongit.eu (warning: squatted domain) deployed to test GitLab with the network team, considered as gitlab.torproject.net but ultimately abandoned
• 2019-02-28: gitlab-01 AKA dip.torproject.org test server setup (issue 29400), following the Brussels meeting
• 2019-07-17: GitLab discussed again at the Stockholm meeting
• 2019-07-29: Formal proposal to deploy GitLab sent to tor-project, no objection
• 2020-03-05: GitLab migrated from gitlab-01 (AKA "dip") to gitlab-02 using the Omnibus package
• 2020-04-27: gitlab-01 retired
• 2020-06-13 19:00UTC: Trac readonly
• 2020-06-13 02:25UTC: Trac tickets migrated (32401 tickets, last ticket id is 34451, first GitLab legacy project ticket id is 40000)
• 2020-06-14 21:22UTC: Trac wiki migrated
• 2020-06-15 18:30UTC: bugs.torproject.org redirects to gitlab
• 2020-06-16 02:15UTC: GitLab launch announced to tor-internal

Design

GitLab is a fairly large program with multiple components. The upstream documentation has a good details of the architecture but this section aims at providing a shorter summary. Here's an overview diagram, first:

The web frontend is Nginx (which we incidentally also use in our cache system) but GitLab wrote their own reverse proxy called GitLab Workhorse which in turn talks to the underlying GitLab Rails application, served by the Unicorn application server. The Rails app stores its data in a postgresql database (although not our own deployment, for now, should be fixed). GitLab also offloads long-term background tasks to a tool called sidekiq.

Those all server HTTP(S) requests but GitLab is of course also accessible over SSH to push/pull git repositories. This is handled by a separate component called gitlab-shell which acts as a shell for the git user.

Workhorse, Rails, sidekiq and gitlab-shell all talk with Redis to store temporary information, caches and session information. They can also communicate with the Gitaly server which handles all communication with the git repositories themselves.

Finally, Git)Lab also features GitLab Pages and Continuous Integration ("pages" and CI, neither of which we do not currently use). CI is handled by GitLab runners which can be deployed by anyone and registered in the Rails app to pull CI jobs. GitLab pages is "a simple HTTP server written in Go, made to serve GitLab Pages with CNAMEs and SNI using HTTP/HTTP2".

Issues

There is no issue tracker specifically for this project, File or search for issues in the gitlab project.

Monitoring and testing

Monitoring right now is minimal: normal host-level metrics like disk space, CPU usage, web port and TLS certificates are monitored by Nagios with our normal infrastructure, as a black box.

Prometheus monitoring is built into the GitLab Omnibus package, so it is not configured through our Puppet like other Prometheus servers. It has still been (manually) integrated in our Prometheus setup and Grafana dashboards (see pager playbook) have been deployed.

More work is underway to improve monitoring in issue 33921.

Backups

There is a cronjob configured via the gitlab puppet monitor that runs every night at 2:00AM UTC. This will basically run $ gitlab-backup and will create a tarball under /srv/gitlab-backup/.

There is also a config backup job (in /etc/cron.d/gitlab-config-backup) that makes sure to backup the content of /var/opt/gitlab/gitlab-rails/etc/ is backed up, because that is not covered by the gitlab-backup command.

Another cron job purges backups older than two days, in /etc/cron.d/gitlab-rotate-backup, so that we don't keep too many copies on the server. It is assumed that the existing backup system will pick up those copies and store them for our normal rotation periods.

Discussion

Overview

The GitLab project at Tor has been a long time coming. If you look at the history section above, you'll see it has been worked on since at least 2016, at which point an external server was setup for the "network team" to do code review. This server was ultimately retired.

The current server has been worked on since 2019, with the master ticket, issue 29400, created in the footsteps of the 2019 Brussels meeting. The service launched some time in June 2020, with a full migration of Trac tickets.

Goals

Must have

• replacement of the Trac issue tracking server
• rough equivalent of Trac features in GitLab

Nice to have

• identical representation of Trac issues in GitLab, including proper issue numbering

Non-Goals

• replacement of Gitolite (git hosting)
• replacement of Gitweb (git hosting)
• replacement of Jenkins (CI)
• replacement of the static site hosting system

Those are not part of the first phase of the project, but it is understood that if one of those features gets used more heavily in GitLab, the original service MUST be eventually migrated into GitLab and turned off. We do not want to run multiple similar services at the same time (for example run both gitolite and gitaly on all git repositories, or run Jenkins and GitLab runners).

Approvals required

The GitLab migration was approved at the 2019 Brussels dev meeting.

Proposed Solution

The solution to the "code review" and "project management" problems are to deploy a GitLab instance which does not aim at managing all source code, in the first stage.

Cost

Staff not evaluated.

In terms of hardware, we start with a single virtual machine and agree that, in the worst case, we can throw a full Hetzner PX62-NVMe node at the problem (~70EUR/mth).

Alternatives considered

GitLab is such a broad project that multiple alternatives exist for different components:

• GitHub
  • Pros:
    • widely used in the open source community
    • Good integration between ticketing system and code
  • Cons
    • It is hosted by a third party (Microsoft!)
    • Closed source
• GitLab:
• Pros:
  • Mostly free software
  • Feature-rich
• Cons:
  • Complex software, high maintenance
  • "Opencore" - some interesting features are closed-source

GitLab command line clients

If you want to do batch operations or integrations with GitLab, you might want to use one of those tools, depending on your environment or prefered programming language:

• bugwarrior (Debian) - support for GitLab, GitHub and other bugtrackers for the taskwarrior database
• git-lab - python commandline client, lists, pulls MR; creates snippets
• GitLab-API-v4 (Debian) - perl library and commandline client
• GitLabracadabra (Debian) - configure a GitLab instance from a YAML configuration, using the API: project settings like labels, admins, etc
• python-gitlab (also known as gitlab-cli in Debian)
• ruby-gitlab (Debian), also includes a commandline client
• salsa (in Debian devscripts) is specifically built for salsa but might be coerced into talking to other GitLab servers

Migration tools

ahf implemente the gitlab using his own home-made tools that talk to the GitLab and Trac API. but there's also tracboat which is designed to migrate from trac to GitLab.

We did not use Tracboat because it uses gitlab's DB directly and thus only works with some very specific version. Each time the database schema changes at GitLab, Tracboat needs to port to it. We prefered to use something that talked with the GitLab API.

We also didn't like the output entirely, so we modified it but still used some of its regular expressions and parser.

We also needed to implement the "ticket movement" hack (with the legacy project) which wasn't implemented in Tracboat.

Finally, we didn't want to do complete user migration, but lazily transfer only some users.