GitLab is a web-based DevOps lifecycle tool that provides a
Git-repository manager providing wiki, issue-tracking and continuous
integration/continuous deployment pipeline features, using an
open-source license, developed by GitLab Inc (Wikipedia). Tor
uses GitLab mainly for issue tracking, wiki hosting and code review
for now, at https://gitlab.torproject.org, after migrating from
You might already have an account! If you were active on Trac, your
account was migrated with the same username and email address as Trac,
unless you have an LDAP account, in which case that was used. So head
over to the password reset page to get access to your account.
As reporting an issue, you first need to figure out which project you
are working on in the GitLab project list. Then, if you are not
familiar with merge requests, you should read the merge requests
introduction in the GitLab documentation. If you are unfamiliar
with merge requests but familiar with GitHub's pull requests, those
Note that we do not necessarily use merge requests in all teams yet,
and Gitolite still has the canonical version of the code. See issue
36 for a followup on this.
Also note that different teams might have different workflows. If a
team has a special workflow that diverges from the one here, it should
be documented here. Those are the workflows we know about:
The "Reply" button only creates a new comment without any quoted text
by default. It seems the solution to that is currently highlighting
the text to quote and then pressing the r-key. See also the other
Alternatively, you can copy-paste the text in question in the comment
form, select the pasted text, and hit the Insert a quote button
which look like a styled, curly, and closing quotation mark ”.
How to migrate a Git repository from legacy to GitLab?
Important: this policy is still being debated. It is not clear if any
or all repositories should be migrated to GitLab, see issue 36
for the discussion on this topic.
As an example of a repository migration, I have moved the wiki from
gitolite to gitlab just now. I have followed the following procedure:
create a project on gitlab (in tpo/tpa/wiki-archive in my case)
push (manually) the latest git references present on git-rw to
gitlab (git push --mirror...)
if the repository is to be archived on GitLab, make it so in
Settings -> General -> Advanced -> Archive project
make an (executable) pre-receive hook in git-rw with an exit
status of 1 warning about the new code location, example:
$ cat /srv/git.torproject.org/repositories/project/help/wiki.git/hooks/pre-receive #!/bin/shcat <<EOFThis repository has been migrated to GitLab:https://gitlab.torproject.org/tpo/tpa/services/-/wikis/homeUpdate your remotes to: email@example.com:tpo/tpa/services.wiki.gitor: https://gitlab.torproject.org/tpo/tpa/services.wiki.gitSee this issue for details:https://gitlab.torproject.org/tpo/tpa/services/-/issues/34437EOFexit 1
or in the case of a fully archived repository (non-writable):
$ cat /srv/git.torproject.org/repositories/project/help/infra.git/hooks/pre-receive #!/bin/shcat <<EOFThis repository has been migrated to GitLab:https://gitlab.torproject.org/tpo/tpa/wiki-infra-archiveWe have migrated away from ikiwiki so it is not necessary anymore.See this issue for details:https://gitlab.torproject.org/tpo/tpa/services/-/issues/34437EOFexit 1
in Gitolite, make the project part of the "Attic", for example
@@ -328,13 +328,13 @@ admin/trac/TracAccountManager "The Tor Project" = "Tor specific changes to Matth repo project/help/infra RW+ = @torproject-admin- config gitweb.category = Infrastructure and Administration-project/help/infra "The Tor Project" = "help.torproject.org infrastructure"+ config gitweb.category = Attic+project/help/infra "The Tor Project" = "help.torproject.org infrastructure (archived to GitLab: https://gitlab.torproject.org/tpo/tpa/wiki-infra-archive')" repo project/help/wiki RW = anarcat- config gitweb.category = Infrastructure and Administration-project/help/wiki "The Tor Project" = "help.torproject.org content"+ config gitweb.category = Attic+project/help/wiki "The Tor Project" = "help.torproject.org content (archived to GitLab: https://gitlab.torproject.org/tpo/tpa/wiki-archive')" repo project/jenkins/jobs RW = @jenkins-admins
The only downside with that approach is that a git clone will not warn
about the project redirection, but I am not sure there's a way to fix
It's possible to add "reaction emojis" to comments and issues and
merge requests in GitLab. Just hit the little smiley face and a dialog
will pop up. You can then browse through the list and pick the right
emoji for how you feel about the comment, but remember to be nice!
By default, new projects do not have notifications setup in
#tor-bots like all the others. To do this, you need to configure a
"Webhook", in the Settings -> Webhooks section of the project. The
URL should be:
... and you should select the notifications you wish to see in
#tor-bots. You can also enable notifications to other channels by
adding more parameters to the URL, like (say)
?channel=%23tor-foo. The parameters are documented the KGB
Note that GitLab admins might be able to configure system-wide
hooks in the admin section, although it's not entirely clear
how does relate to the per-project hooks so those have not been
enabled. Furthermore, it is possible for GitLab admins with root
access to enable webhooks on all projects, with the webhook rake
task. For example, running this on the GitLab server (currently
gitlab-02) will enable the above hook on all repositories:
The current GitLab server was setup in the howto/ganeti cluster in a
regular virtual machine. It was configured with howto/puppet with the
roles::gitlab. That, in turn, relies on a series of profile
elements which configure:
profile::gitlab::web: nginx vhost and TLS cert, depends on
profile::nginx built for the howto/cache service and relying on the
puppet/nginx module from the Forge
profile::gitlab::mail: dovecot and postfix configuration, for
profile::gitlab::database: postgresql configuration, possibly not
used by the Omnibus package, see issue 20
profile::gitlab::app: the core of the configuration of gitlab
itself, uses the puppet/gitlab module from the Forge, with
Prometheus, Grafana, and Nginx support disabled, but Redis,
PostgreSQL, and Prometheus exporters enabled
This installs the GitLab Omnibus distribution which duplicates a
lot of resources we would otherwise manage elsewhere in Puppet,
including (but possibly not limited to):
This therefore leads to a "particular" situation regarding monitoring
and PostgreSQL backups, in particular. See issue 20 for details.
GitLab was migrated from Trac in June 2020, after a few months of
testing. Tests were done first on a server called
dip.torproject.org, a reference to salsa.debian.org, the GitLab
server ran by the Debian project. We identified some problems with
merge requests during the test so the server was reinstalled with
the "GitLab Omnibus" package on the current server, gitlab-02 which
will enter production in the week of June 15th 2020.
We're hoping gitlab will be a good fit because:
Gitlab will allow us to collect our different engineering tools
into a single application: Git repository handling, Wiki, Issue
tracking, Code reviews, and project management tooling.
Gitlab is well-maintained, while Trac plugins are not well
maintained and Trac itself hasn't seen a release for over a year
Gitlab will allow us to build a more modern approach to handling CI
for our different projects. This is going to happen after the
ticket and wiki migration.
(Note that we're only planning to install and use the freely licensed version
of gitlab. There is an "enterprise" version with additional features, but we
prefer to use free software whenever possible.)
The issues and wiki of the "Tor" project are migrated. There are no
other projects in Trac.
Trac issues that remain are really legacy issues, others issues have
been "moved" to the respective projects. @ahf, who did the migration,
created a copy of the mapping for those
looking for their old stuff.
All the tickets that were not moved to their respective projects
have been closed in the first week of july.
We are not migrating away from Gitolite and Jenkins just yet. This
means those services are still fully operational and their equivalent
features in GitLab are not supported (namely Git hosting and
CI). Those services might eventually be migrated to GitLab, but
that's not part of the current migration plan. See issue 36 for
the followup on that.
Again, the canonical copy for source code hosted by git is:
git-rw.torproject - writable git repositories over SSH
We also do not host "GitLab pages", the static site hosting provided
The priority of those features would be:
gitolite replacement and migration
CI deployment, with people migrating their own job from Jenkinks
and TPA shutting down Jenkins on a flag date
GitLab pages replacement and migration from the current static
site hosting system
Those are each large projects and will be undertaken at a later stage,
checklists show up as "X of Y tasks completed"¹
requires conversion from days to hours
RSS, email, ML
Trac sends email to trac-bugs
if users can create projects
no support for custom queries in GitLab³
zwiebelbot has to be patched, other bots to be deployed for notifications⁵
concerns about trusting GitLab with our code
maybe in the future
Trac does not seem well maintained
one big wiki
REST, multiple clients
Drag-and-drop boards seem not to work but the list of issues still can be used.
Trac parent/child issue relationships have been converted into a
simple comment at the beginnning of the ticket linking to the
child/parent tickets. It was originally hoped to use the
"checklists" features but this was not implemented for lack of time.
User registration is perfectly possible in GitLab but since GitLab
instances are frequently attacked by spammers, it is disabled
until we find an alternative. See missing features below for
GitLab, in particular, does not support inline searches, see
Missing features below for details.
The wiki and issue formatting markup is different. Whereas Trac
uses wiki formatting inspired by old wikis like
MoinMoin, a subset of the somewhat standard Wikicreole
markup, GitLab uses Markdown, specifically their own GitLab
version of markdown inspired by GitHub's markdown
extensions. The wiki and issues were automatically converted to
Markdown, but when you file new issues, you will need to use
Markdown, not Creole.
specifically, zwiebelbot now knows about foo#N pointing to issue
N in project foo in GitLab. We need to update (or replace) the
nsa bot in #tor-bots to broadcast announcements to
projects. This could be done with the KGB bot for which we
now have a Puppet module so it could easily be deployed here
because Trac does not allow users to create projects, we have
historically used one gigantic project for everything, which means
we had only one wiki. technically, Trac also supports one wiki per
project, but because project creation requires an admin
intervention, this never concretized.
Ticket fields equivalence
keep the ticket id in legacy project, starts at 40000 in GitLab
use templates to make sure those are filled
Points, in days
/estimate, in hours
boards can sort issues instead of assigning arbitrary keywords
mark only blocker issues to resolve
paid plans also have multiple assignees
issue mentions and checklists
Attachements, per comment
Kanban boards panels
Notice how the Label field is used as a fallback when no equivalent
GitLab does not provide one-to-one feature parity with Trac, but it
comes pretty close. It has issue tracking, wikis, milestones,
keywords, time estimates, and much more.
But one feature it is missing is the advanced ticket query
features of Trac. It's not possible to create "reports" in GitLab to
have pre-cooked issue listings. And it's especially not possible to
embed special searches in wiki pages the same way it is done in Trac.
We suggest people use the "dashboard" feature of GitLab instead. This
featuers follows the Kanban development strategy which is
implemented in GitLab as issue boards. It is also, of course,
possible to link so specific searches from the wiki, but not embed
those tickets in the output.
We do not have a anonymous account (AKA cypherpunks) for
now. GitLab will be in closed registration for now, with users
needing to request approval on a per-person basis for now. Eventually,
we're going to consider other options to work around this (human)
Interesting new features
Using pull requests to your project repositories, and assigning
reviewers on pull requests, rather than using reviewer and
needs_review labels on issues. Issues can refer to pull requests
and vice versa.
Your team can work on using Gitlab boards for handling the
different stages of issue handling. All the way from selection to
finalization with code in a PR. You can have as many boards as you
like: per subproject, per sponsor, per week, all of this is
something we can experiment with.
You can now use time estimation in Gitlab simply by adding a specially
formatted comment in your issues/pull requests instead of using points and
actual_points. See the time tracking documentation for details
Familiarize yourself with new interfaces such as the "to do"
dashboard where you can see what needs your input since last
Create email filters for tickets: Gitlab adds a lot more email
headers to each notification you receive (if you want it via
email), which for example allows you split notifications in your
mail program into different directories.
Bonus info: You will be able to reply via email to the notifications you
receive from Gitlab, and Gitlab will put your responses into the system as
notes on issues :-)
legacy tickets: bugs.torproject.org/N redirects to
new issues: bugs.tpo/PROJECT/N redirects to
merge requests: bugs.tpo/PROJECT!N redirects to
catch all: bugs.tpo/FOO redirects to gitlab.tpo/FOO
ticket list: a bare bugs.tpo redirects to
It used to be that bugs.tpo/N would redirect to issue N the Trac
"tor" project. But unfortunately, there's no global "number space" for
issues in GitLab (or at least not a user-visible one), so N is not
distinct across projects. We therefore need the prefix to
We considered enforcing the tpo prefix there to shorten links, but
we decided against it because it would forbid pointers to
user-specific projects and would make it extremely hard to switch away
from the global tpo group if we ever decide to do that.
Projects are all stored under the over-arching tpo group. This is
done this way to allow project managers to have an overview of all
projects going on at TPO. It also allows us to host other
organisations on our GitLab in a different namespace.
Under the tpo group, each team has its own subgroup and they have
autonomy under that group to manage accesses and projects.
Given the above Team/Group organization, users will be members in
gitlab for the groups/teams they belong to.
Any projects that need to be shared between multiple groups should be
shared using the “Share Project” functionality.
There should be a limited number of members in the Tor Project group,
as these will have access to all subgroups and their
projects. Currently this is limited to Project Managers and Services
Guests: anybody that may need to report issues on a project and/or
make comments on an issue.
Reporter: they can also manage labels
Developer: they can create branches, manage merge requests, force
push to non-protected branches
Maintainer: edit projects, manage runners, edit comments, delete
Owner: we are setting this role for every member in the TPO
team. They can also transfer projects to other name spaces, switch
visilbity level, delete issues.
At group level we have sponsor labels and state labels. The ones that
are used by the whole organization are in the tpo group. Each team
can decide which other labels they add for their projects.
Types of Issue
Related to a project
Other possible keywords needed at group level.
Note that those labels are being worked on ticket 4. We also have
a lot more label than we would like (ticket 3) which makes GitLab
hard to use. Because there are thousands of labels in some projects,
loading the label list can take a second or more on slower links, and
it's really hard to find the label you're looking for, which affects
usability -- and especially discoverability -- quite a bit.
ahf performed a major label cleanup operation on 2020-06-27, following
the specification in the label cleanup repository. It rewrote and
deleted labels in one batch in all projects. When the job was done,
empty labels were removed as well.
Most notably, we're going to have an interruption in the ability to open new
accounts and new tickets. We did not want to migrate without a solution
here; we'll try to have at least a stop-gap solution in place soon, and
something better in the future. For now, we're planning for people that want
to get a new account please send a mail to firstname.lastname@example.org. We
hope to have something else in place once the migration is succesful.
We're not going to migrate long-unused accounts.
Some wiki pages that contained automated listings of tickets will stop
containing those lists: that's a trac feature that gitlab doesn't have. We'll
have to adjust our workflows to work around this. In some cases, we can use
gitlab milestone pages or projects that do not need a wiki page as a work
This information can be extracted back again from the *-meta.warc.gz
(text) files in the above URLs. This was done as part of ticket
40003. There has also been other, independent, crawls of Trac,
which are partly visible in the viewer.
lost in the mists of time: migration from Bugzilla to Flyspray (40
2020-06-13 02:25UTC: Trac tickets migrated (32401 tickets, last
ticket id is 34451, first GitLab legacy project ticket id is
2020-06-14 21:22UTC: Trac wiki migrated
2020-06-15 18:30UTC: bugs.torproject.org redirects to gitlab
2020-06-16 02:15UTC: GitLab launch announced to tor-internal
2020-06-17 12:33UTC: Archivebot starts crawling all tickets of, and
the entire Trac website
2020-06-23: Archivebot completes the full Trac crawl, Trac is fully
archived on the Internet Archive
GitLab is a fairly large program with multiple components. The
upstream documentation has a good details of the architecture but
this section aims at providing a shorter summary. Here's an overview
The web frontend is Nginx (which we incidentally also use in our
howto/cache system) but GitLab wrote their own reverse proxy called
GitLab Workhorse which in turn talks to the underlying GitLab
Rails application, served by the Unicorn application
server. The Rails app stores its data in a howto/postgresql database
(although not our own deployment, for now, should be fixed). GitLab also offloads
long-term background tasks to a tool called sidekiq.
Those all server HTTP(S) requests but GitLab is of course also
accessible over SSH to push/pull git repositories. This is handled by
a separate component called gitlab-shell which acts as a shell
for the git user.
Workhorse, Rails, sidekiq and gitlab-shell all talk with Redis to
store temporary information, caches and session information. They can
also communicate with the Gitaly server which handles all
communication with the git repositories themselves.
Finally, Git)Lab also features GitLab Pages and Continuous Integration
("pages" and CI, neither of which we do not currently use). CI is
handled by GitLab runners which can be deployed by anyone and
registered in the Rails app to pull CI jobs. GitLab pages is "a
simple HTTP server written in Go, made to serve GitLab Pages with
CNAMEs and SNI using HTTP/HTTP2".
Monitoring right now is minimal: normal host-level metrics like disk
space, CPU usage, web port and TLS certificates are monitored by
Nagios with our normal infrastructure, as a black box.
Prometheus monitoring is built into the GitLab Omnibus package, so it
is not configured through our Puppet like other Prometheus
servers. It has still been (manually) integrated in our Prometheus
setup and Grafana dashboards (see pager playbook) have been deployed.
More work is underway to improve monitoring in issue 33921.
There is a cronjob configured via the gitlab puppet monitor that runs
every night at 2:00AM UTC. This will basically run $ gitlab-backup
and will create a tarball under /srv/gitlab-backup/.
There is also a config backup job (in
/etc/cron.d/gitlab-config-backup) that makes sure to backup the
content of /var/opt/gitlab/gitlab-rails/etc/ is backed up, because
that is not covered by the gitlab-backup command.
Another cron job purges backups older than two days, in
/etc/cron.d/gitlab-rotate-backup, so that we don't keep too many
copies on the server. It is assumed that the existing howto/backup
system will pick up those copies and store them for our normal
The GitLab project at Tor has been a long time coming. If you look at
the history section above, you'll see it has been worked on since
at least 2016, at which point an external server was setup for the
"network team" to do code review. This server was ultimately retired.
The current server has been worked on since 2019, with the master
ticket, issue 29400, created in the footsteps of the 2019
Brussels meeting. The service launched some time in June 2020,
with a full migration of Trac tickets.
replacement of the Trac issue tracking server
rough equivalent of Trac features in GitLab
Nice to have
identical representation of Trac issues in GitLab, including proper
replacement of Gitolite (git hosting)
replacement of Gitweb (git hosting)
replacement of Jenkins (CI)
replacement of the static site hosting system
Those are not part of the first phase of the project, but it is
understood that if one of those features gets used more heavily in
GitLab, the original service MUST be eventually migrated into GitLab
and turned off. We do not want to run multiple similar services at
the same time (for example run both gitolite and gitaly on all git
repositories, or run Jenkins and GitLab runners).
The GitLab migration was approved at the 2019 Brussels dev meeting.
The solution to the "code review" and "project management" problems
are to deploy a GitLab instance which does not aim at managing all
source code, in the first stage.
Staff not evaluated.
In terms of hardware, we start with a single virtual machine and agree
that, in the worst case, we can throw a full Hetzner PX62-NVMe node at
the problem (~70EUR/mth).
GitLab is such a broad project that multiple alternatives exist for
widely used in the open source community
Good integration between ticketing system and code
It is hosted by a third party (Microsoft!)
Mostly free software
Complex software, high maintenance
"Opencore" - some interesting features are closed-source
GitLab command line clients
If you want to do batch operations or integrations with GitLab, you
might want to use one of those tools, depending on your environment or
prefered programming language:
salsa (in Debian devscripts) is specifically built for
salsa but might be coerced into talking to other GitLab servers
ahf implemente the gitlab using his own home-made tools that talk to
the GitLab and Trac API. but there's also tracboat which is
designed to migrate from trac to GitLab.
We did not use Tracboat because it uses gitlab's DB directly and thus
only works with some very specific version. Each time the database
schema changes at GitLab, Tracboat needs to port to it. We prefered to
use something that talked with the GitLab API.
We also didn't like the output entirely, so we modified it but still
used some of its regular expressions and parser.
We also needed to implement the "ticket movement" hack (with the legacy
project) which wasn't implemented in Tracboat.
Finally, we didn't want to do complete user migration, but lazily
transfer only some users.