Newer
Older
[GitLab](https://gitlab.com/) is *a web-based DevOps lifecycle tool that provides a
Git-repository manager providing wiki, issue-tracking and continuous
integration/continuous deployment pipeline features, using an
open-source license, developed by GitLab Inc* ([Wikipedia](https://en.wikipedia.org/wiki/GitLab)). Tor
uses GitLab mainly for issue tracking, wiki hosting and code review
for now, at <https://gitlab.torproject.org>, after migrating from
[[trac]].
[[!toc levels=3]]
<!-- note: this template was designed based on multiple sources: -->
<!-- https://www.divio.com/blog/documentation/ -->
<!-- http://opsreportcard.com/section/9-->
<!-- http://opsreportcard.com/section/11 -->
<!-- comments like this one should be removed on instanciation -->
# Tutorial
<!-- simple, brainless step-by-step instructions requiring little or -->
<!-- no technical background -->
## How do I get an account?
Please send a mail to <gitlab-admin@torproject.org>. We hope to have something
else in place once the migration is successful.
## How do I report an issue in Tor software?
For now, just use one of the mailing lists, if it can't wait till the
bugtracker is open again. <tor-dev@lists.torproject.org> is the best
mailing list to use for now.
## Report an issue in the bugtracker itself?
Please hold off till the migration is finished. At that point you can
email <gitlab-admin@torproject.org>.
## Contribute code?
If you have a patch you want to make sure we know about, and it can't
wait till the bugtracker is open again, please use one of the mailing
lists: <tor-dev@lists.torproject.org> would be best.
# How-to
<!-- more in-depth procedure that may require interpretation -->
## Pager playbook
<!-- information about common errors from the monitoring system and -->
<!-- how to deal with them. this should be easy to follow: think of -->
<!-- your future self, in a stressful situation, tired and hungry. -->
* Grafana Dashboards:
* [GitLab overview](https://grafana.torproject.org/d/QrDJktiMz/gitlab-omnibus)
* [Gitaly](https://grafana.torproject.org/d/x6Z50y-iz/gitlab-gitaly)
## Disaster recovery
In case the entire GitLab machine is destroyed, a new server should be
provisionned in the [[ganeti]] cluster (or elsewhere) and backups
should be restored using the below procedure.
### Running an emergency backup
TBD
### baremetal recovery
TBD
# Reference
## Installation
The current GitLab server was setup in the [[ganeti]] cluster in a
regular virtual machine. It was configured with [[puppet]] with the
`roles::gitlab`.
This installs the [GitLab Omnibus](https://docs.gitlab.com/omnibus/) distribution which duplicates a
lot of resources we would otherwise manage elsewhere in Puppet,
including (but possibly not limited to):
* [[prometheus]]
* [[postgresql]]
This therefore leads to a "particular" situation regarding monitoring
and PostgreSQL backups, in particular.
## SLA
<!-- this describes an acceptable level of service for this service -->
## Migration
GitLab was migrated from Trac in June 2020, after a few months of
testing. Tests were done first on a server called
`dip.torproject.org`, a reference to `salsa.debian.org`, the GitLab
server ran by the Debian project. We identified [some problems with
merge requests](https://gitlab.torproject.org/tpo/tpa/services/-/issues/32197) during the test so the server was reinstalled with
the "GitLab Omnibus" package on the current server, `gitlab-02` which
will enter production in the week of June 15th 2020.
### Why migrate?
We're hoping gitlab will be a good fit because:
* Gitlab will allow us to collect our different engineering tools
into a single application: Git repository handling, Wiki, Issue
tracking, Code reviews, and project management tooling.
* Gitlab is well-maintained, while Trac plugins are not well
maintained and Trac itself hasn't seen a release for over a year
(since 2019)
* Gitlab will allow us to build a more modern approach to handling CI
for our different projects. This is going to happen after the
ticket and wiki migration.
(Note that we're only planning to install and use the freely licensed version
of gitlab. There is an "enterprise" version with additional features, but we
prefer to use free software whenever possible.)
### Migrated content
The issues and wiki of the "Tor" project are migrated. There are no
other projects in Trac.
<!-- TODO: details of the migration -->
* Trac wiki: <https://gitlab.torproject.org/legacy/trac>
* Trac issues: <https://gitlab.torproject.org/groups/legacy/-/issues>
Trac issues that remain are really legacy issues, others issues have
been "moved" to the respective projects. @ahf, who did the migration,
maintained a [copy of the mapping](https://anubis.0x90.dk/~ahf/sekrit/tor/gitlab/component-tickets.txt) for those looking for their old
stuff.
We are *not* migrating away from Gitolite and Jenkins just yet. This
means those services are still fully operational and their equivalent
features in GitLab are *not* supported (namely Git hosting and
CI). Those services *might* eventually be migrated to GitLab, but
that's not part of the current migration plan.
Again, the canonical copy for source code hosted by git is:
* `git-rw.torproject` - writable git repositories over SSH
* <https://git.torproject.org/> - readonly clones
* <https://gitweb.torproject.org/> - web interface
We also do not host "GitLab pages", the static site hosting provided
by GitLab.
The priority of those features would be:
1. gitolite replacement and migration
2. CI deployment, with people migrating their own job from Jenkinks
and TPA shutting down Jenkins on a flag date
3. GitLab pages replacement and migration from the current static
site hosting system
Those are each large projects and will be undertaken at a later stage,
progressively.
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
### Feature equivalence
| Feature | Trac | GitLab | Comments |
| ------- | ---- | ------ | -------- |
| Ticket relations | parent/child | checklists | checklists show up as "X of Y tasks completed"¹ |
| Milestones | yes | yes | |
| Estimates | points/actual | estimation/spending | requires conversion from days to hours |
| Private issues | no | yes | |
| Issue subscription | RSS, email, ML | email | Trac sends email to trac-bugs |
| User projects | no | yes | if users can create projects |
| User registration | optional | disabled | ² |
| Search | advanced | basic | no support for custom queries in GitLab³ |
| Markup | WikiCreole | Markdown, GitHub-like | ⁴ |
| IRC bot | yes | yes | zwiebelbot has to be patched, other bots to be deployed for notifications⁵ |
| Git hosting | no, gitolite | yes, builtin | concerns about trusting GitLab with our code |
| CI | no, Jenkins | yes, builtin | maybe in the future |
| Upstream maintenance | slow | fast | Trac does not seem well maintained |
| Wikis | one big wiki | per-project | ⁶ |
| API | XML-RPC | REST, multiple clients | |
| Javascript | optional | required | Drag-and-drop boards seem not to work but the list of issues still can be used. |
Notes:
1. Trac parent/child issue relationships have been converted into a
simple comment at the beginnning of the ticket linking to the
child/parent tickets. It was originally hoped to use the
"checklists" features but this was not implemented for lack of time.
2. User registration is perfectly possible in GitLab but since GitLab
instances are frequently attacked by spammers, it is disabled
until we find an alternative. See missing features below for
details).
3. GitLab, in particular, does not support inline searches, see
Missing features below for details.
4. The wiki and issue formatting markup is different. Whereas Trac
uses [wiki formatting](https://trac.edgewall.org/wiki/WikiFormatting) inspired by old wikis like
[MoinMoin](https://moinmo.in/), a subset of the somewhat standard [Wikicreole](http://www.wikicreole.org/)
markup, GitLab uses [Markdown](https://en.wikipedia.org/wiki/Markdown), specifically their own [GitLab
version of markdown](https://gitlab.com/help/user/markdown) inspired by GitHub's markdown
extensions. The wiki and issues were automatically converted to
Markdown, but when you file new issues, you will need to use
Markdown, not Creole.
5. specifically, zwiebelbot now knows about `foo#N` pointing to issue
N in project `foo` in GitLab. We need to update (or replace) the
`nsa` bot in `#tor-bots` to broadcast announcements to
projects. This could be done with the [KGB](https://salsa.debian.org/kgb-team/kgb/-/wikis/home) bot for which we
now have a [Puppet module](https://gitlab.com/shared-puppet-modules-group/kgb) so it could easily be deployed here
6. because Trac does not allow users to create projects, we have
historically used one gigantic project for everything, which means
we had only one wiki. technically, Trac also supports one wiki per
project, but because project creation requires an admin
intervention, this never concretized.
### Ticket fields equivalence
| Trac | GitLab | Comments |
| ---- | ------ | -------- |
| id | id | keep the ticket id in legacy project, starts at 40000 in GitLab |
| Summary | ? | unused? |
| Reporter | Reporter | |
| Description | Body | |
| Type | Label | use templates to make sure those are filled |
| Milestone | Milestone, Label | |
| Version | Label | |
| Keywords | Label | |
| Points, in days | /estimate, in hours | requires conversion |
| Actual points | /spending | |
| Sponsor | Label | |
| Priority | Board, Label | boards can sort issues instead of assigning arbitrary keywords |
| Component | Subproject, Label | |
| Severity | Label | mark only blocker issues to resolve |
| Cc | @people | paid plans also have multiple assignees |
| Parent issue | #reference | issue mentions and checklists |
| Reviewer | Label | |
| Attachements | Attachements, per comment | |
| Status | Label | Kanban boards panels |
Notice how the `Label` field is used as a fallback when no equivalent
field exists.
### Missing features
GitLab does not provide one-to-one feature parity with Trac, but it
comes pretty close. It has issue tracking, wikis, milestones,
keywords, time estimates, and much more.
But one feature it is missing is the **advanced ticket query**
features of Trac. It's not possible to create "reports" in GitLab to
have pre-cooked issue listings. And it's especially not possible to
embed special searches in wiki pages the same way it is done in Trac.
We suggest people use the "dashboard" feature of GitLab instead. This
featuers follows the [Kanban][] development strategy which is
implemented in GitLab as [issue boards](https://docs.gitlab.com/ee/user/project/issue_board.html). It is also, of course,
possible to link so specific searches from the wiki, but not embed
those tickets in the output.
[Kanban]: https://en.wikipedia.org/wiki/Kanban_(development)
We do *not* have a **anonymous account** (AKA `cypherpunks`) for
now. GitLab will be in **closed registration** for now, with users
needing to request approval on a per-person basis for now. Eventually,
we're going to consider other options to work around this (human)
bottleneck.
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
### Interesting new features
1. Using pull requests to your project repositories, and assigning
reviewers on pull requests, rather than using `reviewer` and
`needs_review` labels on issues. Issues can refer to pull requests
and vice versa.
2. Your team can work on using Gitlab boards for handling the
different stages of issue handling. All the way from selection to
finalization with code in a PR. You can have as many boards as you
like: per subproject, per sponsor, per week, all of this is
something we can experiment with.
3. You can now use time estimation in Gitlab simply by adding a specially
formatted comment in your issues/pull requests instead of using `points` and
`actual_points`. See the [time tracking documentation](https://docs.gitlab.com/ee/user/project/time_tracking.html) for details
4. Familiarize yourself with new interfaces such as the ["to do"
dashboard](https://docs.gitlab.com/ee/user/todos.html) where you can see what needs your input since last
visit
5. Create email filters for tickets: Gitlab adds a lot more [email
headers to each notification](https://docs.gitlab.com/ee/user/profile/notifications.html#filtering-email) you receive (if you want it via
email), which for example allows you split notifications in your
mail program into different directories.
Bonus info: You will be able to reply via email to the notifications you
receive from Gitlab, and Gitlab will put your responses into the system as
notes on issues :-)
### bugs.torproject.org redirections
The <https://bugs.torproject.org> redirection now points at
GitLab. The following rules apply:
1. **legacy tickets**: `bugs.torproject.org/N` redirects to
`gitlab.torproject.org/legacy/trac/-/issues/N`
2. **new issues**: `bugs.tpo/PROJECT/N` redirects to
`gitlab.tpo/PROJECT/-/issues/N`
3. **merge requests**: `bugs.tpo/PROJECT!N` redirects to
`gitlab.tpo/PROJECT/-/merge_requests/N`
4. **catch all**: `bugs.tpo/FOO` redirects to `gitlab.tpo/FOO`
5. **ticket list**: a bare `bugs.tpo` redirects to
`https://gitlab.torproject.org/tpo/-/issues`
It used to be that `bugs.tpo/N` would redirect to issue `N` the Trac
"tor" project. But unfortunately, there's no global "number space" for
issues in GitLab (or at least not a user-visible one), so `N` is not
distinct across projects. We therefore need the prefix to
disambiguate.
We considered enforcing the `tpo` prefix there to shorten links, but
we decided against it because it would forbid pointers to
user-specific projects and would make it extremely hard to switch away
from the global `tpo` group if we ever decide to do that.
1. finish the documentation here, look for `TODO` items in the
document, the `Other considerations` section in the Nextcloud
`PLAN - migration to gitlab.docx` document, look at the `PROPOSAL
of migration to Gitlab.docx` document as well, attach the ahf
[component-tickets.txt](https://anubis.0x90.dk/~ahf/sekrit/tor/gitlab/component-tickets.txt) file to this document
2. Make sure the daily tools we use is ready such as IRC bots.
3. Make sure TPO project issue change notifications are emailed to
the [tor-bugs](https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs) mailing list.
4. Begin handing out some of the accounts to community members that
we have created in their name(s).
5. Begin figuring out how to solve the account creation and anonymous
users in the longer run.
### What will break, and when will you fix it?
Most notably, we're going to have an interruption in the ability to open new
accounts and new tickets. We _did not_ want to migrate without a solution
here; we'll try to have at least a stop-gap solution in place soon, and
something better in the future. For now, we're planning for people that want
to get a new account please send a mail to <gitlab-admin@torproject.org>. We
hope to have something else in place once the migration is succesful.
We're not going to migrate long-unused accounts.
Some wiki pages that contained automated listings of tickets will stop
containing those lists: that's a trac feature that gitlab doesn't have. We'll
have to adjust our workflows to work around this. In some cases, we can use
gitlab milestone pages or projects that do not need a wiki page as a work
around.
* lost in the mists of time: migration from Bugzilla to Flyspray (40
tickets)
* 2010-04-23: [migration from Flyspray to Trac completed](https://lists.torproject.org/pipermail/tor-dev/2010-April/000183.html) (last
Flyspray ticket is [1393](https://bugs.torproject.org/1393), first Trac ticket is [2000](https://bugs.torproject.org/2000))
* 2016-11-29: [first request to setup a GitLab server](https://gitlab.torproject.org/tpo/tpa/services/-/issues/20821)
* ~2017: oniongit.eu (warning: squatted domain) deployed to test
GitLab with the network team, considered as gitlab.torproject.net
but ultimately [abandoned](https://gitlab.torproject.org/tpo/tpa/services/-/issues/21840)
* 2019-02-28: `gitlab-01` AKA dip.torproject.org test server setup
([issue 29400](https://gitlab.torproject.org/tpo/tpa/services/-/issues/29400)), following the [Brussels meeting](https://trac.torproject.org/projects/tor/wiki/org/meetings/2019BrusselsAdminTeamMinutes)
* 2019-07-17: GitLab discussed again at the [Stockholm meeting](https://gitlab.torproject.org/legacy/trac/-/wikis/org/meetings/2019Stockholm/Notes/InternalTooling)
* 2019-07-29: Formal proposal to deploy GitLab [sent to
tor-project](https://lists.torproject.org/pipermail/tor-project/2019-July/002407.html), no objection
* 2020-03-05: GitLab migrated from `gitlab-01` (AKA "dip") to
`gitlab-02` using the Omnibus package
* 2020-04-27: `gitlab-01` retired
* 2020-06-13 19:00UTC: [Trac readonly](https://lists.torproject.org/pipermail/tor-project/2020-June/002872.html)
* 2020-06-13 02:25UTC: Trac tickets migrated (32401 tickets, last
ticket id is [34451](https://bugs.torproject.org/34451), first GitLab legacy project ticket id is
40000)
* 2020-06-14 21:22UTC: Trac wiki migrated
* 2020-06-15 18:30UTC: bugs.torproject.org redirects to gitlab
* 2020-06-16 02:15UTC: GitLab launch announced to tor-internal
## Design
<!-- how this is built -->
<!-- should reuse and expand on the "proposed solution", it's a -->
<!-- "as-built" documented, whereas the "Proposed solution" is an -->
<!-- "architectural" document, which the final result might differ -->
<!-- from, sometimes significantly -->
<!-- a good guide to "audit" an existing project's design: -->
<!-- https://bluesock.org/~willkg/blog/dev/auditing_projects.html -->
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
GitLab is a fairly large program with multiple components. The
[upstream documentation](https://docs.gitlab.com/ee/development/architecture.html) has a good details of the architecture but
this section aims at providing a shorter summary. Here's an overview
diagram, first:
The web frontend is Nginx (which we incidentally also use in our
[[cache]] system) but GitLab wrote their own reverse proxy called
[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse/) which in turn talks to the underlying GitLab
Rails application, served by the [Unicorn](https://yhbt.net/unicorn/) application
server. The Rails app stores its data in a [[postgresql]] database
(although not our own deployment, for now: TODO). GitLab also offloads
long-term background tasks to a tool called [sidekiq](https://github.com/mperham/sidekiq).
Those all server HTTP(S) requests but GitLab is of course also
accessible over SSH to push/pull git repositories. This is handled by
a separate component called [gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell) which acts as a shell
for the `git` user.
Workhorse, Rails, sidekiq and gitlab-shell all talk with Redis to
store temporary information, caches and session information. They can
also communicate with the [Gitaly](https://gitlab.com/gitlab-org/gitaly) server which handles all
communication with the git repositories themselves.
Finally, Git)Lab also features GitLab Pages and Continuous Integration
("pages" and CI, neither of which we do not currently use). CI is
handled by [GitLab runners](https://gitlab.com/gitlab-org/gitlab-runner/) which can be deployed by anyone and
registered in the Rails app to pull CI jobs. [GitLab pages](https://gitlab.com/gitlab-org/gitlab-pages) is "a
simple HTTP server written in Go, made to serve GitLab Pages with
CNAMEs and SNI using HTTP/HTTP2".
## Issues
There is no issue tracker specifically for this project, [File][] or
[search][] for issues in the [gitlab project][search].
[File]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/new
[search]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues
## Monitoring and testing
Monitoring right now is minimal: normal host-level metrics like disk
space, CPU usage, web port and TLS certificates are monitored by
Nagios with our normal infrastructure, as a black box.
Prometheus monitoring is built into the GitLab Omnibus package, so it
is *not* configured through our Puppet like other Prometheus
servers. It has still been (manually) integrated in our Prometheus
setup and Grafana dashboards (see [pager playbook](#Pager_playbook)) have been deployed.
More work is underway to improve monitoring in [issue 33921](https://gitlab.torproject.org/tpo/tpa/services/-/issues/33921).
## Backups
There is a cronjob configured via the gitlab puppet monitor that runs
every night at 2:00AM UTC. This will basically run `$ gitlab-backup `
and will create a tarball under `/srv/gitlab-backup/`.
There is also a config backup job (in
`/etc/cron.d/gitlab-config-backup`) that makes sure to backup the
content of `/var/opt/gitlab/gitlab-rails/etc/` is backed up, because
that is not covered by the `gitlab-backup` command.
Another cron job purges backups older than two days, in
`/etc/cron.d/gitlab-rotate-backup`, so that we don't keep too many
system will pick up those copies and store them for our normal
rotation periods.
# Discussion
## Overview
<!-- describe the overall project. should include a link to a ticket -->
<!-- that has a launch checklist -->
The GitLab project at Tor has been a long time coming. If you look at
the [history](#History) section above, you'll see it has been worked on since
at least 2016, at which point an external server was setup for the
"network team" to do code review. This server was ultimately retired.
The current server has been worked on since 2019, with the master
ticket, [issue 29400](https://gitlab.torproject.org/tpo/tpa/services/-/issues/29400), created in the footsteps of the [2019
Brussels meeting](https://trac.torproject.org/projects/tor/wiki/org/meetings/2019BrusselsAdminTeamMinutes). The service launched some time in June 2020,
with a full migration of Trac tickets.
## Goals
### Must have
* replacement of the Trac issue tracking server
* rough equivalent of Trac features in GitLab
### Nice to have
* identical representation of Trac issues in GitLab, including proper
issue numbering
* replacement of Gitolite (git hosting)
* replacement of Gitweb (git hosting)
* replacement of Jenkins (CI)
* replacement of the static site hosting system
Those are not part of the first phase of the project, but it is
understood that if one of those features gets used more heavily in
GitLab, the original service MUST be eventually migrated into GitLab
and turned off. We do *not* want to run multiple similar services at
the same time (for example run both gitolite and gitaly on all git
repositories, or run Jenkins and GitLab runners).
## Approvals required
The GitLab migration was approved at the 2019 Brussels dev meeting.
## Proposed Solution
The solution to the "code review" and "project management" problems
are to deploy a GitLab instance which does *not* aim at managing all
source code, in the first stage.
Staff not evaluated.
In terms of hardware, we start with a single virtual machine and agree
that, in the worst case, we can throw a full Hetzner PX62-NVMe node at
the problem (~70EUR/mth).
## Alternatives considered
GitLab is such a broad project that multiple alternatives exist for
different components:
* GitHub
* Pros:
* widely used in the open source community
* Good integration between ticketing system and code
* Cons
* It is hosted by a third party (Microsoft!)
* Closed source
* GitLab:
* Pros:
* Mostly free software
* Feature-rich
* Cons:
* Complex software, high maintenance
* "Opencore" - some interesting features are closed-source
If you want to do batch operations or integrations with GitLab, you
might want to use one of those tools, depending on your environment or
prefered programming language:
* [bugwarrior](https://github.com/ralphbean/bugwarrior) ([Debian](https://tracker.debian.org/pkg/bugwarrior)) - support for GitLab, GitHub and
other bugtrackers for the [taskwarrior](http://taskwarrior.org/) database
* [git-lab](https://invent.kde.org/sdk/git-lab) - python commandline client, lists, pulls MR; creates
snippets
* [GitLab-API-v4](https://metacpan.org/release/GitLab-API-v4) ([Debian](https://tracker.debian.org/pkg/libgitlab-api-v4-perl)) - perl library and [commandline
client](https://manpages.debian.org/buster/libgitlab-api-v4-perl/gitlab-api-v4.1p.en.html)
* [GitLabracadabra](https://gitlab.com/gitlabracadabra/gitlabracadabra) ([Debian](https://tracker.debian.org/pkg/gitlabracadabra)) - *configure a GitLab instance
from a YAML configuration, using the API*: project settings like
labels, admins, etc
* [python-gitlab](https://github.com/python-gitlab/python-gitlab) (also known as `gitlab-cli` in [Debian](https://tracker.debian.org/pkg/python-gitlab))
* [ruby-gitlab](https://github.com/narkoz/gitlab) ([Debian](https://tracker.debian.org/pkg/ruby-gitlab)), also includes a [commandline
client](https://narkoz.github.io/gitlab/)
* [salsa](https://manpages.debian.org/buster/devscripts/salsa.1.en.html) (in [Debian devscripts](https://tracker.debian.org/pkg/devscripts)) is specifically built for
salsa but might be coerced into talking to other GitLab servers
### Migration tools
ahf implemente the gitlab using his own home-made tools that talk to
the GitLab and Trac API. but there's also [tracboat](https://github.com/tracboat/tracboat) which is
designed to migrate from trac to GitLab.
We did not use Tracboat because it uses gitlab's DB directly and thus
only works with some very specific version. Each time the database
schema changes at GitLab, Tracboat needs to port to it. We prefered to
use something that talked with the GitLab API.
We also didn't like the output entirely, so we modified it but still
used some of its regular expressions and parser.
We also needed to implement the "ticket movement" hack (with the legacy
project) which wasn't implemented in Tracboat.
Finally, we didn't want to do complete user migration, but lazily
transfer only some users.