|
|
|
[GitLab](https://gitlab.com/) is *a web-based DevOps lifecycle tool that provides a
|
|
|
|
Git-repository manager providing wiki, issue-tracking and continuous
|
|
|
|
integration/continuous deployment pipeline features, using an
|
|
|
|
open-source license, developed by GitLab Inc* ([Wikipedia](https://en.wikipedia.org/wiki/GitLab)). Tor
|
|
|
|
uses GitLab mainly for issue tracking, wiki hosting and code review
|
|
|
|
for now, at <https://gitlab.torproject.org>, after migrating from
|
|
|
|
[howto/trac](howto/trac).
|
|
|
|
|
|
|
|
[[_TOC_]]
|
|
|
|
|
|
|
|
<!-- note: this template was designed based on multiple sources: -->
|
|
|
|
<!-- https://www.divio.com/blog/documentation/ -->
|
|
|
|
<!-- http://opsreportcard.com/section/9-->
|
|
|
|
<!-- http://opsreportcard.com/section/11 -->
|
|
|
|
<!-- comments like this one should be removed on instanciation -->
|
|
|
|
|
|
|
|
# Tutorial
|
|
|
|
|
|
|
|
<!-- simple, brainless step-by-step instructions requiring little or -->
|
|
|
|
<!-- no technical background -->
|
|
|
|
|
|
|
|
## How to get an account?
|
|
|
|
|
|
|
|
You might already *have* an account! If you were active on Trac, your
|
|
|
|
account was migrated with the same username and email address as Trac,
|
|
|
|
unless you have an LDAP account, in which case that was used. So head
|
|
|
|
over to the [password reset page](https://gitlab.torproject.org/users/password/new) to get access to your account.
|
|
|
|
|
|
|
|
If your account was *not* migrated, send a mail to
|
|
|
|
<gitlab-admin@torproject.org> to request a new one.
|
|
|
|
|
|
|
|
We hope to have something else in place for user registration once the
|
|
|
|
migration is successful.
|
|
|
|
|
|
|
|
## How to report an issue in Tor software?
|
|
|
|
|
|
|
|
You first need to figure out which project the issue resides in. The
|
|
|
|
[project list][] is a good place to get started. Here are a few quick
|
|
|
|
links for popular projects:
|
|
|
|
|
|
|
|
[project list]: https://gitlab.torproject.org/tpo
|
|
|
|
|
|
|
|
* [core tor](https://gitlab.torproject.org/tpo/core/tor): [issues](https://gitlab.torproject.org/tpo/core/tor/-/issues), [new issue](https://gitlab.torproject.org/tpo/core/tor/-/issues/new)
|
|
|
|
* [Tor Browser](https://gitlab.torproject.org/tpo/applications/tor-browser): [issues](https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues), [new issue](https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/new)
|
|
|
|
* [gitlab](https://gitlab.torproject.org/tpo/tpa/gitlab): [issues](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues), [new issue](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/new)
|
|
|
|
|
|
|
|
If you do not have a GitLab account or can't figure it out for any
|
|
|
|
reason, you can also use the mailing lists. The
|
|
|
|
<tor-dev@lists.torproject.org> mailing list is the best for now.
|
|
|
|
|
|
|
|
## How to report an issue in the bugtracker itself?
|
|
|
|
|
|
|
|
If you have access to GitLab, you can [file a new issue][File] after
|
|
|
|
you have [searched the GitLab project for similar bugs][search].
|
|
|
|
|
|
|
|
If you do *not* have access to GitLab, you can email
|
|
|
|
<gitlab-admin@torproject.org>.
|
|
|
|
|
|
|
|
## How to contribute code?
|
|
|
|
|
|
|
|
As reporting an issue, you first need to figure out which project you
|
|
|
|
are working on in the GitLab [project list][]. Then, if you are not
|
|
|
|
familiar with merge requests, you should read the [merge requests
|
|
|
|
introduction](https://gitlab.torproject.org/help/user/project/merge_requests/getting_started.md) in the GitLab documentation. If you are unfamiliar
|
|
|
|
with merge requests but familiar with GitHub's pull requests, those
|
|
|
|
are similar.
|
|
|
|
|
|
|
|
Note that we do not necessarily use merge requests in all teams yet,
|
|
|
|
and Gitolite still has the canonical version of the code. See [issue
|
|
|
|
36][] for a followup on this.
|
|
|
|
|
|
|
|
[issue 36]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/36
|
|
|
|
|
|
|
|
Also note that different teams might have different workflows. If a
|
|
|
|
team has a special workflow that diverges from the one here, it should
|
|
|
|
be documented here. Those are the workflows we know about:
|
|
|
|
|
|
|
|
* [Network Team](https://gitlab.torproject.org/tpo/core/tor/-/wikis/NetworkTeam/GitlabReviews)
|
|
|
|
|
|
|
|
If you do not have access to GitLab, please use one of the mailing
|
|
|
|
lists: <tor-dev@lists.torproject.org> would be best.
|
|
|
|
|
|
|
|
## How to quote a comment in a reply?
|
|
|
|
|
|
|
|
The "Reply" button only creates a new comment without any quoted text
|
|
|
|
by default. It seems the solution to that is currently highlighting
|
|
|
|
the text to quote and then pressing the `r`-key. See also the [other
|
|
|
|
keyboard shortcuts](https://docs.gitlab.com/ee/user/shortcuts.html).
|
|
|
|
|
|
|
|
Alternatively, you can copy-paste the text in question in the comment
|
|
|
|
form, select the pasted text, and hit the `Insert a quote` button
|
|
|
|
which look like a styled, curly, and closing quotation mark `”`.
|
|
|
|
|
|
|
|
# How-to
|
|
|
|
|
|
|
|
## How to find the right emoji?
|
|
|
|
|
|
|
|
It's possible to add "reaction emojis" to comments and issues and
|
|
|
|
merge requests in GitLab. Just hit the little smiley face and a dialog
|
|
|
|
will pop up. You can then browse through the list and pick the right
|
|
|
|
emoji for how you feel about the comment, but remember to be nice!
|
|
|
|
|
|
|
|
It's possible you get lost in the list. You can type the name of the
|
|
|
|
emoji to restrict your search, but be warned that some emojis have
|
|
|
|
[particular, non-standard names](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29057) that might not be immediately
|
|
|
|
obvious. For example, `🎉`, `U+1F389 PARTY POPPER`, is found as
|
|
|
|
`tada` in the list! See [this upstream issue for more details](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29057).
|
|
|
|
|
|
|
|
## Hooking up a project with the bots
|
|
|
|
|
|
|
|
By default, new projects do *not* have notifications setup in
|
|
|
|
`#tor-bots` like all the others. To do this, you need to configure a
|
|
|
|
"Webhook", in the `Settings -> Webhooks` section of the project. The
|
|
|
|
URL should be:
|
|
|
|
|
|
|
|
https://kgb-bot.torproject.org/webhook/
|
|
|
|
|
|
|
|
... and you should select the notifications you wish to see in
|
|
|
|
`#tor-bots`. You can also enable notifications to other channels by
|
|
|
|
adding more parameters to the URL, like (say)
|
|
|
|
`?channel=%23tor-foo`. The parameters are documented the [KGB
|
|
|
|
documentation](https://salsa.debian.org/kgb-team/kgb/-/wikis/usage).
|
|
|
|
|
|
|
|
Note that GitLab admins might be able to configure [system-wide
|
|
|
|
hooks](https://gitlab.torproject.org/help/system_hooks/system_hooks) in [the admin section](https://gitlab.torproject.org/admin/hooks), although it's not entirely clear
|
|
|
|
how does relate to the per-project hooks so those have not been
|
|
|
|
enabled. Furthermore, it is possible for GitLab admins with root
|
|
|
|
access to enable webhooks on *all* projects, with the [webhook rake
|
|
|
|
task](https://docs.gitlab.com/ee/raketasks/web_hooks.html#webhooks). For example, running this on the GitLab server (currently
|
|
|
|
`gitlab-02`) will enable the above hook on all repositories:
|
|
|
|
|
|
|
|
sudo gitlab-rake gitlab:web_hook:add URL='https://kgb-bot.torproject.org/webhook/'
|
|
|
|
|
|
|
|
Note that by default, the rake task only enables `Push` events. You
|
|
|
|
need the following patch to enable others:
|
|
|
|
|
|
|
|
modified lib/tasks/gitlab/web_hook.rake
|
|
|
|
@@ -10,7 +10,19 @@ namespace :gitlab do
|
|
|
|
puts "Adding webhook '#{web_hook_url}' to:"
|
|
|
|
projects.find_each(batch_size: 1000) do |project|
|
|
|
|
print "- #{project.name} ... "
|
|
|
|
- web_hook = project.hooks.new(url: web_hook_url)
|
|
|
|
+ web_hook = project.hooks.new(
|
|
|
|
+ url: web_hook_url,
|
|
|
|
+ push_events: true,
|
|
|
|
+ issues_events: true,
|
|
|
|
+ confidential_issues_events: false,
|
|
|
|
+ merge_requests_events: true,
|
|
|
|
+ tag_push_events: true,
|
|
|
|
+ note_events: true,
|
|
|
|
+ confidential_note_events: false,
|
|
|
|
+ job_events: true,
|
|
|
|
+ pipeline_events: true,
|
|
|
|
+ wiki_page_events: true,
|
|
|
|
+ )
|
|
|
|
if web_hook.save
|
|
|
|
puts "added".color(:green)
|
|
|
|
else
|
|
|
|
|
|
|
|
See also the [upstream issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17966) and [our GitLab issue 7](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/7) for
|
|
|
|
details.
|
|
|
|
|
|
|
|
You can also remove a given hook from all repos with:
|
|
|
|
|
|
|
|
sudo gitlab-rake gitlab:web_hook:rm URL='https://kgb-bot.torproject.org/webhook/'
|
|
|
|
|
|
|
|
And, finally, list all hooks with:
|
|
|
|
|
|
|
|
sudo gitlab-rake gitlab:web_hook:list
|
|
|
|
|
|
|
|
## Setting up two-factor authentication (2FA)
|
|
|
|
|
|
|
|
We strongly recommend you enable two-factor authentication on
|
|
|
|
GitLab. This is [well documented in the GitLab manual](https://gitlab.torproject.org/help/user/profile/account/two_factor_authentication.md#two-factor-authentication), but basically:
|
|
|
|
|
|
|
|
1. first, pick a 2FA "app" (and optionally a hardware token) if you
|
|
|
|
don't have one already
|
|
|
|
|
|
|
|
2. head to your [account settings](https://gitlab.torproject.org/profile/account)
|
|
|
|
|
|
|
|
3. register your 2FA app and save the recovery codes somewhere. if
|
|
|
|
you need to enter a URL by hand, you can scan the qrcode with your
|
|
|
|
phone or create one by following this format:
|
|
|
|
|
|
|
|
otpauth://totp/$ACCOUNT?secret=$KEY&issuer=gitlab.torproject.org
|
|
|
|
|
|
|
|
where...
|
|
|
|
|
|
|
|
* `$ACCOUNT` is the `Account` field in the 2FA form
|
|
|
|
* `$KEY` is the `Key` field in the 2FA form, without spaces
|
|
|
|
|
|
|
|
4. register the 2FA hardware token if available
|
|
|
|
|
|
|
|
GitLab requires a 2FA "app" even if you intend to use a hardware
|
|
|
|
token. The 2FA "app" must implement the TOTP protocol, for example the
|
|
|
|
[Google Authenticator](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2) or a free alternative (for example [free OTP
|
|
|
|
plus](https://github.com/helloworld1/FreeOTPPlus/), see also this [list from the Nextcloud project](https://github.com/nextcloud/twofactor_totp#readme)). The
|
|
|
|
hardware token must implement the U2F protocol, which is supported by
|
|
|
|
security tokens like the [YubiKey](https://en.wikipedia.org/wiki/YubiKey), [Nitrokey](https://www.nitrokey.com/), or similar.
|
|
|
|
|
|
|
|
## Pager playbook
|
|
|
|
|
|
|
|
<!-- information about common errors from the monitoring system and -->
|
|
|
|
<!-- how to deal with them. this should be easy to follow: think of -->
|
|
|
|
<!-- your future self, in a stressful situation, tired and hungry. -->
|
|
|
|
|
|
|
|
* Grafana Dashboards:
|
|
|
|
* [GitLab overview](https://grafana.torproject.org/d/QrDJktiMz/gitlab-omnibus)
|
|
|
|
* [Gitaly](https://grafana.torproject.org/d/x6Z50y-iz/gitlab-gitaly)
|
|
|
|
|
|
|
|
## Disaster recovery
|
|
|
|
|
|
|
|
In case the entire GitLab machine is destroyed, a new server should be
|
|
|
|
provisionned in the [howto/ganeti](howto/ganeti) cluster (or elsewhere) and backups
|
|
|
|
should be restored using the below procedure.
|
|
|
|
|
|
|
|
### Running an emergency backup
|
|
|
|
|
|
|
|
TBD
|
|
|
|
|
|
|
|
### baremetal recovery
|
|
|
|
|
|
|
|
TBD
|
|
|
|
|
|
|
|
# Reference
|
|
|
|
|
|
|
|
## Installation
|
|
|
|
|
|
|
|
The current GitLab server was setup in the [howto/ganeti](howto/ganeti) cluster in a
|
|
|
|
regular virtual machine. It was configured with [howto/puppet](howto/puppet) with the
|
|
|
|
`roles::gitlab`. That, in turn, relies on a series of `profile`
|
|
|
|
elements which configure:
|
|
|
|
|
|
|
|
* `profile::gitlab::web`: nginx vhost and TLS cert, depends on
|
|
|
|
`profile::nginx` built for the [howto/cache](howto/cache) service and relying on the
|
|
|
|
[puppet/nginx](https://forge.puppet.com/puppet/nginx) module from the Forge
|
|
|
|
* `profile::gitlab::mail`: dovecot and postfix configuration, for
|
|
|
|
email replies
|
|
|
|
* `profile::gitlab::database`: postgresql configuration, possibly not
|
|
|
|
used by the Omnibus package, see [issue 20](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20)
|
|
|
|
* `profile::gitlab::app`: the core of the configuration of gitlab
|
|
|
|
itself, uses the [puppet/gitlab](https://forge.puppet.com/puppet/gitlab) module from the Forge, with
|
|
|
|
Prometheus, Grafana, and Nginx support disabled, but Redis,
|
|
|
|
PostgreSQL, and Prometheus exporters enabled
|
|
|
|
|
|
|
|
This installs the [GitLab Omnibus](https://docs.gitlab.com/omnibus/) distribution which duplicates a
|
|
|
|
lot of resources we would otherwise manage elsewhere in Puppet,
|
|
|
|
including (but possibly not limited to):
|
|
|
|
|
|
|
|
* [howto/prometheus](howto/prometheus) exorters
|
|
|
|
* [howto/postgresql](howto/postgresql)
|
|
|
|
* redis
|
|
|
|
|
|
|
|
This therefore leads to a "particular" situation regarding monitoring
|
|
|
|
and PostgreSQL backups, in particular. See [issue 20](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20) for details.
|
|
|
|
|
|
|
|
## SLA
|
|
|
|
<!-- this describes an acceptable level of service for this service -->
|
|
|
|
|
|
|
|
## Migration
|
|
|
|
|
|
|
|
GitLab was migrated from Trac in June 2020, after a few months of
|
|
|
|
testing. Tests were done first on a server called
|
|
|
|
`dip.torproject.org`, a reference to `salsa.debian.org`, the GitLab
|
|
|
|
server ran by the Debian project. We identified [some problems with
|
|
|
|
merge requests](https://gitlab.torproject.org/tpo/tpa/services/-/issues/32197) during the test so the server was reinstalled with
|
|
|
|
the "GitLab Omnibus" package on the current server, `gitlab-02` which
|
|
|
|
will enter production in the week of June 15th 2020.
|
|
|
|
|
|
|
|
### Why migrate?
|
|
|
|
|
|
|
|
We're hoping gitlab will be a good fit because:
|
|
|
|
|
|
|
|
* Gitlab will allow us to collect our different engineering tools
|
|
|
|
into a single application: Git repository handling, Wiki, Issue
|
|
|
|
tracking, Code reviews, and project management tooling.
|
|
|
|
* Gitlab is well-maintained, while Trac plugins are not well
|
|
|
|
maintained and Trac itself hasn't seen a release for over a year
|
|
|
|
(since 2019)
|
|
|
|
* Gitlab will allow us to build a more modern approach to handling CI
|
|
|
|
for our different projects. This is going to happen after the
|
|
|
|
ticket and wiki migration.
|
|
|
|
|
|
|
|
(Note that we're only planning to install and use the freely licensed version
|
|
|
|
of gitlab. There is an "enterprise" version with additional features, but we
|
|
|
|
prefer to use free software whenever possible.)
|
|
|
|
|
|
|
|
### Migrated content
|
|
|
|
|
|
|
|
The issues and wiki of the "Tor" project are migrated. There are no
|
|
|
|
other projects in Trac.
|
|
|
|
|
|
|
|
* Trac wiki: <https://gitlab.torproject.org/legacy/trac>
|
|
|
|
* Trac issues: <https://gitlab.torproject.org/legacy/trac/-/issues>
|
|
|
|
|
|
|
|
Trac issues that remain are really legacy issues, others issues have
|
|
|
|
been "moved" to the respective projects. @ahf, who did the migration,
|
|
|
|
created a [copy of the mapping](component-tickets) for those
|
|
|
|
looking for their old stuff.
|
|
|
|
|
|
|
|
All the tickets that were *not* moved to their respective projects
|
|
|
|
have been closed in the first week of july.
|
|
|
|
|
|
|
|
### Not migrated
|
|
|
|
|
|
|
|
We are *not* migrating away from Gitolite and Jenkins just yet. This
|
|
|
|
means those services are still fully operational and their equivalent
|
|
|
|
features in GitLab are *not* supported (namely Git hosting and
|
|
|
|
CI). Those services *might* eventually be migrated to GitLab, but
|
|
|
|
that's not part of the current migration plan. See [issue 36][] for
|
|
|
|
the followup on that.
|
|
|
|
|
|
|
|
Again, the canonical copy for source code hosted by git is:
|
|
|
|
|
|
|
|
* `git-rw.torproject` - writable git repositories over SSH
|
|
|
|
* <https://git.torproject.org/> - readonly clones
|
|
|
|
* <https://gitweb.torproject.org/> - web interface
|
|
|
|
|
|
|
|
We also do not host "GitLab pages", the static site hosting provided
|
|
|
|
by GitLab.
|
|
|
|
|
|
|
|
The priority of those features would be:
|
|
|
|
|
|
|
|
1. gitolite replacement and migration
|
|
|
|
2. CI deployment, with people migrating their own job from Jenkinks
|
|
|
|
and TPA shutting down Jenkins on a flag date
|
|
|
|
3. GitLab pages replacement and migration from the current static
|
|
|
|
site hosting system
|
|
|
|
|
|
|
|
Those are each large projects and will be undertaken at a later stage,
|
|
|
|
progressively.
|
|
|
|
|
|
|
|
### Feature equivalence
|
|
|
|
|
|
|
|
| Feature | Trac | GitLab | Comments |
|
|
|
|
| ------- | ---- | ------ | -------- |
|
|
|
|
| Ticket relations | parent/child | checklists | checklists show up as "X of Y tasks completed"¹ |
|
|
|
|
| Milestones | yes | yes | |
|
|
|
|
| Estimates | points/actual | estimation/spending | requires conversion from days to hours |
|
|
|
|
| Private issues | no | yes | |
|
|
|
|
| Issue subscription | RSS, email, ML | email | Trac sends email to trac-bugs |
|
|
|
|
| User projects | no | yes | if users can create projects |
|
|
|
|
| User registration | optional | disabled | ² |
|
|
|
|
| Search | advanced | basic | no support for custom queries in GitLab³ |
|
|
|
|
| Markup | WikiCreole | Markdown, GitHub-like | ⁴ |
|
|
|
|
| IRC bot | yes | yes | zwiebelbot has to be patched, other bots to be deployed for notifications⁵ |
|
|
|
|
| Git hosting | no, gitolite | yes, builtin | concerns about trusting GitLab with our code |
|
|
|
|
| CI | no, Jenkins | yes, builtin | maybe in the future |
|
|
|
|
| Upstream maintenance | slow | fast | Trac does not seem well maintained |
|
|
|
|
| Wikis | one big wiki | per-project | ⁶ |
|
|
|
|
| API | XML-RPC | REST, multiple clients | |
|
|
|
|
| Javascript | optional | required | Drag-and-drop boards seem not to work but the list of issues still can be used. |
|
|
|
|
|
|
|
|
Notes:
|
|
|
|
|
|
|
|
1. Trac parent/child issue relationships have been converted into a
|
|
|
|
simple comment at the beginnning of the ticket linking to the
|
|
|
|
child/parent tickets. It was originally hoped to use the
|
|
|
|
"checklists" features but this was not implemented for lack of time.
|
|
|
|
|
|
|
|
2. User registration is perfectly possible in GitLab but since GitLab
|
|
|
|
instances are frequently attacked by spammers, it is disabled
|
|
|
|
until we find an alternative. See missing features below for
|
|
|
|
details).
|
|
|
|
|
|
|
|
3. GitLab, in particular, does not support inline searches, see
|
|
|
|
Missing features below for details.
|
|
|
|
|
|
|
|
4. The wiki and issue formatting markup is different. Whereas Trac
|
|
|
|
uses [wiki formatting](https://trac.edgewall.org/wiki/WikiFormatting) inspired by old wikis like
|
|
|
|
[MoinMoin](https://moinmo.in/), a subset of the somewhat standard [Wikicreole](http://www.wikicreole.org/)
|
|
|
|
markup, GitLab uses [Markdown](https://en.wikipedia.org/wiki/Markdown), specifically their own [GitLab
|
|
|
|
version of markdown](https://gitlab.com/help/user/markdown) inspired by GitHub's markdown
|
|
|
|
extensions. The wiki and issues were automatically converted to
|
|
|
|
Markdown, but when you file new issues, you will need to use
|
|
|
|
Markdown, not Creole.
|
|
|
|
|
|
|
|
5. specifically, zwiebelbot now knows about `foo#N` pointing to issue
|
|
|
|
N in project `foo` in GitLab. We need to update (or replace) the
|
|
|
|
`nsa` bot in `#tor-bots` to broadcast announcements to
|
|
|
|
projects. This could be done with the [KGB](https://salsa.debian.org/kgb-team/kgb/-/wikis/home) bot for which we
|
|
|
|
now have a [Puppet module](https://gitlab.com/shared-puppet-modules-group/kgb) so it could easily be deployed here
|
|
|
|
|
|
|
|
6. because Trac does not allow users to create projects, we have
|
|
|
|
historically used one gigantic project for everything, which means
|
|
|
|
we had only one wiki. technically, Trac also supports one wiki per
|
|
|
|
project, but because project creation requires an admin
|
|
|
|
intervention, this never concretized.
|
|
|
|
|
|
|
|
### Ticket fields equivalence
|
|
|
|
|
|
|
|
| Trac | GitLab | Comments |
|
|
|
|
| ---- | ------ | -------- |
|
|
|
|
| id | id | keep the ticket id in legacy project, starts at 40000 in GitLab |
|
|
|
|
| Summary | ? | unused? |
|
|
|
|
| Reporter | Reporter | |
|
|
|
|
| Description | Body | |
|
|
|
|
| Type | Label | use templates to make sure those are filled |
|
|
|
|
| Milestone | Milestone, Label | |
|
|
|
|
| Version | Label | |
|
|
|
|
| Keywords | Label | |
|
|
|
|
| Points, in days | /estimate, in hours | requires conversion |
|
|
|
|
| Actual points | /spending | |
|
|
|
|
| Sponsor | Label | |
|
|
|
|
| Priority | Board, Label | boards can sort issues instead of assigning arbitrary keywords |
|
|
|
|
| Component | Subproject, Label | |
|
|
|
|
| Severity | Label | mark only blocker issues to resolve |
|
|
|
|
| Cc | @people | paid plans also have multiple assignees |
|
|
|
|
| Parent issue | #reference | issue mentions and checklists |
|
|
|
|
| Reviewer | Label | |
|
|
|
|
| Attachements | Attachements, per comment | |
|
|
|
|
| Status | Label | Kanban boards panels |
|
|
|
|
|
|
|
|
Notice how the `Label` field is used as a fallback when no equivalent
|
|
|
|
field exists.
|
|
|
|
|
|
|
|
### Missing features
|
|
|
|
|
|
|
|
GitLab does not provide one-to-one feature parity with Trac, but it
|
|
|
|
comes pretty close. It has issue tracking, wikis, milestones,
|
|
|
|
keywords, time estimates, and much more.
|
|
|
|
|
|
|
|
But one feature it is missing is the **advanced ticket query**
|
|
|
|
features of Trac. It's not possible to create "reports" in GitLab to
|
|
|
|
have pre-cooked issue listings. And it's especially not possible to
|
|
|
|
embed special searches in wiki pages the same way it is done in Trac.
|
|
|
|
|
|
|
|
We suggest people use the "dashboard" feature of GitLab instead. This
|
|
|
|
featuers follows the [Kanban][] development strategy which is
|
|
|
|
implemented in GitLab as [issue boards](https://docs.gitlab.com/ee/user/project/issue_board.html). It is also, of course,
|
|
|
|
possible to link so specific searches from the wiki, but not embed
|
|
|
|
those tickets in the output.
|
|
|
|
|
|
|
|
[Kanban]: https://en.wikipedia.org/wiki/Kanban_(development)
|
|
|
|
|
|
|
|
We do *not* have a **anonymous account** (AKA `cypherpunks`) for
|
|
|
|
now. GitLab will be in **closed registration** for now, with users
|
|
|
|
needing to request approval on a per-person basis for now. Eventually,
|
|
|
|
we're going to consider other options to work around this (human)
|
|
|
|
bottleneck.
|
|
|
|
|
|
|
|
### Interesting new features
|
|
|
|
|
|
|
|
1. Using pull requests to your project repositories, and assigning
|
|
|
|
reviewers on pull requests, rather than using `reviewer` and
|
|
|
|
`needs_review` labels on issues. Issues can refer to pull requests
|
|
|
|
and vice versa.
|
|
|
|
|
|
|
|
2. Your team can work on using Gitlab boards for handling the
|
|
|
|
different stages of issue handling. All the way from selection to
|
|
|
|
finalization with code in a PR. You can have as many boards as you
|
|
|
|
like: per subproject, per sponsor, per week, all of this is
|
|
|
|
something we can experiment with.
|
|
|
|
|
|
|
|
3. You can now use time estimation in Gitlab simply by adding a specially
|
|
|
|
formatted comment in your issues/pull requests instead of using `points` and
|
|
|
|
`actual_points`. See the [time tracking documentation](https://docs.gitlab.com/ee/user/project/time_tracking.html) for details
|
|
|
|
|
|
|
|
4. Familiarize yourself with new interfaces such as the ["to do"
|
|
|
|
dashboard](https://docs.gitlab.com/ee/user/todos.html) where you can see what needs your input since last
|
|
|
|
visit
|
|
|
|
|
|
|
|
5. Create email filters for tickets: Gitlab adds a lot more [email
|
|
|
|
headers to each notification](https://docs.gitlab.com/ee/user/profile/notifications.html#filtering-email) you receive (if you want it via
|
|
|
|
email), which for example allows you split notifications in your
|
|
|
|
mail program into different directories.
|
|
|
|
|
|
|
|
Bonus info: You will be able to reply via email to the notifications you
|
|
|
|
receive from Gitlab, and Gitlab will put your responses into the system as
|
|
|
|
notes on issues :-)
|
|
|
|
|
|
|
|
### bugs.torproject.org redirections
|
|
|
|
|
|
|
|
The <https://bugs.torproject.org> redirection now points at
|
|
|
|
GitLab. The following rules apply:
|
|
|
|
|
|
|
|
1. **legacy tickets**: `bugs.torproject.org/N` redirects to
|
|
|
|
`gitlab.torproject.org/legacy/trac/-/issues/N`
|
|
|
|
2. **new issues**: `bugs.tpo/PROJECT/N` redirects to
|
|
|
|
`gitlab.tpo/PROJECT/-/issues/N`
|
|
|
|
3. **merge requests**: `bugs.tpo/PROJECT!N` redirects to
|
|
|
|
`gitlab.tpo/PROJECT/-/merge_requests/N`
|
|
|
|
4. **catch all**: `bugs.tpo/FOO` redirects to `gitlab.tpo/FOO`
|
|
|
|
5. **ticket list**: a bare `bugs.tpo` redirects to
|
|
|
|
`https://gitlab.torproject.org/tpo/-/issues`
|
|
|
|
|
|
|
|
It used to be that `bugs.tpo/N` would redirect to issue `N` the Trac
|
|
|
|
"tor" project. But unfortunately, there's no global "number space" for
|
|
|
|
issues in GitLab (or at least not a user-visible one), so `N` is not
|
|
|
|
distinct across projects. We therefore need the prefix to
|
|
|
|
disambiguate.
|
|
|
|
|
|
|
|
We considered enforcing the `tpo` prefix there to shorten links, but
|
|
|
|
we decided against it because it would forbid pointers to
|
|
|
|
user-specific projects and would make it extremely hard to switch away
|
|
|
|
from the global `tpo` group if we ever decide to do that.
|
|
|
|
|
|
|
|
### Content organisation
|
|
|
|
|
|
|
|
|
|
|
|
Projects are all stored under the over-arching `tpo` group. This is
|
|
|
|
done this way to allow project managers to have an overview of all
|
|
|
|
projects going on at TPO. It also allows us to host other
|
|
|
|
organisations on our GitLab in a different namespace.
|
|
|
|
|
|
|
|
Under the `tpo` group, each team has its own subgroup and they have
|
|
|
|
autonomy under that group to manage accesses and projects.
|
|
|
|
|
|
|
|
### Permissions
|
|
|
|
|
|
|
|
Given the above Team/Group organization, users will be members in
|
|
|
|
gitlab for the groups/teams they belong to.
|
|
|
|
|
|
|
|
Any projects that need to be shared between multiple groups should be
|
|
|
|
shared using the “Share Project” functionality.
|
|
|
|
|
|
|
|
There should be a limited number of members in the Tor Project group,
|
|
|
|
as these will have access to all subgroups and their
|
|
|
|
projects. Currently this is limited to Project Managers and Services
|
|
|
|
and Sysadmins.
|
|
|
|
|
|
|
|
A reminder of the GitLab [permission system](https://docs.gitlab.com/ee/user/permissions.html) and [types of users](https://about.gitlab.com/handbook/product/#permissions-in-gitlab):
|
|
|
|
|
|
|
|
* Guests: anybody that may need to report issues on a project and/or
|
|
|
|
make comments on an issue.
|
|
|
|
* Reporter: they can also manage labels
|
|
|
|
* Developer: they can create branches, manage merge requests, force
|
|
|
|
push to non-protected branches
|
|
|
|
* Maintainer: edit projects, manage runners, edit comments, delete
|
|
|
|
wiki pages.
|
|
|
|
* Owner: we are setting this role for every member in the TPO
|
|
|
|
team. They can also transfer projects to other name spaces, switch
|
|
|
|
visilbity level, delete issues.
|
|
|
|
|
|
|
|
### Labels
|
|
|
|
|
|
|
|
At group level we have sponsor labels and state labels. The ones that
|
|
|
|
are used by the whole organization are in the `tpo` group. Each team
|
|
|
|
can decide which other labels they add for their projects.
|
|
|
|
|
|
|
|
* Kanban columns
|
|
|
|
* Icebox
|
|
|
|
* Backlog
|
|
|
|
* Next
|
|
|
|
* Doing
|
|
|
|
* Needs Review
|
|
|
|
* Types of Issue
|
|
|
|
* Defect
|
|
|
|
* Enhancement
|
|
|
|
* Task
|
|
|
|
* Related to a project
|
|
|
|
* Scalability
|
|
|
|
* UX
|
|
|
|
* Sponsors
|
|
|
|
* Sponsor X
|
|
|
|
* Keywords
|
|
|
|
* Other possible keywords needed at group level.
|
|
|
|
|
|
|
|
Note that those labels are being worked on [ticket 4](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/4). We also have
|
|
|
|
a lot more label than we would like ([ticket 3](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/3)) which makes GitLab
|
|
|
|
hard to use. Because there are thousands of labels in some projects,
|
|
|
|
loading the label list can take a second or more on slower links, and
|
|
|
|
it's really hard to find the label you're looking for, which affects
|
|
|
|
usability -- and especially discoverability -- quite a bit.
|
|
|
|
|
|
|
|
There is work underway in the [label cleanup](https://gitlab.torproject.org/ahf/label-cleanup/-/tree/main/) repository to
|
|
|
|
rewrite and delete labels in one batch for all the projects.
|
|
|
|
|
|
|
|
### Remaining work
|
|
|
|
|
|
|
|
As of June 17th:
|
|
|
|
|
|
|
|
4. Begin handing out some of the accounts to community members that
|
|
|
|
we have created in their name(s).
|
|
|
|
|
|
|
|
5. Begin figuring out how to solve the account creation and anonymous
|
|
|
|
users in the longer run.
|
|
|
|
|
|
|
|
See also the [list of known issues][search].
|
|
|
|
|
|
|
|
### What will break, and when will you fix it?
|
|
|
|
|
|
|
|
Most notably, we're going to have an interruption in the ability to open new
|
|
|
|
accounts and new tickets. We _did not_ want to migrate without a solution
|
|
|
|
here; we'll try to have at least a stop-gap solution in place soon, and
|
|
|
|
something better in the future. For now, we're planning for people that want
|
|
|
|
to get a new account please send a mail to <gitlab-admin@torproject.org>. We
|
|
|
|
hope to have something else in place once the migration is succesful.
|
|
|
|
|
|
|
|
We're not going to migrate long-unused accounts.
|
|
|
|
|
|
|
|
Some wiki pages that contained automated listings of tickets will stop
|
|
|
|
containing those lists: that's a trac feature that gitlab doesn't have. We'll
|
|
|
|
have to adjust our workflows to work around this. In some cases, we can use
|
|
|
|
gitlab milestone pages or projects that do not need a wiki page as a work
|
|
|
|
around.
|
|
|
|
|
|
|
|
### Trac Archival
|
|
|
|
|
|
|
|
A copy of all Trac web pages were stored in the [Internet
|
|
|
|
Archive](http://archive.org/)'s [Wayback machine](http://web.archive.org/), thanks to [ArchiveBot](https://www.archiveteam.org/index.php?title=ArchiveBot), a tool
|
|
|
|
developed by [ArchiveTeam](https://www.archiveteam.org/), of which anarcat is somewhat a part of.
|
|
|
|
|
|
|
|
First, a list of tickets was created:
|
|
|
|
|
|
|
|
seq 1 40000 | sed 's#^#https://trac.torproject.org/projects/tor/ticket/#'
|
|
|
|
|
|
|
|
This was uploaded to anarcat's pastebin (using [pubpaste](https://gitlab.com/anarcat/pubpaste)) and fed
|
|
|
|
into archivebot with:
|
|
|
|
|
|
|
|
!ao < https://paste.anarc.at/publish/2020-06-17/trac.torproject.org-tickets-1-40000-final.txt
|
|
|
|
!ao https://paste.anarc.at/publish/2020-06-17/trac.torproject.org-tickets-1-40000-final.txt
|
|
|
|
|
|
|
|
This tells ArchiveBot to crawl each ticket individually, and then
|
|
|
|
archive the list itself as well.
|
|
|
|
|
|
|
|
Simultaneously, a full crawl of the entire site (and first level
|
|
|
|
outgoing links) was started, with:
|
|
|
|
|
|
|
|
!a https://trac.torproject.org --explain "migrated to gitlab, readonly" --delay 500
|
|
|
|
|
|
|
|
A list of excludes was added to ignore traps and infinite loops:
|
|
|
|
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://trac\.torproject\.org/projects/tor/query.*[?&]order=(?!priority)
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://trac\.torproject\.org/projects/tor/query.*[&?]desc=1
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://gitweb\.torproject\.org/
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://trac\.torproject\.org/projects/tor/timeline\?
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://trac\.torproject\.org/projects/tor/query\?status=!closed&keywords=
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://trac\.torproject\.org/projects/tor/query\?status=!closed&(version|reporter|owner|cc)=
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://trac\.torproject\.org/projects/tor/query\?(.*&)?(reporter|priority|component|severity|cc|owner|version)=
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://cdn\.media\.ccc\.de/
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://www\.redditstatic\.com/desktop2x/
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://trac\.torproject\.org/projects/tor/report/\d+.*[?&]sort=
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://support\.stripe\.com/
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://cdn\.cms-twdigitalassets\.com/
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://cypherpunks\:writecode@trac\.torproject\.org/
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://login\.blockchain\.com/
|
|
|
|
!ig bpu6j3ucrv87g4aix1zdrhb6k ^https?://dnsprivacy\.org/
|
|
|
|
|
|
|
|
The crawl was slowed down with a 500-1000ms delay to avoid hammering the server:
|
|
|
|
|
|
|
|
!d bpu6j3ucrv87g4aix1zdrhb6k 500 1000
|
|
|
|
|
|
|
|
The results will be accessible in the wayback machine a few days after
|
|
|
|
the crawl. Another crawl was performed back in 2019, so the known full
|
|
|
|
archives of Trac are as follows:
|
|
|
|
|
|
|
|
* [june 2019 ticket crawl](https://archive.fart.website/archivebot/viewer/job/5vytc): 6h30, 29892 files, 1.9 GiB
|
|
|
|
* [june 2020 ticket crawl](https://archive.fart.website/archivebot/viewer/job/c4xu3): 4h30, 33582 files, 1.9GiB
|
|
|
|
* [june 2019 and 2020 full crawls](https://archive.fart.website/archivebot/viewer/job/bpu6j): 5 days, 7h30, 732488 files,
|
|
|
|
105.4 GiB; TBD for the 2020 crawl
|
|
|
|
|
|
|
|
This information can be extracted back again from the `*-meta.warc.gz`
|
|
|
|
(text) files in the above URLs. This was done as part of [ticket
|
|
|
|
40003](https://gitlab.torproject.org/tpo/tpa/services/-/issues/40003). There has also been other, independent, crawls of Trac,
|
|
|
|
which are partly visible [in the viewer](https://archive.fart.website/archivebot/viewer/?q=trac.torproject.org).
|
|
|
|
|
|
|
|
### History
|
|
|
|
|
|
|
|
* lost in the mists of time: migration from Bugzilla to Flyspray (40
|
|
|
|
tickets)
|
|
|
|
* 2010-04-23: [migration from Flyspray to Trac completed](https://lists.torproject.org/pipermail/tor-dev/2010-April/000183.html) (last
|
|
|
|
Flyspray ticket is [1393](https://bugs.torproject.org/1393), first Trac ticket is [2000](https://bugs.torproject.org/2000))
|
|
|
|
* 2016-11-29: [first request to setup a GitLab server](https://gitlab.torproject.org/tpo/tpa/services/-/issues/20821)
|
|
|
|
* ~2017: oniongit.eu (warning: squatted domain) deployed to test
|
|
|
|
GitLab with the network team, considered as gitlab.torproject.net
|
|
|
|
but ultimately [abandoned](https://gitlab.torproject.org/tpo/tpa/services/-/issues/21840)
|
|
|
|
* 2019-02-28: `gitlab-01` AKA dip.torproject.org test server setup
|
|
|
|
([issue 29400](https://gitlab.torproject.org/tpo/tpa/services/-/issues/29400)), following the [Brussels meeting](https://trac.torproject.org/projects/tor/wiki/org/meetings/2019BrusselsAdminTeamMinutes)
|
|
|
|
* 2019-07-17: GitLab discussed again at the [Stockholm meeting](https://gitlab.torproject.org/legacy/trac/-/wikis/org/meetings/2019Stockholm/Notes/InternalTooling)
|
|
|
|
* 2019-07-29: Formal proposal to deploy GitLab [sent to
|
|
|
|
tor-project](https://lists.torproject.org/pipermail/tor-project/2019-July/002407.html), no objection
|
|
|
|
* 2020-03-05: GitLab migrated from `gitlab-01` (AKA "dip") to
|
|
|
|
`gitlab-02` using the Omnibus package
|
|
|
|
* 2020-04-27: `gitlab-01` retired
|
|
|
|
* 2020-06-13 19:00UTC: [Trac readonly](https://lists.torproject.org/pipermail/tor-project/2020-June/002872.html)
|
|
|
|
* 2020-06-13 02:25UTC: Trac tickets migrated (32401 tickets, last
|
|
|
|
ticket id is [34451](https://bugs.torproject.org/34451), first GitLab legacy project ticket id is
|
|
|
|
40000)
|
|
|
|
* 2020-06-14 21:22UTC: Trac wiki migrated
|
|
|
|
* 2020-06-15 18:30UTC: bugs.torproject.org redirects to gitlab
|
|
|
|
* 2020-06-16 02:15UTC: GitLab launch announced to tor-internal
|
|
|
|
* 2020-06-17 12:33UTC: Archivebot starts crawling all tickets of, and
|
|
|
|
the entire Trac website
|
|
|
|
* 2020-06-23: Archivebot completes the full Trac crawl, Trac is fully
|
|
|
|
archived on the Internet Archive
|
|
|
|
|
|
|
|
## Design
|
|
|
|
<!-- how this is built -->
|
|
|
|
<!-- should reuse and expand on the "proposed solution", it's a -->
|
|
|
|
<!-- "as-built" documented, whereas the "Proposed solution" is an -->
|
|
|
|
<!-- "architectural" document, which the final result might differ -->
|
|
|
|
<!-- from, sometimes significantly -->
|
|
|
|
|
|
|
|
<!-- a good guide to "audit" an existing project's design: -->
|
|
|
|
<!-- https://bluesock.org/~willkg/blog/dev/auditing_projects.html -->
|
|
|
|
|
|
|
|
GitLab is a fairly large program with multiple components. The
|
|
|
|
[upstream documentation](https://docs.gitlab.com/ee/development/architecture.html) has a good details of the architecture but
|
|
|
|
this section aims at providing a shorter summary. Here's an overview
|
|
|
|
diagram, first:
|
|
|
|
|
|
|
|
![GitLab's architecture diagram](https://docs.gitlab.com/ee/development/img/architecture_simplified.png)
|
|
|
|
|
|
|
|
The web frontend is Nginx (which we incidentally also use in our
|
|
|
|
[howto/cache](howto/cache) system) but GitLab wrote their own reverse proxy called
|
|
|
|
[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse/) which in turn talks to the underlying GitLab
|
|
|
|
Rails application, served by the [Unicorn](https://yhbt.net/unicorn/) application
|
|
|
|
server. The Rails app stores its data in a [howto/postgresql](howto/postgresql) database
|
|
|
|
(although not our own deployment, for now, [should be fixed](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20)). GitLab also offloads
|
|
|
|
long-term background tasks to a tool called [sidekiq](https://github.com/mperham/sidekiq).
|
|
|
|
|
|
|
|
Those all server HTTP(S) requests but GitLab is of course also
|
|
|
|
accessible over SSH to push/pull git repositories. This is handled by
|
|
|
|
a separate component called [gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell) which acts as a shell
|
|
|
|
for the `git` user.
|
|
|
|
|
|
|
|
Workhorse, Rails, sidekiq and gitlab-shell all talk with Redis to
|
|
|
|
store temporary information, caches and session information. They can
|
|
|
|
also communicate with the [Gitaly](https://gitlab.com/gitlab-org/gitaly) server which handles all
|
|
|
|
communication with the git repositories themselves.
|
|
|
|
|
|
|
|
Finally, Git)Lab also features GitLab Pages and Continuous Integration
|
|
|
|
("pages" and CI, neither of which we do not currently use). CI is
|
|
|
|
handled by [GitLab runners](https://gitlab.com/gitlab-org/gitlab-runner/) which can be deployed by anyone and
|
|
|
|
registered in the Rails app to pull CI jobs. [GitLab pages](https://gitlab.com/gitlab-org/gitlab-pages) is "a
|
|
|
|
simple HTTP server written in Go, made to serve GitLab Pages with
|
|
|
|
CNAMEs and SNI using HTTP/HTTP2".
|
|
|
|
|
|
|
|
## Issues
|
|
|
|
|
|
|
|
There is no issue tracker specifically for this project, [File][] or
|
|
|
|
[search][] for issues in the [gitlab project][search].
|
|
|
|
|
|
|
|
[File]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/new
|
|
|
|
[search]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues
|
|
|
|
|
|
|
|
## Monitoring and testing
|
|
|
|
|
|
|
|
Monitoring right now is minimal: normal host-level metrics like disk
|
|
|
|
space, CPU usage, web port and TLS certificates are monitored by
|
|
|
|
Nagios with our normal infrastructure, as a black box.
|
|
|
|
|
|
|
|
Prometheus monitoring is built into the GitLab Omnibus package, so it
|
|
|
|
is *not* configured through our Puppet like other Prometheus
|
|
|
|
servers. It has still been (manually) integrated in our Prometheus
|
|
|
|
setup and Grafana dashboards (see [pager playbook](#Pager_playbook)) have been deployed.
|
|
|
|
|
|
|
|
More work is underway to improve monitoring in [issue 33921](https://gitlab.torproject.org/tpo/tpa/services/-/issues/33921).
|
|
|
|
|
|
|
|
## Backups
|
|
|
|
|
|
|
|
There is a cronjob configured via the gitlab puppet monitor that runs
|
|
|
|
every night at 2:00AM UTC. This will basically run `$ gitlab-backup `
|
|
|
|
and will create a tarball under `/srv/gitlab-backup/`.
|
|
|
|
|
|
|
|
There is also a config backup job (in
|
|
|
|
`/etc/cron.d/gitlab-config-backup`) that makes sure to backup the
|
|
|
|
content of `/var/opt/gitlab/gitlab-rails/etc/` is backed up, because
|
|
|
|
that is not covered by the `gitlab-backup` command.
|
|
|
|
|
|
|
|
Another cron job purges backups older than two days, in
|
|
|
|
`/etc/cron.d/gitlab-rotate-backup`, so that we don't keep too many
|
|
|
|
copies on the server. It is assumed that the existing [howto/backup](howto/backup)
|
|
|
|
system will pick up those copies and store them for our normal
|
|
|
|
rotation periods.
|
|
|
|
|
|
|
|
# Discussion
|
|
|
|
|
|
|
|
## Overview
|
|
|
|
|
|
|
|
<!-- describe the overall project. should include a link to a ticket -->
|
|
|
|
<!-- that has a launch checklist -->
|
|
|
|
|
|
|
|
The GitLab project at Tor has been a long time coming. If you look at
|
|
|
|
the [history](#History) section above, you'll see it has been worked on since
|
|
|
|
at least 2016, at which point an external server was setup for the
|
|
|
|
"network team" to do code review. This server was ultimately retired.
|
|
|
|
|
|
|
|
The current server has been worked on since 2019, with the master
|
|
|
|
ticket, [issue 29400](https://gitlab.torproject.org/tpo/tpa/services/-/issues/29400), created in the footsteps of the [2019
|
|
|
|
Brussels meeting](https://trac.torproject.org/projects/tor/wiki/org/meetings/2019BrusselsAdminTeamMinutes). The service launched some time in June 2020,
|
|
|
|
with a full migration of Trac tickets.
|
|
|
|
|
|
|
|
## Goals
|
|
|
|
|
|
|
|
### Must have
|
|
|
|
|
|
|
|
* replacement of the Trac issue tracking server
|
|
|
|
* rough equivalent of Trac features in GitLab
|
|
|
|
|
|
|
|
### Nice to have
|
|
|
|
|
|
|
|
* identical representation of Trac issues in GitLab, including proper
|
|
|
|
issue numbering
|
|
|
|
|
|
|
|
### Non-Goals
|
|
|
|
|
|
|
|
* replacement of Gitolite (git hosting)
|
|
|
|
* replacement of Gitweb (git hosting)
|
|
|
|
* replacement of Jenkins (CI)
|
|
|
|
* replacement of the static site hosting system
|
|
|
|
|
|
|
|
Those are not part of the first phase of the project, but it is
|
|
|
|
understood that if one of those features gets used more heavily in
|
|
|
|
GitLab, the original service MUST be eventually migrated into GitLab
|
|
|
|
and turned off. We do *not* want to run multiple similar services at
|
|
|
|
the same time (for example run both gitolite and gitaly on all git
|
|
|
|
repositories, or run Jenkins and GitLab runners).
|
|
|
|
|
|
|
|
## Approvals required
|
|
|
|
|
|
|
|
The GitLab migration was approved at the 2019 Brussels dev meeting.
|
|
|
|
|
|
|
|
## Proposed Solution
|
|
|
|
|
|
|
|
The solution to the "code review" and "project management" problems
|
|
|
|
are to deploy a GitLab instance which does *not* aim at managing all
|
|
|
|
source code, in the first stage.
|
|
|
|
|
|
|
|
## Cost
|
|
|
|
|
|
|
|
Staff not evaluated.
|
|
|
|
|
|
|
|
In terms of hardware, we start with a single virtual machine and agree
|
|
|
|
that, in the worst case, we can throw a full Hetzner PX62-NVMe node at
|
|
|
|
the problem (~70EUR/mth).
|
|
|
|
|
|
|
|
## Alternatives considered
|
|
|
|
|
|
|
|
GitLab is such a broad project that multiple alternatives exist for
|
|
|
|
different components:
|
|
|
|
|
|
|
|
* GitHub
|
|
|
|
* Pros:
|
|
|
|
* widely used in the open source community
|
|
|
|
* Good integration between ticketing system and code
|
|
|
|
* Cons
|
|
|
|
* It is hosted by a third party (Microsoft!)
|
|
|
|
* Closed source
|
|
|
|
* GitLab:
|
|
|
|
* Pros:
|
|
|
|
* Mostly free software
|
|
|
|
* Feature-rich
|
|
|
|
* Cons:
|
|
|
|
* Complex software, high maintenance
|
|
|
|
* "Opencore" - some interesting features are closed-source
|
|
|
|
|
|
|
|
### GitLab command line clients
|
|
|
|
|
|
|
|
If you want to do batch operations or integrations with GitLab, you
|
|
|
|
might want to use one of those tools, depending on your environment or
|
|
|
|
prefered programming language:
|
|
|
|
|
|
|
|
* [bugwarrior](https://github.com/ralphbean/bugwarrior) ([Debian](https://tracker.debian.org/pkg/bugwarrior)) - support for GitLab, GitHub and
|
|
|
|
other bugtrackers for the [taskwarrior](http://taskwarrior.org/) database
|
|
|
|
* [git-lab](https://invent.kde.org/sdk/git-lab) - python commandline client, lists, pulls MR; creates
|
|
|
|
snippets
|
|
|
|
* [GitLab-API-v4](https://metacpan.org/release/GitLab-API-v4) ([Debian](https://tracker.debian.org/pkg/libgitlab-api-v4-perl)) - perl library and [commandline
|
|
|
|
client](https://manpages.debian.org/buster/libgitlab-api-v4-perl/gitlab-api-v4.1p.en.html)
|
|
|
|
* [GitLabracadabra](https://gitlab.com/gitlabracadabra/gitlabracadabra) ([Debian](https://tracker.debian.org/pkg/gitlabracadabra)) - *configure a GitLab instance
|
|
|
|
from a YAML configuration, using the API*: project settings like
|
|
|
|
labels, admins, etc
|
|
|
|
* [python-gitlab](https://github.com/python-gitlab/python-gitlab) (also known as `gitlab-cli` in [Debian](https://tracker.debian.org/pkg/python-gitlab))
|
|
|
|
* [ruby-gitlab](https://github.com/narkoz/gitlab) ([Debian](https://tracker.debian.org/pkg/ruby-gitlab)), also includes a [commandline
|
|
|
|
client](https://narkoz.github.io/gitlab/)
|
|
|
|
* [salsa](https://manpages.debian.org/buster/devscripts/salsa.1.en.html) (in [Debian devscripts](https://tracker.debian.org/pkg/devscripts)) is specifically built for
|
|
|
|
salsa but might be coerced into talking to other GitLab servers
|
|
|
|
|
|
|
|
### Migration tools
|
|
|
|
|
|
|
|
ahf implemente the gitlab using his own home-made tools that talk to
|
|
|
|
the GitLab and Trac API. but there's also [tracboat](https://github.com/tracboat/tracboat) which is
|
|
|
|
designed to migrate from trac to GitLab.
|
|
|
|
|
|
|
|
We did not use Tracboat because it uses gitlab's DB directly and thus
|
|
|
|
only works with some very specific version. Each time the database
|
|
|
|
schema changes at GitLab, Tracboat needs to port to it. We prefered to
|
|
|
|
use something that talked with the GitLab API.
|
|
|
|
|
|
|
|
We also didn't like the output entirely, so we modified it but still
|
|
|
|
used some of its regular expressions and parser.
|
|
|
|
|
|
|
|
We also needed to implement the "ticket movement" hack (with the legacy
|
|
|
|
project) which wasn't implemented in Tracboat.
|
|
|
|
|
|
|
|
Finally, we didn't want to do complete user migration, but lazily
|
|
|
|
transfer only some users.
|
|
|
|
|