Skip to content
Snippets Groups Projects
Verified Commit 20dc8901 authored by anarcat's avatar anarcat
Browse files

final edit

parent f372412f
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
policy/tpa-rfc-36-gitolite-gitweb-retirement.md
+ 189
− 139
View file @ 20dc8901
......@@ -65,7 +65,7 @@ years now, so it's time to move forward.
## Gitolite and GitWeb inventory
As of 2023-05-11, there are 566 git repositories on disk on the
As of 2023-05-11, there are 566 Git repositories on disk on the
Gitolite server (`cupani`), but oddly only 539 in the Gitolite
configuration file. 358 of those repositories are in the `user/`
namespace, which leaves us 208 "normal" repositories. Out of those, 65
......@@ -134,9 +134,41 @@ infrastructure, all new repositories MUST be created on GitLab.
## Redirections
For backwards compatibility, web redirections will permanently set in
the static mirror system. This will included a limited set of URLs
that GitLab can support in a meaningful way. It will *not* include an
SSH endpoint, which will start failing at the end of the migration.
the static mirror system.
This will include a limited set of URLs that GitLab can support in a
meaningful way, but some URLs *will* break. The following cgit URLs
notably do not have an equivalence in GitLab:
| cgit | note |
|------------|--------------------------------------------|
| `atom` | needs a feed token, user must be logged in |
| `blob` | no direct equivalent |
| `info` | not working on main cgit website? |
| `ls_cache` | not working, irrelevant? |
| `objects` | undocumented? |
| `snapshot` | pattern too hard to match on cgit's side |
The supported URLs are:
| cgit | note |
|-----------|------------------------------------------------------------------------------|
| `summary` | |
| `about` | |
| `commit` | |
| `diff` | incomplete: cgit can diff arbitrary refs and not GitLab, hard to parse |
| `patch` | |
| `rawdiff` | incomplete: which GitLab can't diff individual files |
| `log` | |
| `atom` | |
| `refs` | incomplete: GitLab has separate pages tags and branches, redirecting to tags |
| `tree` | incomplete: has no good default in GitLab, defaulting to HEAD |
| `plain` | |
| `blame` | incomplete: same default as tree above |
| `stats` | |
Redirections also do *not* include SSH (`ssh://`) remotes, which will
start failing at the end of the migration.
## Per-repository particularities
......@@ -150,12 +182,12 @@ can be included in this proposal.
Repositories that did not have any new commit in the last two years
are considered "idled" and should be migrated or archived to GitLab by
their owners. Failing that, TPA will *archive* the repositories in the
`legacy/` namespace at the final deadline.
GitLab `legacy/` namespace before final deadline.
### user repositories
There are 358 repositories under the `user/` namespace, owned by a
distinct 70 users.
There are 358 repositories under the `user/` namespace, owned by 70
distinct users.
Those repositories must be migrated to their corresponding user on the
GitLab side.
......@@ -166,8 +198,8 @@ in GitLab, owned by the GitLab admin doing the migration.
### "mirror" and "extern" repositories
Those repositories will be archived to GitLab within a month of the
adoption of this proposal.
Those repositories will be migrated to, and archived in, GitLab within
a month of the adoption of this proposal.
### Applications team repositories
......@@ -179,43 +211,22 @@ gitlab.torproject.org (Gitlab) repos."
The following redirections will be deployed shortly:
| Gitolite | gitlab | fate |
|----------------------------|--------------------------------------|---------|
| builders/tor-browser-build | tpo/applications/tor-browser-build | migrate |
| builders/rbm | tpo/applications/rbm | migrate |
| tor-android-service | tpo/applications/tor-android-service | migrate |
| tor-browser | tpo/applications/tor-browser/ | migrate |
| tor-browser-spec | tpo/applications/tor-browser-spec | migrate |
| tor-launcher | tpo/applications/tor-launcher | archive |
| torbutton | tpo/applications/torbutton | archive |
| Gitolite | gitlab | fate |
|------------------------------|----------------------------------------|---------|
| `builders/tor-browser-build` | `tpo/applications/tor-browser-build` | migrate |
| `builders/rbm` | `tpo/applications/rbm` | migrate |
| `tor-android-service` | `tpo/applications/tor-android-service` | migrate |
| `tor-browser` | `tpo/applications/tor-browser/` | migrate |
| `tor-browser-spec` | `tpo/applications/tor-browser-spec` | migrate |
| `tor-launcher` | `tpo/applications/tor-launcher` | archive |
| `torbutton` | `tpo/applications/torbutton` | archive |
See [tpo/tpa/team#41181][] for the ticket tracking this work.
[tpo/tpa/team#41181]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41181
### Hooks
There are 11 git hooks are currently deployed on `cupani`, the
Gitolite server. Here are how they will be managed:
This is a good example of how a team can migrate to GitLab and submit
a list of redirections to TPA.
| hook | GitLab equivalence |
|--------------------------------------------------------------------------------|-----------------------------------------|
| `post-receive.d/00-sync-to-mirror` | [Static shim][] |
| `post-receive.d/git-multimail` | No equivalence, see [issue gitlab#71][] |
| `post-receive.d/github-push` | [Native mirroring][] |
| `post-receive.d/gitlab-push` | N/A |
| `post-receive.d/irc-message` | [Web hooks][] |
| `post-receive.d/per-repo-hook` | N/A, trigger for later hooks |
| `post-receive-per-repo.d/admin%dns%auto-dns` | TPA-specific, see below |
| `post-receive-per-repo.d/admin%dns%domains/trigger-dns-server` | TPA-specific, see below |
| `post-receive-per-repo.d/admin%letsencrypt-domains/trigger-letsencrypt-server` | TPA-specific, see below |
| `post-receive-per-repo.d/admin%tor-nagios/trigger-nagios-build` | TPA-specific, see below |
| `post-receive-per-repo.d/tor-cloud/trigger-staticiforme-cloud` | ignored, discontinued in 2015 |
[Native mirroring]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab#how-to-mirror-a-git-repository-from-gitlab-to-github
[Web hooks]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab#publishing-notifications-on-irc
[Static shim]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/static-shim
[issue gitlab#71]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/71
[tpo/tpa/team#41181]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41181
### TPA repositories
......@@ -229,70 +240,96 @@ Many of those repositories have hooks that trigger all sorts of
actions on the infrastructure and will need to be converted in GitLab
CI actions or similar.
Note that TPA also has git repositories on the Puppet server
(`tor-puppet.git`) and LDAP server (`account-keyring.git`), but those
are not managed by Gitolite and do not fall inside the scope of this
proposal.
Those repositories are particularly problematic and will need special
work before being migrated to GitLab. Here's the list of repositories
and their proposed fate.
| Repository | data | Problem | Fate |
|-----------------------------------|--------------------------------------|-----------------------------------------|--------------------------------------|
| `account-keyring.git` | OpenPGP keyrings | hooks into the static mirror system | convert to GitLab CI |
| `buildbot-conf.git` | old buildbot config? | obsolete | archive |
| `dip.git` | GitLab ansible playbooks? | duplicate of `services/gitlab/dip.git`? | archive? |
| `dns/auto-dns.git` | DNS zones source used by LDAP server | security | check OpenPGP signatures |
| `dns/dns-helpers.git` | DNSSEC generator used on DNS master | security | check OpenPGP signatures |
| `dns/domains.git` | DNS zones source used by LDAP server | security | check OpenPGP signatures |
| `dns/mini-nag.git` | monitoring on DNS primary | security | check OpenPGP signatures |
| `letsencrypt-domains.git` | TLS certificates generation | security | ??? GitLab CI + OpenPGP signatures?? |
| `puppet/puppet-ganeti.git` | puppet-ganeti fork | misplaced | destroy |
| `services/gettor.git` | ansible playbook for gettor | obsolete | archive |
| `services/gitlab/dip-configs.git` | GitLab ansible playbooks? | obsolete | archive |
| `services/gitlab/dip.git` | GitLab ansible playbooks? | duplicate of `dip.git`? | archive? |
| `services/gitlab/ldapsync.git` | LDAP to GitLab script, unused | obsolete | archive |
| `static-builds.git` | Jenkins static sites build scripts | obsolete | archive |
| `tor-jenkins.git` | Jenkins build scripts | obsolete | archive |
| `tor-nagios.git` | Icinga configuration | confidentiality? | abolish? see also [TPA-RFC-33][] |
| `tor-passwords.git` | password manager | confidentiality | migrate? |
| `tor-virt.git` | libvirt VM configuration | obsolete | destroy |
| `trac/TracAccountManager.git` | Trac tools | obsolete | archive |
| `trac/trac-email.git` | Trac tools | obsolete | archive |
| `tsa-misc.git` | miscellaneous scripts | none | migrate |
| `userdir-ldap-cgi.git` | fork of DSA's repository | none | migrate |
| `userdir-ldap.git` | fork of DSA's repository | none | migrate |
The following repositories are particularly problematic and will need
special work to migrate. Here's the list of repositories and their
proposed fate.
| Repository | data | Problem | Fate |
|-------------------------------|--------------------------------------|-------------------------------------|----------------------------------|
| `account-keyring` | OpenPGP keyrings | hooks into the static mirror system | convert to GitLab CI |
| `buildbot-conf` | old buildbot config? | obsolete | archive |
| `dip` | GitLab ansible playbooks? | duplicate of `services/gitlab/dip`? | archive? |
| `dns/auto-dns` | DNS zones source used by LDAP server | security | check OpenPGP signatures |
| `dns/dns-helpers` | DNSSEC generator used on DNS master | security | check OpenPGP signatures |
| `dns/domains` | DNS zones source used by LDAP server | security | check OpenPGP signatures |
| `dns/mini-nag` | monitoring on DNS primary | security | check OpenPGP signatures |
| `letsencrypt-domains` | TLS certificates generation | security | move to Puppet? |
| `puppet/puppet-ganeti` | puppet-ganeti fork | misplaced | destroy |
| `services/gettor` | ansible playbook for gettor | obsolete | archive |
| `services/gitlab/dip-configs` | GitLab ansible playbooks? | obsolete | archive |
| `services/gitlab/dip` | GitLab ansible playbooks? | duplicate of `dip`? | archive? |
| `services/gitlab/ldapsync` | LDAP to GitLab script, unused | obsolete | archive |
| `static-builds` | Jenkins static sites build scripts | obsolete | archive |
| `tor-jenkins` | Jenkins build scripts | obsolete | archive |
| `tor-nagios` | Icinga configuration | confidentiality? | abolish? see also [TPA-RFC-33][] |
| `tor-passwords` | password manager | confidentiality | migrate? |
| `tor-virt` | libvirt VM configuration | obsolete | destroy |
| `trac/TracAccountManager` | Trac tools | obsolete | archive |
| `trac/trac-email` | Trac tools | obsolete | archive |
| `tsa-misc` | miscellaneous scripts | none | migrate |
| `userdir-ldap-cgi` | fork of DSA's repository | none | migrate |
| `userdir-ldap` | fork of DSA's repository | none | migrate |
[TPA-RFC-33]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-33-monitoring
The most critical repositories are the ones marked `security`. A
solution will be decided on a case-by-case basis but in general, the
solution will be decided on a case-by-case basis. In general, the
approach taken will be to pull changes from GitLab (maybe with a
webhook to kick the pull) and check the integrity of the repository
with OpenPGP signatures to avoid abuse and reset the trust anchor.
with OpenPGP signatures as a trust anchor.
Note that TPA also has Git repositories on the Puppet server
(`tor-puppet.git`) and LDAP server (`account-keyring.git`), but those
are not managed by Gitolite and are out of scope for this proposal.
## Hooks
There are 11 Git hooks are currently deployed on the Gitolite
server.
| hook | GitLab equivalence |
|--------------------------------------------------------------------------------|-----------------------------------------|
| `post-receive.d/00-sync-to-mirror` | [Static shim][] |
| `post-receive.d/git-multimail` | No equivalence, see [issue gitlab#71][] |
| `post-receive.d/github-push` | [Native mirroring][] |
| `post-receive.d/gitlab-push` | N/A |
| `post-receive.d/irc-message` | [Web hooks][] |
| `post-receive.d/per-repo-hook` | N/A, trigger for later hooks |
| `post-receive-per-repo.d/admin%dns%auto-dns` | TPA-specific, see above |
| `post-receive-per-repo.d/admin%dns%domains/trigger-dns-server` | TPA-specific, see above |
| `post-receive-per-repo.d/admin%letsencrypt-domains/trigger-letsencrypt-server` | TPA-specific, see above |
| `post-receive-per-repo.d/admin%tor-nagios/trigger-nagios-build` | TPA-specific, see above |
| `post-receive-per-repo.d/tor-cloud/trigger-staticiforme-cloud` | ignored, discontinued in 2015 |
[Native mirroring]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab#how-to-mirror-a-git-repository-from-gitlab-to-github
[Web hooks]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab#publishing-notifications-on-irc
[Static shim]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/static-shim
[issue gitlab#71]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/71
## Timeline
The migration will happen in four stages:
1. voluntary migration
1. now and for the next 6 months: voluntary migration
2. 6 months later: evaluation and idle repositories locked down
3. 9 months later: TPA enforced migration
4. 12 months later: Gitolite and GitWeb server retirements
### T: proposal adopted, voluntary migration encouraged
Once this proposal is adopted, Gitolite users are strongly encouraged
to start migrating their content to GitLab, following the [migration procedure][].
Once this proposal is standard (see the [deadline][] below), Gitolite
users are strongly advised to migrate to GitLab, following the
[migration procedure][].
[deadline]: #deadline
### T+6 months: evaluation and idle repositories locked down
After 6 months, an evaluation of the progress will be performed and
reminders will be sent to users still needing to migrate.
After 6 months, TPA will evaluate the migration progress and send
reminders to users still needing to migrate.
Repositories still without any changes in the last two years will be
locked in Gitolite, preventing any further changes.
TPA will lock Gitolite repositories without any changes in the last
two years, preventing any further change.
### T+9 months: TPA enforces migration
......@@ -301,11 +338,12 @@ repositories will be moved or archived to GitLab by TPA itself, with a
completion after 12 months.
Once all repositories are migrated, the redirections will be moved to
the static mirror system. The retirement procedure for the two hosts
(`cupani` for Gitolite and `vineale` for GitWeb) will be started which
involves shutting down the machines and removing them from
monitoring. Disks will not be destroyed for a few more months for
safety reasons.
the static mirror system.
The retirement procedure for the two hosts (`cupani` for Gitolite and
`vineale` for GitWeb) will be started which involves shutting down the
machines and removing them from monitoring. Disks will not be
destroyed for three more months.
### T+12 months: complete Gitolite and GitWeb server retirement
......@@ -329,26 +367,31 @@ requirement came out:
# Personas
Here we collect some "personas", fictitious characters that try to
cover most of the current use cases. The goal is to see how the
changes will affect them. If you are not represented by one of those
personas, please let us know and describe your use case.
## Arthur, the user
Arthur, the average Tor user, will likely not notice any change from
this migration. Arthur rarely interacts with our git servers; if at
all, it might be through some link to a specification hidden deep
inside one of our applications or website documentation. Redirections
will ensure those will keep working at least partially.
this migration.
If Arthur ever becomes really motivated, they will become Barbara,
drive-by contributor.
Arthur rarely interacts with our Git servers: if at all, it would be
through some link to a specification hidden deep inside one of our
applications documentation or a website. Redirections will ensure
those will keep working at least partially.
## Barbara, the drive-by contributor
Barbara is a drive-by contributor, which finds and reports bugs in our
software or our documentation. Previously, Barbara was lost when she
would previously find git repositories because it was not clear where
or how to contribute.
Barbara is a drive-by contributor, who finds and reports bugs in our
software or our documentation. Previously, Barbara would sometimes get
lost when she would find Git repositories, because it was not clear
where or how to contribute to those projects.
Now, if Barbara finds the old git repositories, she will be redirected
to GitLab where she can make awesome contributions.
Now, if Barbara finds the old Git repositories, she will be redirected
to GitLab where she can make awesome contributions, by reporting
issues or merge requests in the right projects.
## Charlie, the old-timer
......@@ -357,8 +400,10 @@ Tor. He knows by heart proposal numbers and magic redirections like
<https://spec.torproject.org/>.
Charlie will be slightly disappointed because some deep links to line
numbers in GitWeb will break. Charlie is concerned about the attack
surface in GitLab, but will look at the [mitigation strategies][].
numbers in GitWeb will break. In particular, line number anchors might
not work correctly. Charlie is also concerned about the attack surface
in GitLab, but will look at the [mitigation strategies][] to see if
something might solve that concern.
Otherwise Charlie should be generally unaffected by the change.
......@@ -371,9 +416,9 @@ rejected in the process.
## Keeping Gitolite and GitWeb
One alternative that has been considered (and, indeed, has been the
de-facto solution for almost three years now) is to keep Gitolite and
GitWeb running indefinitely.
One alternative is to keep Gitolite and GitWeb running
indefinitely. This has been the de-facto solution for almost three
years now.
In a [October 2020 tools meeting][], it was actually decided to
replace Gitolite with GitLab by 2021 or 2022. The alternative of
......@@ -382,33 +427,33 @@ imposes too much burden on the TPA team while draining valuable
resources away from improving GitLab hosting, all the while providing
a false sense of security.
We want to extend a warm thank you to the good people who setup and
managed those (c)git(web) and Gitolite servers for all that time:
thanks!
That said, we want to extend a warm thank you to the good people who
setup and managed those (c)git(web) and Gitolite servers for all that
time: thanks!
[October 2020 tools meeting]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40472#note_2756345
## Keeping Gitolite only for problem repositories
One [suggestion][] that was made was to keep Gitolite for problematic
repositories and keep a mirror to avoid having to migrate those to
GitLab.
One [suggestion][] is to keep Gitolite for problematic repositories
and keep a mirror to avoid having to migrate those to GitLab.
After some research, it seems like only TPA is affected by those
problems. In effect, we have already implemented part of that
suggestion by keeping Gitolite around for so long. Now, we're actually
going to find solutions to migrate those hooks into GitLab reliably.
It seems like only TPA is affected by those problems. We're taking it
upon ourselves to cleanup this legacy and pivot to a more neutral,
less powerful Git hosting system that relies less on custom (and
legacy) Git hooks. Instead, we'll design a more standard system based
on web hooks or other existing solutions (e.g. Puppet).
[suggestion]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40472#note_2756340
## Concerns about GitLab's security
During the discussions surrounding the GitLab migration, one of the
concerns that was raised was, in general terms, "how do we protect our
code against the larger attack surface of GitLab?
concerns raised was, in general terms, "how do we protect our code
against the larger attack surface of GitLab?
A summary of those discussions that happened in tpo/tpa/gitlab#36 and
tpo/tpa/gitlab#81 was posted in the [Security Concerns][] section of
A summary of those discussions that happened in [tpo/tpa/gitlab#36][] and
[tpo/tpa/gitlab#81][] was posted in the [Security Concerns][] section of
our internal Gitolite documentation.
The conclusion of that discussion was:
......@@ -429,47 +474,51 @@ The conclusion of that discussion was:
[Security Concerns]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git#security-concerns
[put it]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81#note_2732337
[tpo/tpa/gitlab#36]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/36
[tpo/tpa/gitlab#81]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81#note_2732337
## git:// protocol redirections
We do not currently support cloning repositories over the `git://`
protocol and therefore do not have to worry about redirecting those, thankfully.
protocol and therefore do not have to worry about redirecting those,
thankfully.
## GitWeb to cgit redirections
Once a upon a time, the web interface to the git repositories was
Once a upon a time, the web interface to the Git repositories was
running GitWeb. It was, at some point, migrated to cgit, which changed
a bunch of URLs and broke many URLs in the process.
a bunch of URLs and broke many URLs in the process. See [this
discussion for examples][].
Those URLs have been broken for years and will not be fixed in this
migration. TPA is not opposed to fixing them, but we find our energy
is best spent redirecting currently working URLs to GitLab than
already broken ones.
[this discussion for examples]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40472#note_2866806
## GitLab hosting improvement plans
This proposal explicitly does not cover possible improvements to
GitLab hosting, which will be discussed separately.
GitLab hosting.
It is understood, however, that GitLab will need more resources, but
in terms of hardware and staff, and the retirement of the old Git
infrastructure might provide a little slack that could be used for
that purpose.
That said, GitLab will need more resources, both in terms of hardware
and staff. The retirement of the old Git infrastructure might provide
a little slack for exactly that purpose.
## Other forges
There are many other forges around. We have used Trac in the past (see
[our Trac documentation][]) and projects like [Gitea][] or
[Sourcehut][] are around as well.
There are many other "forges" like GitLab around. We have used Trac in
the past (see [our Trac documentation][]) and projects like [Gitea][]
or [Sourcehut][] are around as well.
Other than Trac, no serious evaluation of alternative Git forges was
performed before we migrated to GitLab in 2020. Now, we feel it's too
late to put that put that into question.
Migrating to other forges is therefore considered out of scope as far
as Gitolite's retirement is concerned, but TPA doesn't permanently
exclude evaluating other solutions than GitLab in the future if
requirements are not being fulfilled correctly.
as Gitolite's retirement is concerned. But TPA doesn't permanently
exclude evaluating other solutions than GitLab in the future.
[Gitea]: https://gitea.io/
[Sourcehut]: https://sr.ht/
......@@ -485,8 +534,8 @@ TPA and tor-internal.
# Deadline
This proposal will be marked as adopted on June 15th unless an
objection is raised.
This proposal will be standard and put in motion on June 15th unless
an objection is raised.
# Status
......@@ -494,7 +543,8 @@ This proposal is currently in the `proposed` state.
# References
This proposal is discussed in [issue tpo/tpa/team#40472][]. It is a
long ticket and you do not need to read its entirety
This proposal was established in [issue tpo/tpa/team#40472][] but
further discussions should happen in [tpo/tpa/team#41180][].
[issue tpo/tpa/team#40472]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40472
[tpo/tpa/team#41180]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41180
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment