TPA team issueshttps://gitlab.torproject.org/tpo/tpa/team/-/issues2024-03-26T20:49:37Zhttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41213lock down legacy git infrastructure2024-03-26T20:49:37Zanarcatlock down legacy git infrastructureAs part of the Gitolite retirement procedure (TPA-RFC-36, #41180), lock Gitolite repositories without any changes in the last
two years, preventing any further change.As part of the Gitolite retirement procedure (TPA-RFC-36, #41180), lock Gitolite repositories without any changes in the last
two years, preventing any further change.legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcat2024-01-31https://gitlab.torproject.org/tpo/tpa/team/-/issues/41214review gitolite retirement progress and send a reminder2024-03-20T18:56:12Zanarcatreview gitolite retirement progress and send a reminderAs part of the Gitolite retirement procedure (TPA-RFC-36, #41180), review the progress of the migration and send a reminder:
- [x] how many repositories are left to migrate, populating #41215 with the result
- [x] did any repository get...As part of the Gitolite retirement procedure (TPA-RFC-36, #41180), review the progress of the migration and send a reminder:
- [x] how many repositories are left to migrate, populating #41215 with the result
- [x] did any repository get changes since the deprecation notice on 2023-06-08
- [x] send a reminder, similar to #41212legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcat2024-01-24https://gitlab.torproject.org/tpo/tpa/team/-/issues/41220Fix up gitweb links on https://spec.torproject.org/2024-01-11T21:34:29ZGeorg KoppenFix up gitweb links on https://spec.torproject.org/As part of the the process in TPA-RFC-36 the links to gitweb need to get replaced on spec.torproject.org.
(Someone with the right powers might want to link this ticket to #41180 and set the right milestone)As part of the the process in TPA-RFC-36 the links to gitweb need to get replaced on spec.torproject.org.
(Someone with the right powers might want to link this ticket to #41180 and set the right milestone)legacy Git infrastructure retirement (TPA-RFC-36)https://gitlab.torproject.org/tpo/tpa/team/-/issues/40472draft TPA-RFC-36: establish policy on git repository mirroring, hosting and, ...2024-01-10T21:56:45Zanarcatdraft TPA-RFC-36: establish policy on git repository mirroring, hosting and, ultimately migration from gitoliteWe have already started mirroring (gitlab#18, gitlab#35) repositories from gitolite to GitLab. We need to decide how and/or if we will accept such requests in the future, and, in particular, whether we want to host all our git repositori...We have already started mirroring (gitlab#18, gitlab#35) repositories from gitolite to GitLab. We need to decide how and/or if we will accept such requests in the future, and, in particular, whether we want to host all our git repositories on GitLab in the long term.
If so, we need to come up with a migration plan on how the old repositories on gitolite will "map" to the ones in GitLab. This is particularly complicated by the fact that the namespace established on GitLab does not necessarily reflect the one in use on Gitolite, so we are very likely to have to come up with some rewrite rules to handle those redirections.
But at the very least, we need a plan, and we need it fast, because I am worried this migration will happen organically and we will then have to maintain *two* git hosting systems in parallel. This is similar to the problem of "hosting both trac and gitlab in parallel" that we have (succesfully, i think) avoided, but it was a near hit. ;)
TL;DR: this is the current state of the non-official policy:
* gitolite and gitweb will eventually be retired, probably in 2022
* new repositories are created on GitLab
* repositories can be [mirrored between gitolite and GitLab](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git#mirroring-a-gitolite-repository-to-gitlab)
* repositories ("only small ones") can be [*migrated* to GitLab](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git#how-to-migrate-a-git-repository-from-legacy-to-gitlab) as well
* an RFC will be written before the final migration is started, and discussed here
By "small ones", I think we meant "not tor browser",
We need to define the following policies:
1. [x] do we keep gitolite around forever? **no**. gitolite and gitweb will be replaced by GitLab eventually.
2. [x] if we do, do we keep the old codebase or upgrade? **N/A**
3. [x] if we do not, when do we retire git-rw (cupani) and gitweb (vineale)? **within 1 or 2 years, that is 2021 or 2022**
4. [x] if we do not, how do we protect our code against the larger attack surface of GitLab? <del>opened gitlab#81 for that discussion</del> this roadblock is removed. it's the responsability of teams to implement commit signing or other integrity measures they need.
5. [x] where do people create new git repositories? gitolite or gitlab? **new repositories are created on gitlab**
6. [x] can people **mirror** their git repositories from gitolite to gitlab? **yes, in a limited way** there are known issues with protected branches (gitlab#38)
7. [x] how do we **mirror** a repo from gitolite to gitlab? **documented**, see [this section](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git#github-and-gitlab-mirrors)
8. [x] can people **migrate** their git repositories from gitolite to gitlab? **yes, but only small projects right now**, [documented here](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab/#how-to-migrate-a-git-repository-from-legacy-to-gitlab), missing support for redirection on clone
9. [x] how do we **migrate** a repo from gitolite to gitlab? using the [migration procedure](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab/#how-to-migrate-a-git-repository-from-legacy-to-gitlab)
10. [x] how do we redirect users from gitolite to gitlab? using the above migration procedure, although there are issues with SSH clones (which don't fire a hook) and HTTP clones need webserver-level redirection
Remaining task list:
* [x] figure out how to do redirections on `git clone`
* [x] figure out gitweb redirection patterns
* [ ] propose a migration plan for retiring the legacy gitolite infrastructure (cupani/git-rw, vineale/gitweb)
Those can be executed in parallel.legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/40642Remove torflow and pytorctl from {git,gitweb}.tpo2024-01-09T16:58:29ZjugaRemove torflow and pytorctl from {git,gitweb}.tpoBoth project are deprecated and have been moved to https://gitlab.torproject.org/tpo/network-health/torflow and https://gitlab.torproject.org/tpo/network-health/pytorctl respectively.
Thanks!
(cc @gk)Both project are deprecated and have been moved to https://gitlab.torproject.org/tpo/network-health/torflow and https://gitlab.torproject.org/tpo/network-health/pytorctl respectively.
Thanks!
(cc @gk)Alexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41425move gitaly backups to object storage2023-12-14T21:16:52Zanarcatmove gitaly backups to object storageIn #40518, I've evaluated the situation with GitLab backups, and concluded we need a better backup system for Gitaly backups, as:
> they are "*not* covered by current script, assumed bacula works, but [actually this is problematic](http...In #40518, I've evaluated the situation with GitLab backups, and concluded we need a better backup system for Gitaly backups, as:
> they are "*not* covered by current script, assumed bacula works, but [actually this is problematic](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#back-up-repository-data-separately): gitaly needs to be stopped before backups can be performed consistently, so we actually need to either do that (!) or re-enable the script... there's actually a contradiction in documentation about this, i [filed a ticket](https://gitlab.com/gitlab-org/gitlab/-/issues/432743). it looks like the solution here is to use [object storage to do server-side repository backups](https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#create-server-side-repository-backups)." ([source](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40518#note_2968836))
The problem with object storage backups is that object storage itself is *not* backed up (https://gitlab.torproject.org/tpo/tpa/team/-/issues/41415), so we need to figure out how to handle this problem.
This is higher priority to backing up minio itself, if there is another solution, that's possibly a better way forward.
Checklist:
- [x] configure a bucket and accesses
- [x] configure gitlab (gitaly?) to access it
- [x] test a backup (works!)
- [x] test a nightly (works, but takes +200GiB!! oops.)
- [x] test INCREMENTAL backups (works!)
- [x] purge previous extra 200GiB backup
- [ ] figure out expiration policies ([upstream](https://gitlab.com/gitlab-org/gitlab/-/issues/435265))
- [ ] document the design of server-side backups ([upstream](https://gitlab.com/gitlab-org/gitlab/-/issues/435266))
- [ ] document and test restore procedures
- [ ] document install procedures: how gitaly backups were setup, mainly from this issue
- [ ] disable bacula backups once we're sure everything is in orderanarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41347Move tor-pristine-upstream.git to GitLab2023-11-29T22:24:59ZJérôme Charaouilavamind@torproject.orgMove tor-pristine-upstream.git to GitLabDuring a work session with @weasel today about releasing new tor versions to our Debian repository we agreed we should move over `tor-pristine-upstream.git` to GitLab, as it currently lives only on git-rw.tpo.During a work session with @weasel today about releasing new tor versions to our Debian repository we agreed we should move over `tor-pristine-upstream.git` to GitLab, as it currently lives only on git-rw.tpo.legacy Git infrastructure retirement (TPA-RFC-36)Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41180discuss TPA-RFC-36: gitolite and gitweb migration to GitLab2023-08-23T07:32:26Zanarcatdiscuss TPA-RFC-36: gitolite and gitweb migration to GitLabIn #40472 and gitlab#36, we had lengthy discussions on how we could possibly migrate everything to GitLab already, and came up with a proposal.
Those tickets are very long, however, and basically summarized in the TPA-RFC-36 proposal. T...In #40472 and gitlab#36, we had lengthy discussions on how we could possibly migrate everything to GitLab already, and came up with a proposal.
Those tickets are very long, however, and basically summarized in the TPA-RFC-36 proposal. This is therefore a new ticket aimed at receiving feedback on the proposal.
the full text of the proposal was sent to tor-internal and is also available in here:
https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-36-gitolite-gitweb-retirement
Summary: Gitolite (`git.torproject.org` and `git-rw.torproject.org`) and
GitWeb (<https://gitweb.torproject.org>) will be fully retired within
9 to 12 months (by the end of Q2 2024). TPA will implement
redirections on the web interfaces to maintain limited backwards
compatibility for the old URLs. Start migrating your repositories now
by following the [migration procedure][].
[migration procedure]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git#how-to-migrate-a-git-repository-from-legacy-to-gitlab
checklist of tasks to be done after this is adopted:
- [x] create issues for each step in the timeline
- [x] update wiki page to reflect the adoptionlegacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcat2023-06-08https://gitlab.torproject.org/tpo/tpa/team/-/issues/41248Retire torproject-pusher GitHub account2023-08-02T18:04:56ZJérôme Charaouilavamind@torproject.orgRetire torproject-pusher GitHub accountWe currently have a `torproject-pusher` account that allows pushing commits from Gitolite (`cupani`) to GitHub. It used to also allow mirroring from tpo/web/lego and tpo/web/manual, but I migrated those to project-specifc deploy keys in ...We currently have a `torproject-pusher` account that allows pushing commits from Gitolite (`cupani`) to GitHub. It used to also allow mirroring from tpo/web/lego and tpo/web/manual, but I migrated those to project-specifc deploy keys in #41246.
In light of the migration off Gitolite, we should retire this account.legacy Git infrastructure retirement (TPA-RFC-36)Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2023-08-03https://gitlab.torproject.org/tpo/tpa/team/-/issues/41189Migrating (etc) nickm's gitolite repositories.2023-07-04T13:58:51ZNick MathewsonMigrating (etc) nickm's gitolite repositories.Hello!
I believe that the following should be *archived*:
* `user/nickm/calltool`
* `user/nickm/tor`
* `user/nickm/tor-ideas`
* `user/nickm/tor-roadmaps`
* `user/nickm/torspec`
I believe that all other `user/nickm` repositories s...Hello!
I believe that the following should be *archived*:
* `user/nickm/calltool`
* `user/nickm/tor`
* `user/nickm/tor-ideas`
* `user/nickm/tor-roadmaps`
* `user/nickm/torspec`
I believe that all other `user/nickm` repositories should be destroyed, namely:
```
@ W user/nickm/bridgedb
@ W user/nickm/githax
R W user/nickm/libevent-ideas
@ W user/nickm/trunnel
```legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcat2023-06-09https://gitlab.torproject.org/tpo/tpa/team/-/issues/41236gitolite: Redirect tor.git to Gitlab2023-06-26T16:15:35ZDavid Gouletdgoulet@torproject.orggitolite: Redirect tor.git to GitlabHello,
Network team is ready to have `tor.git` on Gitolite to be redirected to Gitlab and thus become canonical repository:
https://gitweb.torproject.org/tor.git -> https://gitlab.torproject.org/tpo/core/tor
Thanks!Hello,
Network team is ready to have `tor.git` on Gitolite to be redirected to Gitlab and thus become canonical repository:
https://gitweb.torproject.org/tor.git -> https://gitlab.torproject.org/tpo/core/tor
Thanks!legacy Git infrastructure retirement (TPA-RFC-36)Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41235Redirect and destroy stem.git gitolite to Gitlab2023-06-26T16:15:01ZjugaRedirect and destroy stem.git gitolite to GitlabWe've just migrated stem to gitlab (https://gitlab.torproject.org/tpo/network-health/team/-/issues/307), so you can now destroy the one at https://gitweb.torproject.org/stem.git and redirect it to https://gitlab.torproject.org/tpo/networ...We've just migrated stem to gitlab (https://gitlab.torproject.org/tpo/network-health/team/-/issues/307), so you can now destroy the one at https://gitweb.torproject.org/stem.git and redirect it to https://gitlab.torproject.org/tpo/network-health/stem.git
Thanks!legacy Git infrastructure retirement (TPA-RFC-36)https://gitlab.torproject.org/tpo/tpa/team/-/issues/41232gitolite: Redirect or close several projects for Network Team2023-06-26T16:14:38ZDavid Gouletdgoulet@torproject.orggitolite: Redirect or close several projects for Network TeamGreetings,
The following can simply be **closed** without any redirections:
- https://gitweb.torproject.org/testnet.git/
- https://gitweb.torproject.org/tor-rust-dependencies.git/
The following should be **redirected**:
- https://git...Greetings,
The following can simply be **closed** without any redirections:
- https://gitweb.torproject.org/testnet.git/
- https://gitweb.torproject.org/tor-rust-dependencies.git/
The following should be **redirected**:
- https://gitweb.torproject.org/fallback-scripts.git/ -> https://gitlab.torproject.org/tpo/core/fallback-scripts
- https://gitweb.torproject.org/chutney.git/ -> https://gitlab.torproject.org/tpo/core/chutney
- https://gitweb.torproject.org/fuzzing-corpora.git/ -> https://gitlab.torproject.org/tpo/core/fuzzing-corpora
- https://gitweb.torproject.org/trunnel.git/ -> https://gitlab.torproject.org/tpo/core/trunnellegacy Git infrastructure retirement (TPA-RFC-36)https://gitlab.torproject.org/tpo/tpa/team/-/issues/41212announce gitolite/gitweb deprecation to the community2023-06-08T18:11:37Zanarcatannounce gitolite/gitweb deprecation to the communityin #41180 we decided to migrate fully to GitLab, make people aware.
there will be a banner on the site (#41211 ) but we should also do an email announcement.in #41180 we decided to migrate fully to GitLab, make people aware.
there will be a banner on the site (#41211 ) but we should also do an email announcement.legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41211add a banner on legacy git servers to announce deprecation and migration2023-06-08T18:02:25Zanarcatadd a banner on legacy git servers to announce deprecation and migrationthere should be a banner of some sort on the git servers to announce their deprecation.there should be a banner of some sort on the git servers to announce their deprecation.legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcat2023-06-08https://gitlab.torproject.org/tpo/tpa/team/-/issues/41195More anti-censorship repositories to migrate2023-06-08T17:42:47Zmeskiomeskio@torproject.orgMore anti-censorship repositories to migrate| gitolite | gitlab | fate |
| ------ | ------ | -----|
| bridgedb.git | tpo/anti-censorship/bridgedb | migrate |
| project/bridges/bridgedb-admin.git | tpo/anti-censorship/bridgedb-admin | migrate || gitolite | gitlab | fate |
| ------ | ------ | -----|
| bridgedb.git | tpo/anti-censorship/bridgedb | migrate |
| project/bridges/bridgedb-admin.git | tpo/anti-censorship/bridgedb-admin | migrate |legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41193Redirect pluggable-transports to GitLab2023-06-08T17:42:47Zmeskiomeskio@torproject.orgRedirect pluggable-transports to GitLabI've migrated these repositories:
| gitolite | gitlab | fate |
| ------ | ------ | -----|
| pluggable-transports/httpsproxy.git | tpo/anti-censorship/pluggable-transports/httpsproxy | archive |
| pluggable-transports/obfs4.git | tpo/an...I've migrated these repositories:
| gitolite | gitlab | fate |
| ------ | ------ | -----|
| pluggable-transports/httpsproxy.git | tpo/anti-censorship/pluggable-transports/httpsproxy | archive |
| pluggable-transports/obfs4.git | tpo/anti-censorship/pluggable-transports/lyrebird | migrate |
| pluggable-transports/obfsproxy.git | tpo/anti-censorship/pluggable-transports/obfsproxy | archive |
| pluggable-transports/obfsproxy-legacy.git | tpo/anti-censorship/pluggable-transports/obfsproxy-legacy | archive |
| pluggable-transports/pyptlib.git | tpo/anti-censorship/pluggable-transports/pyptlib | archive |
| pluggable-transports/snowflake.git | tpo/anti-censorship/pluggable-transports/snowflake | migrate |
| pluggable-transports/snowflake-mobile.git | tpo/anti-censorship/pluggable-transports/snowflake-mobile | migrate |
| pluggable-transports/snowflake-webext.git | tpo/anti-censorship/pluggable-transports/snowflake-webext | migrate |
| pluggable-transports/websocket.git | tpo/anti-censorship/pluggable-transports/websocket | archive |legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41190Redirect pluggable-transports/{bundle.git,fog.git} to GitLab2023-06-08T17:42:47ZDavid Fifielddcf@torproject.orgRedirect pluggable-transports/{bundle.git,fog.git} to GitLab(Following [migration instructions](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git?version_id=1f73e6c85b6645782b2348fa368dc278d182bb56#user-part-importing-the-repository-into-gitlab).)
In addition to those in tpo/tpa/team#...(Following [migration instructions](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git?version_id=1f73e6c85b6645782b2348fa368dc278d182bb56#user-part-importing-the-repository-into-gitlab).)
In addition to those in tpo/tpa/team#41182,
I've also migrated these repositories I am responsible for.
Both of them have been archived.
|gitolite|gitlab|fate|
|---|---|---|
|/pluggable-transports/bundle.git|/tpo/anti-censorship/pt-bundle|archive|
|/pluggable-transports/fog.git|/tpo/anti-censorship/pluggable-transports/fog|archive|legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41182Redirect goptlib, meek, flashproxy repositories to GitLab2023-06-08T17:42:47ZDavid Fifielddcf@torproject.orgRedirect goptlib, meek, flashproxy repositories to GitLab(Following [migration instructions](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git?version_id=1f73e6c85b6645782b2348fa368dc278d182bb56#user-part-importing-the-repository-into-gitlab).)
These repositories that I am responsi...(Following [migration instructions](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git?version_id=1f73e6c85b6645782b2348fa368dc278d182bb56#user-part-importing-the-repository-into-gitlab).)
These repositories that I am responsible for have been moved to GitLab and need redirects.
The flashproxy respository has additionally been "archived" in GitLab.
|gitolite|gitlab|fate|
|---|---|---|
|/pluggable-transports/goptlib.git|/tpo/anti-censorship/pluggable-transports/goptlib|migrate|
|/pluggable-transports/meek.git|/tpo/anti-censorship/pluggable-transports/meek|migrate|
|/flashproxy.git|/tpo/anti-censorship/pluggable-transports|archive|legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41181redirect applications team repositories to GitLab2023-06-08T17:42:47Zanarcatredirect applications team repositories to GitLabin 2020, the applications team [announced their gitolite repositories were deprecated](https://lists.torproject.org/pipermail/tor-project/2022-December/003518.html). those are the repositories that need to be redirected by TPA and have a...in 2020, the applications team [announced their gitolite repositories were deprecated](https://lists.torproject.org/pipermail/tor-project/2022-December/003518.html). those are the repositories that need to be redirected by TPA and have already been imported in GitLab:
| gitolite | gitlab | fate |
|----------------------------|--------------------------------------|---------|
| builders/tor-browser-build | tpo/applications/tor-browser-build | migrate |
| builders/rbm | tpo/applications/rbm | migrate |
| tor-android-service | tpo/applications/tor-android-service | migrate |
| tor-browser | tpo/applications/tor-browser/ | migrate |
| tor-browser-spec | tpo/applications/tor-browser-spec | migrate |
| tor-launcher | tpo/applications/tor-launcher | archive |
| torbutton | tpo/applications/torbutton | archive |legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcat