RETIRED
The Gitolite and Gitweb have been retired and repositories migrated to GitLab. See TPA-RFC-36 for the decision and the [legacy Git infrastructure retirement milestone](%legacy Git infrastructure retirement (TPA-RFC-36)) for progress.
Original documentation
Our git setup consists of three interdependent services:
-
git-rw.torproject.org
: ssh accessible and writeable git repositories - https://git.torproject.org: read-only anonymous access
- https://gitweb.torproject.org/: web browsing repositories
When a developer pushes to git-rw, the repository is mirrored to git and so made available via the gitweb service.
- RETIRED
- Original documentation
-
Howto
- Regular repositories
- User repositories
- Learning what git repos you can read/write
- Commit hooks
- Migrating a repository to GitLab
- Migration to other servers
- Destroying a repository
- Mirroring a gitolite repository to GitLab
- Mirroring a private git repository to GitLab's
- Archiving a repository
- GitHub and GitLab Mirrors implementation details
- Pager playbook
- Reference
Howto
Regular repositories
Creating a new repository
Creating a new top-level repository is not something that should be done often. The top-level repositories are all shown on the gitweb, and we'd like to keep the noise down. If you're not sure if you need a top-level repository then perhaps request a user repository first, and use that until you know you need a top-level repository.
Some projects, for example pluggable-transports, have a path hierarchy for their repositories. This should be encouraged to help keep this organised.
A request for a new top-level repository should include: the users that should have access to it, the repository name (including any folder it should live in), and a short description. If the users that should have access to this repository should be kept in sync with some other repository, a group might be created or re-used as part of the request.
For example:
Please create a new repository metrics/awesome-pipeline.git.
This should be accessible by the same set of users that have access to the
metrics-cloud repository.
The description for the repository is: Tor Metrics awesome pipeline repository.
This message was signed for trac.torproject.org on 2018-10-16 at 19:00:00 UTC.
The git team may ask for additional information to clarify the request if necessary, and may ask for replies to that information to be signed if they would affect the access to the repository. In the case that replies are to be signed, include the ticket number in the signed text to avoid replay attacks.
The git team member will edit the gitolite configuration to add a new block (alphabetically sorted within the configuration file) that looks like the following:
repo metrics-cloud
RW = @metrics-cloud
config hooks.email-enabled = true
config hooks.mailinglist = tor-commits@lists.torproject.org
config hooks.irc-enabled = true
config hooks.ircproject = or
config hooks.githuburl = torproject/metrics-cloud
config hooks.gitlaburl = torproject/metrics/metrics-cloud
metrics-cloud "The Tor Project" = "Configurations for Tor Metrics cloud orchestration"
Deconstructing this:
repo metrics-cloud
Starts a repository block.
RW = @metrics-cloud
Allows non-destructive read/write but not branch/tag deletion or non-fast-forward pushes. Alternatives would include "R" for read-only, or "RW+" to allow for destructive actions. We only allow destructive actions for user's personal repositories.
In this case, the permissions are delegated to a group (starting with @) and not an individual user.
config hooks.email-enabled = true
config hooks.mailinglist = tor-commits@lists.torproject.org
This enables the email hook to send one email per commit to the commits list. For all top-level repositories, the mailing list should be tor-commits@lists.torproject.org.
config hooks.irc-enabled = true
config hooks.ircproject = or
This enables the IRC hook to send one message per commit to an IRC channel. If the project is set to "or" the messages will be sent to #tor-bots.
config hooks.githuburl = torproject/metrics-cloud
config hooks.gitlaburl = torproject/metrics/metrics-cloud
These enable pushing a mirror to external services. The external service will have to be configured to accept these pushes, and we should avoid adding mirror URLs where things aren't configured yet so we don't trigger any IPS or abuse detection system by making loads of bad push attempts.
metrics-cloud "The Tor Project" = "Configurations for Tor Metrics cloud orchestration"
The last line of this file is what is used to provide configuration to gitweb. Starting with the path, then the owner, then the short description.
Upon push, the new repository will be created. It may take some minutes to appear on the gitweb. Do not fear, the old list that did not yet include the new repository has just been cached.
Push takes ages. Don't Ctrl-C it or you can end up in an inconsistent state. Just let it run. A future git team member might work on backgrounding the sync task.
Groups are defined at the top of the file, again in alphabetical order (not part of the repository block):
@metrics-cloud = karsten irl
Adding developers to a repository
If you want access to an existing repository please have somebody who already has access to ask that you be added by filing a trac ticket. This should be GPG signed as above.
Request a user be added to an existing repository
The git team member will either add a permissions line to the configuration for the repository or will add a username to the group, depending on how the repository is configured.
Deleting accidentally pushed tags/branches
These requests are for a destructive action and should be signed. You should also sanity check the request and not just blindly copy/paste the list of branch names.
The git team member will need to:
- Edit the gitolite configuration to allow RW+ access for the specified branch or tag.
- Push an empty reference to the remote reference to delete it. In doing this, all the hooks will run ensuring that the gitweb mirror and all other external mirrors are kept in sync.
- Revert the commit that gave the git team member this access.
The additional permission line will look something like:
RW+ refs/heads/travis-ci = irl
RW+ refs/tags/badtag-v1.0 = irl
This is to protect the git team member from accidentally deleting everything,
do not just give yourself RW+
permissions for the whole repository unless you
are feeling brave, even when someone has accidentally pushed their entire
history of personal branches to the canonical repository.
User repositories
Developers who have a tpo LDAP account can request personal git repositories be
created on our git infrastructure. Please file a ticket in Trac using the link
below. User repositories have the path user/<username>/<repository>.git
.
This request should contain: username, repository name, and a short description. Here is an example where irl is requesting a new example repository:
Please create a new user repository user/irl/example.git.
The description for the repository is: Iain's example repository.
This message was signed for trac.torproject.org on 2018-10-16 at 19:00:00 UTC.
Please use GPG to clearsign this text, it will be checked against the GPG key that you have linked to you in our LDAP. Additionally, ensure that it is wrapped as a code block (within !{{{ }}}).
There have not yet been any cases where user repositories have allowed access by other users than the owner. Let's keep it that way or this will get complicated.
Users will have full access to their own repos and can therefore delete branches, tags, and perform non-fast-forward pushes.
Learning what git repos you can read/write
Once you have an LDAP account and have an ssh key set up for it, run:
ssh git@git-rw.torproject.org
and it will tell you what bits you have on which repos. The first column is who can read (@ for everybody, R for you, blank for not you), and the second column is who can write (@ for everybody, W for you, blank for not you).
Commit hooks
There are a variety of commit hooks that are easy to add for your git repo, ranging from irc notifications to email notifications to github auto-syncing. Clone the gitolite-admin repo and look at the "config hooks" lines for examples. You can request changes by filing a trac ticket as described above, or just request the hooks when you first ask for your repo to be set up.
Hooks are stored in /srv/git.torproject.org/git-helpers
on the
server.
Standard Commit Hooks for Canonical Repositories
Changes to most repositories are reported to:
-
the #tor-bots IRC channel (or #tor-internal for private admin repositories)
-
Some repositories have a dedicated mailing list for commits at https://lists.torproject.org
Migrating a repository to GitLab
Moving a repository from Gitolite to GitLab proceeds in two parts. One part can be done by any user with access to GitLab. The second part needs to be done by TPA.
User part: importing the repository into GitLab
This is the part you need to do as a user to move to GitLab:
-
import the Gitolite repository in GitLab:
- create a new project
- pick the "Import project" button
- pick the "Repo by URL" button
- copy-paste the
https://git.torproject.org/...
Git Repository URL - pick a project name and namespace (should ideally match the original project as close as possible)
- add a description (again, matching the original from gitweb/gitolite)
- pick the "Create project" button
This will import the git repository into a new GitLab project.
-
if the repository is to be archived on GitLab, make it so in
Settings
->General
->Advanced
->Archive project
-
file a ticket with TPA to request a redirection. make sure you mention both the path to the gitolite and GitLab repositories
That's it, you are done! The remaining steps will be executed by TPA. (Note, if you are TPA, see the next section.)
Note that you can migrate multiple repositories at once by following those steps multiple times. In that case, create a single ticket for TPA with the before/after names, and how they should be handled.
For example, here's the table of repositories migrated by the applications team:
Gitolite | GitLab | fate |
---|---|---|
builders/tor-browser-build | tpo/applications/tor-browser-build | migrated |
builders/rbm | tpo/applications/rbm | migrated |
tor-android-service | tpo/applications/tor-android-service | migrated |
tor-browser | tpo/applications/tor-browser | migrated |
tor-browser-spec | tpo/applications/tor-browser-spec | migrated |
tor-launcher | tpo/applications/tor-launcher | archived |
torbutton | tpo/applications/torbutton | archived |
The above shows 5 repositories that have been migrated to GitLab and are still active, two that have be migrated and archived. There's a third possible fate that is "destroy" in which case TPA will simply mark the repository as inactive and will not migrate it.
Note the verb tense matters here: if the repository is marked as "migrated" or "archived", TPA will assume the repository has already been migrated and/or archived! It is your responsibility to do that migration, unless otherwise noted.
So if you do want TPA to actually migrate the repositories for you, please make that explicit in the issue and use the proper verb tenses.
See issue tpo/tpa/team#41181 for an example issue as well, although that one doesn't use the proper verb tenses
TPA part: lock down the repository and add redirections
This part handles the server side of things. It will import the
repository to GitLab, optionally archive it, install a pre-receive
hook in the Git repository to forbid pushes, redirections in the Git
web interfaces, and document the change in Gitolite.
This one fabric command should do it all:
fab -H cupani.torproject.org \
gitolite.migrate-repo \
--name "$PROJECT_NAME" \
--description "$PROJECT_DESCRIPTION"
--issue-url=$ISSUE_URL
--import-project \
$GITOLITE_REPO \
$GITLAB_PROJECT \
Example:
fab -H cupani.torproject.org \
gitolite.migrate-repo \
--name "letsencrypt-domains" \
--description "torproject letsencrypt domains"
--issue-url=https://gitlab.torproject.org/tpo/tpa/team/-/issues/41574 \
--import-project \
admin/letsencrypt-domains \
tpo/tpa/letsencrypt-domains \
If the repository is to be archived, you can also pass the --archive
flag.
Manual procedures
NOTE: This procedure is deprecated and replaced by the above "all in one" procedure.
The procedure is this simple two-step process:
-
(optional) triage the ticket with the labels Git and Gitweb, and the milestone %legacy Git infrastructure retirement (TPA-RFC-36)
-
run the following Fabric task:
fab -H cupani.torproject.org gitolite.migrate-repo \ $GITOLITE_REPO \ $GITLAB_PROJECT \ --issue-url=$GITLAB_ISSUE
For example, this is how the
gotlib
project was marked as migrated:fab -H cupani.torproject.org gitolite.migrate-repo \ pluggable-transports/goptlib \ tpo/anti-censorship/pluggable-transports/goptlib \ --issue-url=https://gitlab.torproject.org/tpo/tpa/team/-/issues/41182
The following changes are done by the Fabric task:
-
make an (executable)
pre-receive
hook ingit-rw
with an exit status of1
warning about the new code location -
in Puppet, add a line for this project in
modules/profile/files/git/gitolite2gitlab.txt
(intor-puppet.git
), for example:pluggable-transports/goptlib tpo/anti-censorship/pluggable-transports/goptlib
This ensures proper redirects are deployed on the Gitolite and GitWeb servers.
-
in Gitolite, mark the project as "Migrated to GitLab", for example
@@ -715,7 +715,7 @@ repo debian/goptlib config hooks.irc-enabled = true config hooks.ircproject = or config hooks.projectname = debian-goptlib - config gitweb.category = Packaging + config gitweb.category = Migrated to GitLab debian/goptlib "The Tor Project" = "Debian packaging for the goptlib pluggable transport library" repo debian/torproject-keyring
We were then manually importing the repository in GitLab with:
fab gitlab.create-project \
-p $GITLAB_PROJECT \
--name "$GITLAB_PROJECT_NAME" \
--import-url https://git.torproject.org/$GITOLITE_REPO.git \
--description "Archive from Gitolite: $GITOLITE_DESCRIPTION"
If the repository is to be archived in GitLab, also provide the
--archive
flag.
For example, this is an actual run:
fab gitlab.create-project \
-p tpo/tpa/dip \
--name "dip" \
--import-url https://git.torproject.org/admin/services/gitlab/dip.git \
--archive \
--description "Archive from Gitolite: Ansible recipe for running dip from debian salsa"
Migration to other servers
Some repositories were found to be too sensitive for GitLab. While some of the issues could be mitigated through Git repository integrity tricks, this was considered to be too time-consuming to respect the migration deadline.
So a handful of repositories were migrated directly to the affected servers. Those are:
- DNS services, moved to
nevii
, in/srv/dns.torproject.org/repositories/
-
dns/auto-dns
: DNS zones source used by LDAP server -
dns/dns-helpers
: DNSSEC generator used on DNS master -
dns/domains
: DNS zones source used by LDAP server -
dns/mini-nag
: monitoring on DNS primary
-
- Let's Encrypt, moved to
nevii
, in/srv/letsencrypt.torproject.org/repositories/
-
admin/letsencrypt-domains
: TLS certificates generation
-
- Monitoring, moved to
nagios
:-
tor-nagios
: Icinga configuration
-
- Passwords, moved to
pauli
:-
tor-passwords
: password manager
-
When the repositories required some action to happen on push (which is
all repositories except the password manager), a post-receive
hook
was implemented to match the original configuration.
They are all actual git repositories with working trees (as opposed to
bare repositories) to simplify the configuration (and avoid an
intermediate bare repository). Local changes are strongly discouraged,
the work tree is updated thanks to the
receive.denyCurrentBranch=updateInstead
configuration setting.
Destroying a repository
Instead of migrating a repository to GitLab, you might want to simply get rid of it. This can be relevant in case the repository is a duplicate, or it's a fork and all branches were merged, for example.
We generally prefer to archive repositories that said, so in general you should follow the migration procedure instead.
To destroy a repository:
-
file a ticket with TPA to request the destruction of the repository or repositories. make sure to explain why you believe the repositories can be destroyed.
-
if you're not TPA, you're done, wait for a response or requests for clarification. the rest of this procedure is relevant only for TPA
-
if you're TPA, examine the request thoroughly. make sure that:
-
the GitLab user requesting the destruction has access to the Gitolite repository. normally, usernames should generally match as LDAP users were imported when GitLab was created, but it's good to watch out for homograph attacks, for example
-
there's a reasonable explanation for the destruction, e.g. that no important data will actually be lost when the repository is destroyed
-
-
install a redirection and schedule destruction of the repository, with the command:
fab -H cupani.torproject.org gitolite.destroy-repo-scheduled --issue-url=$URL $REPOSITORY
for example, this is how the
user/nickm/githax
repository was disabled and scheduled for destruction:anarcat@angela:fabric-tasks$ fab -H cupani.torproject.org gitolite.destroy-repo-scheduled --issue-url=https://gitlab.torproject.org/tpo/tpa/team/-/issues/41219 admin/tor-virt.git INFO: preparing destroying of Gitolite repository admin/tor-virt in /srv/git.torproject.org/repositories/admin/tor-virt.git INFO: uploading 468 bytes to /srv/git.torproject.org/repositories/admin/tor-virt.git/hooks/pre-receive INFO: making /srv/git.torproject.org/repositories/admin/tor-virt.git/hooks/pre-receive executable INFO: scheduling destruction of /srv/git.torproject.org/repositories/admin/tor-virt.git in 30 days on cupani.torproject.org INFO: scheduling rm -rf "/srv/git.torproject.org/repositories/admin/tor-virt.git" to run on cupani.torproject.org in 30 days warning: commands will be executed using /bin/sh job 20 at Fri Apr 19 19:01:00 2024 INFO: scheduling destruction of /srv/gitweb.torproject.org/repositories/admin/tor-virt.git in 30 days on vineale.torproject.org INFO: scheduling rm -rf "/srv/gitweb.torproject.org/repositories/admin/tor-virt.git" to run on cupani.torproject.org in 30 days warning: commands will be executed using /bin/sh job 21 at Fri Apr 19 19:01:00 2024 INFO: modifying gitolite.conf to add "config gitweb.category = Scheduled for destruction" INFO: rewriting gitolite config /home/anarcat/src/tor/gitolite-admin/conf/gitolite.conf to change project admin/tor-virt to category Scheduled for destruction diff --git i/conf/gitolite.conf w/conf/gitolite.conf index dd3a79e..822be3e 100644 --- i/conf/gitolite.conf +++ w/conf/gitolite.conf @@ -1420,7 +1420,7 @@ repo admin/tor-virt #RW = @torproject-admin config hooks.irc-enabled = true config hooks.ircproject = tor-admin - config gitweb.category = Attic + config gitweb.category = Scheduled for destruction admin/tor-virt "The Tor Project" = "torproject's libvirt configuration" repo admin/buildbot-conf commit and push above changes in /home/anarcat/src/tor/gitolite-admin? [control-c abort, enter to continue] INFO: committing conf/gitolite.conf [master bd49f71] Repository admin/tor-virt scheduled for destruction 1 file changed, 1 insertion(+), 1 deletion(-) INFO: pushing in /home/anarcat/src/tor/gitolite-admin [...]
The very long gitolite output has been stripped above.
Mirroring a gitolite repository to GitLab
This procedure is DEPRECATED. Instead, consider migrating the repository to GitLab permanently or simply destroying the repository if its data is worthless.
This procedure is kept for historical purposes only.
-
import the Gitolite repository in GitLab:
- create a new project
- pick the "Import project" button
- pick the "Repo by URL" button
- copy-paste the
https://git.torproject.org/...
Git Repository URL - pick a project name and namespace (should ideally match the original project as close as possible)
- add a description (again, matching the original from gitweb/gitolite)
- pick the "Create project" button
This will import the git repository into a new GitLab project.
-
grant
Developer
access to the gitolite-merge-bot user in the project -
in Gitolite, add the GitLab project URL to enable the mirror hook, for example:
modified conf/gitolite.conf @@ -1502,6 +1502,7 @@ repo translation RW+ = emmapeel config hooks.irc-enabled = true config hooks.ircproject = or + config hooks.gitlaburl = tpo/web/translation translation "The Tor Project" = "Translations, one branch per project" repo translation-tools
In that example, the
translation.git
repository will push to thetpo/web/translation
mirror.
Mirroring a private git repository to GitLab's
If a repository is, for some reason (typically security), not hosted on GitLab, it can still be mirrored there. A typical example is the Puppet repository (see TPA-RFC-76).
The following instructions assume you are mirroring a private
repository from a host (alberti.torproject.org
in this case) where
users typically push in a sandbox user (git
in this case). We also
assume you have a local clone of the repository you can operate from.
-
Create the repository in GitLab, possibly private itself, this can be done by adding a remote and pushing from the local clone:
git remote add gitlab ssh://git@gitlab.torproject.org/tpo/tpa/account-keyring.git git push gitlab --mirror
-
Add the GitLab remote on the private repository (in this case on
alberti
, running asgit
:git remote add origin ssh://git@gitlab.torproject.org/tpo/tpa/account-keyring.git
-
Create a deploy key on the server (again, as
git@alberti
):ssh-keygen -t ed25519
-
Add the deploy key to the GitLab repository, in Settings, Repository, Deploy keys, make sure it has write access, and name it after the user on the mirrored host (e.g.
git@alberti.torproject.org
in this case) -
Protect the branch, in Settings, Repository, Protected branches:
- Allowed to merge: no one
- Allowed to push and merge: no one, and add the deploy key
-
Disable merge requests (in Settings, General) or set them to be "fast-forward only" (in Settings, Merge requests)
-
On the mirrored repository, add a
post-receive
hook like:
#!/bin/sh
echo "Pushing to GitLab..."
git push --mirror
If there's already a `post-receive` hook, add the `git` command to
the end of it.
- Test pushing to the mirrored repository, commits should end up on the GitLab mirror.
See also #41977 (closed) for an example where multiple repos were configured as such.
Archiving a repository
IMPORTANT: this procedure is DEPRECATED. Repositories archived on Gitolite still will be migrated to GitLab, follow the migration procedure instead. Note that even repositories that should be archived in Gitolite MUST be migrated to GitLab and then archived.
If a repository is not to be migrated or mirrored to GitLab (see below) but just archived, use the following procedure.
-
make an (executable)
pre-receive
hook ingit-rw
with an exit status of1
warning about the new code location, example:$ cat /srv/git.torproject.org/repositories/project/help/wiki.git/hooks/pre-receive #!/bin/sh cat <<EOF This repository has been archived and should not be used anymore. See this issue for details: https://gitlab.torproject.org/tpo/tpa/services/-/issues/TODO EOF exit 1
-
Make sure the hook is executable:
chmod +x hooks/pre-receive
-
in Gitolite, make the project part of the "Attic", for example
repo project/foo RW = anarcat - config gitweb.category = Old category -project/foo "The Tor Project" = "foo project" + config gitweb.category = Attic +project/foo "The Tor Project" = "foo project (deprecated)" repo project/bar RW = @jenkins-admins
The description
file in the repository should also be updated
similarly.
GitHub and GitLab Mirrors implementation details
Some repositories are mirrored to https://github.com/torproject organization and to the https://gitlab.torproject.org/ server, through gitolite hooks. See above on how to migrate and mirror such repositories to GitLab.
This used to be through a git push --mirror $REMOTE
command, but now
we do a git push --force $REMOTE '+refs/*:refs/*'
, because the
--mirror
argument was destroying merge requests on the GitLab
side. This, for example, is what you get with --mirror
:
user@tor-dev:~/src/gitlab.torproject.org/xxx/xxx$ git push --mirror git@gitlab.torproject.org:ahf/test-push-mirror.git --dry-run
To gitlab.torproject.org:ahf/test-push-mirror.git
dd75357..964d4c0 master -> master
- [deleted] test-branch
- [deleted] refs/merge-requests/1/head
- [deleted] refs/merge-requests/1/merge
This is exactly what we want to avoid: it correctly moves the master
branch forward, but the mirroring deletes the refs/merge-requests/*
content at the destination.
Instead with just --force
:
user@tor-dev:~/src/gitlab.torproject.org/xxx/xxx$ git push --force git@gitlab.torproject.org:ahf/test-push-mirror.git '+refs/*:refs/*' --dry-run
To gitlab.torproject.org:ahf/test-push-mirror.git
dd75357..964d4c0 master -> master
Here master gets moved forward properly, but we do not delete anything at the destination that is unknown at the source.
Adding --prune here would give the same behavior as git push --mirror:
user@tor-dev:~/src/gitlab.torproject.org/xxx/xxx$ git push --prune --force git@gitlab.torproject.org:ahf/test-push-mirror.git '+refs/*:refs/*' --dry-run
To gitlab.torproject.org:ahf/test-push-mirror.git
dd75357..964d4c0 master -> master
- [deleted] test-branch
- [deleted] refs/merge-requests/1/head
- [deleted] refs/merge-requests/1/merge
Since we move everything under refs/*
with the refspec we pass, this should include tags as well as branches.
The only downside of this approach is this: if a person pushes to Gitlab a branch that does not not exist on Gitolite, the branch will remain on Gitlab until it's manually deleted. That is fine: if the branch does exist, it will simply be overwritten next time Gitolite pushes to Gitlab.
See also bug 41 for a larger discussion on this solution.
Pager playbook
gitweb out of sync
If vineale is down for an extended period of time, it's a good idea to trigger a re-sync of all the repositories to ensure that the latest version is available to clone from the anonymous endpoints.
Create an empty commit in the gitolite-admin.git repository using:
git commit -m "trigger resync" --allow-empty
and push this commit. This will run through the post-commit hook that includes syncing everything.
Reference
Design
git-rw.torproject.org
, the writable git repository hosting, runs on
cupani.torproject.org
as the git user. Users in the gitolite (gid
1504) group can become the git user. The gitolite installation is
contained inside /srv/git.torproject.org
with the repositories being
found in the repositories
folder there.
The gitolite installation itself is not from Debian packages. It's a
manual install, in /srv/git.torproject.org/gitolite/src
, of an
extremely old version (v0.95-38-gb0ce84d
, december 2009).
Anonymous git and gitweb run on vineale.torproject.org
and as the gitweb
user. Users in the gitweb (gid 1505) group can become the gitweb user.
Data for these services can be found in /srv/gitweb.torproject.org
.
The gitolite configuration is found at
git@git-rw.torproject.org:gitolite-admin.git
and is not mirrored to gitweb.
The gitolite
group on the git-rw
server defined in LDAP and has
total control of the gitolite installation, as its members can sudo
to git.
The git
user gets redirected through the
/srv/git.torproject.org/gitolite/src/gl-auth-command
through the
/etc/ssh/userkeys/git
authorized_keys
file. This, in turn, gets
generated from LDAP, somewhere inside the ud-generate
command,
because exportOptions
is set to GITOLITE
on the cupani
host. All
users with a valid LDAP account get their SSH key added to the list
and only gitolite configuration restricts further access.
When a repository is pushed to, it gets synchronised to the gitweb
host on a post-receive
hook
(/srv/git.torproject.org/git-helpers/post-receive.d/00-sync-to-mirror
),
which calls .../git-helpers/tools/sync-repository
which just
rsync
's the repository over, if and only if the
git-daemon-export-ok
flag file is present. If it isn't, an empty
repository (/srv/git.torproject.org/empty-repository
) is
synchronized over, deleting the repository from the gitweb mirror.
Access to push to this repository is controlled by the
gitolite-admin
repository entry in the gitolite configuration file,
and not by LDAP groups.
Note that there is a /srv/git.torproject.org/projects.list
file that
contains a list of repositories. That file is defined in
/srv/git.torproject.org/etc/gitolite.rc
and is, in theory, the
entire list of projects managed by gitolite. In practice, it's not:
some (private?) projects are missing in there, but it's not clear why
exactly (for example, admin/trac/TracAccountManager
is not in there
even though it's got the git-daemon-export-ok
flag and is listed in
the gitolite.conf
file). This might be because of access controls
specifications in the gitolite.conf
file.
GitLab migration
As mentioned in the lead, the gitolite/gitweb infrastructure is, as of May 2021, considered legacy and users are encouraged to create new repositories, and migrate old ones to GitLab. In the intermediate period, repositories can be mirrored between gitolite and GitLab as well.
Security concerns
This section is a summary of the discussions that happened in tpo/tpa/gitlab#36 and tpo/tpa/gitlab#81.
Some developers expressed concerns about using GitLab as a canonical location for source code repositories, mainly because of the much broader attack surface GitLab provides, compared to the legacy, gitolite-based infrastructure, especially considering that the web application basically has write access to everything.
Of course, GitLab is larger, and if there's an unauthenticated attack against GitLab, that could compromise our repositories. And there is a stead flow of new vulnerabilities in GitLab (sorted by priority), including remote code execution. And although none of those provide unauthenticated code execution, our anonymous portal provides a bypass to that protection, so this is a real threat that must be addressed.
When we think about authenticated users, however, gitolite has a problem: our current gitolite install is pretty old, and (deliberately) does not follow new upstream releases. Great care has been taken to run a gitolite version that is specifically older, to ensure a smaller attack surface, because it has less features than newer gitolite versions. That's why it's such a weird version.
It is worrisome that we use an old version of the software that is essentially unmaintained. It is technical debt that makes maintenance harder. It's true that this old gitolite has a much smalller attack surface than gitlab (or even more recent gitolite), but the chosen approach to fix this problem has to do with having other mechanisms to ensure code integrity (code signing and supply chain integrity) or secrecy (ie. encrypted repositories) than trusting the transport.
We are actively maintaining gitlab, following upstream releases quite closely. Upstream is actively auditing their code base, and many vulnerabilities published are actually a result of those internal audits.
If we are worried about trust in our supply chain, GitLab security is only part of the problem. It's a problem that currently exists with Gitolite. For example, what happens if a developer's laptop gets compromised? How do we audit changes to gitolite repositories, assuming it's not compromised? GitLab provides actually more possibilities for such audits. Solutions like code reviews, signed commits, reproducible builds, and transparency logs provide better, long-term and service-agnostic solutions to those problems.
In the end, it came up to a trade-off: GitLab is much easier to use. Convenience won over hardened security, especially considering the cost of running two services in parallel. Or, as Nick Mathewson put it:
I'm proposing that, since this is an area where the developers would need to shoulder most of the burden, the development teams should be responsible for coming up with solutions that work for them on some reasonable timeframe, and that this shouldn't be admin's problem assuming that the timeframe is long enough.
For now, the result of that discussion is a summary of git repository integrity solutions, which is therefore delegated to teams.
Migration roadmap
TODO.
Issues
There is no issue tracker specifically for this project, File or search for issues in the team issue tracker, with the ~Git label.