git.md

git.torproject.org and associated services

• git.torproject.org and associated services
  • Overview
  • Regular repositories
    • Creating a new repository
    • Adding developers to a repository
    • Deleting accidentally pushed tags/branches
  • User repositories
  • Learning what git repos you can read/write
  • Commit hooks
    • Standard Commit Hooks for Canonical Repositories
    • GitHub and GitLab Mirrors
  • Archiving a repository
  • Admin details
  • Potential problems to solve
    • gitweb out of sync

Overview

Our git setup consists of three interdependent services:

• git-rw.torproject.org: ssh accessible and writeable git repositories
• git.torproject.org: https:// read-only anonymous access
• gitweb.torproject.org: web browsing repositories

When a developer pushes to git-rw, the repository is mirrored to git and so made available via the gitweb service.

Regular repositories

Creating a new repository

Creating a new top-level repository is not something that should be done often. The top-level repositories are all shown on the gitweb, and we'd like to keep the noise down. If you're not sure if you need a top-level repository then perhaps request a user repository first, and use that until you know you need a top-level repository.

Some projects, for example pluggable-transports, have a path hierachy for their repositories. This should be encouraged to help keep this organised.

A request for a new top-level repository should include: the users that should have access to it, the repository name (including any folder it should live in), and a short description. If the users that should have access to this repository should be kept in sync with some other repository, a group might be created or re-used as part of the request.

For example:

Please create a new repository metrics/awesome-pipeline.git.

This should be accessible by the same set of users that have access to the
metrics-cloud repository.

The description for the repository is: Tor Metrics awesome pipeline repository.

This message was signed for trac.torproject.org on 2018-10-16 at 19:00:00 UTC.

The git team may ask for additional information to clarify the request if necessary, and may ask for replies to that information to be signed if they would affect the access to the repository. In the case that replies are to be signed, include the ticket number in the signed text to avoid replay attacks.

The git team member will edit the gitolite configuration to add a new block (alphabetically sorted within the configuration file) that looks like the following:

repo metrics-cloud
    RW                                       = @metrics-cloud
    config hooks.email-enabled               = true
    config hooks.mailinglist                 = tor-commits@lists.torproject.org
    config hooks.irc-enabled                 = true
    config hooks.ircproject                  = or
    config hooks.githuburl                   = torproject/metrics-cloud
    config hooks.gitlaburl                   = torproject/metrics/metrics-cloud
metrics-cloud "The Tor Project" = "Configurations for Tor Metrics cloud orchestration"

Deconstructing this:

repo metrics-cloud

Starts a repository block.

    RW                                       = @metrics-cloud

Allows non-destructive read/write but not branch/tag deletion or non-fast-forward pushes. Alternatives would include "R" for read-only, or "RW+" to allow for destructive actions. We only allow destructive actions for user's personal repositories.

In this case, the permissions are delegated to a group (starting with @) and not an individual user.

    config hooks.email-enabled               = true
    config hooks.mailinglist                 = tor-commits@lists.torproject.org

This enables the email hook to send one email per commit to the commits list. For all top-level repositories, the mailing list should be tor-commits@lists.torproject.org.

    config hooks.irc-enabled                 = true
    config hooks.ircproject                  = or

This enables the IRC hook to send one message per commit to an IRC channel. If the project is set to "or" the messages will be sent to #tor-bots.

    config hooks.githuburl                   = torproject/metrics-cloud
    config hooks.gitlaburl                   = torproject/metrics/metrics-cloud

These enable pushing a mirror to external services. The external service will have to be configured to accept these pushes, and we should avoid adding mirror URLs where things aren't configured yet so we don't trigger any IPS or abuse detection system by making loads of bad push attempts.

metrics-cloud "The Tor Project" = "Configurations for Tor Metrics cloud orchestration"

The last line of this file is what is used to provide configuration to gitweb. Starting with the path, then the owner, then the short description.

Upon push, the new repository will be created. It may take some minutes to appear on the gitweb. Do not fear, the old list that did not yet include the new repository has just been cached.

Push takes ages. Don't Ctrl-C it or you can end up in an inconsistent state. Just let it run. A future git team member might work on backgrounding the sync task.

Groups are defined at the top of the file, again in alphabetical order (not part of the repository block):

@metrics-cloud                               = karsten irl

Adding developers to a repository

If you want access to an existing repository please have somebody who already has access to ask that you be added by filing a trac ticket. This should be GPG signed as above.

Request a user be added to an existing repository

The git team member will either add a permissions line to the configuration for the repository or will add a username to the group, depending on how the repository is configured.

Deleting accidentally pushed tags/branches

These requests are for a destructive action and should be signed. You should also sanity check the request and not just blindly copy/paste the list of branch names.

The git team member will need to:

1. Edit the gitolite configuration to allow RW+ access for the specified branch or tag.
2. Push an empty reference to the remote reference to delete it. In doing this, all the hooks will run ensuring that the gitweb mirror and all other external mirrors are kept in sync.
3. Revert the commit that gave the git team member this access.

The additional permission line will look something like:

    RW+ refs/heads/travis-ci                = irl
    RW+ refs/tags/badtag-v1.0               = irl

This is to protect the git team member from accidentally deleting everything, do not just give yourself RW+ permissions for the whole repository unless you are feeling brave, even when someone has accidentally pushed their entire history of personal branches to the canonical repository.

User repositories

Developers who have a tpo LDAP account can request personal git repositories be created on our git infrastructure. Please file a ticket in Trac using the link below. User repositories have the path user/<username>/<repository>.git.

Request a new user repository

This request should contain: username, repository name, and a short description. Here is an example where irl is requesting a new example repository:

Please create a new user repository user/irl/example.git.

The description for the repository is: Iain's example repository.

This message was signed for trac.torproject.org on 2018-10-16 at 19:00:00 UTC.

Please use GPG to clearsign this text, it will be checked against the GPG key that you have linked to you in our LDAP. Additionally, ensure that it is wrapped as a code block (within !{{{ }}}).

There have not yet been any cases where user repositories have allowed access by other users than the owner. Let's keep it that way or this will get complicated.

Users will have full access to their own repos and can therefore delete branches, tags, and perform non-fast-forward pushes.

Learning what git repos you can read/write

Once you have an LDAP account and have an ssh key set up for it, run:

ssh git@git-rw.torproject.org

and it will tell you what bits you have on which repos. The first column is who can read (@ for everybody, R for you, blank for not you), and the second column is who can write (@ for everybody, W for you, blank for not you).

Commit hooks

There are a variety of commit hooks that are easy to add for your git repo, ranging from irc notifications to email notifications to github auto-syncing. Clone the gitolite-admin repo and look at the "config hooks" lines for examples. You can request changes by filing a trac ticket as described above, or just request the hooks when you first ask for your repo to be set up.

Hooks are stored in /srv/git.torproject.org/git-helpers on the server.

Standard Commit Hooks for Canonical Repositories

Changes to most repositories are reported to:

• the #tor-bots IRC channel (or #tor-internal for private admin repositories)

• Some repositories have a dedicated mailing list for commits at https://lists.torproject.org

GitHub and GitLab Mirrors

Some repositories are mirrored to https://github.com/torproject organization and to the https://gitlab.torproject.org/ server, through gitlite hooks.

This used to be through a git push --mirror $REMOTE command, but now we do a git push --force $REMOTE '+refs/*:refs/*', because the --mirror argument was destroying merge requests on the GitLab side. This, for example, is what you get with --mirror:

user@tor-dev:~/src/gitlab.torproject.org/xxx/xxx$ git push --mirror git@gitlab.torproject.org:ahf/test-push-mirror.git --dry-run
To gitlab.torproject.org:ahf/test-push-mirror.git
   dd75357..964d4c0  master -> master
 - [deleted]         test-branch
 - [deleted]         refs/merge-requests/1/head
 - [deleted]         refs/merge-requests/1/merge

This is exactly what we want to avoid: it correctly moves the master branch forward, but the mirroring deletes the refs/merge-requests/* content at the destination.

Instead with just --force:

user@tor-dev:~/src/gitlab.torproject.org/xxx/xxx$ git push --force git@gitlab.torproject.org:ahf/test-push-mirror.git '+refs/*:refs/*' --dry-run
To gitlab.torproject.org:ahf/test-push-mirror.git
   dd75357..964d4c0  master -> master

Here master gets moved forward properly, but we do not delete anything at the destination that is unknown at the source.

Adding --prune here would give the same behavior as git push --mirror:

user@tor-dev:~/src/gitlab.torproject.org/xxx/xxx$ git push --prune --force git@gitlab.torproject.org:ahf/test-push-mirror.git '+refs/*:refs/*' --dry-run
To gitlab.torproject.org:ahf/test-push-mirror.git
   dd75357..964d4c0  master -> master
 - [deleted]         test-branch
 - [deleted]         refs/merge-requests/1/head
 - [deleted]         refs/merge-requests/1/merge

Since we move everything under refs/* with the refspec we pass, this should include tags as well as branches.

The only downside of this approach is this: if a person pushes to Gitlab a branch that does not not exist on Gitolite, the branch will remain on Gitlab until it's manually deleted. That is fine: if the branch does exist, it will simply be overwritten next time Gitolite pushes to Gitlab.

See also bug 41 for a larger discussion on this solution.

Archiving a repository

To archive a repository, it must first be deactivated by adding a pre-receive hook, e.g. in:

/srv/git.torproject.org/repositories/PROJECT.git/hooks/pre-receive 

The script should display some sort of warning and exit with an error, for example:

#!/bin/sh

cat <<EOF
This repository is deprecated and should not be used anymore.

See this issue for details:

https://gitlab.torproject.org/tpo/tpa/team/-/issues/9999999
EOF

exit 1

Then in Gitolite, the repository must be put "in the Attic", using a patch that looks something like this:

 repo project/foo
     RW                                       = anarcat
-    config gitweb.category                   = Old category
-project/foo "The Tor Project" = "foo project"
+    config gitweb.category                   = Attic
+project/foo "The Tor Project" = "foo project (deprecated)"
 
 repo project/bar
     RW                                       = @jenkins-admins

The description file in the repository should also be updated similarly.

Note that repos can also be migrated (and archived) to GitLab, see the GitLab migration procedure for details.

Admin details

git-rw runs on cupani.torproject.org and runs as the git user. Users in the gitolite (gid 1504) group can become the git user. The gitolite installation is contained inside /srv/git.torproject.org with the repositories being found in the repositories folder there.

The gitolite installation itself is not from Debian packages. It's a manual install, in /srv/git.torproject.org/gitolite/src, of an extremely old version (v0.95-38-gb0ce84d, december 2009).

Anonymous git and gitweb run on vineale.torproject.org and as the gitweb user. Users in the gitweb (gid 1505) group can become the gitweb user. Data for these services can be found in /srv/gitweb.torproject.org.

The gitolite configuration is found at git@git-rw.torproject.org:gitolite-admin.git and is not mirrored to gitweb.

The gitolite group on the git-rw server defined in LDAP and has total control of the gitolite installation, as its members can sudo to git.

The git user gets redirected through the /srv/git.torproject.org/gitolite/src/gl-auth-command through the /etc/ssh/userkeys/git authorized_keys file. This, in turn, gets generated from LDAP, somewhere inside the ud-generate command, because exportOptions is set to GITOLITE on the cupani host. All users with a valid LDAP account get their SSH key added to the list and only gitolite configuration restricts further access.

Access to push to this repository is controlled by the gitolite-admin repository entry in the gitolite configuration file, and not by LDAP groups.

Potential problems to solve

gitweb out of sync

If vineale is down for an extended period of time, it's a good idea to trigger a re-sync of all the repositories to ensure that the latest version is available to clone from the anonymous endpoints.

Create an empty commit in the gitolite-admin.git repository using:

git commit -m "trigger resync" --allow-empty

and push this commit. This will run through the post-commit hook that includes syncing everything.