Our git setup consists of three interdependent services:

 * `git-rw.torproject.org`:  ssh accessible and writeable git repositories
 * <https://git.torproject.org>: read-only anonymous access
 * <https://gitweb.torproject.org/>: web browsing repositories

When a developer pushes to git-rw, the repository is mirrored to git and so
made available via the gitweb service.

Some repositories are also mirrored to [GitLab](howto/gitlab) and we are
[considering migrating all repositories there](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/36). At the time of
writing (May 2021), the git infrastructure documented in this wiki
page is considered **legacy** and new repositories should instead be
created on GitLab.

[[_TOC_]]

# Howto

## Regular repositories

### Creating a new repository

Creating a new top-level repository is not something that should be done often.
The top-level repositories are all shown on the gitweb, and we'd like to keep
the noise down. If you're not sure if you need a top-level repository then
perhaps request a user repository first, and use that until you know you need
a top-level repository.

Some projects, for example pluggable-transports, have a path hierachy for their
repositories. This should be encouraged to help keep this organised.

A request for a new top-level repository should include: the users that should
have access to it, the repository name (including any folder it should live in),
and a short description. If the users that should have access to this repository
should be kept in sync with some other repository, a group might be created or
re-used as part of the request.

For example:

```
Please create a new repository metrics/awesome-pipeline.git.

This should be accessible by the same set of users that have access to the
metrics-cloud repository.

The description for the repository is: Tor Metrics awesome pipeline repository.

This message was signed for trac.torproject.org on 2018-10-16 at 19:00:00 UTC.
```

The git team may ask for additional information to clarify the request if
necessary, and may ask for replies to that information to be signed if they
would affect the access to the repository. In the case that replies are to be
signed, include the ticket number in the signed text to avoid replay attacks.

The git team member will edit the gitolite configuration to add a new block
(alphabetically sorted within the configuration file) that looks like the
following:

```
repo metrics-cloud
    RW                                       = @metrics-cloud
    config hooks.email-enabled               = true
    config hooks.mailinglist                 = tor-commits@lists.torproject.org
    config hooks.irc-enabled                 = true
    config hooks.ircproject                  = or
    config hooks.githuburl                   = torproject/metrics-cloud
    config hooks.gitlaburl                   = torproject/metrics/metrics-cloud
metrics-cloud "The Tor Project" = "Configurations for Tor Metrics cloud orchestration"
```

Deconstructing this:

```
repo metrics-cloud
```

Starts a repository block.

```
    RW                                       = @metrics-cloud
```

Allows non-destructive read/write but not branch/tag deletion or
non-fast-forward pushes. Alternatives would include "R" for read-only, or
"RW+" to allow for destructive actions. We only allow destructive actions for
user's personal repositories.

In this case, the permissions are delegated to a group (starting with @) and
not an individual user.

```
    config hooks.email-enabled               = true
    config hooks.mailinglist                 = tor-commits@lists.torproject.org
```

This enables the email hook to send one email per commit to the commits list.
For all top-level repositories, the mailing list should be
tor-commits@lists.torproject.org.

```
    config hooks.irc-enabled                 = true
    config hooks.ircproject                  = or
```

This enables the IRC hook to send one message per commit to an IRC channel. If
the project is set to "or" the messages will be sent to #tor-bots.

```
    config hooks.githuburl                   = torproject/metrics-cloud
    config hooks.gitlaburl                   = torproject/metrics/metrics-cloud
```

These enable pushing a mirror to external services. The external service will
have to be configured to accept these pushes, and we should avoid adding
mirror URLs where things aren't configured yet so we don't trigger any IPS or
abuse detection system by making loads of bad push attempts.

```
metrics-cloud "The Tor Project" = "Configurations for Tor Metrics cloud orchestration"
```

The last line of this file is what is used to provide configuration to gitweb.
Starting with the path, then the owner, then the short description.

Upon push, the new repository will be created. It may take some minutes to
appear on the gitweb. Do not fear, the old list that did not yet include the
new repository has just been cached.

Push takes ages. Don't Ctrl-C it or you can end up in an inconsistent state.
Just let it run. A future git team member might work on backgrounding the
sync task.

Groups are defined at the top of the file, again in alphabetical order
(not part of the repository block):

```
@metrics-cloud                               = karsten irl
```

### Adding developers to a repository

If you want access to an existing repository please have somebody who already
has access to ask that you be added by filing a trac ticket. This should be GPG
signed as above.

[Request a user be added to an existing repository](https://trac.torproject.org/projects/tor/newticket?summary=Please%20add%20%3Cusername%3E%20to%20repository%20%3Crepository%3E.git&component=Internal%20Services/Service%20-%20git&type=task)

The git team member will either add a permissions line to the configuration for
the repository or will add a username to the group, depending on how the
repository is configured.

### Deleting accidentally pushed tags/branches

These requests are for a destructive action and should be signed. You should
also sanity check the request and not just blindly copy/paste the list of
branch names.

The git team member will need to:

1. Edit the gitolite configuration to allow RW+ access for the specified branch
   or tag.
2. Push an empty reference to the remote reference to delete it. In doing this,
   all the hooks will run ensuring that the gitweb mirror and all other external
   mirrors are kept in sync.
3. Revert the commit that gave the git team member this access.

The additional permission line will look something like:

```
    RW+ refs/heads/travis-ci                = irl
    RW+ refs/tags/badtag-v1.0               = irl
```

This is to protect the git team member from accidentally deleting everything,
do not just give yourself `RW+` permissions for the whole repository unless you
are feeling brave, even when someone has accidentally pushed their entire
history of personal branches to the canonical repository.

## User repositories

Developers who have a tpo LDAP account can request personal git repositories be
created on our git infrastructure. Please file a ticket in Trac using the link
below. User repositories have the path `user/<username>/<repository>.git`.

[Request a new user repository](https://trac.torproject.org/projects/tor/newticket?summary=Please%20create%20new%20repository%20user/%3Cusername%3E/%3Crepository%3E.git&component=Internal%20Services/Service%20-%20git&type=task)

This request should contain: username, repository name, and a short
description. Here is an example where irl is requesting a new example
repository:

```
Please create a new user repository user/irl/example.git.

The description for the repository is: Iain's example repository.

This message was signed for trac.torproject.org on 2018-10-16 at 19:00:00 UTC.
```

Please use GPG to clearsign this text, it will be checked against the GPG key
that you have linked to you in our LDAP. Additionally, ensure that it is
wrapped as a code block (within !{{{ }}}).

There have not yet been any cases where user repositories have allowed access
by other users than the owner. Let's keep it that way or this will get
complicated.

Users will have full access to their own repos and can therefore delete
branches, tags, and perform non-fast-forward pushes.

## Learning what git repos you can read/write

Once you have an LDAP account and have an ssh key set up for it, run:

```
ssh git@git-rw.torproject.org
```

and it will tell you what bits you have on which repos. The first column is who
can read (@ for everybody, R for you, blank for not you), and the second column
is who can write (@ for everybody, W for you, blank for not you).

## Commit hooks

There are a variety of commit hooks that are easy to add for your git repo,
ranging from irc notifications to email notifications to github auto-syncing.
Clone the gitolite-admin repo and look at the "config hooks" lines for
examples. You can request changes by filing a trac ticket as described above,
or just request the hooks when you first ask for your repo to be set up.

Hooks are stored in `/srv/git.torproject.org/git-helpers` on the
server.

### Standard Commit Hooks for Canonical Repositories

Changes to most repositories are reported to:

 * the #tor-bots IRC channel (or #tor-internal for private admin repositories)

 * Some repositories have a dedicated mailing list for commits at https://lists.torproject.org

### How to migrate a Git repository from legacy to GitLab?

Important: this policy is still being debated. It is not clear if any
or all repositories should be migrated to GitLab, see [issue 36][]
for the discussion on this topic.

As an example of a repository migration, I have moved the wiki from
gitolite to gitlab just now. I have followed the following procedure:

 1. import the Gitolite repository in GitLab:
 
    * [create a new project](https://gitlab.torproject.org/projects/new)
    * pick the "Import project" button
    * pick the "Repo by URL" button
    * copy-paste the `https://git.torproject.org/...` Git Repository
      URL
    * pick a project name and namespace (should ideally match the
      original project as close as possible)
    * add a description (again, matching the original from
      gitweb/gitolite)
    * pick the "Create project" button
 
    This will import the git repository into a new GitLab project.

 3. if the repository is to be archived on GitLab, make it so in
    `Settings` -> `General` -> `Advanced` -> `Archive project`

 4. make an (executable) `pre-receive` hook in `git-rw` with an exit
    status of `1` warning about the new code location, example:

        $ cat /srv/git.torproject.org/repositories/project/help/wiki.git/hooks/pre-receive 
        #!/bin/sh

        cat <<EOF
        This repository has been migrated to GitLab:

        https://gitlab.torproject.org/tpo/tpa/services/-/wikis/home

        Update your remotes to:

            git@gitlab.torproject.org:tpo/tpa/services.wiki.git

        or:

            https://gitlab.torproject.org/tpo/tpa/services.wiki.git

        See this issue for details:

        https://gitlab.torproject.org/tpo/tpa/services/-/issues/34437
        EOF

        exit 1

    or in the case of a fully archived repository (non-writable):
    
        $ cat /srv/git.torproject.org/repositories/project/help/infra.git/hooks/pre-receive 
        #!/bin/sh

        cat <<EOF
        This repository has been migrated to GitLab:

        https://gitlab.torproject.org/tpo/tpa/wiki-infra-archive

        We have migrated away from ikiwiki so it is not necessary anymore.

        See this issue for details:

        https://gitlab.torproject.org/tpo/tpa/services/-/issues/34437
        EOF

        exit 1

 4. in Gitolite, make the project part of the "Attic", for example

        @@ -328,13 +328,13 @@ admin/trac/TracAccountManager "The Tor Project" = "Tor specific changes to Matth
         
         repo project/help/infra
             RW+                                      = @torproject-admin
        -    config gitweb.category                   = Infrastructure and Administration
        -project/help/infra "The Tor Project" = "help.torproject.org infrastructure"
        +    config gitweb.category                   = Attic
        +project/help/infra "The Tor Project" = "help.torproject.org infrastructure (archived to GitLab: https://gitlab.torproject.org/tpo/tpa/wiki-infra-archive')"
         
         repo project/help/wiki
             RW                                       = anarcat
        -    config gitweb.category                   = Infrastructure and Administration
        -project/help/wiki "The Tor Project" = "help.torproject.org content"
        +    config gitweb.category                   = Attic
        +project/help/wiki "The Tor Project" = "help.torproject.org content (archived to GitLab: https://gitlab.torproject.org/tpo/tpa/wiki-archive')"
         
         repo project/jenkins/jobs
             RW                                       = @jenkins-admins

The only downside with that approach is that a git clone will not warn
about the project redirection, but I am not sure there's a way to fix
that.

See [issue 36][] for further discussion.

### Mirroring a gitolite repository to GitLab

Important: this policy is still being debated. It is not clear if any
or all repositories should be mirror or just migrated to GitLab, see
[issue 36][] for the discussion on this topic.

[issue 36]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40472

Also consider [migrating the repository to GitLab permanently](#how-to-migrate-a-git-repository-from-legacy-to-gitlab).

 1. import the Gitolite repository in GitLab:

    * [create a new project](https://gitlab.torproject.org/projects/new)
    * pick the "Import project" button
    * pick the "Repo by URL" button
    * copy-paste the `https://git.torproject.org/...` Git Repository
      URL
    * pick a project name and namespace (should ideally match the
      original project as close as possible)
    * add a description (again, matching the original from
      gitweb/gitolite)
    * pick the "Create project" button

    This will import the git repository into a new GitLab project.

 2. grant `Developer` access to the [gitolite-merge-bot](https://gitlab.torproject.org/gitolite-merge-bot) user in
    the project

 2. in Gitolite, add the GitLab project URL to enable the mirror hook,
    for example:

        modified   conf/gitolite.conf
        @@ -1502,6 +1502,7 @@ repo translation
             RW+                                      = emmapeel
             config hooks.irc-enabled                 = true
             config hooks.ircproject                  = or
        +    config hooks.gitlaburl                   = tpo/web/translation
         translation "The Tor Project" = "Translations, one branch per project"
         
         repo translation-tools

    In that example, the `translation.git` repository will push to the
    `tpo/web/translation` mirror.

### GitHub and GitLab Mirrors implementation details

Some repositories are mirrored to https://github.com/torproject
organization and to the https://gitlab.torproject.org/ server, through
gitolite hooks. See above on how to [migrate](#how-to-migrate-a-git-repository-from-legacy-to-gitlab) and [mirror](#how-to-migrate-a-git-repository-from-legacy-to-gitlab) such
repositories to GitLab.

This used to be through a `git push --mirror $REMOTE` command, but now
we do a `git push --force $REMOTE '+refs/*:refs/*'`, because the
`--mirror` argument was destroying merge requests on the GitLab
side. This, for example, is what you get with `--mirror`:

    user@tor-dev:~/src/gitlab.torproject.org/xxx/xxx$ git push --mirror git@gitlab.torproject.org:ahf/test-push-mirror.git --dry-run
    To gitlab.torproject.org:ahf/test-push-mirror.git
       dd75357..964d4c0  master -> master
     - [deleted]         test-branch
     - [deleted]         refs/merge-requests/1/head
     - [deleted]         refs/merge-requests/1/merge

This is exactly what we want to avoid: it correctly moves the master
branch forward, but the mirroring deletes the `refs/merge-requests/*`
content at the destination.

Instead with just `--force`:

    user@tor-dev:~/src/gitlab.torproject.org/xxx/xxx$ git push --force git@gitlab.torproject.org:ahf/test-push-mirror.git '+refs/*:refs/*' --dry-run
    To gitlab.torproject.org:ahf/test-push-mirror.git
       dd75357..964d4c0  master -> master

Here master gets moved forward properly, but we do not delete anything
at the destination that is unknown at the source.

Adding --prune here would give the same behavior as git push --mirror:

    user@tor-dev:~/src/gitlab.torproject.org/xxx/xxx$ git push --prune --force git@gitlab.torproject.org:ahf/test-push-mirror.git '+refs/*:refs/*' --dry-run
    To gitlab.torproject.org:ahf/test-push-mirror.git
       dd75357..964d4c0  master -> master
     - [deleted]         test-branch
     - [deleted]         refs/merge-requests/1/head
     - [deleted]         refs/merge-requests/1/merge

Since we move everything under `refs/*` with the refspec we pass, this should include tags as well as branches.

The only downside of this approach is this: if a person pushes to
Gitlab a branch that does not not exist on Gitolite, the branch will
remain on Gitlab until it's manually deleted. That is fine: if the
branch does exist, it will simply be overwritten next time Gitolite
pushes to Gitlab.

See also [bug 41](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/41#note_2685994) for a larger discussion on this solution.

## Archiving a repository

To archive a repository, it must first be deactivated by adding a
`pre-receive` hook, e.g. in:

    /srv/git.torproject.org/repositories/PROJECT.git/hooks/pre-receive 

The script should display some sort of warning and exit with an error,
for example:

```
#!/bin/sh

cat <<EOF
This repository is deprecated and should not be used anymore.

See this issue for details:

https://gitlab.torproject.org/tpo/tpa/team/-/issues/9999999
EOF

exit 1
```

Make sure the hook is executable:

    chmod +x hooks/pre-receive

Then in Gitolite, the repository must be put "in the Attic", using a
patch that looks something like this:

```
 repo project/foo
     RW                                       = anarcat
-    config gitweb.category                   = Old category
-project/foo "The Tor Project" = "foo project"
+    config gitweb.category                   = Attic
+project/foo "The Tor Project" = "foo project (deprecated)"
 
 repo project/bar
     RW                                       = @jenkins-admins
```

The `description` file in the repository should also be updated
similarly.

Note that repos can also be migrated (and archived) to GitLab, see
[the GitLab migration procedure](howto/gitlab/#how-to-migrate-a-git-repository-from-legacy-to-gitlab) for details.

## Pager playbook

### gitweb out of sync

If vineale is down for an extended period of time, it's a good idea to trigger
a re-sync of all the repositories to ensure that the latest version is available
to clone from the anonymous endpoints.

Create an empty commit in the gitolite-admin.git repository using:

```
git commit -m "trigger resync" --allow-empty
```

and push this commit. This will run through the post-commit hook that includes
syncing everything.

# Reference

## Design

git-rw runs on `cupani.torproject.org` and runs as the git user. Users in the
gitolite (gid 1504) group can become the git user. The gitolite installation
is contained inside `/srv/git.torproject.org` with the repositories being found
in the `repositories` folder there.

The gitolite installation itself is *not* from Debian packages. It's a
manual install, in `/srv/git.torproject.org/gitolite/src`, of an
extremely old version (`v0.95-38-gb0ce84d`, december 2009).

Anonymous git and gitweb run on `vineale.torproject.org` and as the gitweb
user. Users in the gitweb (gid 1505) group can become the gitweb user.
Data for these services can be found in `/srv/gitweb.torproject.org`.

The gitolite configuration is found at
`git@git-rw.torproject.org:gitolite-admin.git` and is not mirrored to gitweb.

The `gitolite` group on the `git-rw` server defined in LDAP and has
total control of the gitolite installation, as its members can `sudo`
to git.

The `git` user gets redirected through the
`/srv/git.torproject.org/gitolite/src/gl-auth-command` through the
`/etc/ssh/userkeys/git` `authorized_keys` file. This, in turn, gets
generated from LDAP, somewhere inside the `ud-generate` command,
because `exportOptions` is set to `GITOLITE` on the `cupani` host. All
users with a valid LDAP account get their SSH key added to the list
and only gitolite configuration restricts further access.

Access to push to this repository is controlled by the
`gitolite-admin` repository entry in the gitolite configuration file,
and not by LDAP groups.

## GitLab migration

As mentioned in the lead, the gitolite/gitweb infrastructure is, as of
May 2021, considered *legacy* and users are encouraged to create new
repositories, and migrate old ones to GitLab. In the intermediate
period, repositories can be [mirrored between gitolite and GitLab](#github-and-gitlab-mirrors)
as well.

### Security concerns

> This section is a summary of the discussions that happened in [tpo/tpa/gitlab#36](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/36) and [tpo/tpa/gitlab#81](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81).

Some developers expressed concerns about using GitLab as a canonical
location for source code repositories, mainly because of the much
broader attack surface GitLab provides, compared to the legacy,
[gitolite-based infrastructure](howto/git), especially considering that the
web application basically has write access to everything.

Of course, GitLab is larger, and if there's an unauthenticated attack
against GitLab, that could compromise our respositories. And there is
a stead flow of new [vulnerabilities in GitLab](https://www.cvedetails.com/vulnerability-list/vendor_id-13074/Gitlab.html) ([sorted by
priority](https://www.cvedetails.com/vulnerability-list.php?vendor_id=13074&product_id=&version_id=&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&month=0&cweid=0&order=3&trc=178&sha=fe38ca18c40b857201e9ae9283dea03b71b724f0)), including remote code execution. And although none of
those provide *unauthenticated* code execution, our anonymous portal
provides a bypass to that protection, so this is a real threat that
must be addressed.

When we think about authenticated users, however, gitolite has a
problem: our current gitolite install is pretty old, and
(deliberately) does not follow new upstream releases. Great care has
been taken to run a gitolite version that is specifically older, to
ensure a smaller attack surface, because it has less features than
newer gitolite versions. That's why it's such a weird version.

It is worrisome that we use an old version of the software that is
essentially unmaintained. It is technical debt that makes maintenance
harder. It's true that this old gitolite has a much smalller attack
surface than gitlab (or even more recent gitolite), but the chosen
approach to fix this problem has to do with having other mechanisms to
ensure code integrity (code signing and supply chain integrity) or
secrecy (ie. encrypted repositories) than trusting the transport.

We are actively maintaining gitlab, following upstream releases quite
closely. Upstream is actively auditing their code base, and many
vulnerabilities published are actually a result of those internal
audits. 

If we are worried about trust in our supply chain, GitLab security is
only part of the problem. It's a problem that currently exists with
Gitolite. For example, what happens if a developer's laptop gets
compromised? How do we audit changes to gitolite repositories,
assuming it's not compromised? GitLab provides actually more
possibilities for such audits. Solutions like code reviews, signed
commits, reproducible builds, and transparency logs provide better,
long-term and service-agnostic solutions to those problems.

In the end, it came up to a trade-off: GitLab is much easier to
use. Convenience won over hardened security, especially considering
the cost of running two services in parallel. Or, as Nick Mathewson
[put it](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81#note_2732337):

> I'm proposing that, since this is an area where the developers would
> need to shoulder most of the burden, the development teams should be
> responsible for coming up with solutions that work for them on some
> reasonable timeframe, and that this shouldn't be admin's problem
> assuming that the timeframe is long enough.

For now, the result of that discussion is a [summary of git repository
integrity solutions](howto/gitlab#git-repository-integrity-solutions), which is therefore delegated to teams.

### Migration roadmap

TODO.
