howto/gitlab.md

@@ -1697,6 +1697,129 @@ here:
 
 <https://docs.gitlab.com/ee/administration/packages/container_registry.html>
 
+### Gitaly
+
+[Gitaly](https://gitlab.com/gitlab-org/gitaly) is GitLab's Git frontend server. It's a GRPC API that
+allows for sharding and high availability (with Praefect), although we
+only plan on using the sharding for now. Again, we have decided to
+*not* use the full high-availability solution, called [Gitaly
+Cluster](https://docs.gitlab.com/administration/gitaly/#gitaly-cluster) as its architecture is way to complicated: it involves a
+load balancer (Praefect) with a PostgreSQL database *cluster* to keep
+track of state.
+
+A new server (`gitaly-01.torproject.org`) was configured
+([tpo/tpa/team#42225](https://gitlab.torproject.org/tpo/tpa/team/-/issues/42225)) to reduce the load on the main GitLab
+server, as part of [scaling GitLab to more users](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40479).
+
+Gitaly is installed with The `profile::gitaly` Puppet class. It should
+support installing a new server, but it was not tested on a second
+server yet.
+
+It's running inside a `podman` container, deployed
+[`podman-systemd.unit`](https://manpages.debian.org/testing/podman/podman-systemd.unit.5.en.html) so that the container definition is shipped
+inside a unit file, which takes care of supervising the service and
+upgrades. A container was chosen because the other options were to
+deploy the huge Omnibus Debian package, the [Omnibus Docker
+container](https://docs.gitlab.com/install/docker/) or [building from source](https://docs.gitlab.com/install/installation/#install-gitaly) [at each release](https://docs.gitlab.com/update/upgrading_from_source/#update-gitaly). The
+former seemed to add too much administrative overhead, and we wanted
+to experiment with running that service inside a container (without
+having to jump fully in Kubernetes and the [Helm chart](https://docs.gitlab.com/charts/installation/deployment/) just
+yet).
+
+This led to some oddities like having to chase minor releases in the
+tag (see upstream issue [gitlab-org/build/CNG#2223](https://gitlab.com/gitlab-org/build/CNG/-/issues/2223)). The source of
+the container image is in the [upstream CNG project](https://gitlab.com/gitlab-org/build/CNG/-/tree/578ae04e7515f5fde1a18dbe50b6bfddbd4a9719/gitaly).
+
+Configuration on the host is inside `/etc/gitaly/config.toml`, which
+includes secrets. This and `/home/git/repositories` are bind-mounted
+inside the container, which runs as the `git` user inside the
+container and on the host (but not in rootless mode), in "host"
+network mode (so ports are exposed directly inside the VM).
+
+The Gitaly container has a health check which essentially checks that
+a process named `gitaly` listens on the network inside the
+container. This overrides the upstream checks which only checks on the
+plain text port, which we have disabled, as we use our normal Let's
+Encrypt certificates for TLS to communicate between Gitaly and its
+clients. You can run the health check manually with:
+
+    podman healthcheck run systemd-gitaly; echo $?
+
+If it prints nothing and returns zero, it's healthy, otherwise it will
+print `unhealthy`.
+
+You can do a manual check of the configuration with:
+
+    podman run  --rm  --entrypoint /usr/local/bin/gitaly --network host  --user git:git -v /home/git/repositories:/home/git/repositories -v /etc/gitaly/config.toml:/etc/gitaly/config.toml -v /etc/ssl/private/gitaly-01.torproject.org.key:/etc/gitlab/ssl/key.pem -v /etc/ssl/torproject/certs/gitaly-01.torproject.org.crt-chained:/etc/gitlab/ssl/cert.pem registry.gitlab.com/gitlab-org/build/cng/gitaly:18-1-stable check /etc/gitaly/config.toml
+
+The commandline is derived from the `ExecStart` you can find in:
+
+    systemctl cat gitaly | grep ExecStart
+
+Gitaly has multiple clients: the GitLab rails app, Sidekiq, and so
+on. From our perspective, there's "the gitlab server" (`gitlab-02`)
+and "Gitaly" (`gitaly-01`), however. More details on the architecture
+we're using are available in the [network architecture](https://docs.gitlab.com/administration/gitaly/configure_gitaly/#network-architecture) section of
+the [upstream Gitaly configuration documentation](https://docs.gitlab.com/administration/gitaly/configure_gitaly/).
+
+GitLab authenticates to Gitaly using what we call the
+`gitaly_auth_token` (`auth.token` in Gitaly's `config.toml` and
+`gitlab_rails.repositories_storage.$STORAGE.gitaly_token` in
+`/etc/gitlab/gitlab.rb` on GitLab) and Gitaly authenticates to GitLab
+using the `gitlab_shell_secret_token` (`gitlab.secret` in Gitaly's
+`config.toml` and `gitlab_shell.secret_token` in
+`/etc/gitlab/gitlab-secrets.json` on GitLab).
+
+The `gitlab_shell_secret_token` is (currently) global to all GitLab
+rails instances, but the `gitaly_auth_token` is unique per Gitaly
+instance.
+
+Once a Gitaly server has been configured in GitLab, look in the
+[gitaly section of the admin interface](https://gitlab.torproject.org/admin/gitaly_servers ) to see if it works
+correctly. If it gives a 500 error, look in the GitLab rails
+application logs (`/var/log/gitlab/gitlab-rails/production.log`) for
+the error message. If you get a `permission denied: wrong hmac
+signature`, it's because the `auth.token` Gitaly setting doesn't match
+the secret configured on the GitLab server, see [this
+question](https://support.gitlab.com/hc/en-us/articles/20336430607260-500-error-on-GitLab-homepage-or-project-page-and-permission-denied-wrong-hmac-signature-error-in-Gitaly-Praefect-logs). Note that the secret needs to be configured in the
+`repositories_storages` setting, *not* the `gitaly['configuration'] =
+{ auth: ... }` section.
+
+Use `gitlab-rake gitlab:gitaly:check` on the GitLab server to check
+the Gitaly configuration, here's an example of a working configuration:
+
+```
+root@gitlab-02:~# gitlab-rake gitlab:gitaly:check
+Checking Gitaly ...
+
+Gitaly: ... default ... OK
+storage1 ... OK
+
+Checking Gitaly ... Finished
+```
+
+
+Repositories are *sharded* across servers, that is a repository is
+stored only on *one* server and *not* replicated across the fleet. The
+[repository weight](https://gitlab.torproject.org/help/administration/repository_storage_paths.md#configure-where-new-repositories-are-stored) determines the odds of a repository ending up
+on a given Gitaly server. As of this writing, the gitaly server is
+only in testing, so its weight is `0`, which means repositories are
+not automatically assigned to it, but [repositories can be moved](https://docs.gitlab.com/administration/operations/moving_repositories/)
+individually or in batch, through the GitLab API.
+
+Weights can be configured in the [repositories section of the GitLab
+admin interface](https://gitlab.torproject.org/admin/application_settings/repository#js-repository-storage-settings).
+
+#### Troubleshooting
+
+https://docs.gitlab.com/administration/gitaly/troubleshooting/
+
+disk use
+
+```
+Gitlab::GitalyClient::ServerService.new("default").storage_disk_statistics
+```
+
 ## Upgrades
 
 GitLab upgrades are generally done automatically through
@@ -1729,6 +1852,17 @@ code.
 If you have trouble during the upgrade, follow the [upstream
 troubleshooting guide](https://docs.gitlab.com/update/package/package_troubleshooting/).
 
+Gitaly requires special handling, see below.
+
+### Gitaly
+
+Gitaly's container follows a minor release and needs to be updated
+when new minor releases come out. We've [asked upstream to improve on
+this](https://gitlab.com/gitlab-org/build/CNG/-/issues/2223), but for now this requires some manual work.
+
+Podman should automatically upgrade containers on that minor release
+branch, however.
+
 ## SLA
 
 <!-- this describes an acceptable level of service for this service -->