Skip to content
Snippets Groups Projects
Verified Commit 905d21a7 authored by anarcat's avatar anarcat
Browse files

document the gitlab registry (gitlab#89) 

I have briefly considered making a new service document for this, but
I'm tired and I'm not sure it's worth it. I did try to at least nail
something in the installation and pager playbook, along with a basic
howto.
parent 458aa5ab
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 198 additions and 0 deletions
howto/gitlab.md
+ 198
− 0
View file @ 905d21a7
......@@ -110,6 +110,38 @@ which look like a styled, curly, and closing quotation mark `”`.
All CI documentation resides in a different document see
[service/ci](service/ci).
## Container registry operations
### Logging in
To upload content to the registry, you first need to login. This can
be done with the `login` command:
podman login
This will ask you for your GitLab username and a password, for which
you should use a [personal access token](https://gitlab.torproject.org/-/profile/personal_access_tokens).
### Uploading an image
Assuming you already have an image built (below we have it labeled
with `containers.torproject.org/anarcat/test/airsonic-test`), you can
upload it with:
podman push containers.torproject.org/anarcat/test/airsonic-test containers.torproject.org/anarcat/test
Notice the two arguments: the first is the label of the image to
upload and the second is *where* to upload it, or "destination". The
destination is made of two parts, the first component is the host name
of the container registry (in our case `containers.torproject.org`)
and the second part is the path to the project to upload into (in our
case [`anarcat/test`](https://gitlab.torproject.org/anarcat/test).
The uploaded container image should appear under Deploy -> Container
Registry in your project. In the above case, it is in:
<https://gitlab.torproject.org/anarcat/test/container_registry/4>
## Email interactions
You can interact with GitLab by email too.
......@@ -615,6 +647,10 @@ To limit this to `job.log`, of course, you can do:
find -name "job.log" -mtime +14 -print0 | du --files0-from=- -c -h | tee find-mtime+14-joblog-du.log
If we ran out of space on the object storage because of the GitLab
registry, consider [purging untagged manifests](https://docs.gitlab.com/ee/administration/packages/container_registry.html#removing-untagged-manifests-and-unreferenced-layers) by tweaking the
cron job defined in `profile::gitlab::app` in Puppet.
### Incoming email routing
Incoming email get routed through either eugeni or the submission
......@@ -697,6 +733,108 @@ Delivered mail 641c797273ba1_86be948d03829@gitlab-02.mail (7.2ms)
[issue 139]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/139
### Gitlab registry troubleshooting
If something goes with the GitLab Registry feature, you should first
look at the logs in:
tail -f /var/log/gitlab/registry/current /var/log/gitlab/nginx/gitlab_registry_*.log /var/log/gitlab/gitlab-rails/production.log
The first one might be the one with more relevant information, but is
the hardest to parse, as it's this weird "date {JSONBLOB}" format that
no human or machine can parse.
You can restart *just* the registry with:
gitlab-ctl restart registry
A misconfiguration of the object storage backend will look like this
when uploading a container:
Error: trying to reuse blob sha256:61581d479298c795fa3cfe95419a5cec510085ec0d040306f69e491a598e7707 at destination: pinging container registry containers.torproject.org: invalid status code from registry 503 (Service Unavailable)
The registry logs might have something like this:
```
2023-07-18_21:45:26.21751 time="2023-07-18T21:45:26.217Z" level=info msg="router info" config_http_addr="127.0.0.1:5000" config_http_host= config_http_net= config_http_prefix= config_http_relative_urls=true correlation_id=01H5NFE6E94A566P4EZG2ZMFMT go_version=go1.19.8 method=HEAD path="/v2/anarcat/test/blobs/sha256:61581d479298c795fa3cfe95419a5cec510085ec0d040306f69e491a598e7707" root_repo=anarcat router=gorilla/mux vars_digest="sha256:61581d479298c795fa3cfe95419a5cec510085ec0d040306f69e491a598e7707" vars_name=anarcat/test version=v3.76.0-gitlab
2023-07-18_21:45:26.21774 time="2023-07-18T21:45:26.217Z" level=info msg="authorized request" auth_project_paths="[anarcat/test]" auth_user_name=anarcat auth_user_type=personal_access_token correlation_id=01H5NFE6E94A566P4EZG2ZMFMT go_version=go1.19.8 root_repo=anarcat vars_digest="sha256:61581d479298c795fa3cfe95419a5cec510085ec0d040306f69e491a598e7707" vars_name=anarcat/test version=v3.76.0-gitlab
2023-07-18_21:45:26.30401 time="2023-07-18T21:45:26.303Z" level=error msg="unknown error" auth_project_paths="[anarcat/test]" auth_user_name=anarcat auth_user_type=personal_access_token code=UNKNOWN correlation_id=01H5NFE6CZBE49BZ6KBK4EHSJ1 detail="SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.\n\tstatus code: 403, request id: 17731468F69A0F79, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8" error="unknown: unknown error" go_version=go1.19.8 host=containers.torproject.org method=HEAD remote_addr=64.18.183.94 root_repo=anarcat uri="/v2/anarcat/test/blobs/sha256:a55f9a4279c12800590169f7782b956e5c06ec88ec99c020dd111a7a1dcc7eac" user_agent="containers/5.23.1 (github.com/containers/image)" vars_digest="sha256:a55f9
```
If you suspect the object storage backend to be the problem, you
should try to communicate with the MinIO server by configuring the
`rclone` client on the GitLab server and trying to manipulate the
server. Look for the access token in `/etc/gitlab/gitlab.rb` and use
it to configure `rclone` like this:
rclone config create minio s3 provider Minio endpoint https://minio.torproject.org:9000/ region dallas access_key_id gitlab-registry secret_access_key REDACTED
Then you can list the registry bucket:
rclone ls minio:gitlab-registry/
See how to [Use rclone as an object storage client](service/minio#use-rclone-as-an-object-storage-client) for more ideas.
The above may reproduce the above error from the registry:
SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
That is either due to an incorrect access key or bucket. An error that
was made during the original setup was to treat `gitlab/registry` as a
bucket, while it's a subdirectory... This was fixed by switching to
`gitlab-registry` as a bucket name. Another error we had was to use
`endpoint` instead of `regionendpoint`.
Another tweak that was done was to set a region in MinIO. Before the
right region was set and matching in the configuration, we had this
error in the registry logs:
2023-07-18_21:04:57.46099 time="2023-07-18T21:04:57.460Z" level=fatal msg="configuring application: 1 error occurred:\n\t* validating region provided: dallas\n\n"
As a last resort, you can revert back to the [filesystem storage](https://docs.gitlab.com/ee/administration/packages/container_registry.html#use-file-system)
by commenting out the `storage => { ... 's3' ... }` block in
`profile::gitlab::app` and adding a line in the `gitlab_rails` blob
like:
registry_path => '/var/opt/gitlab/gitlab-rails/shared/registry',
Note that this is a risky operation, as you might end up with a "split
brain" where some images are on the filesystem, and some on object
storage. Warning users with maintenance announcement on the GitLab
site might be wise.
In the same section, you can [disable the registry by default](https://docs.gitlab.com/ee/administration/packages/container_registry.html#disable-container-registry-for-new-projects-site-wide) on
all projects with:
gitlab_default_projects_features_container_registry => false,
... or [disable it site-wide](https://docs.gitlab.com/ee/administration/packages/container_registry.html#disable-container-registry-site-wide) with:
registry => {
enable => false
# [...]
}
Note that the `registry` configuration is stored inside the Docker
Registry `config.yaml` file as a single line that looks like JSON. You
*may* think it's garbled and the reason why things don't work, but it
isn't, that is valid YAML, just harder to parse. Blame `gitlab-ctl`'s
Chef cookbook on that... A non-mangled version of the working config
would look like:
```
storage:
s3:
accesskey: gitlab-registry
secretkey: REDACTED
region: dallas
regionendpoint: https://minio.torproject.org:9000/
bucket: gitlab-registry
```
Another option that was explored while setting up the registry is
enabling the [debug server](https://docs.gitlab.com/ee/administration/packages/container_registry.html#enable-the-registry-debug-server).
## Disaster recovery
In case the entire GitLab machine is destroyed, a new server should be
......@@ -1058,6 +1196,66 @@ Puppet class. The following GitLab settings were added:
The virtual host for the `pages.torproject.net` domain was configured
through the `profile::gitlab::web` class.
### GitLab registry
The GitLab registry was setup first by deploying an object storage
server (see [minio](service/minio)). An access key was created with:
mc admin user svcacct add admin gitlab --access-key gitlab-registry
... and the secret key stored in Trocla.
Then the config was injected in the `profile::gitlab::app` class,
mostly inline. The registry itself is configured through the
`profile::gitlab::registry` class, so that it could possibly be moved
onto its own host.
That configuration was filled with many perils, partly documented in
[tpo/tpa/gitlab#89](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/89). One challenge was to get everything working at
once. The software itself is the [Docker Registry](https://docs.docker.com/registry/) [shipped with
GitLab Omnibus](https://docs.gitlab.com/ee/administration/packages/container_registry.html), and it's configured through Puppet, which passes
the value to the `/etc/gitlab/gitlab.rb` file which *then* writes the
final configuration into `/var/opt/gitlab/registry/config.yml`.
We take the [separate bucket](https://docs.gitlab.com/ee/administration/object_storage.html#use-separate-buckets) approach in that each service using
object storage has its own bucket assigned. This required a special
policy to be applied to the `gitlab` MinIO user:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::gitlab*"
],
"Sid": "BucketAccessForUser"
}
]
}
That is the policy called `gitlab-star-bucket-policy` which grants
access to all buckets prefixed with `gitlab` (as opposed to only the
`gitlab` bucket itself).
It might be possible to manage the Docker registry software and
configuration directly from Puppet, with Debian package, but that
configuration is actually [deprecated since 15.8 and unsupported in
GitLab 16](https://docs.gitlab.com/ee/administration/packages/container_registry.html#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). I explained our rationale on why this could be
interesting in the [relevant upstream issue](https://gitlab.com/gitlab-org/container-registry/-/issues/958#note_1476978502).
We have created a `registry` user on the host because that's what
GitLab expects, but it might be possible to use a different, less
generic username by following [this guide](https://docs.gitlab.com/omnibus/settings/configuration.html#specify-numeric-user-and-group-identifiers).
A cron job runs every Saturday to clean up unreferenced
layers. [Untagged manifests](https://docs.gitlab.com/ee/administration/packages/container_registry.html#removing-untagged-manifests-and-unreferenced-layers) are *not* purged even if invisible, as
we feel maybe those would result in needless double-uploads. If we do
run out of disk space on images, that is a policy we could implement.
## SLA
<!-- this describes an acceptable level of service for this service -->
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment