... | @@ -110,6 +110,38 @@ which look like a styled, curly, and closing quotation mark `”`. |
... | @@ -110,6 +110,38 @@ which look like a styled, curly, and closing quotation mark `”`. |
|
All CI documentation resides in a different document see
|
|
All CI documentation resides in a different document see
|
|
[service/ci](service/ci).
|
|
[service/ci](service/ci).
|
|
|
|
|
|
|
|
## Container registry operations
|
|
|
|
|
|
|
|
### Logging in
|
|
|
|
|
|
|
|
To upload content to the registry, you first need to login. This can
|
|
|
|
be done with the `login` command:
|
|
|
|
|
|
|
|
podman login
|
|
|
|
|
|
|
|
This will ask you for your GitLab username and a password, for which
|
|
|
|
you should use a [personal access token](https://gitlab.torproject.org/-/profile/personal_access_tokens).
|
|
|
|
|
|
|
|
### Uploading an image
|
|
|
|
|
|
|
|
Assuming you already have an image built (below we have it labeled
|
|
|
|
with `containers.torproject.org/anarcat/test/airsonic-test`), you can
|
|
|
|
upload it with:
|
|
|
|
|
|
|
|
podman push containers.torproject.org/anarcat/test/airsonic-test containers.torproject.org/anarcat/test
|
|
|
|
|
|
|
|
Notice the two arguments: the first is the label of the image to
|
|
|
|
upload and the second is *where* to upload it, or "destination". The
|
|
|
|
destination is made of two parts, the first component is the host name
|
|
|
|
of the container registry (in our case `containers.torproject.org`)
|
|
|
|
and the second part is the path to the project to upload into (in our
|
|
|
|
case [`anarcat/test`](https://gitlab.torproject.org/anarcat/test).
|
|
|
|
|
|
|
|
The uploaded container image should appear under Deploy -> Container
|
|
|
|
Registry in your project. In the above case, it is in:
|
|
|
|
|
|
|
|
<https://gitlab.torproject.org/anarcat/test/container_registry/4>
|
|
|
|
|
|
## Email interactions
|
|
## Email interactions
|
|
|
|
|
|
You can interact with GitLab by email too.
|
|
You can interact with GitLab by email too.
|
... | @@ -615,6 +647,10 @@ To limit this to `job.log`, of course, you can do: |
... | @@ -615,6 +647,10 @@ To limit this to `job.log`, of course, you can do: |
|
|
|
|
|
find -name "job.log" -mtime +14 -print0 | du --files0-from=- -c -h | tee find-mtime+14-joblog-du.log
|
|
find -name "job.log" -mtime +14 -print0 | du --files0-from=- -c -h | tee find-mtime+14-joblog-du.log
|
|
|
|
|
|
|
|
If we ran out of space on the object storage because of the GitLab
|
|
|
|
registry, consider [purging untagged manifests](https://docs.gitlab.com/ee/administration/packages/container_registry.html#removing-untagged-manifests-and-unreferenced-layers) by tweaking the
|
|
|
|
cron job defined in `profile::gitlab::app` in Puppet.
|
|
|
|
|
|
### Incoming email routing
|
|
### Incoming email routing
|
|
|
|
|
|
Incoming email get routed through either eugeni or the submission
|
|
Incoming email get routed through either eugeni or the submission
|
... | @@ -697,6 +733,108 @@ Delivered mail 641c797273ba1_86be948d03829@gitlab-02.mail (7.2ms) |
... | @@ -697,6 +733,108 @@ Delivered mail 641c797273ba1_86be948d03829@gitlab-02.mail (7.2ms) |
|
|
|
|
|
[issue 139]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/139
|
|
[issue 139]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/139
|
|
|
|
|
|
|
|
### Gitlab registry troubleshooting
|
|
|
|
|
|
|
|
If something goes with the GitLab Registry feature, you should first
|
|
|
|
look at the logs in:
|
|
|
|
|
|
|
|
tail -f /var/log/gitlab/registry/current /var/log/gitlab/nginx/gitlab_registry_*.log /var/log/gitlab/gitlab-rails/production.log
|
|
|
|
|
|
|
|
The first one might be the one with more relevant information, but is
|
|
|
|
the hardest to parse, as it's this weird "date {JSONBLOB}" format that
|
|
|
|
no human or machine can parse.
|
|
|
|
|
|
|
|
You can restart *just* the registry with:
|
|
|
|
|
|
|
|
gitlab-ctl restart registry
|
|
|
|
|
|
|
|
A misconfiguration of the object storage backend will look like this
|
|
|
|
when uploading a container:
|
|
|
|
|
|
|
|
Error: trying to reuse blob sha256:61581d479298c795fa3cfe95419a5cec510085ec0d040306f69e491a598e7707 at destination: pinging container registry containers.torproject.org: invalid status code from registry 503 (Service Unavailable)
|
|
|
|
|
|
|
|
The registry logs might have something like this:
|
|
|
|
|
|
|
|
```
|
|
|
|
2023-07-18_21:45:26.21751 time="2023-07-18T21:45:26.217Z" level=info msg="router info" config_http_addr="127.0.0.1:5000" config_http_host= config_http_net= config_http_prefix= config_http_relative_urls=true correlation_id=01H5NFE6E94A566P4EZG2ZMFMT go_version=go1.19.8 method=HEAD path="/v2/anarcat/test/blobs/sha256:61581d479298c795fa3cfe95419a5cec510085ec0d040306f69e491a598e7707" root_repo=anarcat router=gorilla/mux vars_digest="sha256:61581d479298c795fa3cfe95419a5cec510085ec0d040306f69e491a598e7707" vars_name=anarcat/test version=v3.76.0-gitlab
|
|
|
|
2023-07-18_21:45:26.21774 time="2023-07-18T21:45:26.217Z" level=info msg="authorized request" auth_project_paths="[anarcat/test]" auth_user_name=anarcat auth_user_type=personal_access_token correlation_id=01H5NFE6E94A566P4EZG2ZMFMT go_version=go1.19.8 root_repo=anarcat vars_digest="sha256:61581d479298c795fa3cfe95419a5cec510085ec0d040306f69e491a598e7707" vars_name=anarcat/test version=v3.76.0-gitlab
|
|
|
|
2023-07-18_21:45:26.30401 time="2023-07-18T21:45:26.303Z" level=error msg="unknown error" auth_project_paths="[anarcat/test]" auth_user_name=anarcat auth_user_type=personal_access_token code=UNKNOWN correlation_id=01H5NFE6CZBE49BZ6KBK4EHSJ1 detail="SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.\n\tstatus code: 403, request id: 17731468F69A0F79, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8" error="unknown: unknown error" go_version=go1.19.8 host=containers.torproject.org method=HEAD remote_addr=64.18.183.94 root_repo=anarcat uri="/v2/anarcat/test/blobs/sha256:a55f9a4279c12800590169f7782b956e5c06ec88ec99c020dd111a7a1dcc7eac" user_agent="containers/5.23.1 (github.com/containers/image)" vars_digest="sha256:a55f9
|
|
|
|
```
|
|
|
|
|
|
|
|
If you suspect the object storage backend to be the problem, you
|
|
|
|
should try to communicate with the MinIO server by configuring the
|
|
|
|
`rclone` client on the GitLab server and trying to manipulate the
|
|
|
|
server. Look for the access token in `/etc/gitlab/gitlab.rb` and use
|
|
|
|
it to configure `rclone` like this:
|
|
|
|
|
|
|
|
rclone config create minio s3 provider Minio endpoint https://minio.torproject.org:9000/ region dallas access_key_id gitlab-registry secret_access_key REDACTED
|
|
|
|
|
|
|
|
Then you can list the registry bucket:
|
|
|
|
|
|
|
|
rclone ls minio:gitlab-registry/
|
|
|
|
|
|
|
|
See how to [Use rclone as an object storage client](service/minio#use-rclone-as-an-object-storage-client) for more ideas.
|
|
|
|
|
|
|
|
The above may reproduce the above error from the registry:
|
|
|
|
|
|
|
|
SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
|
|
|
|
|
|
|
|
That is either due to an incorrect access key or bucket. An error that
|
|
|
|
was made during the original setup was to treat `gitlab/registry` as a
|
|
|
|
bucket, while it's a subdirectory... This was fixed by switching to
|
|
|
|
`gitlab-registry` as a bucket name. Another error we had was to use
|
|
|
|
`endpoint` instead of `regionendpoint`.
|
|
|
|
|
|
|
|
Another tweak that was done was to set a region in MinIO. Before the
|
|
|
|
right region was set and matching in the configuration, we had this
|
|
|
|
error in the registry logs:
|
|
|
|
|
|
|
|
2023-07-18_21:04:57.46099 time="2023-07-18T21:04:57.460Z" level=fatal msg="configuring application: 1 error occurred:\n\t* validating region provided: dallas\n\n"
|
|
|
|
|
|
|
|
As a last resort, you can revert back to the [filesystem storage](https://docs.gitlab.com/ee/administration/packages/container_registry.html#use-file-system)
|
|
|
|
by commenting out the `storage => { ... 's3' ... }` block in
|
|
|
|
`profile::gitlab::app` and adding a line in the `gitlab_rails` blob
|
|
|
|
like:
|
|
|
|
|
|
|
|
registry_path => '/var/opt/gitlab/gitlab-rails/shared/registry',
|
|
|
|
|
|
|
|
Note that this is a risky operation, as you might end up with a "split
|
|
|
|
brain" where some images are on the filesystem, and some on object
|
|
|
|
storage. Warning users with maintenance announcement on the GitLab
|
|
|
|
site might be wise.
|
|
|
|
|
|
|
|
In the same section, you can [disable the registry by default](https://docs.gitlab.com/ee/administration/packages/container_registry.html#disable-container-registry-for-new-projects-site-wide) on
|
|
|
|
all projects with:
|
|
|
|
|
|
|
|
gitlab_default_projects_features_container_registry => false,
|
|
|
|
|
|
|
|
... or [disable it site-wide](https://docs.gitlab.com/ee/administration/packages/container_registry.html#disable-container-registry-site-wide) with:
|
|
|
|
|
|
|
|
registry => {
|
|
|
|
enable => false
|
|
|
|
# [...]
|
|
|
|
}
|
|
|
|
|
|
|
|
Note that the `registry` configuration is stored inside the Docker
|
|
|
|
Registry `config.yaml` file as a single line that looks like JSON. You
|
|
|
|
*may* think it's garbled and the reason why things don't work, but it
|
|
|
|
isn't, that is valid YAML, just harder to parse. Blame `gitlab-ctl`'s
|
|
|
|
Chef cookbook on that... A non-mangled version of the working config
|
|
|
|
would look like:
|
|
|
|
|
|
|
|
```
|
|
|
|
storage:
|
|
|
|
s3:
|
|
|
|
accesskey: gitlab-registry
|
|
|
|
secretkey: REDACTED
|
|
|
|
region: dallas
|
|
|
|
regionendpoint: https://minio.torproject.org:9000/
|
|
|
|
bucket: gitlab-registry
|
|
|
|
```
|
|
|
|
|
|
|
|
Another option that was explored while setting up the registry is
|
|
|
|
enabling the [debug server](https://docs.gitlab.com/ee/administration/packages/container_registry.html#enable-the-registry-debug-server).
|
|
|
|
|
|
## Disaster recovery
|
|
## Disaster recovery
|
|
|
|
|
|
In case the entire GitLab machine is destroyed, a new server should be
|
|
In case the entire GitLab machine is destroyed, a new server should be
|
... | @@ -1058,6 +1196,66 @@ Puppet class. The following GitLab settings were added: |
... | @@ -1058,6 +1196,66 @@ Puppet class. The following GitLab settings were added: |
|
The virtual host for the `pages.torproject.net` domain was configured
|
|
The virtual host for the `pages.torproject.net` domain was configured
|
|
through the `profile::gitlab::web` class.
|
|
through the `profile::gitlab::web` class.
|
|
|
|
|
|
|
|
### GitLab registry
|
|
|
|
|
|
|
|
The GitLab registry was setup first by deploying an object storage
|
|
|
|
server (see [minio](service/minio)). An access key was created with:
|
|
|
|
|
|
|
|
mc admin user svcacct add admin gitlab --access-key gitlab-registry
|
|
|
|
|
|
|
|
... and the secret key stored in Trocla.
|
|
|
|
|
|
|
|
Then the config was injected in the `profile::gitlab::app` class,
|
|
|
|
mostly inline. The registry itself is configured through the
|
|
|
|
`profile::gitlab::registry` class, so that it could possibly be moved
|
|
|
|
onto its own host.
|
|
|
|
|
|
|
|
That configuration was filled with many perils, partly documented in
|
|
|
|
[tpo/tpa/gitlab#89](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/89). One challenge was to get everything working at
|
|
|
|
once. The software itself is the [Docker Registry](https://docs.docker.com/registry/) [shipped with
|
|
|
|
GitLab Omnibus](https://docs.gitlab.com/ee/administration/packages/container_registry.html), and it's configured through Puppet, which passes
|
|
|
|
the value to the `/etc/gitlab/gitlab.rb` file which *then* writes the
|
|
|
|
final configuration into `/var/opt/gitlab/registry/config.yml`.
|
|
|
|
|
|
|
|
We take the [separate bucket](https://docs.gitlab.com/ee/administration/object_storage.html#use-separate-buckets) approach in that each service using
|
|
|
|
object storage has its own bucket assigned. This required a special
|
|
|
|
policy to be applied to the `gitlab` MinIO user:
|
|
|
|
|
|
|
|
{
|
|
|
|
"Version": "2012-10-17",
|
|
|
|
"Statement": [
|
|
|
|
{
|
|
|
|
"Action": [
|
|
|
|
"s3:*"
|
|
|
|
],
|
|
|
|
"Effect": "Allow",
|
|
|
|
"Resource": [
|
|
|
|
"arn:aws:s3:::gitlab*"
|
|
|
|
],
|
|
|
|
"Sid": "BucketAccessForUser"
|
|
|
|
}
|
|
|
|
]
|
|
|
|
}
|
|
|
|
|
|
|
|
That is the policy called `gitlab-star-bucket-policy` which grants
|
|
|
|
access to all buckets prefixed with `gitlab` (as opposed to only the
|
|
|
|
`gitlab` bucket itself).
|
|
|
|
|
|
|
|
It might be possible to manage the Docker registry software and
|
|
|
|
configuration directly from Puppet, with Debian package, but that
|
|
|
|
configuration is actually [deprecated since 15.8 and unsupported in
|
|
|
|
GitLab 16](https://docs.gitlab.com/ee/administration/packages/container_registry.html#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). I explained our rationale on why this could be
|
|
|
|
interesting in the [relevant upstream issue](https://gitlab.com/gitlab-org/container-registry/-/issues/958#note_1476978502).
|
|
|
|
|
|
|
|
We have created a `registry` user on the host because that's what
|
|
|
|
GitLab expects, but it might be possible to use a different, less
|
|
|
|
generic username by following [this guide](https://docs.gitlab.com/omnibus/settings/configuration.html#specify-numeric-user-and-group-identifiers).
|
|
|
|
|
|
|
|
A cron job runs every Saturday to clean up unreferenced
|
|
|
|
layers. [Untagged manifests](https://docs.gitlab.com/ee/administration/packages/container_registry.html#removing-untagged-manifests-and-unreferenced-layers) are *not* purged even if invisible, as
|
|
|
|
we feel maybe those would result in needless double-uploads. If we do
|
|
|
|
run out of disk space on images, that is a policy we could implement.
|
|
|
|
|
|
## SLA
|
|
## SLA
|
|
|
|
|
|
<!-- this describes an acceptable level of service for this service -->
|
|
<!-- this describes an acceptable level of service for this service -->
|
... | | ... | |