Enable s3 storage for runners

Now that minio s3 storage is setup it would probably be worth setting up the runner cache to use the s3 storage, which will make the cache available to all runners.

Its a fairly simple change to the runner's config.toml, something like the following works:

[[runners]]
  name = "xxx"
  output_limit = 10240
  url = "https://xxx"
  id = 0
  token = "xxx"
  token_obtained_at = 0001-01-01T00:00:00Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  pre_clone_script = "umask 022"
  pre_get_sources_script = "umask 022"
  [runners.cache]
    Type = "s3"
    Shared = true
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
      ServerAddress = "objects.torproject.org"
      AccessKey = "gitlab"
      SecretKey = "xxx"
      BucketName = "gitlab-runner"
      BucketLocation = "seattle"
  [runners.docker]
    tls_verify = false
    image = "docker:stable"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache", "/run/docker.sock:/run/docker.sock"]
    shm_size = 0

closed

reopened

assigned to @lavamind

changed the description

added Backlog label

We discussed this at our checking yesterday and decided against deploying this immediately as TPA will be running with reduced staff until the end of august and this is a change that has a not-insignificant probability of throwing up surprises.

Definitely on our to-do list, however!

added Gitlab label

mentioned in issue #40931 (closed)

mentioned in issue #41295 (closed)

added Next label and removed Backlog label

added Doing label and removed Next label

mentioned in issue #40964 (closed)

I wrote to OSUOSL support to ask if they could configure the runners they provide for us with the shared cache configuration we're deploying on our own runners.

They can configure S3 cache on those runners for us, I sent them the necessary bits.

They deployed the configuration on both runners, ci-runner-x86-09 and -10, and I tested both runners are able to pull/push caches from our MinIO instance.

The configuration changes have been implemented manually on TPA runners and I can confirm they're now uploading cache to the MinIO instance.

All done here!

closed

maybe you've done so already, but could you make sure this is documented... somewhere? maybe service/ci?

also, could we get an idea of the impact on disk space, whether or not that worked and if we need to change disk sizes for future runners? maybe keep this open for a week and see what it looks like before/after in grafana?

finally, if it does have an impact, maybe we can tweak the cron jobs?

i mean maybe all this can be in a followup issue too, but it seems this wouldn't be complete without a little followup. :)

reopened

mentioned in commit wiki-replica@891038d5

maybe you've done so already, but could you make sure this is documented... somewhere? maybe service/ci?

Ok, done.

also, could we get an idea of the impact on disk space, whether or not that worked and if we need to change disk sizes for future runners? maybe keep this open for a week and see what it looks like before/after in grafana?

Right, thanks for reminding me: I did look into this, and I think that enabling the s3/distributed caching mechanism, in itself, does not reduce the disk space used by volumes. Runners are set up this way by default, intentionally, to minimize network traffic to/from object storage. What happens is after a job, a cache volume containing the cache.zip file is left, and when the cache is needed again, it compares it to what's available in object storage and only downloads the cache if the version in object storage is newer.

I think we can disable this behavior, forcing the runners to download caches from object storage every time, by enabling the disable_cache setting in the gitlab-runner configuration, as documented here.

I haven't tested this, and am a little reluctant because I think it's possible some jobs might be using the /cache volume directly (eg. with ci scripts dumping files in there directly), and this change might affect them. We could, however, directly look into these volumes and see if that's the case or not.

I've enhanced tpa-du-gl-volumes to show the distinction between /builds and /cache volumes on the runners. This distinction is made by looking at the last component of the volume names, which are md5 hashes of the path.

I think we should close this issue and revisit the issue around runner disk space, if needed, at a later time.

I've written some notes in this ticket as a starting point.

closed