... | ... | @@ -438,15 +438,31 @@ the course of its regular work. Those tend to pile up, unless they get |
|
|
cleaned. [Upstream suggests](https://docs.gitlab.com/runner/executors/docker.html#clearing-docker-cache) a [fairly naive shell script](https://gitlab.com/gitlab-org/gitlab-runner/blob/master/packaging/root/usr/share/gitlab-runner/clear-docker-cache) to do
|
|
|
this cleanup, but it has a number of issues:
|
|
|
|
|
|
1. it is noisy ([patched locally with this MR](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2711))
|
|
|
1. it is noisy (tried to [patch this locally with this MR](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2711), but
|
|
|
was refused upstream)
|
|
|
2. it might be too aggressive
|
|
|
|
|
|
So we only run it weekly, and instead run a more "gentle" `docker
|
|
|
system prune` command to cleanup orphaned stuff after 3 days.
|
|
|
|
|
|
Also note that documentation on this inside GitLab runner is
|
|
|
inconsistent at best, see [this other MR](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2711) and [this issue](https://gitlab.com/gitlab-org/gitlab-runner-docker-cleanup/-/issues/21).
|
|
|
|
|
|
So we're not using the upstream cleanup script, and we suspect
|
|
|
upstream itself is not using it at all (i.e. on `gitlab.com`) because
|
|
|
it's fundamentally ineffective.
|
|
|
|
|
|
Instead, we have a set of cron jobs (in
|
|
|
`profile::gitlab::runner::docker`) which does the following:
|
|
|
|
|
|
1. clear all volumes and dead containers, daily (equivalent of the
|
|
|
upstream clear-docker-cache for volumes, basically)
|
|
|
2. clear images older than 30 days, daily (unless used by a running
|
|
|
container)
|
|
|
3. clear all dangling (ie. untagged) images, daily
|
|
|
4. clear all "nightly" images, daily
|
|
|
|
|
|
Note that this documentation might be out of date and the Puppet code
|
|
|
should be considered authoritative on this policy, as we've frequently
|
|
|
had to tweak this to deal with out of disk issues.
|
|
|
|
|
|
### rootless containers
|
|
|
|
|
|
We are considering [podman](https://podman.io/) for running containers more securely:
|
... | ... | |