gitlab containers aren't removed after their job has finished

It appears that gitlab keeps the containers around after the corresponding job finishes.

It's not entirely clear why this is, since every job gets a fresh container, and artifacts are pushed to and pulled from the central gitlab instance. Maybe they implement job caches as a pointer into a previously run container instead of copying the data out and storing it separately?

Whatever the reason, this results in a lot of wasted storage. e.g. arti rust builds have gigabytes of downloaded and compiled dependencies.

In the short term, it'd be a good idea for jobs to use after_script to clean up any significant storage they're not explicitly persisting in artifacts or cache. For jobs that build rust code this may involve e.g. deleting $HOME/.cargo and^1 the compiled dependencies in the target directory.

^1: EDIT: according to #note_2846251 only the project dir is preserved, so e.g. $HOME/.cargo is already cleaned up automatically

In the longer term it'd be better to configure/fix gitlab to simply not persist these containers. The most relevant gitlab-side issue I could find is https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27332. It looks like using setting disable_cache=true in the runner config might do the trick, but looks like it'd also disable local caching. (Maybe that's an acceptable tradeoff in the short term?)