... | ... | @@ -148,39 +148,9 @@ adjusted to cache less. |
|
|
|
|
|
### Disk full on GitLab server
|
|
|
|
|
|
If the main GitLab server is running out of space, then it's projects
|
|
|
that are taking up space. We've typically had trouble with artifacts
|
|
|
taking up space (tpo/tpa/team#40615, tpo/tpa/team#40517).
|
|
|
|
|
|
You can see the largest disk users in the GitLab admin area in
|
|
|
[Overview -> Projects -> Sort by: Largest repository](https://gitlab.torproject.org/admin/projects?sort=storage_size_desc).
|
|
|
|
|
|
Note that, although it's unlikely, it's technically possible that an
|
|
|
archived project takes up space, so make sure you check the "Show
|
|
|
archived projects" option in the "Sort by" drop down.
|
|
|
|
|
|
In the past, we have worked around that problem by reducing the
|
|
|
default artifact retention period from 4 to 2 weeks
|
|
|
(tpo/tpa/team#40516) but obviously does not take effect
|
|
|
immediately.
|
|
|
|
|
|
More recently, we have tried to tweak individual project's retention
|
|
|
policies and scheduling strategies (details in tpo/tpa/team#40615).
|
|
|
|
|
|
Please be aware of the [known upstream issues](#known-upstream-issues) that affect those
|
|
|
diagnostics as well.
|
|
|
|
|
|
To see if expiration policies work (or if "kept" artifacts or
|
|
|
old `job.log` are a problem), use this command (which takes a while to
|
|
|
run):
|
|
|
|
|
|
find -mtime +14 -print0 | du --files0-from=- -c -h | tee find-mtime+14-du.log
|
|
|
|
|
|
To limit this to `job.log`, of course, you can do:
|
|
|
|
|
|
find -name "job.log" -mtime +14 -print0 | du --files0-from=- -c -h | tee find-mtime+14-joblog-du.log
|
|
|
|
|
|
TODO: document how to safely remove old artifacts and `job.log` files.
|
|
|
Similar to the above, but typically happens on the GitLab
|
|
|
server. Documented in the GitLab documentation, see [Disk full on
|
|
|
GitLab server](#disk-full-on-gitlab-server).
|
|
|
|
|
|
### DNS resolution failures
|
|
|
|
... | ... | |