Loading policy.md +1 −0 Original line number Diff line number Diff line Loading @@ -14,6 +14,7 @@ and add it to the above list. * [TPA-RFC-3: tools](policy/tpa-rfc-3-tools) * [TPA-RFC-11: SVN retirement](policy/tpa-rfc-11-svn-retirement) * [TPA-RFC-14: GitLab artifacts expiry](policy/tpa-rfc-14-gitlab-artifacts) ## Proposed Loading policy/tpa-rfc-14-gitlab-artifacts.md 0 → 100644 +80 −0 Original line number Diff line number Diff line --- title: TPA-RFC-14: GitLab artifacts expiry change --- [[_TOC_]] Summary: GitLab artifacts used to be deleted after 30 days. Now they will be deleted after 14 days. Latest artifacts are always kept. That expiry period can be changed with the `artifacts:expire_in` field in `.gitlab-ci.yml`. # What We will soon change the retention period for artifacts produced by GitLab CI jobs. By default, GitLab keeps artifacts to 30 days (~four weeks), but we will lower this to 14 days (two weeks). Latest artifacts for all pipelines are kept indefinitely regardless of this change. Artifacts marked "keep" will also still be kept. # Why Artifacts are using a *lot* of disk space. At last count we had 300GB of artifacts and were gaining 3GB per *day*, with no improvement in sight. We have *already* grown the GitLab server's disk space to accommodate that growth, but it has already filled up. It is our hope that this change will allow us to avoid growing the disk indefinitely and will make it easier for TPA to manage the growing GitLab infrastructure in the short term. # How The [default artifacts expiration][] timeout will be changed from 30 days to 14 days in the GitLab administration panel. If you wish to override that setting, you can add a [artifacts:expire_in][] setting in your `.gitlab-ci.yml` file. This will only affect new jobs. Artifacts of jobs created before the change will expire after 30 days, as before. [artifacts:expire_in]: https://gitlab.torproject.org/help/ci/yaml/index.md#artifactsexpire_in [default artifacts expiration]: https://gitlab.torproject.org/help/user/admin_area/settings/continuous_integration.md#default-artifacts-expiration It is speculated that the Jenkins migration is at least partly responsible for the growth in disk usage. It is our hope that the disk usage growth will slow down as that migration completes, but we are conscious that GitLab is being used more and more by all teams and that it's entirely reasonable that the artifacts storage will keep growing indefinitely. We already have disk space available to allocate to this problem in the mid-term. This is mostly to give us some time to breathe before we throw all the hardware we have left at the problem. We are also looking at [long-term storage problems][] and [GitLab scalability issues][] in parallel with this. [GitLab scalability issues]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40479 [long-term storage problems]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40478 # Deadline This proposal will be implemented in a week unless someone strongly objects, either here or in [ticket 40516][]. # Status This proposal is in the `draft` state. # References * [ticket 40516][]: bug report about artifacts filling up disks * [GitLab scalability issues][] * [long-term storage problems][] * [artifacts:expire_in][] setting * [default artifacts expiration][] setting [ticket 40516]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40516 Loading
policy.md +1 −0 Original line number Diff line number Diff line Loading @@ -14,6 +14,7 @@ and add it to the above list. * [TPA-RFC-3: tools](policy/tpa-rfc-3-tools) * [TPA-RFC-11: SVN retirement](policy/tpa-rfc-11-svn-retirement) * [TPA-RFC-14: GitLab artifacts expiry](policy/tpa-rfc-14-gitlab-artifacts) ## Proposed Loading
policy/tpa-rfc-14-gitlab-artifacts.md 0 → 100644 +80 −0 Original line number Diff line number Diff line --- title: TPA-RFC-14: GitLab artifacts expiry change --- [[_TOC_]] Summary: GitLab artifacts used to be deleted after 30 days. Now they will be deleted after 14 days. Latest artifacts are always kept. That expiry period can be changed with the `artifacts:expire_in` field in `.gitlab-ci.yml`. # What We will soon change the retention period for artifacts produced by GitLab CI jobs. By default, GitLab keeps artifacts to 30 days (~four weeks), but we will lower this to 14 days (two weeks). Latest artifacts for all pipelines are kept indefinitely regardless of this change. Artifacts marked "keep" will also still be kept. # Why Artifacts are using a *lot* of disk space. At last count we had 300GB of artifacts and were gaining 3GB per *day*, with no improvement in sight. We have *already* grown the GitLab server's disk space to accommodate that growth, but it has already filled up. It is our hope that this change will allow us to avoid growing the disk indefinitely and will make it easier for TPA to manage the growing GitLab infrastructure in the short term. # How The [default artifacts expiration][] timeout will be changed from 30 days to 14 days in the GitLab administration panel. If you wish to override that setting, you can add a [artifacts:expire_in][] setting in your `.gitlab-ci.yml` file. This will only affect new jobs. Artifacts of jobs created before the change will expire after 30 days, as before. [artifacts:expire_in]: https://gitlab.torproject.org/help/ci/yaml/index.md#artifactsexpire_in [default artifacts expiration]: https://gitlab.torproject.org/help/user/admin_area/settings/continuous_integration.md#default-artifacts-expiration It is speculated that the Jenkins migration is at least partly responsible for the growth in disk usage. It is our hope that the disk usage growth will slow down as that migration completes, but we are conscious that GitLab is being used more and more by all teams and that it's entirely reasonable that the artifacts storage will keep growing indefinitely. We already have disk space available to allocate to this problem in the mid-term. This is mostly to give us some time to breathe before we throw all the hardware we have left at the problem. We are also looking at [long-term storage problems][] and [GitLab scalability issues][] in parallel with this. [GitLab scalability issues]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40479 [long-term storage problems]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40478 # Deadline This proposal will be implemented in a week unless someone strongly objects, either here or in [ticket 40516][]. # Status This proposal is in the `draft` state. # References * [ticket 40516][]: bug report about artifacts filling up disks * [GitLab scalability issues][] * [long-term storage problems][] * [artifacts:expire_in][] setting * [default artifacts expiration][] setting [ticket 40516]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40516
