add tpa-rfc-74 draft

policy.md

@@ -29,6 +29,7 @@ the Git repository for this wiki, run the command:
  * [TPA-RFC-47: Email account retirement](policy/tpa-rfc-47-email-account-retirement)
  * [TPA-RFC-66: Migrate to Gitlab Ultimate Edition](policy/tpa-rfc-66-gitlab-ultimate-program)
  * [TPA-RFC-73: Tails infra merge roadmap](policy/tpa-rfc-73-tails-infra-merge-roadmap)
+ * [TPA-RFC-74: GitLab CI retention policy](policy/tpa-rfc-74-gitlab-ci-retention-policy)
 
 ## Proposed
 

policy/tpa-rfc-74-gitlab-ci-retention-policy.md

+---
+title: TPA-RFC-74: GitLab CI retention policy
+costs: n/a
+approval: sysadmins and team leads
+affected users: gitlab users
+deadline: none
+status: draft
+discussion: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41874
+---
+
+[[_TOC_]]
+
+Summary: a proposal to limit the retention of GitLab CI data to 1 year
+
+# Background
+
+As more and more Tor projects moved to GitLab and embraced its continuous
+integration features, managing the ensuing storage requirements has been a
+challenge.
+
+We regularly deal with near filesystem saturation incidents on the GitLab
+server, especially involving CI artifact storage, such as tpo/tpa/team#41402
+and recently, tpo/tpa/team#41861
+
+Previously, [TPA-RFC-14][] was implemented to reduce the default artifact
+retention period from 30 to 14 days. This, and CI optimization of individual
+projects has provided relief, but the long-term issue has not been definitively
+addressed since the retention period doesn't apply to some artifacts such as
+job logs, which are kept indefinitely by default.
+
+[TPA-RFC-14]: tpa-rfc-14-gitlab-artifacts
+
+# Proposal
+
+Implement a daily GitLab maintenance task to delete CI pipelines older than 1
+year in *all* projects hosted on our instance. This will:
+
+ * Purge old CI pipeline and job records for the GitLab database
+ * Delete associated CI job artifacts, even those "kept" either:
+   * When [manually prevented from expiring][] ("Keep" button)
+   * When they're the [latest successful pipeline artifact][]
+ * Delete old CI job log artifacts
+
+[manually prevented from expiring]: https://gitlab.torproject.org/help/ci/jobs/job_artifacts#with-an-expiry
+[latest pipeline artifact]: https://gitlab.torproject.org/help/ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs
+
+## Goals
+
+This is expected to significantly reduce the growth rate of CI-related storage
+usage, and of the GitLab service in general.
+
+## Affected users
+
+All users of GitLab CI will be impacted by this change.
+
+## Timeline
+
+Barring the need to further discussion, this will be implemented on Monday,
+December 9th.
+
+## Costs estimates
+
+### Hardware
+
+This is expected to reduce future requirements in terms of storage hardware.
+
+### Staff
+
+This will reduce the amount of TPA labor needed to deal with filesystem
+saturation incidents.
+
+# Alternatives considered
+
+A "CI housekeeping" script is already in place, which scrubs job logs daily in
+a hard-coded list of key projects such as c-tor packaging, which runs an
+elaborate CI pipeline on a daily basis, and triage-bot, which runs it CI
+pipeline on a schedule, every 15 minutes.
+
+Although it has helped up until now, this approach is not able to deal with the
+increasing use of personal fork projects which are used for development.
+
+It's possible to define a different retention policy based on a project's
+namespace. For example, projects under the `tpo` namespace could have a longer
+retention period, while others (personal projects) could have a shorter one.
+This isn't part of the proposal currently as it could violate the principle of
+least surprise.
+
+# References
+
+ * Discussion ticket: tpo/tpa/team#41861
+ * [Make It Ephemeral: Software Should Decay and Lose Data](https://lucumr.pocoo.org/2024/10/30/make-it-ephemeral/)