From 7d4f36a80fea195afebbe7bbeaff670da46be5b7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= <anarcat@debian.org>
Date: Tue, 20 Feb 2024 12:52:34 -0500
Subject: [PATCH] document a bit more how secrets are managed in minio
(tpo/tpa/team#41530)
---
service/object-storage.md | 28 +++++++++++++++++++++++++++-
1 file changed, 27 insertions(+), 1 deletion(-)
diff --git a/service/object-storage.md b/service/object-storage.md
index 66d8ae62..c00e6d59 100644
--- a/service/object-storage.md
+++ b/service/object-storage.md
@@ -239,6 +239,22 @@ namely:
systemctl restart container-minio.service
+### Access keys
+
+Access keys secrets cannot be reset: the key must be deleted and a new
+one must be created in its place.
+
+A better way to do this is to create a *new* key and mark the old one
+as expiring. To rotate the GitLab secrets, for example, a new key
+named `gitlab-registry-24` was created (`24` being the year, but it
+could be anything), and the `gitlab-registry` key was marked as
+expiring 24h after. The new key was stored in Trocla and the key name,
+in Puppet.
+
+The runner cache token is more problematic, as the Puppet module
+doesn't update it automatically once the runner is registered. That
+needs to be modified by hand.
+
## Pager playbook
### Restarting the service
@@ -591,6 +607,9 @@ The normal user accounts are typically accessed with tokens saved as
aliases on the main `minio-01` server. If that access is lost, you can
use the [password reset](#password-reset) procedures to recover.
+Finally, there's a secret token to access the MinIO statistics that's
+[generated on the fly](https://min.io/docs/minio/linux/operations/monitoring/collect-minio-metrics-using-prometheus.html). See the [monitoring and metrics section](#monitoring-and-metrics).
+
## Implementation
MinIO is implemented in Golang, as a single binary.
@@ -656,7 +675,14 @@ This is not seen as a deal-breaker in using MinIO for TPA.
## Monitoring and metrics
The main [Prometheus](howto/prometheus) server is configured to scrape metrics
-directly from the `minio-01` server.
+directly from the `minio-01` server. This was done by running the
+following command on the server:
+
+ mc admin prometheus generate admin
+
+... and copying the bearer token into the Prometheus configuration
+(`profile::::prometheus::server::internal` in Puppet). Look for
+`minio_prometheus_jwt_secret`.
The [upstream monitoring metrics](https://min.io/docs/minio/linux/operations/monitoring/collect-minio-metrics-using-prometheus.html#minio-metrics-collect-using-prometheus) do not mention it, but there's a
[range of Grafana dashboards](https://www.startpage.com/do/metasearch.pl?query=inurl:https://grafana.com/grafana/dashboards minio) as well. Unfortunately, we couldn't
--
GitLab