Access keys secrets cannot be reset: the key must be deleted and a new
one must be created in its place.
A better way to do this is to create a *new* key and mark the old one
as expiring. To rotate the GitLab secrets, for example, a new key
named `gitlab-registry-24` was created (`24` being the year, but it
could be anything), and the `gitlab-registry` key was marked as
expiring 24h after. The new key was stored in Trocla and the key name,
in Puppet.
The runner cache token is more problematic, as the Puppet module
doesn't update it automatically once the runner is registered. That
needs to be modified by hand.
## Pager playbook
### Restarting the service
...
...
@@ -591,6 +607,9 @@ The normal user accounts are typically accessed with tokens saved as
aliases on the main `minio-01` server. If that access is lost, you can
use the [password reset](#password-reset) procedures to recover.
Finally, there's a secret token to access the MinIO statistics that's
[generated on the fly](https://min.io/docs/minio/linux/operations/monitoring/collect-minio-metrics-using-prometheus.html). See the [monitoring and metrics section](#monitoring-and-metrics).
## Implementation
MinIO is implemented in Golang, as a single binary.
...
...
@@ -656,7 +675,14 @@ This is not seen as a deal-breaker in using MinIO for TPA.
## Monitoring and metrics
The main [Prometheus](howto/prometheus) server is configured to scrape metrics
directly from the `minio-01` server.
directly from the `minio-01` server. This was done by running the
following command on the server:
mc admin prometheus generate admin
... and copying the bearer token into the Prometheus configuration
(`profile::::prometheus::server::internal` in Puppet). Look for
`minio_prometheus_jwt_secret`.
The [upstream monitoring metrics](https://min.io/docs/minio/linux/operations/monitoring/collect-minio-metrics-using-prometheus.html#minio-metrics-collect-using-prometheus) do not mention it, but there's a
[range of Grafana dashboards](https://www.startpage.com/do/metasearch.pl?query=inurl:https://grafana.com/grafana/dashboards minio) as well. Unfortunately, we couldn't
