gitlab-02 running out of disk space

From: nagios@hetzner-hel1-01.torproject.org
Subject: ** PROBLEM Service Alert: gitlab-02/disk usage on /var/opt is CRITICAL **
To: anarcat+rapports@orangeseeds.org
Date: Thu, 21 Oct 2021 16:09:39 +0000

***** Icinga *****

Notification Type: PROBLEM

Service: disk usage on /var/opt
Host: gitlab-02
Address: 116.202.120.180
State: CRITICAL

Date/Time: Thu Oct 21 16:09:39 UTC 2021

Additional Info:

DISK CRITICAL - free space: /var/opt 8752 MB (4% inode=95%):

Solution picked (see below): grow the disk.

It will require two downtimes:

1. a reboot (done)
2. a period during which artifacts are moved between the two disks

The next steps are:

1. wait for the disk to grow
2. reboot the VM from the outside
3. pvresize /dev/sdd
4. lvcreate -n gitlab-shared vg_gitlab-02_hdd -L 250G
5. mkfs -t ext4 -j /dev/vg_gitlab-02_hdd/gitlab-shared
6. echo /dev/vg_gitlab-02_hdd/gitlab-share /srv/gitlab-shared ext4 defaults 0 2 >> /etc/fstab
7. mount /srv/gitlab-shared
8. rsync -aSHAXx --info=progress2 --partial /var/opt/gitlab/gitlab-rails/shared/artifacts/ /srv/gitlab-shared/artifacts/
9. stop gitlab?
10. again: rsync -aSHAXx --info=progress2 --partial /var/opt/gitlab/gitlab-rails/shared/artifacts/ /srv/gitlab-shared/artifacts/
11. mount bind or symlink?
12. restart gitlab
13. close this ticket. :)