Skip to content
Snippets Groups Projects

CRITICAL disk usage on metrics-psqlts-01

    • Closed Incident created by Kez @kez

    Icinga is reporting a critical issue for disk usage - all since 2022-09-29 22:57:01

    DISK CRITICAL - free space: / 561 MB (5% inode=87%): /dev 3962 MB (100% inode=99%): /dev/shm 3978 MB (99% inode=99%): /run 795 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /tmp 3978 MB (100% inode=99%): /run/credentials 795 MB (99% inode=99%): /var/tmp 561 MB (5% inode=87%):
    Edited by Kez

    Child items ...

    • Activity

    • Kez changed the description

      changed the description

    • Kez marked this issue as related to #40478 (closed)

      marked this issue as related to #40478 (closed)

    • anarcat changed issue type to incident

      changed issue type to incident

    • anarcat changed the severity to Low - S4

      changed the severity to Low - S4

    • anarcat
      anarcat @anarcat ·
      Owner

      @hiro that's your box right? do you need more space or is this temporary?

    • anarcat added Metrics Needs Information labels

      added Metrics Needs Information labels

    • Hiro
      Hiro @hiro ·
      Reporter

      @anarcat this is temporary. I download the files and then delete them and download again. Maybe a little more space would make the alert go away. If we have space that would be great.

    • Hiro
      Hiro @hiro ·
      Reporter

      I see stuff are crashing :( means effectively my plan to download and delete files didn't work. Can I get some more space? If that's no biggie ...

    • anarcat assigned to @anarcat

      assigned to @anarcat

    • anarcat added Doing label and removed Needs Information label

      added Doing label and removed Needs Information label

    • anarcat
      anarcat @anarcat ·
      Owner

      @hiro how much do you need? you only have a stock 10G for the system on this thing, i could easily attach another 20G SSD on there, would that be enough?

    • anarcat
      anarcat @anarcat ·
      Owner

      hiro confirmed a 20G would be okay. turns out this is on gnt-chi so it's not SSD at all, it's those SAS drives.

      we also didn't have room on the primary, so i flipped to secondary and reallocated disks elsewhere:

      root@chi-node-01:~# gnt-instance migrate metrics-psqlts-01.torproject.org
Instance metrics-psqlts-01.torproject.org will be migrated. Note that
migration might impact the instance if anything goes wrong (e.g. due
to bugs in the hypervisor). Continue?
y/[n]/?: y
Mon Oct  3 15:02:49 2022 Migrating instance metrics-psqlts-01.torproject.org
Mon Oct  3 15:02:50 2022 * checking disk consistency between source and target
Mon Oct  3 15:02:51 2022 * closing instance disks on node chi-node-10.torproject.org
Mon Oct  3 15:02:52 2022 * changing into standalone mode
Mon Oct  3 15:02:53 2022 * changing disks into dual-master mode
Mon Oct  3 15:02:54 2022 * wait until resync is done
Mon Oct  3 15:02:55 2022 * opening instance disks on node chi-node-11.torproject.org in shared mode
Mon Oct  3 15:02:55 2022 * opening instance disks on node chi-node-10.torproject.org in shared mode
Mon Oct  3 15:02:56 2022 * preparing chi-node-10.torproject.org to accept the instance
Mon Oct  3 15:02:56 2022 * migrating instance to chi-node-10.torproject.org
Mon Oct  3 15:02:57 2022 * starting memory transfer
Mon Oct  3 15:03:08 2022 * memory transfer progress: 14.56 %
Mon Oct  3 15:03:19 2022 * memory transfer progress: 29.38 %
Mon Oct  3 15:03:29 2022 * memory transfer progress: 44.02 %
Mon Oct  3 15:03:40 2022 * memory transfer progress: 58.63 %
Mon Oct  3 15:03:51 2022 * memory transfer progress: 73.26 %
Mon Oct  3 15:04:02 2022 * memory transfer progress: 87.76 %
Mon Oct  3 15:04:10 2022 * memory transfer complete
Mon Oct  3 15:04:10 2022 * closing instance disks on node chi-node-11.torproject.org
Mon Oct  3 15:04:11 2022 * wait until resync is done
Mon Oct  3 15:04:12 2022 * changing into standalone mode
Mon Oct  3 15:04:12 2022 * changing disks into single-master mode
Mon Oct  3 15:04:14 2022 * wait until resync is done
Mon Oct  3 15:04:14 2022 * done
root@chi-node-01:~# gnt-instance modify --disk add:size=20g metrics-psqlts-01.torproject.org
Failure: command execution error:
Can't create block device <LogicalVolume(/dev/vg_ganeti/8053521e-336a-42a2-967a-76f5b0b91032.disk2_data, not visible, size=20480m)> on node chi-node-10.torproject.org for instance metrics-psqlts-01.torproject.org: Can't create block device: Not enough free space: required 20480, available 15108.0
-node-01:~# gnt-instance replace-disks -I . metrics-psqlts-01.torproject.org
Mon Oct  3 15:10:40 2022  - INFO: Selected new secondary for instance '280b14af-c029-4255-873b-48d2e6ba716a': chi-node-03.torproject.org
Mon Oct  3 15:10:40 2022 Replacing disk(s) 0, 1 for instance 'metrics-psqlts-01.torproject.org'
Mon Oct  3 15:10:40 2022 Current primary node: chi-node-10.torproject.org
Mon Oct  3 15:10:40 2022 Current secondary node: chi-node-11.torproject.org
Mon Oct  3 15:10:40 2022 STEP 1/6 Check device existence
Mon Oct  3 15:10:40 2022  - INFO: Checking disk/0 on chi-node-10.torproject.org
Mon Oct  3 15:10:40 2022  - INFO: Checking disk/1 on chi-node-10.torproject.org
Mon Oct  3 15:10:41 2022  - INFO: Checking volume groups
Mon Oct  3 15:10:41 2022 STEP 2/6 Check peer consistency
Mon Oct  3 15:10:41 2022  - INFO: Checking disk/0 consistency on node chi-node-10.torproject.org
Mon Oct  3 15:10:42 2022  - INFO: Checking disk/1 consistency on node chi-node-10.torproject.org
Mon Oct  3 15:10:42 2022 STEP 3/6 Allocate new storage
Mon Oct  3 15:10:42 2022  - INFO: Adding new local storage on chi-node-03.torproject.org for disk/0
Mon Oct  3 15:10:44 2022  - INFO: Adding new local storage on chi-node-03.torproject.org for disk/1
Mon Oct  3 15:10:46 2022 STEP 4/6 Changing drbd configuration
Mon Oct  3 15:10:46 2022  - INFO: activating a new drbd on chi-node-03.torproject.org for disk/0
Mon Oct  3 15:10:49 2022  - INFO: activating a new drbd on chi-node-03.torproject.org for disk/1
Mon Oct  3 15:10:52 2022  - INFO: Shutting down drbd for disk/0 on old node
Mon Oct  3 15:10:53 2022  - INFO: Shutting down drbd for disk/1 on old node
Mon Oct  3 15:10:53 2022  - INFO: Detaching primary drbds from the network (=> standalone)
Mon Oct  3 15:10:54 2022  - INFO: Updating instance configuration
Mon Oct  3 15:10:54 2022  - INFO: Attaching primary drbds to new secondary (standalone => connected)
Mon Oct  3 15:10:56 2022 STEP 5/6 Sync devices
Mon Oct  3 15:10:56 2022  - INFO: Waiting for instance metrics-psqlts-01.torproject.org to sync disks
Mon Oct  3 15:10:57 2022  - INFO: - device disk/0:  0.20% done, 33m 35s remaining (estimated)
Mon Oct  3 15:10:57 2022  - INFO: - device disk/1:  0.60% done, 4m 8s remaining (estimated)
Mon Oct  3 15:11:57 2022  - INFO: - device disk/0: 17.40% done, 4m 22s remaining (estimated)
Mon Oct  3 15:11:57 2022  - INFO: - device disk/1: 82.50% done, 11s remaining (estimated)
Mon Oct  3 15:12:09 2022  - INFO: - device disk/0: 21.60% done, 3m 34s remaining (estimated)
Mon Oct  3 15:13:10 2022  - INFO: - device disk/0: 44.10% done, 2m 23s remaining (estimated)
Mon Oct  3 15:14:10 2022  - INFO: - device disk/0: 66.50% done, 1m 31s remaining (estimated)
Mon Oct  3 15:15:11 2022  - INFO: - device disk/0: 89.20% done, 28s remaining (estimated)
Mon Oct  3 15:15:39 2022  - INFO: - device disk/0: 99.80% done, 0s remaining (estimated)
Mon Oct  3 15:15:40 2022  - INFO: - device disk/0: 100.00% done, 0s remaining (estimated)
Mon Oct  3 15:15:41 2022  - INFO: Instance metrics-psqlts-01.torproject.org's disks are in sync
Mon Oct  3 15:15:41 2022 STEP 6/6 Removing old storage
Mon Oct  3 15:15:41 2022  - INFO: Remove logical volumes for 0
Mon Oct  3 15:15:42 2022  - INFO: Remove logical volumes for 1

      migrating to a new secondary:

      root@chi-node-01:~# gnt-instance migrate metrics-psqlts-01.torproject.org
Instance metrics-psqlts-01.torproject.org will be migrated. Note that
migration might impact the instance if anything goes wrong (e.g. due
to bugs in the hypervisor). Continue?
y/[n]/?: y
Mon Oct  3 15:17:59 2022 Migrating instance metrics-psqlts-01.torproject.org
Mon Oct  3 15:17:59 2022 * checking disk consistency between source and target
Mon Oct  3 15:18:01 2022 * closing instance disks on node chi-node-03.torproject.org
Mon Oct  3 15:18:02 2022 * changing into standalone mode
Mon Oct  3 15:18:02 2022 * changing disks into dual-master mode
Mon Oct  3 15:18:04 2022 * wait until resync is done
Mon Oct  3 15:18:05 2022 * opening instance disks on node chi-node-10.torproject.org in shared mode
Mon Oct  3 15:18:05 2022 * opening instance disks on node chi-node-03.torproject.org in shared mode
Mon Oct  3 15:18:06 2022 * preparing chi-node-03.torproject.org to accept the instance
Mon Oct  3 15:18:06 2022 * migrating instance to chi-node-03.torproject.org
Mon Oct  3 15:18:06 2022 * starting memory transfer
Mon Oct  3 15:18:17 2022 * memory transfer progress: 14.61 %
Mon Oct  3 15:18:28 2022 * memory transfer progress: 29.51 %
Mon Oct  3 15:18:39 2022 * memory transfer progress: 44.07 %
Mon Oct  3 15:18:50 2022 * memory transfer progress: 58.71 %
Mon Oct  3 15:19:01 2022 * memory transfer progress: 73.45 %
Mon Oct  3 15:19:12 2022 * memory transfer progress: 88.33 %
Mon Oct  3 15:19:19 2022 * memory transfer has switched to postcopy
Mon Oct  3 15:19:20 2022 * memory transfer complete
Mon Oct  3 15:19:20 2022 * closing instance disks on node chi-node-10.torproject.org
Mon Oct  3 15:19:21 2022 * wait until resync is done
Mon Oct  3 15:19:22 2022 * changing into standalone mode
Mon Oct  3 15:19:22 2022 * changing disks into single-master mode
Mon Oct  3 15:19:24 2022 * wait until resync is done
Mon Oct  3 15:19:25 2022 * done

      reallocating secondary:

      ot@chi-node-01:~# gnt-instance replace-disks -I . metrics-psqlts-01.torproject.org
Mon Oct  3 15:25:27 2022  - INFO: Selected new secondary for instance '280b14af-c029-4255-873b-48d2e6ba716a': chi-node-11.torproject.org
Mon Oct  3 15:25:27 2022 Replacing disk(s) 0, 1 for instance 'metrics-psqlts-01.torproject.org'
Mon Oct  3 15:25:27 2022 Current primary node: chi-node-03.torproject.org
Mon Oct  3 15:25:27 2022 Current secondary node: chi-node-10.torproject.org
Mon Oct  3 15:25:27 2022 STEP 1/6 Check device existence
Mon Oct  3 15:25:27 2022  - INFO: Checking disk/0 on chi-node-03.torproject.org
Mon Oct  3 15:25:27 2022  - INFO: Checking disk/1 on chi-node-03.torproject.org
Mon Oct  3 15:25:27 2022  - INFO: Checking volume groups
Mon Oct  3 15:25:28 2022 STEP 2/6 Check peer consistency
Mon Oct  3 15:25:28 2022  - INFO: Checking disk/0 consistency on node chi-node-03.torproject.org
Mon Oct  3 15:25:28 2022  - INFO: Checking disk/1 consistency on node chi-node-03.torproject.org
Mon Oct  3 15:25:29 2022 STEP 3/6 Allocate new storage
Mon Oct  3 15:25:29 2022  - INFO: Adding new local storage on chi-node-11.torproject.org for disk/0
Mon Oct  3 15:25:31 2022  - INFO: Adding new local storage on chi-node-11.torproject.org for disk/1
Mon Oct  3 15:25:32 2022 STEP 4/6 Changing drbd configuration
Mon Oct  3 15:25:32 2022  - INFO: activating a new drbd on chi-node-11.torproject.org for disk/0
Mon Oct  3 15:25:35 2022  - INFO: activating a new drbd on chi-node-11.torproject.org for disk/1
Mon Oct  3 15:25:37 2022  - INFO: Shutting down drbd for disk/0 on old node
Mon Oct  3 15:25:38 2022  - INFO: Shutting down drbd for disk/1 on old node
Mon Oct  3 15:25:39 2022  - INFO: Detaching primary drbds from the network (=> standalone)
Mon Oct  3 15:25:40 2022  - INFO: Updating instance configuration
Mon Oct  3 15:25:40 2022  - INFO: Attaching primary drbds to new secondary (standalone => connected)
Mon Oct  3 15:25:42 2022 STEP 5/6 Sync devices
Mon Oct  3 15:25:42 2022  - INFO: Waiting for instance metrics-psqlts-01.torproject.org to sync disks
Mon Oct  3 15:25:42 2022  - INFO: - device disk/0:  0.40% done, 5m 22s remaining (estimated)
Mon Oct  3 15:25:42 2022  - INFO: - device disk/1:  2.00% done, 55s remaining (estimated)
Mon Oct  3 15:26:38 2022  - INFO: - device disk/0: 20.80% done, 3m 32s remaining (estimated)
Mon Oct  3 15:27:41 2022  - INFO: - device disk/0: 37.70% done, 4m 25s remaining (estimated)
Mon Oct  3 15:28:45 2022  - INFO: - device disk/0: 50.90% done, 4m 48s remaining (estimated)
Mon Oct  3 15:29:46 2022  - INFO: - device disk/0: 65.50% done, 3m 11s remaining (estimated)
Mon Oct  3 15:30:49 2022  - INFO: - device disk/0: 74.50% done, 3m 38s remaining (estimated)
Mon Oct  3 15:32:02 2022  - INFO: - device disk/0: 82.00% done, 6m 49s remaining (estimated)
Mon Oct  3 15:33:03 2022  - INFO: Instance metrics-psqlts-01.torproject.org's disks are in sync
Mon Oct  3 15:33:04 2022 STEP 6/6 Removing old storage
Mon Oct  3 15:33:04 2022  - INFO: Remove logical volumes for 0
Mon Oct  3 15:33:04 2022  - INFO: Remove logical volumes for 1

      next up is adding the disk now.

      Edited by anarcat
    • anarcat
      anarcat @anarcat ·
      Owner

      @hiro ETA: 1h

      adding the disk:

      root@chi-node-01:~# gnt-instance modify --disk add:size=20g metrics-psqlts-01.torproject.org
Mon Oct  3 15:35:59 2022  - INFO: Waiting for instance metrics-psqlts-01.torproject.org to sync disks
Mon Oct  3 15:35:59 2022  - INFO: - device disk/2:  0.10% done, 51m 23s remaining (estimated)
    • anarcat
      anarcat @anarcat ·
      Owner

      almost done, that was wildly over-estimating.

      root@chi-node-01:~# gnt-instance modify --disk add:size=20g metrics-psqlts-01.torproject.org
Mon Oct  3 15:35:59 2022  - INFO: Waiting for instance metrics-psqlts-01.torproject.org to sync disks
Mon Oct  3 15:35:59 2022  - INFO: - device disk/2:  0.10% done, 51m 23s remaining (estimated)
Mon Oct  3 15:37:02 2022  - INFO: - device disk/2: 10.40% done, 9m 16s remaining (estimated)
Mon Oct  3 15:38:02 2022  - INFO: - device disk/2: 21.60% done, 6m 44s remaining (estimated)
Mon Oct  3 15:39:03 2022  - INFO: - device disk/2: 31.90% done, 7m 48s remaining (estimated)
Mon Oct  3 15:40:03 2022  - INFO: - device disk/2: 42.50% done, 6m 14s remaining (estimated)
Mon Oct  3 15:41:04 2022  - INFO: - device disk/2: 52.10% done, 5m 7s remaining (estimated)
Mon Oct  3 15:42:04 2022  - INFO: - device disk/2: 61.70% done, 3m 28s remaining (estimated)
Mon Oct  3 15:43:05 2022  - INFO: - device disk/2: 69.90% done, 3m 21s remaining (estimated)
Mon Oct  3 15:44:06 2022  - INFO: - device disk/2: 76.60% done, 3m 23s remaining (estimated)
Mon Oct  3 15:45:07 2022  - INFO: - device disk/2: 84.30% done, 3m 20s remaining (estimated)
Mon Oct  3 15:46:11 2022  - INFO: - device disk/2: 87.20% done, 6m 57s remaining (estimated)
Mon Oct  3 15:47:17 2022  - INFO: - device disk/2: 89.00% done, 6m 8s remaining (estimated)
Mon Oct  3 15:48:18 2022  - INFO: - device disk/2: 92.60% done, 2m 4s remaining (estimated)
Mon Oct  3 15:49:21 2022  - INFO: - device disk/2: 97.10% done, 1m 17s remaining (estimated)
Mon Oct  3 15:50:21 2022  - INFO: Instance metrics-psqlts-01.torproject.org's disks are in sync
Modified instance metrics-psqlts-01.torproject.org
 - disk/2 -> add:size=20480,mode=rw
Please don't forget that most parameters take effect only at the next (re)start of the instance initiated by ganeti; restarting from within the instance will not be enough.
      Edited by anarcat
    • anarcat
      anarcat @anarcat ·
      Owner

      i rebooted the server for the change to take effect, formatted /dev/sdc in ext4+journaling, mounted it on /srv (after moving the parser thing out of there), and bind-mounted /var/lib/postgresql in there.

      restarted postgresql, this seems resolved. let me know if you find any problem, @hiro

    • anarcat closed

      closed

    • anarcat changed the incident status to Resolved by closing the incident

      changed the incident status to Resolved by closing the incident

    • Kez
      Kez @kez ·
      Author

      The disk has already filled up again: DISK WARNING - free space: / 1820 MB (19% inode=88%):

    • Kez reopened

      reopened

    • anarcat
      anarcat @anarcat ·
      Owner

      it looks like postgresql has a 4.5GB /var/log/postgresql/postgresql-13-main.log file... this might be related to the fact that DNS is hosed because unbound is crashed, possibly because we ran out of disk. fun times.

    • anarcat
      anarcat @anarcat ·
      Owner

      unbound was crashed because it created empty anchor files in /var/lib/unbound. i removed them, put a dummy external nameserver in resolv.conf, reran puppet (so it restores the anchor files) and it restarted everything properly (including bacula as well).

      now i'm doing this:

      root@metrics-psqlts-01:/var/log/postgresql# logrotate -f /etc//logrotate.d/postgresql-common

      to rotate that huge logfile out of existence.

    • anarcat
      anarcat @anarcat ·
      Owner 
      root@metrics-psqlts-01:/var/log/postgresql# logrotate -f /etc//logrotate.d/postgresql-common
error: error writing to /var/log/postgresql/postgresql-13-main.log.1: No space left on device
error: error copying /var/log/postgresql/postgresql-13-main.log to /var/log/postgresql/postgresql-13-main.log.1: No space left on device

      i just cleared that log file for now.

      fwiw, that log file had a lot of warnings about being unable to connect to the backup server because DNS was done, it could be why it filled up the disk after the disk filled up on its own...

      now, it looks like everything is back in order. i did see a lot of queries logged in there, so this might happen again, but that would be @hiro's problem then. :)

    • anarcat closed

      closed

    • anarcat changed the incident status to Resolved by closing the incident

      changed the incident status to Resolved by closing the incident

    • anarcat mentioned in commit wiki-replica@e465b607

      mentioned in commit wiki-replica@e465b607

    Please register or sign in to reply