Skip to content
Snippets Groups Projects

backup server (bungei) volume filled up

    • Closed Incident created by Jérôme Charaoui @lavamind

    Bacula backups have started failing today, amid warnings about free space on the server.

    @anarcat investigated and found out that archive-01 that has grown seemingly out of control:

    yep, so archive-01's /srv/archive.torproject.org/htdocs/tor-package-archive/torbrowser is 5TB

    Space reservation was reduced from 0.5 to 0.01 percent which should give enough room for the upcoming backup to succeed.

    Next steps:

    • reduce reserved space
    • grow logical volume
    • unlock mounts
    • make sure the archive-01 job completes
    • make sure all backups have ran in the last 24h (ie. make nagios green again)
    • open a ticket for or just make a plan for the next time
    • document this in the backup pager playbook
    ✓ 7 of 7 checklist items completed · Edited

    Child items ...

    • Activity

    • Jérôme Charaoui added Backup Doing Emergency labels

      added Backup Doing Emergency labels

    • Jérôme Charaoui assigned to @anarcat

      assigned to @anarcat

    • anarcat
      anarcat @anarcat ·
      Owner

      some notes from IRC:

      15:39:32 <anarcat> our global disk usage went up by 5TiB in the last *day*
15:40:21 <anarcat> if i read this right: https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&from=now-7d&to=now
15:41:24 <anarcat> like wtf happened here https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&from=now-7d&to=now&viewPanel=1137
15:43:21 <anarcat> well fuck
15:47:13 <anarcat> i ran ncdu on bungei
15:47:18 <anarcat> archive-01 is using 27TiB
15:48:05 <anarcat> it could be the *rotation* on there, bacula would still have the older versions of the files and things got overwritten on archive-01, so 
                   we don't see it rising, but it's taking up more space in bacula
15:48:56 <anarcat> yeah, it's archive-01
15:49:13 <anarcat> it's been pounding one new 500GB volume after the other for 24h now
15:49:21 <anarcat> -rw-r-----  1 bacula bacula  500G Oct 17 23:33 torproject-archive-01.torproject.org-full.2023-10-17_20:18
15:49:24 <anarcat> -rw-r-----  1 bacula bacula  500G Oct 18 01:33 torproject-archive-01.torproject.org-full.2023-10-17_23:34
15:49:29 <anarcat> ...
15:49:29 <anarcat> -rw-r-----  1 bacula bacula  500G Oct 18 12:01 torproject-archive-01.torproject.org-full.2023-10-18_10:15
15:49:33 <anarcat> -rw-r-----  1 bacula bacula  349G Oct 18 13:13 torproject-archive-01.torproject.org-full.2023-10-18_12:01
15:51:02 <anarcat> i'm going to tune down the reserved block size
15:51:33 <anarcat> it seems we have 2.7TB reserved
15:51:52 <anarcat> which is half a percent. i think
15:52:07 <anarcat> Setting reserved blocks percentage to 0.01% (1530082 blocks)
15:53:56 <anarcat> Filesystem                           Size  Used Avail Use% Mounted on
15:53:57 <anarcat> /dev/mapper/vg_bulk-backups--bacula   57T   55T  2.5T  96% /srv/backups/bacula
15:55:22 <anarcat> yep, so archive-01's /srv/archive.torproject.org/htdocs/tor-package-archive/torbrowser is 5TB
15:58:32 <anarcat> i ran the "mount" command to unblock the backup jobs for relay-01 and archive-01
15:58:36 <anarcat> we might run out of disk space again
16:00:56 <anarcat> i suspect this might work better with borg
16:01:04 <anarcat> but i've never scaled borg up to ~100TiB
16:03:10 <anarcat> oh dear
16:03:10 <anarcat>  JobId  Level      Files    Bytes   Status   Finished        Name 
16:03:10 <anarcat> ====================================================================
16:03:15 <anarcat> 246156  Full     684,887    4.665 T  Other    18-Oct-23 13:15 archive-01.torproject.org
16:03:21 <anarcat> so it looks like the full failed
16:03:27 <anarcat> and it's going to RETRY IT
16:03:32 <anarcat> so we're going to full up *another* 5TB
16:07:22 <anarcat> i cleared the files from the previous, failed full backup
16:07:42 <anarcat> it's possible this damage the files from the current full backup, but that beats disabling backups for the entire infrastructure
16:07:52 <anarcat> we now have 6.7T left

      so tldr: it seems the TB archive is 5TB and was fully rotated with the 13 release, which meant the full added a 5TiB of storage on the backup server. the archive-01 backup job failed, then when i allocated more diskspace (by tuning the free space), the job could resume (with the mount command) but then in a new set volume, hoping that the new ~2TB of disk space would be sufficient.

    • anarcat
      anarcat @anarcat ·
      Owner

      and update this morning: it wasn't. server is still out of disk space and all nightlies have failed.

    • anarcat changed the severity to High - S2

      changed the severity to High - S2

    • anarcat changed the incident status to Acknowledged

      changed the incident status to Acknowledged

    • anarcat
      anarcat @anarcat ·
      Owner

      actually, the server is not out of disk space:

      root@bungei:/srv/backups/bacula# df -h 
Filesystem                           Size  Used Avail Use% Mounted on
udev                                  63G     0   63G   0% /dev
tmpfs                                 13G  1.2M   13G   1% /run
/dev/mapper/croot                     20G  4.6G   14G  25% /
tmpfs                                 63G  104K   63G   1% /dev/shm
tmpfs                                5.0M     0  5.0M   0% /run/lock
tmpfs                                4.0G     0  4.0G   0% /tmp
/dev/md125                           486M  152M  305M  34% /boot
/dev/mapper/vg_bulk-backups--pg      3.0T  1.6T  1.3T  56% /srv/backups/pg
/dev/mapper/vg_bulk-backups--bacula   57T   56T  1.7T  98% /srv/backups/bacula
tmpfs                                 13G     0   13G   0% /run/user/0
tmpfs                                 13G     0   13G   0% /run/user/1529

      ... but maybe bacula is unhappy because there isn't enough free space to go around (@lavamind's idea). so let's just give that poor server more space, we still have some wiggle room.

      resizing logical volume:

      root@bungei:/srv/backups/bacula# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  vg_bulk   1   2   0 wz--n- 72.60t 12.60t
root@bungei:/srv/backups/bacula# lvs
  LV             VG      Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  backups-bacula vg_bulk -wi-ao---- 57.00t                                                    
  backups-pg     vg_bulk -wi-ao----  3.00t                                                    
root@bungei:/srv/backups/bacula# lvextend vg_bulk/backups-bacula -L +10T
  Size of logical volume vg_bulk/backups-bacula changed from 57.00 TiB (14942208 extents) to 67.00 TiB (17563648 extents).
  Logical volume vg_bulk/backups-bacula successfully resized.
root@bungei:/srv/backups/bacula# lvs
  LV             VG      Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  backups-bacula vg_bulk -wi-ao---- 67.00t                                                    
  backups-pg     vg_bulk -wi-ao----  3.00t                                                    
root@bungei:/srv/backups/bacula# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  vg_bulk   1   2   0 wz--n- 72.60t 2.60t
root@bungei:/srv/backups/bacula# resize2fs /dev/mapper/vg_bulk-backups--bacula 
resize2fs 1.46.2 (28-Feb-2021)
Filesystem at /dev/mapper/vg_bulk-backups--bacula is mounted on /srv/backups/bacula; on-line resizing required
old_desc_blocks = 7296, new_desc_blocks = 8576

      last command still in progress as of right now.

    • anarcat
      anarcat @anarcat ·
      Owner

      command finally completed with:

      The filesystem on /dev/mapper/vg_bulk-backups--bacula is now 17985175552 (4k) blocks long.

      and we're now at 83% usage:

      Filesystem                           Size  Used Avail Use% Mounted on
/dev/mapper/vg_bulk-backups--bacula   67T   56T   12T  83% /srv/backups/bacula

      we need to long at other options here, probably throwing hardware at this for now.

      Edited by anarcat
    • anarcat
      anarcat @anarcat ·
      Owner

      i've tried unblocking all jobs:

       JobId  Type Level     Files     Bytes  Name              Status
======================================================================
246219  Back Full    723,866    5.763 T archive-01.torproject.org is running
246221  Back Incr          0         0  onionoo-backend-01.torproject.org is running
246222  Back Incr          0         0  dangerzone-01.torproject.org is waiting for a mount request
246223  Back Incr          0         0  ns5.torproject.org is waiting for a mount request
246224  Back Incr          0         0  tb-build-05.torproject.org is waiting for a mount request
246225  Back Incr          0         0  crm-ext-01.torproject.org is waiting for a mount request
246226  Back Incr          0         0  media-01.torproject.org is waiting for a mount request
246227  Back Incr          0         0  weather-01.torproject.org is waiting for a mount request
246228  Back Incr          0         0  neriniflorum.torproject.org is waiting for a mount request
246229  Back Incr          0         0  tb-build-02.torproject.org is waiting for a mount request
246230  Back Incr          0         0  survey-01.torproject.org is waiting for a mount request
====

Terminated Jobs:
 JobId  Level      Files    Bytes   Status   Finished        Name 
====================================================================
246241  Incr       1,748    204.2 M  OK       18-Oct-23 22:37 fsn-node-02.torproject.org
246242  Incr         208    65.54 M  OK       18-Oct-23 22:43 tb-tester-01.torproject.org
246244  Incr         646    450.3 M  OK       18-Oct-23 22:48 rude.torproject.org
246245  Incr         302    249.2 M  OK       18-Oct-23 23:04 perdulce.torproject.org
246246  Incr       1,712    176.3 M  OK       18-Oct-23 23:09 fsn-node-05.torproject.org
246247  Incr         345    6.928 G  OK       18-Oct-23 23:23 probetelemetry-01.torproject.org
246248  Incr         407    6.120 G  OK       18-Oct-23 23:33 minio-01.torproject.org
246249  Incr         303    56.57 M  OK       18-Oct-23 23:34 dal-rescue-01.torproject.org
246250  Incr      29,385    62.60 G  OK       19-Oct-23 01:14 tb-build-06.torproject.org
246220  Incr         180    282.8 M  OK       19-Oct-23 15:16 ci-runner-x86-02.torproject.org

====
*mount jobid=246222
3001 OK mount requested. Device="FileStorage-dangerzone-01.torproject.org" (/srv/backups/bacula/dangerzone-01.torproject.org)
*mount jobid=246223
3001 OK mount requested. Device="FileStorage-ns5.torproject.org" (/srv/backups/bacula/ns5.torproject.org)
*mount jobid=246224
3001 OK mount requested. Device="FileStorage-tb-build-05.torproject.org" (/srv/backups/bacula/tb-build-05.torproject.org)
*mount jobid=246225
3001 OK mount requested. Device="FileStorage-crm-ext-01.torproject.org" (/srv/backups/bacula/crm-ext-01.torproject.org)
^[[A*mount jobid=246226
3001 OK mount requested. Device="FileStorage-media-01.torproject.org" (/srv/backups/bacula/media-01.torproject.org)
*mount jobid=246227
3001 OK mount requested. Device="FileStorage-weather-01.torproject.org" (/srv/backups/bacula/weather-01.torproject.org)
*mount jobid=246228
3001 OK mount requested. Device="FileStorage-neriniflorum.torproject.org" (/srv/backups/bacula/neriniflorum.torproject.org)
*mount jobid=246229
3001 OK mount requested. Device="FileStorage-tb-build-02.torproject.org" (/srv/backups/bacula/tb-build-02.torproject.org)
*mount jobid=246230
3001 OK mount requested. Device="FileStorage-survey-01.torproject.org" (/srv/backups/bacula/survey-01.torproject.org)
*
    • anarcat
      anarcat @anarcat ·
      Owner

      and it looks like that worked, suspiciously quickly:

       JobId  Level      Files    Bytes   Status   Finished        Name 
====================================================================
246250  Incr      29,385    62.60 G  OK       19-Oct-23 01:14 tb-build-06.torproject.org
246220  Incr         180    282.8 M  OK       19-Oct-23 15:16 ci-runner-x86-02.torproject.org
246226  Incr         237    83.55 M  OK       19-Oct-23 15:17 media-01.torproject.org
246222  Incr         259    208.7 M  OK       19-Oct-23 15:17 dangerzone-01.torproject.org
246227  Incr         284    150.1 M  OK       19-Oct-23 15:17 weather-01.torproject.org
246225  Incr       1,296    249.0 M  OK       19-Oct-23 15:17 crm-ext-01.torproject.org
246223  Incr         220    1.181 G  OK       19-Oct-23 15:17 ns5.torproject.org
246228  Incr         230    1.596 G  OK       19-Oct-23 15:17 neriniflorum.torproject.org
246230  Incr         304    80.13 M  OK       19-Oct-23 15:17 survey-01.torproject.org
246229  Incr         264    191.4 M  OK       19-Oct-23 15:17 tb-build-02.torproject.org
    • anarcat added an incident timeline event

      added an incident timeline event

    • anarcat edited the event time/date and text on incident timeline event

      edited the event time/date and text on incident timeline event

    • anarcat added an incident timeline event

      added an incident timeline event

    • anarcat added an incident timeline event

      added an incident timeline event

    • anarcat edited the event time/date and text on incident timeline event

      edited the event time/date and text on incident timeline event

    • anarcat edited the event time/date and text on incident timeline event

      edited the event time/date and text on incident timeline event

    • anarcat added an incident timeline event

      added an incident timeline event

    • anarcat edited the event time/date and text on incident timeline event

      edited the event time/date and text on incident timeline event

    • anarcat added an incident timeline event

      added an incident timeline event

    • anarcat added an incident timeline event

      added an incident timeline event

    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    Please register or sign in to reply