PostgreSQL backups are failing

assigned to @anarcat

changed the incident status to Acknowledged

marked this incident as related to #41388 (closed)

[materculae, main] WAL-MISSING-AFTER: materculae/main.WAL.00000001000005C3000000FC

okay, those are due to the #41388 (closed) outage: when the /srv/backups/pg directory was unmounted, all those WAL files were written to... well, who the hell knows, nowhere? under the mountpoint? rghl.

that was indeed the problem, and easy to recover from:

root@bungei:/srv/backups/pg/materculae# mount --bind / /mnt
root@bungei:/srv/backups/pg/materculae# df
Filesystem                            1K-blocks        Used   Available Use% Mounted on
udev                                   65870652           0    65870652   0% /dev
tmpfs                                  13179448        1132    13178316   1% /run
/dev/mapper/croot                      20447824    18601700      782088  96% /
tmpfs                                  65897232         368    65896864   1% /dev/shm
tmpfs                                      5120           0        5120   0% /run/lock
tmpfs                                   4194304           0     4194304   0% /tmp
/dev/md125                               496658      154973      311404  34% /boot
/dev/mapper/vg_bulk-backups--bacula 71653916048 55066360972 16581418364  77% /srv/backups/bacula
tmpfs                                  13179444           0    13179444   0% /run/user/0
/dev/mapper/vg_bulk-backups--pg      3169547176  1893676212  1136270276  63% /srv/backups/pg
root@bungei:/srv/backups/pg/materculae# ls /mnt/srv/backups/pg/
bacula-director-01  materculae  metricsdb-01  polyanthum  puppetdb-01  rude  weather-01
root@bungei:/srv/backups/pg/materculae# find /mnt/srv/backups/pg/
/mnt/srv/backups/pg/
/mnt/srv/backups/pg/.nobackup
/mnt/srv/backups/pg/puppetdb-01
/mnt/srv/backups/pg/puppetdb-01/main.WAL.0000000100000003000000BB
/mnt/srv/backups/pg/puppetdb-01/main.WAL.0000000100000003000000BA
/mnt/srv/backups/pg/weather-01
/mnt/srv/backups/pg/weather-01/main.WAL.00000001000000000000002D
/mnt/srv/backups/pg/bacula-director-01
/mnt/srv/backups/pg/bacula-director-01/main.WAL.00000001000009EF0000005E
/mnt/srv/backups/pg/materculae
/mnt/srv/backups/pg/materculae/main.WAL.00000001000005C3000000FE
/mnt/srv/backups/pg/materculae/main.WAL.00000001000005C3000000FD
/mnt/srv/backups/pg/metricsdb-01
/mnt/srv/backups/pg/metricsdb-01/main.WAL.0000000100000344000000CC
/mnt/srv/backups/pg/metricsdb-01/main.WAL.0000000100000344000000C9
/mnt/srv/backups/pg/metricsdb-01/main.WAL.0000000100000344000000CB
/mnt/srv/backups/pg/metricsdb-01/main.WAL.0000000100000344000000CA
/mnt/srv/backups/pg/metricsdb-01/main.WAL.0000000100000344000000C8
/mnt/srv/backups/pg/rude
/mnt/srv/backups/pg/rude/main.WAL.0000000200000103000000E3
/mnt/srv/backups/pg/polyanthum
/mnt/srv/backups/pg/polyanthum/main.WAL.000000010000003600000015
/mnt/srv/backups/pg/polyanthum/main.WAL.00000001000000360000000F
/mnt/srv/backups/pg/polyanthum/main.WAL.000000010000003600000012
/mnt/srv/backups/pg/polyanthum/main.WAL.00000001000000360000000B
/mnt/srv/backups/pg/polyanthum/main.WAL.000000010000003600000010
/mnt/srv/backups/pg/polyanthum/main.WAL.000000010000003600000016
/mnt/srv/backups/pg/polyanthum/main.WAL.000000010000003600000017
/mnt/srv/backups/pg/polyanthum/main.WAL.000000010000003600000014
/mnt/srv/backups/pg/polyanthum/main.WAL.00000001000000360000000D
/mnt/srv/backups/pg/polyanthum/main.WAL.00000001000000360000000E
/mnt/srv/backups/pg/polyanthum/main.WAL.000000010000003600000011
/mnt/srv/backups/pg/polyanthum/main.WAL.000000010000003600000013
/mnt/srv/backups/pg/polyanthum/main.WAL.00000001000000360000000C
root@bungei:/srv/backups/pg/materculae# rsync -r -t -v /mnt/srv/backups/pg/ /srv/backups/pg/
sending incremental file list
./
.nobackup
bacula-director-01/
bacula-director-01/main.WAL.00000001000009EF0000005E
materculae/
materculae/main.WAL.00000001000005C3000000FD
materculae/main.WAL.00000001000005C3000000FE
metricsdb-01/
metricsdb-01/main.WAL.0000000100000344000000C8
metricsdb-01/main.WAL.0000000100000344000000C9
metricsdb-01/main.WAL.0000000100000344000000CA
metricsdb-01/main.WAL.0000000100000344000000CB
metricsdb-01/main.WAL.0000000100000344000000CC
polyanthum/
polyanthum/main.WAL.00000001000000360000000B
polyanthum/main.WAL.00000001000000360000000C
polyanthum/main.WAL.00000001000000360000000D
polyanthum/main.WAL.00000001000000360000000E
polyanthum/main.WAL.00000001000000360000000F
polyanthum/main.WAL.000000010000003600000010
polyanthum/main.WAL.000000010000003600000011
polyanthum/main.WAL.000000010000003600000012
polyanthum/main.WAL.000000010000003600000013
polyanthum/main.WAL.000000010000003600000014
polyanthum/main.WAL.000000010000003600000015
polyanthum/main.WAL.000000010000003600000016
polyanthum/main.WAL.000000010000003600000017
puppetdb-01/
puppetdb-01/main.WAL.0000000100000003000000BA
puppetdb-01/main.WAL.0000000100000003000000BB
rude/
rude/main.WAL.0000000200000103000000E3
weather-01/
weather-01/main.WAL.00000001000000000000002D

sent 419,534,890 bytes  received 573 bytes  119,867,275.14 bytes/sec
total size is 419,430,400  speedup is 1.00

that fixes almost all warnings:

Click to expand

root@bungei:/srv/backups/pg/materculae# /usr/lib/nagios/plugins/dsa-check-backuppg  -e
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000060
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000005F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000005E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000005D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000005C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000005B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000005A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000059
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000058
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000057
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000056
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000055
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000054
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000053
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000052
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000051
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000050
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000004F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000004E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000004D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000004C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000004B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000004A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000049
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000048
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000047
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000046
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000045
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000044
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000043
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000042
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000041
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000040
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000003F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000003E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000003D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000003C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000003B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000003A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000039
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000038
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000037
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000036
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000035
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000034
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000033
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000032
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000031
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000030
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000002F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000002E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000002D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000002C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000002B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000002A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000029
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000028
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000027
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000026
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000025
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000024
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000023
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000022
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000021
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000020
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000001F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000001E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000001D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000001C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000001B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000001A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000019
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000018
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000017
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000016
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000015
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000014
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000013
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000012
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000011
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000010
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000000F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000000E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000000D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000000C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000000B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001040000000A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000009
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000008
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000007
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000006
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000005
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000004
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000003
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000002
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000001
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010400000000
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000FF
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000FE
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000FD
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000FC
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000FB
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000FA
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000F9
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000F8
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000F7
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000F6
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000F5
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000F4
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000F3
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000F2
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000F1
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000F0
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000EF
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000EE
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000ED
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000EC
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000EB
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000EA
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000E9
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000E8
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000E7
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000E6
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000E5
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000E4
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000E3
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000E2
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000E1
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000E0
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000DF
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000DE
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000DD
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000DC
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000DB
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000DA
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000D9
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000D8
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000D7
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000D6
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000D5
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000D4
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000D3
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000D2
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000D1
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000D0
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000CF
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000CE
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000CD
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000CC
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000CB
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000CA
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000C9
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000C8
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000C7
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000C6
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000C5
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000C4
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000C3
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000C2
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000C1
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000C0
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000BF
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000BE
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000BD
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000BC
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000BB
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000BA
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000B9
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000B8
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000B7
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000B6
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000B5
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000B4
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000B3
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000B2
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000B1
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000B0
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000AF
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000AE
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000AD
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000AC
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000AB
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000AA
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000A9
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000A8
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000A7
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000A6
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000A5
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000A4
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000A3
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000A2
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000A1
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000103000000A0
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000009F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000009E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000009D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000009C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000009B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000009A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000099
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000098
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000097
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000096
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000095
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000094
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000093
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000092
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000091
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000090
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000008F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000008E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000008D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000008C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000008B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000008A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000089
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000088
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000087
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000086
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000085
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000084
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000083
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000082
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000081
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000080
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000007F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000007E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000007D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000007C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000007B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000007A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000079
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000078
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000077
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000076
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000075
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000074
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000073
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000072
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000071
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000070
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000006F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000006E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000006D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000006C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000006B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000006A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000069
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000068
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000067
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000066
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000065
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000064
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000063
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000062
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000061
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000060
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000005F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000005E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000005D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000005C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000005B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000005A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000059
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000058
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000057
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000056
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000055
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000054
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000053
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000052
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000051
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000050
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000004F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000004E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000004D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000004C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000004B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000004A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000049
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000048
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000047
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000046
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000045
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000044
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000043
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000042
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000041
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000040
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000003F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000003E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000003D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000003C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000003B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000003A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000039
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000038
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000037
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000036
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000035
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000034
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000033
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000032
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000031
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000030
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000002F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000002E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000002D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000002C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000002B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000002A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000029
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000028
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000027
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000026
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000025
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000024
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000023
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000022
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000021
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000020
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000001F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000001E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000001D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000001C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000001B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000001A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000019
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000018
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000017
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000016
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000015
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000014
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000013
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000012
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000011
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000010
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000000F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000000E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000000D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000000C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000000B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001030000000A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000009
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000008
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000007
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000006
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000005
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000004
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000003
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000002
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000001
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010300000000
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000FF
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000FE
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000FD
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000FC
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000FB
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000FA
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000F9
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000F8
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000F7
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000F6
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000F5
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000F4
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000F3
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000F2
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000F1
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000F0
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000EF
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000EE
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000ED
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000EC
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000EB
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000EA
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000E9
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000E8
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000E7
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000E6
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000E5
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000E4
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000E3
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000E2
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000E1
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000E0
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000DF
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000DE
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000DD
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000DC
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000DB
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000DA
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000D9
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000D8
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000D7
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000D6
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000D5
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000D4
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000D3
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000D2
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000D1
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000D0
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000CF
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000CE
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000CD
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000CC
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000CB
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000CA
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000C9
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000C8
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000C7
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000C6
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000C5
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000C4
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000C3
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000C2
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000C1
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000C0
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000BF
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000BE
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000BD
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000BC
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000BB
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000BA
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000B9
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000B8
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000B7
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000B6
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000B5
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000B4
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000B3
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000B2
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000B1
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000B0
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000AF
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000AE
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000AD
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000AC
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000AB
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000AA
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000A9
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000A8
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000A7
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000A6
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000A5
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000A4
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000A3
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000A2
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000A1
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.0000000200000102000000A0
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000009F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000009E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000009D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000009C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000009B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000009A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000099
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000098
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000097
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000096
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000095
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000094
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000093
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000092
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000091
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000090
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000008F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000008E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000008D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000008C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000008B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000008A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000089
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000088
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000087
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000086
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000085
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000084
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000083
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000082
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000081
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000080
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000007F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000007E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000007D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000007C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000007B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000007A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000079
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000078
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000077
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000076
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000075
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000074
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000073
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000072
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000071
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000070
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000006F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000006E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000006D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000006C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000006B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000006A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000069
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000068
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000067
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000066
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000065
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000064
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000063
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000062
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000061
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000060
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000005F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000005E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000005D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000005C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000005B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000005A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000059
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000058
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000057
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000056
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000055
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000054
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000053
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000052
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000051
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000050
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000004F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000004E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000004D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000004C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000004B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000004A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000049
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000048
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000047
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000046
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000045
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000044
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000043
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000042
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000041
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000040
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000003F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000003E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000003D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000003C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000003B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000003A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000039
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000038
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000037
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000036
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000035
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000034
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000033
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000032
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000031
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000030
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000002F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000002E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000002D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000002C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000002B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000002A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000029
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000028
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000027
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000026
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000025
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000024
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000023
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000022
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000021
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000020
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000001F
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000001E
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000001D
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000001C
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000001B
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.00000002000001020000001A
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000019
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000018
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000017
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000016
[rude, main] UNEXPECTED-TIMELINE: rude/main.WAL.000000020000010200000015
[rude, main] CANNOT-PARSE-WAL: rude/main.WAL.00000002.history
[rude, main] WAL-MISSING-AFTER: rude/main.WAL.000000010000010200000010
[rude, main] WAL-IS-OLD: latest wal file is too old
[weather-01, main] WAL-MISSING-AFTER: weather-01/main.WAL.000000010000000000000002
NOT-CONFIGURED: survey-01-pg13
NOT-CONFIGURED: rude-13

remains:

rude's WAL-IS-OLD and more
weather's WAL-MISSING-AFTER
NOT-CONFIGURED warnings (harmless)

changed the description

[rude, main] CANNOT-PARSE-WAL: rude/main.WAL.00000002.history

this, at least, is easy to fix:

root@bungei:/srv/backups/pg/rude# rm main.WAL.00000002.history

less didn't keep a backlog, but it was a warning about a failed recovery attempt, which I judged harmless to remove.

i think all other problems stem from the structure of the first WAL files:

total 24705868
-rw-------  1 torbackup torbackup   16777216 Nov  1 18:49 main.WAL.00000001000001020000000B
-rw-------  1 torbackup torbackup   16777216 Nov  1 18:50 main.WAL.00000001000001020000000C
-rw-------  1 torbackup torbackup   16777216 Nov  1 18:50 main.WAL.000000010000000000000001
-rw-------  1 torbackup torbackup   16777216 Nov  1 18:50 main.WAL.000000010000010200000010
-rw-------  1 torbackup torbackup   16777216 Nov  1 18:50 main.WAL.000000010000010200000012
-rw-------  1 torbackup torbackup   16777216 Nov  1 18:52 main.WAL.000000010000010200000013
-rw-------  1 torbackup torbackup   16777216 Nov  1 18:54 main.WAL.000000010000010200000014
-rw-------  1 torbackup torbackup        394 Nov  1 18:54 main.WAL.000000010000010200000014.00000028.backup
-rw-------  1 torbackup torbackup 3744234270 Nov  1 18:54 main.BASE.bungei.torproject.org-20231101-185240-rude.torproject.org-main-15-backup.tar.gz
-rw-------  1 torbackup torbackup   16777216 Nov  1 19:54 main.WAL.000000010000010200000015

notice the gaps in the meginning and the weird main.WAL.000000010000000000000001 file there... i'll remove a bunch of those:

rm main.WAL.00000001000001020000000B
rm main.WAL.00000001000001020000000C
rm main.WAL.000000010000000000000001
rm main.WAL.000000010000010200000010

but the UNEXPECTED-TIMELINE and WAL-IS-OLD warnings remain.

i think i found what the problem is. all the files the script is complaining about are named main.WAL.00000002*, as opposed to main.WAL.00000001*. That's because we're archiving a separate timeline. That is probably because of the way @lavamind performed a restore on the database server, but i'm not exactly sure how or why.

that, it turns out:

root@bungei:/srv/backups/pg/rude# rm main.WAL.00000002.history

... is when the timeline was split, I think.

i have deleted that file on the backup server, but it's still present on rude, and says:

root@rude:/var/lib/postgresql/15/main# cat ./pg_wal/00000002.history
1	102/15000000	no recovery target specified

i'm not sure how to resolve this or why this matters, still investigating.

for weather, the fix was easy:

root@bungei:/srv/backups/pg# rm weather-01/main.WAL.000000010000000000000001
root@bungei:/srv/backups/pg# rm weather-01/main.WAL.000000010000000000000002

... and now the backup script doesn't complain about it anymore.

i fixed rude by configuring the timeline id in the configuration file, and documented how to deal with this in the future in the wiki.

but before figuring that out, i moved the rude/ directory out of there and made a new base backup a few times. so now we have another leftover directory, which I've queued for removal using a generous schedule, after our findings in #41388 (closed):

root@bungei:/srv/backups/pg# at now + 28day
warning: commands will be executed using /bin/sh
at> rm -r /srv/backup/pg/rude-old
at> <EOT>
job 115 at Mon Dec 11 21:54:00 2023

i think this is otherwise fixed now, the check script doesn't complain about anything else anymore:L

root@bungei:/srv/backups/pg# /usr/lib/nagios/plugins/dsa-check-backuppg  -e
NOT-CONFIGURED: rude-old
NOT-CONFIGURED: survey-01-pg13
NOT-CONFIGURED: rude-13

closed

changed the incident status to Resolved by closing the incident

mentioned in commit wiki-replica@eed52f36

mentioned in issue #41411 (closed)

PostgreSQL backups are failing

Child items ...

Activity