GitLab

It looks like around Jan 19th we couldn't talk to our (singular) destination:

grep "The last 9 times the destination" * | cut -d ":"  -f 2 | sort | uniq -c
    507 Jan 19 20
   3132 Jan 19 21
   3121 Jan 19 22
   2809 Jan 19 23
   3120 Jan 20 00
   3110 Jan 20 01
   2800 Jan 20 02
   3128 Jan 20 03
   3130 Jan 20 04
   3139 Jan 20 05
   2834 Jan 20 06
   3151 Jan 20 07
   3158 Jan 20 08
   2844 Jan 20 09
   3174 Jan 20 10
   3180 Jan 20 11
   2862 Jan 20 12
   3192 Jan 20 13
   3200 Jan 20 14
   3212 Jan 20 15
   2878 Jan 20 16
   3180 Jan 20 17
   3181 Jan 20 18
   2846 Jan 20 19
   3136 Jan 20 20
   3130 Jan 20 21
   2823 Jan 20 22
   3150 Jan 20 23
   3150 Jan 21 00
   2835 Jan 21 01
   3150 Jan 21 02
   3152 Jan 21 03
   3162 Jan 21 04
   2854 Jan 21 05
   3180 Jan 21 06
   3180 Jan 21 07
   2862 Jan 21 08
   3190 Jan 21 09
   3190 Jan 21 10
   2862 Jan 21 11
   3180 Jan 21 12
   3171 Jan 21 13
   2853 Jan 21 14
   3171 Jan 21 15
   3170 Jan 21 16
   2846 Jan 21 17
   3153 Jan 21 18
   3136 Jan 21 19
   2805 Jan 21 20
   3110 Jan 21 21
   3110 Jan 21 22
   2794 Jan 21 23
   3100 Jan 22 00
   3100 Jan 22 01
   2790 Jan 22 02
   3108 Jan 22 03
   3124 Jan 22 04
   2817 Jan 22 05
   3136 Jan 22 06
   3150 Jan 22 07
   2837 Jan 22 08
   3160 Jan 22 09
   3162 Jan 22 10
   2853 Jan 22 11
   3170 Jan 22 12
   3172 Jan 22 13
   2871 Jan 22 14
   3190 Jan 22 15
   3190 Jan 22 16
   2871 Jan 22 17
   3190 Jan 22 18
   3154 Jan 22 19
   2835 Jan 22 20
   3142 Jan 22 21
   3140 Jan 22 22
   2812 Jan 22 23
   3120 Jan 23 00
   3120 Jan 23 01
   2814 Jan 23 02
   3130 Jan 23 03
   2812 Jan 23 04
   3130 Jan 23 05
   2198 Jan 23 06

However we aren't retrying it properly (and the log message isn't complete):

grep "was not tried" scanner.log* | cut -d ":" -f 2 | sort | uniq -c
      1 Jan 19 20
      3 Jan 19 21
      1 Jan 19 22
      1 Jan 19 23
      1 Jan 20 02
      1 Jan 20 07
      1 Jan 20 18
      1 Jan 21 15

grep "was not tried" scanner.log*
scanner.log.18:Jan 21 15:32:03 INFO Thread-2 destination.py:251 - is_functional - The destination %s was not tried for %s hours, it is going to by tried again.
scanner.log.28:Jan 20 18:15:50 INFO Thread-2 destination.py:251 - is_functional - The destination %s was not tried for %s hours, it is going to by tried again.
scanner.log.32:Jan 20 07:32:22 INFO Thread-2 destination.py:251 - is_functional - The destination %s was not tried for %s hours, it is going to by tried again.
scanner.log.35:Jan 20 02:10:50 INFO Thread-4 destination.py:251 - is_functional - The destination %s was not tried for %s hours, it is going to by tried again.
scanner.log.36:Jan 19 23:30:03 INFO Thread-4 destination.py:251 - is_functional - The destination %s was not tried for %s hours, it is going to by tried again.
scanner.log.37:Jan 19 20:55:41 INFO Thread-3 destination.py:251 - is_functional - The destination %s was not tried for %s hours, it is going to by tried again.
scanner.log.37:Jan 19 21:01:49 INFO Thread-3 destination.py:251 - is_functional - The destination %s was not tried for %s hours, it is going to by tried again.
scanner.log.37:Jan 19 21:14:09 INFO Thread-2 destination.py:251 - is_functional - The destination %s was not tried for %s hours, it is going to by tried again.
scanner.log.37:Jan 19 21:32:37 INFO Thread-4 destination.py:251 - is_functional - The destination %s was not tried for %s hours, it is going to by tried again.
scanner.log.37:Jan 19 22:15:55 INFO Thread-4 destination.py:251 - is_functional - The destination %s was not tried for %s hours, it is going to by tried again.