Skip to content
Snippets Groups Projects

gitlab account takeover audit

    • Closed Incident created by anarcat @anarcat

    Today's GitLab release include a fix for a full account takeover based on a failed password reset mechanism.

    This issue was introduced in GitLab 16.1, released on May 1, 2023. We need to verify whether this vulnerability was exploited on our server.

    Child items ...

    • Activity

    • anarcat added Doing Emergency Gitlab Security labels

      added Doing Emergency Gitlab Security labels

    • anarcat assigned to @anarcat

      assigned to @anarcat

    • anarcat
      anarcat @anarcat ·
      Author Owner

      According to APT's history.log, we have deployed the vulnerable version (16.1.0) on 2023-06-24 06:58:10, so that's our start time. Logs go back only to 2023-12-14 on disk, so we'll have to restore from backups, and will probably miss some.

      I wrote an audit script that does one of the two checks suggested in the advisory:

      Check gitlab-rails/production_json.log for HTTP requests to the /users/password path with params.value.email consisting of a JSON array with multiple email addresses.

      it's unclear whether both checks need to be performed, I'm assuming this one is sufficient.

      example run:

      root@gitlab-02:~# ~/audit-gitlab-logs.py /var/log/gitlab/gitlab-rails/production_json.log*                                                                   INFO:root:parsing /var/log/gitlab/gitlab-rails/production_json.log
INFO:root:parsing /var/log/gitlab/gitlab-rails/production_json.log.1.gz
[...]
INFO:root:email: 'example@mail.com'

      that email is considered not vulnerable as it's a string and not a list. but we are not quite sure of the data structure pattern here, so we're actually dumping all email values there as a safety precaution.

      so far, logs up to December 14th are clean. i'm in the process of restoring logs past that date with bacula and, amazingly, it's now been writing 43GiB to /srv/gitlab-backup/bacula-restores/2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.3.gz, and it's still going. so something's really off with that logfile...

      in any case, this rolls us back to October 23rd, as far as logs are concerned. before that, we have some crumbs we can tap into:

      *list job=gitlab-02.torproject.org
+---------+--------------------------+---------------------+------+-------+-----------+-----------------+-----------+
| jobid   | name                     | starttime           | type | level | jobfiles  | jobbytes        | jobstatus |
+---------+--------------------------+---------------------+------+-------+-----------+-----------------+-----------+
| 203,940 | gitlab-02.torproject.org | 2022-08-08 19:27:09 | B    | F     |         0 |               0 | f         |
| 245,144 | gitlab-02.torproject.org | 2023-10-07 08:09:08 | B    | F     | 1,629,553 | 336,632,852,099 | T         |
| 248,827 | gitlab-02.torproject.org | 2023-11-15 18:48:27 | B    | I     |         0 |               0 | f         |
| 248,859 | gitlab-02.torproject.org | 2023-11-15 19:09:49 | B    | I     |         0 |               0 | A         |
| 249,327 | gitlab-02.torproject.org | 2023-11-20 18:21:23 | B    | D     |   892,988 | 264,468,338,327 | T         |
| 249,379 | gitlab-02.torproject.org | 2023-11-21 08:15:46 | B    | F     | 1,335,086 | 468,988,525,649 | T         |
| 250,036 | gitlab-02.torproject.org | 2023-11-28 09:57:33 | B    | D     |    87,822 | 111,508,968,913 | T         |
| 250,786 | gitlab-02.torproject.org | 2023-12-06 13:06:08 | B    | I     |     8,094 |  22,869,198,180 | f         |

      so i've a restore of the log files as of 2023-10-07 08:09:08, which leaves us a gap between 2023-10-07 and 2023-10-23. we should be able to recover those with the 2023-11-28 09:57:33 backup as well.

      if i count this right, this will give us full logs back to early august, leaving a gap of about a month between 2023-06-24 and 2023-08-01 or so.

    • anarcat
      anarcat @anarcat ·
      Author Owner

      for some reason, the .3.gz log files all have garbage after the file and restore to seemingly unlimited file sizes. i canceled the first restore after 100GB and the second after 1GB.

      i then uncompressed and recompressed the files to restore them to a sane size.

      now i'm restoring a .7.gz file missing from one of the backups.

    • anarcat
      anarcat @anarcat ·
      Author Owner

      i have also queued the 2023-11-28 09:57:33 restore job.

    • anarcat
      anarcat @anarcat ·
      Author Owner

      i'm struggling to restore that missing .7.gz file. here's what i actually have now:

      root@gitlab-02:/srv/gitlab-backup/bacula-restores# ls -ltr */var/log/gitlab/gitlab-rails/*
-rw------- 1 bacula root 190918578 Sep  8 00:46 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.30.gz
-rw------- 1 bacula root 153182315 Sep  9 00:47 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.29.gz
-rw------- 1 bacula root 118855199 Sep 10 00:48 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.28.gz
-rw------- 1 bacula root 156067714 Sep 11 00:48 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.27.gz
-rw------- 1 bacula root 230139283 Sep 12 00:49 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.26.gz
-rw------- 1 bacula root 244607528 Sep 13 00:50 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.25.gz
-rw------- 1 bacula root 260289024 Sep 14 00:52 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.24.gz
-rw------- 1 bacula root 227283391 Sep 15 00:09 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.23.gz
-rw------- 1 bacula root 202312058 Sep 16 00:11 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.22.gz
-rw------- 1 bacula root 168395530 Sep 17 00:13 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.21.gz
-rw------- 1 bacula root 143639185 Sep 18 00:14 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.20.gz
-rw------- 1 bacula root 210448328 Sep 19 00:29 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.19.gz
-rw------- 1 bacula root 207472224 Sep 20 00:30 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.18.gz
-rw------- 1 bacula root 204973430 Sep 21 00:31 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.17.gz
-rw------- 1 bacula root 239074377 Sep 22 00:32 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.16.gz
-rw------- 1 bacula root 206433162 Sep 23 00:34 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.15.gz
-rw------- 1 bacula root 150147595 Sep 24 00:03 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.14.gz
-rw------- 1 bacula root 126508959 Sep 25 00:04 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.13.gz
-rw------- 1 bacula root 173472797 Sep 26 00:05 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.12.gz
-rw------- 1 bacula root 178841226 Sep 27 00:07 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.11.gz
-rw------- 1 bacula root 174483886 Sep 28 00:08 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.10.gz
-rw------- 1 bacula root 179356377 Sep 29 00:09 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.9.gz
-rw------- 1 bacula root 162559578 Sep 30 00:11 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.8.gz
-rw------- 1 bacula root 173311306 Oct  1 00:12 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.7.gz
-rw------- 1 bacula root 192072171 Oct  2 00:13 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.6.gz
-rw------- 1 bacula root 194416152 Oct  3 00:14 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.5.gz
-rw------- 1 bacula root 196515172 Oct  4 00:14 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.4.gz
-rw------- 1 bacula root 207350191 Oct  6 00:17 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.2.gz
-rw------- 1 bacula root 216223748 Oct  7 00:18 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.1.gz
-rw------- 1 bacula root 748738510 Oct  7 08:10 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log
-rw------- 1 bacula root 141085760 Oct 23 00:53 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.30.gz
-rw------- 1 bacula root 141085760 Oct 23 00:53 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.30.gz
-rw------- 1 bacula root 165841410 Oct 24 00:33 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.29.gz
-rw------- 1 bacula root 165841410 Oct 24 00:33 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.29.gz
-rw------- 1 bacula root 177242941 Oct 25 00:34 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.28.gz
-rw------- 1 bacula root 177242941 Oct 25 00:34 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.28.gz
-rw------- 1 bacula root 222244953 Oct 26 00:35 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.27.gz
-rw------- 1 bacula root 222244953 Oct 26 00:35 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.27.gz
-rw------- 1 bacula root 198320549 Oct 27 00:36 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.26.gz
-rw------- 1 bacula root 198320549 Oct 27 00:36 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.26.gz
-rw------- 1 bacula root 154567451 Oct 28 00:37 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.25.gz
-rw------- 1 bacula root 154567451 Oct 28 00:37 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.25.gz
-rw------- 1 bacula root 147009092 Oct 29 00:38 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.24.gz
-rw------- 1 bacula root 147009092 Oct 29 00:38 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.24.gz
-rw------- 1 bacula root 137416242 Oct 30 00:39 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.23.gz
-rw------- 1 bacula root 137416242 Oct 30 00:39 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.23.gz
-rw------- 1 bacula root 150385950 Oct 31 00:39 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.22.gz
-rw------- 1 bacula root 150385950 Oct 31 00:39 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.22.gz
-rw------- 1 bacula root 177694690 Nov  1 00:40 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.21.gz
-rw------- 1 bacula root 177694690 Nov  1 00:40 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.21.gz
-rw------- 1 bacula root 169974681 Nov  2 00:44 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.20.gz
-rw------- 1 bacula root 169974681 Nov  2 00:44 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.20.gz
-rw------- 1 bacula root 151018665 Nov  3 00:45 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.19.gz
-rw------- 1 bacula root 151018665 Nov  3 00:45 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.19.gz
-rw------- 1 bacula root 160206335 Nov  4 00:46 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.18.gz
-rw------- 1 bacula root 160206335 Nov  4 00:46 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.18.gz
-rw------- 1 bacula root 170837712 Nov  5 00:47 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.17.gz
-rw------- 1 bacula root 170837712 Nov  5 00:47 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.17.gz
-rw------- 1 bacula root 193217709 Nov  6 00:48 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.16.gz
-rw------- 1 bacula root 193217709 Nov  6 00:48 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.16.gz
-rw------- 1 bacula root 189824061 Nov  7 00:49 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.15.gz
-rw------- 1 bacula root 189824061 Nov  7 00:49 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.15.gz
-rw------- 1 bacula root 207636659 Nov  8 00:50 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.14.gz
-rw------- 1 bacula root 207636659 Nov  8 00:50 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.14.gz
-rw------- 1 bacula root 196572384 Nov  9 00:52 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.13.gz
-rw------- 1 bacula root 196572384 Nov  9 00:52 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.13.gz
-rw------- 1 bacula root 223754038 Nov 10 00:53 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.12.gz
-rw------- 1 bacula root 223754038 Nov 10 00:53 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.12.gz
-rw------- 1 bacula root 160846831 Nov 11 00:54 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.11.gz
-rw------- 1 bacula root 160846831 Nov 11 00:54 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.11.gz
-rw------- 1 bacula root 133461290 Nov 12 00:55 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.10.gz
-rw------- 1 bacula root 133461290 Nov 12 00:55 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.10.gz
-rw------- 1 bacula root 126234966 Nov 13 00:56 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.9.gz
-rw------- 1 bacula root 126234966 Nov 13 00:56 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.9.gz
-rw------- 1 bacula root 152778127 Nov 14 00:56 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.8.gz
-rw------- 1 bacula root 152778127 Nov 14 00:56 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.8.gz
-rw------- 1 bacula root 171437917 Nov 16 00:08 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.6.gz
-rw------- 1 bacula root 171437917 Nov 16 00:08 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.6.gz
-rw------- 1 bacula root 195091597 Nov 17 00:09 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.5.gz
-rw------- 1 bacula root 195091597 Nov 17 00:09 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.5.gz
-rw------- 1 bacula root 160796161 Nov 18 00:06 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.4.gz
-rw------- 1 bacula root 160796161 Nov 18 00:06 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.4.gz
-rw------- 1 bacula root 147141799 Nov 20 00:08 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.2.gz
-rw------- 1 bacula root 147141799 Nov 20 00:08 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.2.gz
-rw------- 1 bacula root 178482788 Nov 21 00:09 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.1.gz
-rw------- 1 bacula root 178482788 Nov 21 00:09 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.1.gz
-rw------- 1 bacula root 623627343 Nov 21 08:16 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log
-rw------- 1 bacula root 623627343 Nov 21 08:16 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log
-rw-r--r-- 1 root   root 152162871 Jan 12 04:46 2023-12-13T01:48:48/var/log/gitlab/gitlab-rails/production_json.log.3.gz
-rw------- 1 bacula root 176847920 Jan 12 04:51 2023-10-07T08:09:08/var/log/gitlab/gitlab-rails/production_json.log.3.gz
-rw------- 1 bacula root 152162893 Jan 12 05:12 2023-11-28T09:57:33/var/log/gitlab/gitlab-rails/production_json.log.3.gz

      notice how we still have that gap in october. i suspect i may have done a bad restore for the 2023-11-28T09:57:33 backup.

    • anarcat
      anarcat @anarcat ·
      Author Owner

      well shit, i can't figure this out. this is the third restore i'm attempting to close the gap between nov 21 and dec 14 and i keep failing. it keeps restoring from the full backups instead of the incrementals. those were probably purged by our retention policies?

      it's also possible this is related to that damn .3.gz file which garbles everything. but i've tried to skip restoring that file (using the pattern /var/log/gitlab/gitlab-rails/production_json.log.[^3].*) and bacula trips on some other file (the .25 log file). really odd.

      i looked into those files and, indeed, it looks like the end of the file is not compressed. but what it looks like is an actual bacula raw volume, streamed straight down from the storage server.

      so that is clearly worrisome.

      but not directly related to the audit at hand. i'll just declare those old logs lost and start processing the logs i could lay my hands on.

    • anarcat
      anarcat @anarcat ·
      Author Owner

      one thing to keep in mind here is that 2fa accounts are not vulnerable.

      so we might want to broaden the enforcement of that policy. right now it's only on tpo/tpa, but we could extend to (say) tpo/web or tpo/core...

    • anarcat
      anarcat @anarcat ·
      Author Owner

      here's all the .log files bacula seem to know about, notice how the oldest full is missing:

      bacula=# select jobid, job.name,type,level,starttime, path.path || filename.name AS path from path JOIN file USING (pathid) JOIN filename USING (filenameid) JOIN job USING (jobid) WHERE path.path='/var/log/gitlab/gitlab-rails/' AND filename.name = 'production_json.log' ORDER BY starttime DESC limit 100;
 jobid  |           name           | type | level |      starttime      |                       path                       
--------+--------------------------+------+-------+---------------------+--------------------------------------------------
 254122 | gitlab-02.torproject.org | B    | I     | 2024-01-11 03:18:34 | /var/log/gitlab/gitlab-rails/production_json.log
 254033 | gitlab-02.torproject.org | B    | I     | 2024-01-10 03:34:02 | /var/log/gitlab/gitlab-rails/production_json.log
 253930 | gitlab-02.torproject.org | B    | I     | 2024-01-09 02:03:58 | /var/log/gitlab/gitlab-rails/production_json.log
 253830 | gitlab-02.torproject.org | B    | D     | 2024-01-08 00:21:10 | /var/log/gitlab/gitlab-rails/production_json.log
 253772 | gitlab-02.torproject.org | B    | I     | 2024-01-07 08:51:10 | /var/log/gitlab/gitlab-rails/production_json.log
 253676 | gitlab-02.torproject.org | B    | I     | 2024-01-06 07:39:12 | /var/log/gitlab/gitlab-rails/production_json.log
 253582 | gitlab-02.torproject.org | B    | I     | 2024-01-05 07:45:13 | /var/log/gitlab/gitlab-rails/production_json.log
 253493 | gitlab-02.torproject.org | B    | I     | 2024-01-04 09:06:29 | /var/log/gitlab/gitlab-rails/production_json.log
 253402 | gitlab-02.torproject.org | B    | I     | 2024-01-03 09:27:41 | /var/log/gitlab/gitlab-rails/production_json.log
 253313 | gitlab-02.torproject.org | B    | I     | 2024-01-02 11:00:27 | /var/log/gitlab/gitlab-rails/production_json.log
 253219 | gitlab-02.torproject.org | B    | I     | 2024-01-01 09:54:12 | /var/log/gitlab/gitlab-rails/production_json.log
 253120 | gitlab-02.torproject.org | B    | F     | 2023-12-31 08:18:22 | /var/log/gitlab/gitlab-rails/production_json.log
 253064 | gitlab-02.torproject.org | B    | I     | 2023-12-30 19:09:25 | /var/log/gitlab/gitlab-rails/production_json.log
 252973 | gitlab-02.torproject.org | B    | I     | 2023-12-29 21:06:48 | /var/log/gitlab/gitlab-rails/production_json.log
 252887 | gitlab-02.torproject.org | B    | I     | 2023-12-28 22:24:09 | /var/log/gitlab/gitlab-rails/production_json.log
 252799 | gitlab-02.torproject.org | B    | I     | 2023-12-27 23:54:35 | /var/log/gitlab/gitlab-rails/production_json.log
 252705 | gitlab-02.torproject.org | B    | D     | 2023-12-26 22:45:44 | /var/log/gitlab/gitlab-rails/production_json.log
 252700 | gitlab-02.torproject.org | B    | I     | 2023-12-26 21:09:03 | /var/log/gitlab/gitlab-rails/production_json.log
 252610 | gitlab-02.torproject.org | B    | I     | 2023-12-25 21:33:14 | /var/log/gitlab/gitlab-rails/production_json.log
 252507 | gitlab-02.torproject.org | B    | I     | 2023-12-24 19:42:21 | /var/log/gitlab/gitlab-rails/production_json.log
 252411 | gitlab-02.torproject.org | B    | I     | 2023-12-23 19:12:27 | /var/log/gitlab/gitlab-rails/production_json.log
 252317 | gitlab-02.torproject.org | B    | I     | 2023-12-22 18:18:37 | /var/log/gitlab/gitlab-rails/production_json.log
 252213 | gitlab-02.torproject.org | B    | I     | 2023-12-21 18:19:00 | /var/log/gitlab/gitlab-rails/production_json.log
 252120 | gitlab-02.torproject.org | B    | I     | 2023-12-20 17:36:20 | /var/log/gitlab/gitlab-rails/production_json.log
 252031 | gitlab-02.torproject.org | B    | I     | 2023-12-19 18:51:06 | /var/log/gitlab/gitlab-rails/production_json.log
 251940 | gitlab-02.torproject.org | B    | I     | 2023-12-18 19:39:17 | /var/log/gitlab/gitlab-rails/production_json.log
 251854 | gitlab-02.torproject.org | B    | D     | 2023-12-17 21:21:33 | /var/log/gitlab/gitlab-rails/production_json.log
 251775 | gitlab-02.torproject.org | B    | I     | 2023-12-17 01:42:11 | /var/log/gitlab/gitlab-rails/production_json.log
 251673 | gitlab-02.torproject.org | B    | I     | 2023-12-16 00:48:36 | /var/log/gitlab/gitlab-rails/production_json.log
 251578 | gitlab-02.torproject.org | B    | I     | 2023-12-15 00:15:18 | /var/log/gitlab/gitlab-rails/production_json.log
 251481 | gitlab-02.torproject.org | B    | I     | 2023-12-14 00:24:50 | /var/log/gitlab/gitlab-rails/production_json.log
(31 rows)
    • anarcat
      anarcat @anarcat ·
      Author Owner

      what could be happening is that .gz files are excluded from incrementals:

      bacula=# SELECT jobid, job.name,type,level,starttime, path.path || filename.name AS path FROM path
      JOIN file USING (pathid)
      JOIN filename USING (filenameid)
      JOIN job USING (jobid)
      WHERE jobid=251481 and filename.name like 'production_json.log%'
      ORDER BY starttime DESC
      LIMIT 10;
 jobid  |           name           | type | level |      starttime      |                       path
--------+--------------------------+------+-------+---------------------+--------------------------------------------------
 251481 | gitlab-02.torproject.org | B    | I     | 2023-12-14 00:24:50 | /var/log/gitlab/gitlab-rails/production_json.log
(1 row)
      • anarcat
        anarcat @anarcat ·
        Author Owner

        i can't find anything that matches the vague description provided of a possible compromise.

        next steps:

        1. open an issue to enforce 2fa more broadly, discuss
        2. evaluate wtf is going on with backups: do we have corruption? why do we get multi-gigabyte files with trailing garbage?
        3. why can't we restore past log files? are they excluded?
        4. document audit script, possibly publicly

        anything else i'm missing?

        Edited by anarcat
      • anarcat
        anarcat @anarcat ·
        Author Owner
        1. open an issue to enforce 2fa more broadly, discuss

        #41473 (closed)

        1. evaluate wtf is going on with backups: do we have corruption? why do we get multi-gigabyte files with trailing garbage?
        2. why can't we restore past log files? are they excluded?

        to investigate, probably in separate issue.

        1. document audit script, possibly publicly

        just pushed to fabric-tasks@686dde76 for now.

      • Please register or sign in to reply
    • anarcat mentioned in commit fabric-tasks@686dde76

      mentioned in commit fabric-tasks@686dde76

    • anarcat
      anarcat @anarcat ·
      Author Owner

      the recovered backups were destroyed, a copy of the output of the script was kept in /root/gitlab-audit-tpo-tpa-team-41470.txt on gitlab-02 and the script was added to fabric-tasks@686dde76

    • anarcat mentioned in issue #41473 (closed)

      mentioned in issue #41473 (closed)

    • anarcat marked this incident as related to #41473 (closed)

      marked this incident as related to #41473 (closed)

    • anarcat marked this incident as related to #41469 (closed)

      marked this incident as related to #41469 (closed)

    • anarcat mentioned in issue #41474 (closed)

      mentioned in issue #41474 (closed)

    • anarcat marked this incident as related to #41474 (closed)

      marked this incident as related to #41474 (closed)

    • anarcat
      anarcat @anarcat ·
      Author Owner

      opened #41474 (closed) to followup on backups, closing incident.

    • anarcat closed

      closed

    • anarcat changed the incident status to Resolved by closing the incident

      changed the incident status to Resolved by closing the incident

    • anarcat made the incident visible to everyone

      made the incident visible to everyone

    Please register or sign in to reply