TPA issueshttps://gitlab.torproject.org/groups/tpo/tpa/-/issues2024-03-18T23:29:44Zhttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41549BTCPayServer is Down2024-03-18T23:29:44ZSusanBTCPayServer is DownI am unable to connect to the btcpay.torproject.org. It says the site cannot be reached. I believe this means that donors cannot use it to donate either.I am unable to connect to the btcpay.torproject.org. It says the site cannot be reached. I believe this means that donors cannot use it to donate either.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41470gitlab account takeover audit2024-01-22T16:13:39Zanarcatgitlab account takeover auditToday's GitLab release include a fix for a [full account takeover](https://about.gitlab.com/releases/2024/01/11/critical-security-release-gitlab-16-7-2-released/#account-takeover-via-password-reset-without-user-interactions) based on a f...Today's GitLab release include a fix for a [full account takeover](https://about.gitlab.com/releases/2024/01/11/critical-security-release-gitlab-16-7-2-released/#account-takeover-via-password-reset-without-user-interactions) based on a failed password reset mechanism.
This issue was introduced in GitLab 16.1, released on May 1, 2023. We need to verify whether this vulnerability was exploited on our server.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41429outage in gnt-fsn cluster (gitlab, collector, other services affected)2023-12-14T03:51:28Zanarcatoutage in gnt-fsn cluster (gitlab, collector, other services affected)GitLab was not responsive this morning, and people also reported performance issues on collector.
Status site incident: https://status.torproject.org/issues/2023-12-06-gitlab-collector-outage/
Seems like this was an issue with the back...GitLab was not responsive this morning, and people also reported performance issues on collector.
Status site incident: https://status.torproject.org/issues/2023-12-06-gitlab-collector-outage/
Seems like this was an issue with the backend network that led to DRBD disk inconsistencies and I/O timeouts. The status-site update procedures without gitlab were tested and slightly updated.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41409gitlab notes delivered by email are lost2023-11-22T15:06:14Zanarcatgitlab notes delivered by email are lostThis morning @boklm reported:
```
12:05:06 <boklm> I sent an email to gitlab to create a new ticket, but I don't see it being created
12:05:16 <boklm> is there some issue with emails for gitlab?
12:05:49 <boklm> (I also sent an email to...This morning @boklm reported:
```
12:05:06 <boklm> I sent an email to gitlab to create a new ticket, but I don't see it being created
12:05:16 <boklm> is there some issue with emails for gitlab?
12:05:49 <boklm> (I also sent an email to change assignee for a ticket 30 minutes ago and it was not done)
```
I also just realized that I sent a response in a ticket (https://gitlab.torproject.org/tpo/tpa/team/-/issues/41405#note_2968173) on *Sunday* but it never landed in the ticket (the aforementioned response was copy-pasted today).
So it looks like the email bridge is losing traffic, oops.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41320duplicate GID between tagtor and metrics-api2023-09-14T15:35:09Zanarcatduplicate GID between tagtor and metrics-apithe metrics-api and tagtor groups have the same GID, which raises this error in `ud-replicate`, even on unrelated servers:
```
root@cdn-backend-sunet-02:~# ud-replicate
receiving file list ... done
sent 19 bytes received 837 bytes 1,...the metrics-api and tagtor groups have the same GID, which raises this error in `ud-replicate`, even on unrelated servers:
```
root@cdn-backend-sunet-02:~# ud-replicate
receiving file list ... done
sent 19 bytes received 837 bytes 1,712.00 bytes/sec
total size is 226,362 speedup is 264.44
makedb:cdn-backend-sunet-02.torproject.org/group.tdb:69: duplicate key
```
That's because both groups have the same GID (`2196`):
```
571 gid=tagtor,ou=users,dc=torproject,dc=org
gid: tagtor
gidNumber: 2196
objectClass: top
objectClass: debianGroup
[...]
587 gid=metrics-api,ou=users,dc=torproject,dc=org
gid: metrics-api
gidNumber: 2196
objectClass: top
objectClass: debianGroup
```
it's the same for the metrics-api and tagtor users: they share the same UID.
there are two problems here:
1. duplicate UID/GIDs - in general that should be avoided, but this is particularly a problem because both users/groups pairs are deployed on the same server (`metricsdb-01`), it's a miracle any of this worked at all
2. this shouldn't error *everywhere* and I worry this is a sign that users are not syncing properly
This could be related to the botched LDAP upgrade I did yesterday (#40693), but I suspect the duplication has been there since August 25th. We haven't noticed then possibly because `makedb` was less strict or some other thing changed... unclear.
In any case, we need to split those UIDs. @hiro what should those UIDs be? what files should be owned by who?anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41317gitlab is slow(er than usual)2023-09-11T18:09:44Zanarcatgitlab is slow(er than usual)this morning gitlab slowed to a crawl. multiple users reported the site being barely usable, and i didn't even file an incident here at the time because it was so slow, filing this as a post-mortem.this morning gitlab slowed to a crawl. multiple users reported the site being barely usable, and i didn't even file an incident here at the time because it was so slow, filing this as a post-mortem.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41169Error 503 All backends failed or unhealthy2023-05-11T15:58:30ZrichardError 503 All backends failed or unhealthyWhen attempting to fetch tor-browser updates (for example: https://cdn-fastly.torproject.org/aus1/torbrowser/12.5a5/tor-browser-linux64-12.5a5_ALL.mar ) we get this in response:
![image](/uploads/8a230be054a2944469fab85c99cf73bf/image.png)When attempting to fetch tor-browser updates (for example: https://cdn-fastly.torproject.org/aus1/torbrowser/12.5a5/tor-browser-linux64-12.5a5_ALL.mar ) we get this in response:
![image](/uploads/8a230be054a2944469fab85c99cf73bf/image.png)anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41148renew and transfer tor.network2023-05-17T17:57:25Zanarcatrenew and transfer tor.network/cc @arma/cc @armaanarcatanarcat2023-05-25https://gitlab.torproject.org/tpo/tpa/team/-/issues/40981TPA-RFC-44: Email emergency recovery2022-12-15T21:54:24ZanarcatTPA-RFC-44: Email emergency recoveryTo respond to the bouncing email crisis (tpo/web/civicrm#74), I've drafted a new proposal to implement emergency measures but also a long term plan to host our own email properly:
https://gitlab.torproject.org/tpo/tpa/team/-/wikis/polic...To respond to the bouncing email crisis (tpo/web/civicrm#74), I've drafted a new proposal to implement emergency measures but also a long term plan to host our own email properly:
https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-44-email-emergency-recovery
This ticket provides a space to review the proposal, express dissent, encouragement, or any other comments.
next steps:
2. [x] make tickets for the work to be done in emergency
3. [x] update the status page (https://status.torproject.org/issues/2022-11-30-mail-delivery/)
4. [x] SPF (hard), DKIM and DMARC (soft) records on CiviCRM (#40986)
6. [x] DKIM signatures on eugeni and submission (#40988)
7. [x] DKIM signature on all mail hosts (#40989)
8. [x] Deploy SPF ~~(hard)~~, DKIM, and DMARC records for all of torproject.org (#40990)
5. [x] ~~Deploy a new, sender-rewriting, mail exchanger (#40987)~~ postponed to next year, followup in #41009
9. [x] update the status page (https://status.torproject.org/issues/2022-11-30-mail-delivery/)
10. [x] update the documentation in howto/submission and service/email.md
11. [x] ~~split long-term parts of TPA-RFC-44 *out* of it into a new proposal (yes, editing the standard, omg)~~ followup in #41009improve mail servicesanarcatanarcat2022-12-07