TPA issueshttps://gitlab.torproject.org/groups/tpo/tpa/-/issues2024-02-19T15:10:38Zhttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41524Sending mail to travel@torproject.org does not forward to gmail2024-02-19T15:10:38ZSebastian HahnSending mail to travel@torproject.org does not forward to gmailI wrote an email to travel@torproject.org, which worked a few days ago, but now eugeni informed me it couldn't deliver to gmail:
> This is the mail system at host eugeni.torproject.org.
>
> I'm sorry to have to inform you that your mes...I wrote an email to travel@torproject.org, which worked a few days ago, but now eugeni informed me it couldn't deliver to gmail:
> This is the mail system at host eugeni.torproject.org.
>
> I'm sorry to have to inform you that your message could not
> be delivered to one or more recipients. It's attached below.
>
> For further assistance, please send mail to postmaster.
>
> If you do so, please include this problem report. You can
> delete your own text from the attached returned message.
>
> The mail system
>
> <<handle>@gmail.com>: host
> gmail-smtp-in.l.google.com[2a00:1450:400c:c0c::1a] said: 550-5.7.26 This
> mail has been blocked because the sender is unauthenticated. 550-5.7.26
> Gmail requires all senders to authenticate with either SPF or DKIM.
> 550-5.7.26 550-5.7.26 Authentication results: 550-5.7.26 DKIM = did not
> pass 550-5.7.26 SPF [<domain>] with ip:
> [<ip> 550-5.7.26 8] = did not pass 550-5.7.26
> 550-5.7.26 For instructions on setting up authentication, go to 550 5.7.26
> https://support.google.com/mail/answer/81126#authentication
> g7-20020a05600c4c8700b004107664bbe8si4522016wmp.61 - gsmtp (in reply to end
> of DATA command)
> ...
At the same time, there was a CC directly to gmail from my own mailserver, which gmail accepted:
> postfix/smtp[3961572]: F067B38802FE: to=<<handle>@gmail.com>, relay=gmail-smtp-in.l.google.com[<ip>]:25, delay=0.97, delays=0.13/0.01/0.41/0.43, dsn=2.0.0, status=sent (250 2.0.0 OK 1707803587 c8-20020adffb48000000b0033add90c729si3955987wrs.862 - gsmtp)improve mail serviceshttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41523document donate-review deployment process and project in general2024-02-14T21:09:13Zanarcatdocument donate-review deployment process and project in generalin tpo/tpa/team#41519, we have identified that donate-review lacks documentation. #41518 is a task for @lavamind to review that project, but this is for @kez to document it as much as they can.in tpo/tpa/team#41519, we have identified that donate-review lacks documentation. #41518 is a task for @lavamind to review that project, but this is for @kez to document it as much as they can.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41522TPA-RFC-62: migrate tor-passwords to password-store2024-02-21T19:49:00ZanarcatTPA-RFC-62: migrate tor-passwords to password-storeIn #29677, we have reviewed a bunch of password managers. Bitwarden seems to be emerging as a possible candidate for an organisation-wide password management service, but in the short term however, we do not want to make any major change...In #29677, we have reviewed a bunch of password managers. Bitwarden seems to be emerging as a possible candidate for an organisation-wide password management service, but in the short term however, we do not want to make any major changes to our workflow. There's also an argument to be made that TPA should *not* be using a global password manager and is best protecting those secrets with a a different mechanism.
In any case, during a recent offboarding process (tpo/tpa/team#41519), it became very clear that our *current* password manager (pwstore) has major flaws:
1. key management: in this case, @hiro's key was expired and had to be manually removed from the user's list. this would be similar in pass, except that the keyid file is easier to manage, as its signature is managed automatically by `pass init`, provided that the `PASSWORD_STORE_SIGNING_KEY` variable is set
2. password rotation: because multiple passwords are stored in the same file, it's hard or impossible to actually see the last rotation on a single password
3. conflicts: because multiple passwords are stored in the same file, we frequently get conflicts when making changes, which is particularly painful if we need to distribute the "rotation" work
4. abandonware: a [pull request to fix Debian bookworm / Ruby 3.1 support](https://github.com/weaselp/pwstore/pull/8) has been ignored for more than a year at this point
5. counter-intuitive interface: there's no command to extract a password, you're presumably supposed to use `gpg -d` to read the password files, yet you can't use other tools to directly manipulate the password files because the target encryption keys are specified in a meta file (that latter issue is shared with pass, to be fair)
6. not packaged: pwstore is not in Debian, flatpak, or anything else
The main downside to pass is the .gpg-id system is less secure than pwstore: its signature is not enforced unless the environment variable is set, which is a bit brittle. It's also relying on the global GPG key store although in theory it should be possible to rely on another keyring by passing different options to GnuPG.
Finally, by splitting secrets into different files, we disclose **which** accounts we have access to, but I consider this a reasonable tradeoff for the benefits it brings.
Update: the above was put in an actual proposal, see https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-62-tpa-password-manageranarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41535http://check.torproject.org and IPv62024-02-27T06:40:22Zvapsianhttp://check.torproject.org and IPv6The website http://check.torproject.org is not reachable via IPv6, and the DNS entry has on an A record.
As the tor project is starting to have IPv6 capabilities, the website http://check.torproject.org should also be reachable via IPv6...The website http://check.torproject.org is not reachable via IPv6, and the DNS entry has on an A record.
As the tor project is starting to have IPv6 capabilities, the website http://check.torproject.org should also be reachable via IPv6 to help configurations (socks proxy), and indicate IPv6 and IPv4 tor connectivity status.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41521install redis on donate-review-012024-02-13T17:35:37ZKezinstall redis on donate-review-01@stephen needs a redis server available for testing donate-neo review apps. the easiest way to set that up would be to add the redis package to the machine in puppet. redis shouldn't need any additional configuration, i believe it should...@stephen needs a redis server available for testing donate-neo review apps. the easiest way to set that up would be to add the redis package to the machine in puppet. redis shouldn't need any additional configuration, i believe it should "just work" out of the box. the most configuring it could need is allowing all connections from localhost.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/anon_ticket/-/issues/64User landing page doesn't show Gitlab Account requests2024-02-13T09:47:03ZjugaUser landing page doesn't show Gitlab Account requestsBut the text shown when creating a new one says "you will not be able to check it from your landing page", so we should show the Account requests or, if we prefer more privacy, remove that text.But the text shown when creating a new one says "you will not be able to check it from your landing page", so we should show the Account requests or, if we prefer more privacy, remove that text.https://gitlab.torproject.org/tpo/tpa/team/-/issues/41520Intermittent GitLab CI runner failures: "network already exists"2024-03-25T21:30:51ZJérôme Charaouilavamind@torproject.orgIntermittent GitLab CI runner failures: "network already exists"Since enabling the `FF_NETWORK_PER_BUILD` on our Podman CI runners, there have been a number of intermittent errors like [this one](https://gitlab.torproject.org/tpo/tpa/ci-test/-/jobs/473518):
```
Running with gitlab-runner 16.8.0 (c72...Since enabling the `FF_NETWORK_PER_BUILD` on our Podman CI runners, there have been a number of intermittent errors like [this one](https://gitlab.torproject.org/tpo/tpa/ci-test/-/jobs/473518):
```
Running with gitlab-runner 16.8.0 (c72a09b6)
on ci-runner-x86-02-main __hc2zXq, system ID: s_39a8ec4bc83a
feature flags: FF_NETWORK_PER_BUILD:true
Preparing the "docker" executor 00:11
Using Docker executor with image debian:latest ...
ERROR: Preparation failed: Error response from daemon: container d4bbbaa38009ad974fa78664c59a1e28536096505fbf5c9dcbf99675343d50c3 does not exist in database: no such container (manager.go:81:1s)
Will be retried in 3s ...
Using Docker executor with image debian:latest ...
ERROR: Preparation failed: Error response from daemon: network name runner-hc2zxq-project-1144-concurrent-0-job-473518-network already used: network already exists (manager.go:67:0s)
Will be retried in 3s ...
Using Docker executor with image debian:latest ...
ERROR: Preparation failed: Error response from daemon: network name runner-hc2zxq-project-1144-concurrent-0-job-473518-network already used: network already exists (manager.go:67:0s)
Will be retried in 3s ...
ERROR: Job failed (system failure): Error response from daemon: network name runner-hc2zxq-project-1144-concurrent-0-job-473518-network already used: network already exists (manager.go:67:0s)
```
The issue has been documented in this GitLab ticket: [Podman. preparation failed, sometimes](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28971). The gist is that it's been identified as an issue in Podman 4.4 (which we run), and the fix is to upgrade the runners to Podman 4.5, which isn't straightforward because that's not available in Debian stable currently.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2024-04-06https://gitlab.torproject.org/tpo/tpa/team/-/issues/41517retire majus2024-03-27T15:22:59Zanarcatretire majusin #40692, we upgraded majus to Debian bookworm, yay!
but unfortunately, the transifex-client package has been removed from debian, so it won't be covered by security support and has not been upgraded to the latest release.
so we need...in #40692, we upgraded majus to Debian bookworm, yay!
but unfortunately, the transifex-client package has been removed from debian, so it won't be covered by security support and has not been upgraded to the latest release.
so we need to find a solution for this. i favor completely retiring the host and @emmapeel seemed opened to the idea in https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252#note_2993038
so what's the next step here, @emmapeel you were saying there were some leftovers to garbage-collect?
checklist:
1. [x] announcement
2. [x] nagios
3. [x] retire the host in fabric
4. [x] remove from LDAP with `ldapvi`
5. [x] power-grep
6. [x] remove from tor-passwords
7. [x] remove from DNSwl
8. [x] remove from docs
9. [x] remove from racks
10. [x] remove from reverse DNSold service retirement 2023anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41516metricsdb-01 root filesystem is full2024-02-05T20:09:05ZJérôme Charaouilavamind@torproject.orgmetricsdb-01 root filesystem is fullFor over a week, the root filesystem on `metricsdb-01` has been filled to 100%.
The cause seems to be related to logs lines such as this being added tens (even hundreds) of thousands of times every day:
Feb 05 04:05:37 metricsdb-01...For over a week, the root filesystem on `metricsdb-01` has been filled to 100%.
The cause seems to be related to logs lines such as this being added tens (even hundreds) of thousands of times every day:
Feb 05 04:05:37 metricsdb-01 run[3664186]: 2024-02-05 04:05:37,453 WARN o.t.m.d.p.WebStatsParser:114 ERROR: duplicate key value violates unique constraint "log_line_pkey"
Feb 05 04:05:37 metricsdb-01 run[3664186]: Detail: Key (digest)=(g4tX2M7Beig0hqfn2OaUHKGTpXTjel+p8wrfWoTzK+8) already exists.HiroHirohttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41515meronense OOM2024-02-05T19:52:19Zanarcatmeronense OOMtoday, metrics.tpo went down because the OOM killer was invoked. not sure what happened. i restarted both metrics-r and metrics-web.service, pending further investigation.
this happened before, of course. we bumped the memory on that bo...today, metrics.tpo went down because the OOM killer was invoked. not sure what happened. i restarted both metrics-r and metrics-web.service, pending further investigation.
this happened before, of course. we bumped the memory on that box to 20GB in #41335 and had issues after the bullseye upgrade as well (#40814), both incidents should be investigated. those are just the incidents that pop up in the gitlab "Similar issues", further investigation in other issues probably warranted.
possibly related with the bookworm upgrade, of course (#41252).anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41514metricsdb-01 is out of disk space on /2024-02-14T15:38:44ZKezmetricsdb-01 is out of disk space on /Roger reported metrics.tpo as being down (website returning 503). I checked nagios, and it looks like metricsdb-01 is out of disk space on the root partition. No other metrics-related issues are being reported in nagios, so I assume this...Roger reported metrics.tpo as being down (website returning 503). I checked nagios, and it looks like metricsdb-01 is out of disk space on the root partition. No other metrics-related issues are being reported in nagios, so I assume this is what's causing the metrics.tpo outage.HiroHirohttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41513I need a sanitized private/settings.local.php file2024-02-05T15:58:16ZKezI need a sanitized private/settings.local.php fileWhile working on the unit tests for #41511, I ran into an issue where some of the tests rely on the `private/settings.local.php` file. I don't have a copy of this, and there's no example copy I can work with. I'll need the settings.local...While working on the unit tests for #41511, I ran into an issue where some of the tests rely on the `private/settings.local.php` file. I don't have a copy of this, and there's no example copy I can work with. I'll need the settings.local.php file from the crm-ext-01 box so I can fix any issues with the unit tests, and document the config file. **This file contains private keys** so the file will need to be sanitized before it's posted in this ticket or checked into git. This file should be located at `crm-ext-01.torproject.org:/srv/donate-api.torproect.org/htdocs-staging`.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41512Simplify onionoo architecture2024-03-26T15:45:11ZHiroSimplify onionoo architectureCurrently onionoo is a service comprised of 4 VMs: two backends with the onionoo java apps serving and updating the data, and two frontends.
At the time the service was launched this architecture made a lot of sense, but I think now we ...Currently onionoo is a service comprised of 4 VMs: two backends with the onionoo java apps serving and updating the data, and two frontends.
At the time the service was launched this architecture made a lot of sense, but I think now we could simplify its maintenance by reducing it to a backend with a web server (like nginx) with some aggressive caching.
I was hoping that we would get sooner to the point where onionoo would be retired, but given the current pace of development of the metrics pipeline, I personally think it makes sense to reduce this service now so that it is easier to maintain for metrics and tpa.
What do you think?HiroHirohttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41511upgrade crm-ext-01 to php 8 or retire2024-02-20T20:04:45Zanarcatupgrade crm-ext-01 to php 8 or retirein https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252#note_2990644, @kez tested crm-ext-01 after I upgraded it and found the donate site completely broken by the PHP 8.2 upgrade. apparently, `implode` completely changed signature ...in https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252#note_2990644, @kez tested crm-ext-01 after I upgraded it and found the donate site completely broken by the PHP 8.2 upgrade. apparently, `implode` completely changed signature in PHP and the old signature was dropped in PHP 8, which breaks a *lot* of things.
exactly how much is unclear, @kez estimated just the work to estimate that work to be a few hours of work.
for now i rolled back to the php 7.4 package from bullseye, and added it to the sources.list file (although puppet might have killed the .list file already). we need to figure out a plan to go forward, either port the code, or retire the box, which is the ultimate goal once donate-neo goes to production.Redesign donate.torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41510Handle GitLab access token mandatory expiration2024-03-27T14:35:56ZJérôme Charaouilavamind@torproject.orgHandle GitLab access token mandatory expirationGitLab recently introduced a maximum lifetime for *all* access tokens. The change is discussed in a [blog post](https://about.gitlab.com/blog/2023/10/25/access-token-lifetime-limits/) from last October. Most importantly:
> As of the 16...GitLab recently introduced a maximum lifetime for *all* access tokens. The change is discussed in a [blog post](https://about.gitlab.com/blog/2023/10/25/access-token-lifetime-limits/) from last October. Most importantly:
> As of the 16.0 milestone (May 2023), we applied an expiration date of May 14, 2024, to any personal, group, or project access token that previously didn't have one.
We should provide a convenient way of dealing with token renewal for the many tokens used across this instance which are now set to expire soon.
Unfortunately, it seems like we're more or less to our own devices to figure out which tokens/projects are affected by this change.anarcatanarcat2024-04-30https://gitlab.torproject.org/tpo/tpa/team/-/issues/41509upgrade libapache2-mod-qos to the debian trixie version on check-012024-03-12T00:05:22ZKezupgrade libapache2-mod-qos to the debian trixie version on check-01after upgrading check-01 (#41252), apache broke because of a bug in bookworm's libapache2-mod-qos. the version from trixie is fixed, and according to the debian mailing list it can be used on bookworm https://bugs.debian.org/cgi-bin/bugr...after upgrading check-01 (#41252), apache broke because of a bug in bookworm's libapache2-mod-qos. the version from trixie is fixed, and according to the debian mailing list it can be used on bookworm https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1000072
i'll be upgrading mod-qos to the trixie version and making sure apache starts on check.Debian 13 trixie upgradeJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/status-site/-/issues/40check.tpo is down (likely due to bookworm upgrade)2024-01-31T16:06:21ZGeorg Koppencheck.tpo is down (likely due to bookworm upgrade)See: https://gitlab.torproject.org/tpo/network-health/metrics/tor-check/-/issues/40017 and https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252#note_2990660 for more context.See: https://gitlab.torproject.org/tpo/network-health/metrics/tor-check/-/issues/40017 and https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252#note_2990660 for more context.Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41508Export rdsys based HTTPS distributor as HTTPS service at rdsys-test-01 with a...2024-02-01T14:02:40ZshelikhooExport rdsys based HTTPS distributor as HTTPS service at rdsys-test-01 with an non-production domain nameI would like to request exporting service hosted at `http://127.0.0.1:7200` at rdsys-test-01 as a https website with an non-production domain name.
Ideally, it should sanitize and supply `X-Forwarded-For` to reflect client IP address.
...I would like to request exporting service hosted at `http://127.0.0.1:7200` at rdsys-test-01 as a https website with an non-production domain name.
Ideally, it should sanitize and supply `X-Forwarded-For` to reflect client IP address.
See also: https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/191 as a part of S150.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41507ExoneraTor suffering from possible denial of service2024-03-07T14:22:30ZJérôme Charaouilavamind@torproject.orgExoneraTor suffering from possible denial of serviceSince approximately the new year, it seems like ExoneraTor (hosted on `materculae`) is suffering from an unusually high load:
![Capture_d_écran_du_2024-01-30_18-26-01](/uploads/6d9c76f835b53626b2d6cb0b13628ee0/Capture_d_écran_du_2024-01...Since approximately the new year, it seems like ExoneraTor (hosted on `materculae`) is suffering from an unusually high load:
![Capture_d_écran_du_2024-01-30_18-26-01](/uploads/6d9c76f835b53626b2d6cb0b13628ee0/Capture_d_écran_du_2024-01-30_18-26-01.png)
Many requests appear to be timing out with a 502 error.HiroHirohttps://gitlab.torproject.org/tpo/tpa/anon_ticket/-/issues/63This anonticket service gives Server error 500 when I click on my previous Is...2024-03-21T15:12:16ZcypherpunksThis anonticket service gives Server error 500 when I click on my previous Issue #40324This anonticket service gives Server error 500 when I click on my previous Issue #40324
I want to edit or add more info to that thread but every time I try (on different days) I get that "server error 500"This anonticket service gives Server error 500 when I click on my previous Issue #40324
I want to edit or add more info to that thread but every time I try (on different days) I get that "server error 500"