TPA team issueshttps://gitlab.torproject.org/tpo/tpa/team/-/issues2024-03-28T13:14:07Zhttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41565Puppet fighting over /etc/apt/sources.list2024-03-28T13:14:07ZJérôme Charaouilavamind@torproject.orgPuppet fighting over /etc/apt/sources.listOur current Puppet is set up to purge (delete) `/etc/apt/sources.list`.
The problem is, APT::Periodic (`apt-daily.service`) is recreating it every day: the file's creation time concurs with the schedule of`apt-daily.timer` (as seen with...Our current Puppet is set up to purge (delete) `/etc/apt/sources.list`.
The problem is, APT::Periodic (`apt-daily.service`) is recreating it every day: the file's creation time concurs with the schedule of`apt-daily.timer` (as seen with `systemctl list-timers apt-daily.timer`). The creation of the file can also be reproduced manually:
```
# rm /var/lib/apt/periodic/*-stamp
# /usr/lib/apt/apt.systemd.daily update
# ls -l /etc/apt/sources.list
```
The file being purged every day leads to frequent and unnecessary triggers of `apt update`, and this sometimes even causes Puppet run failures which show up in monitoring:
```
puppet-agent[3791032]: (/Stage[main]/Apt/File[sources.list]/ensure) removed (corrective)
puppet-agent[3791032]: (/Stage[main]/Apt::Update/Exec[apt_update]/returns) Reading package lists...
puppet-agent[3791032]: (/Stage[main]/Apt::Update/Exec[apt_update]/returns) E: Could not get lock /var/lib/apt/lists/lock. It is held by process 3791201 (apt-get)
puppet-agent[3791032]: (/Stage[main]/Apt::Update/Exec[apt_update]/returns) E: Unable to lock directory /var/lib/apt/lists/
puppet-agent[3791032]: (/Stage[main]/Apt::Update/Exec[apt_update]) Failed to call refresh: '/usr/bin/apt-get update' returned 100 instead of one of [0]
puppet-agent[3791032]: (/Stage[main]/Apt::Update/Exec[apt_update]) '/usr/bin/apt-get update' returned 100 instead of one of [0]
puppet-agent[3791032]: Applied catalog in 14.75 seconds
systemd[1]: puppet-run.service: Main process exited, code=exited, status=6/NOTCONFIGURED
systemd[1]: puppet-run.service: Failed with result 'exit-code'.
```Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41564Install make and newer golang in rdsys-test-012024-03-26T18:18:36Zmeskiomeskio@torproject.orgInstall make and newer golang in rdsys-test-01I need make and golang>=1.21 (is available in bookworm-backports) to build some binaries for testing on the server Can you install it?
Thank you.I need make and golang>=1.21 (is available in bookworm-backports) to build some binaries for testing on the server Can you install it?
Thank you.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41561Update sarthikg's PGP key2024-03-25T13:53:37ZGeorg KoppenUpdate sarthikg's PGP key```
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Sarthik's laptop with the old PGP key on it died the other day, thus he needed
to create a new one:
C63B F870 B219 08AD 2A58 2C85 A347 8949 750F 08BE
which can be found at:
https:/...```
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Sarthik's laptop with the old PGP key on it died the other day, thus he needed
to create a new one:
C63B F870 B219 08AD 2A58 2C85 A347 8949 750F 08BE
which can be found at:
https://sarthikg.com/gpg.txt
I verified the fingerprint via a call which is probably better than nothing... :)
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEvoH49M1O0+io5Dsi74N5QXAIgZAFAmYBSsIACgkQ74N5QXAI
gZBpxA//etoEfrRnWRy6bnzcmuHKhlyENn4u07UjGNM0KTfmHPanh3Lh3ATfMcYk
eflJ62YveJWliRjTywNLqdlFT25f72AfpVvDF7K67w8CqhncpyZrdQ6h4ZCTpEln
eyiDhI1dcD/+2bpG6EX+SSdTvuMroS9haPKzBozL6WpjJXVC48wOz20ixKHvu8u6
G/S6IJ2dGrJarISfKuY+/ltfGjKS9yjU+ahXosLfAYoVmxWB3/bXKLo8MoKO3aUM
kgBviqiGr/nn9b9n4nxDmuuFjSIEKEvDYrzEyZUSa6Zb3l/7ywWuG6DDeBi7kG6Y
+Fub+bCVV18BBkTIQLMiMvq3VmLNrMNdLwlIzuBvH5Ipvo0K6zIsQ/8rJlc71jr2
ep7+muHkEtB5UQ9MeZ2VEVkQZsfltnZ88+L8QaUkHHM2Twrg1fj9AIHeQ1pgYRSv
KrckrcXbbk7ebLdFZ6NMj1uFQb/1WmqgKnaMwtGt9Qwy7IkBD+fBDs2JgvVsqPnS
59R3VOfXc2NW+qB8wGN0IHOlVtZkK21w9w7Jdz/SxAxEz8/7t7fZAxPf6gU3U+Zs
1FNBgnsZDnsfU9cvt2vzYS6+ViRn6AZUbDkVt+iVYjoxAZ4Uj4g98wXV7JP4JNsT
FwBsIo2rrBZd3LsVOAQk3SAqELcueagb9dUb0sQIsTVkphyccvA=
=WdtS
-----END PGP SIGNATURE-----
```
/cc @sarthikgJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41557order and setup new backup server (bungei 2 AKA bacula-storage-02)2024-03-28T12:44:38Zanarcatorder and setup new backup server (bungei 2 AKA bacula-storage-02)The new backup server (#41364) has been approved.
@lavamind can you settle the specs, and order the box? i'm happy to review and approve the hardware if you don't feel fully confident, of course...
i also put the server setup in this i...The new backup server (#41364) has been approved.
@lavamind can you settle the specs, and order the box? i'm happy to review and approve the hardware if you don't feel fully confident, of course...
i also put the server setup in this issue, but we can also spin that out in a separate one if we want to... one thing that's for sure is we want to move to barman for the psql backups (#40950), but one ... uh... problem with that approach is that we actually want cross-backups, so, technically, we *can't* actually deploy *that* on the new server ... oops?
so maybe we need to move other psql databases and we'll have to resize the partition anyways? urghl.
thanks!Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41556Deploy Tor Weather 2.02024-03-25T13:18:01ZGeorg KoppenDeploy Tor Weather 2.0@sarthikg has re-written Tor Weather (yay!) and we want to deploy the 2.0 version now. There are architectural changes as well that e.g. do not need any onionoo-service and onionoo-timer jobs anymore. Morevover, IIRC we need to do some d...@sarthikg has re-written Tor Weather (yay!) and we want to deploy the 2.0 version now. There are architectural changes as well that e.g. do not need any onionoo-service and onionoo-timer jobs anymore. Morevover, IIRC we need to do some database migrations as well. So, this will be exciting I guess. :)Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41555failed disk on fsn-node-022024-03-12T19:23:34ZJérôme Charaouilavamind@torproject.orgfailed disk on fsn-node-02One of the 10GB HDDs on fsn-node-02 has failed over the weekend. The raid-1 volume below `vg_ganeti_hdd` is thus degraded but otherwise healthy.One of the 10GB HDDs on fsn-node-02 has failed over the weekend. The raid-1 volume below `vg_ganeti_hdd` is thus degraded but otherwise healthy.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41552Grant cohosh developer access to the blog project2024-03-07T02:44:43ZanarcatGrant cohosh developer access to the blog projectFollowing the [instructions on the blog wiki page](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/blog#1-navigate-to-the-gitlab-blog-project-at-httpsgitlabtorprojectorgtpowebblog) led me here :) Do you need me to sign this re...Following the [instructions on the blog wiki page](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/blog#1-navigate-to-the-gitlab-blog-project-at-httpsgitlabtorprojectorgtpowebblog) led me here :) Do you need me to sign this request?anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41551Grant cohosh developer access to the blog project2024-03-07T02:44:24ZCecylia BocovichGrant cohosh developer access to the blog projectFollowing the [instructions on the blog wiki page](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/blog#1-navigate-to-the-gitlab-blog-project-at-httpsgitlabtorprojectorgtpowebblog) led me here :) Do you need me to sign this re...Following the [instructions on the blog wiki page](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/blog#1-navigate-to-the-gitlab-blog-project-at-httpsgitlabtorprojectorgtpowebblog) led me here :) Do you need me to sign this request?anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41549BTCPayServer is Down2024-03-18T23:29:44ZSusanBTCPayServer is DownI am unable to connect to the btcpay.torproject.org. It says the site cannot be reached. I believe this means that donors cannot use it to donate either.I am unable to connect to the btcpay.torproject.org. It says the site cannot be reached. I believe this means that donors cannot use it to donate either.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41546GitLab CI object storage cache is broken2024-03-06T20:34:17ZJérôme Charaouilavamind@torproject.orgGitLab CI object storage cache is brokenAll our GitLab CI jobs seem to be failing to upload caches to the MinIO object storage bucket:
```
Uploading cache.zip to https://minio.torproject.org:9000/gitlab-ci-runner-cache/project/2302/default-non_protected
FATAL: received: 403 ...All our GitLab CI jobs seem to be failing to upload caches to the MinIO object storage bucket:
```
Uploading cache.zip to https://minio.torproject.org:9000/gitlab-ci-runner-cache/project/2302/default-non_protected
FATAL: received: 403 Forbidden
Failed to create cache
```
This is probably related to the recent rotation of credentials.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41539Create an operations email list2024-03-28T01:14:48Zal smithCreate an operations email listThe operations team needs an email list to coordinate its work. (This will help with our grants@torproject.org email issues, as we'll be able to reduce the number of people using that alias once the operations list is established.)
**Re...The operations team needs an email list to coordinate its work. (This will help with our grants@torproject.org email issues, as we'll be able to reduce the number of people using that alias once the operations list is established.)
**Requirements**
1. Does **not** require a moderation queue
2. Allows people who are not subscribed to the list to send email to the list **without friction**
3. Is not archived (for anyone, including members of the list)
4. Is not displayed on lists.torproject.org
Is that something a list can do?
If so, we request `tor-operations@` to be created. :smile:
Note: It's possible that an operations list exits already, per this ticket from 8 years ago, but I don't think so based on my quick test. Just adding for due diligence since I noticed it: https://gitlab.torproject.org/tpo/tpa/team/-/issues/15992Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2024-03-31https://gitlab.torproject.org/tpo/tpa/team/-/issues/41536Draft specs and estimates for new backup storage server2024-03-13T21:04:06ZanarcatDraft specs and estimates for new backup storage server(next) cluster scalingJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41534Install package libcapture-tiny-perl on tbb-nightlies-master2024-02-20T18:19:26ZboklmInstall package libcapture-tiny-perl on tbb-nightlies-masterAfter tor-browser-build#41067, we need the package
`libcapture-tiny-perl` to be installed on
`tbb-nightlies-master.torproject.org`.
Thanks!After tor-browser-build#41067, we need the package
`libcapture-tiny-perl` to be installed on
`tbb-nightlies-master.torproject.org`.
Thanks!Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41533update meskio's openpgp key2024-02-21T15:35:32Zmeskiomeskio@torproject.orgupdate meskio's openpgp key```
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
I have extended the expirate date of my openpgp key. Can it be updated in my account?
The key is the same one already present in the account's database:
pub rsa4096 2013-07-23 [SC] ...```
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
I have extended the expirate date of my openpgp key. Can it be updated in my account?
The key is the same one already present in the account's database:
pub rsa4096 2013-07-23 [SC] [expires: 2025-04-14]
07948FFA64160A425BCD27EAC732B1D1C28F4E2F
It can be downloaded from:
https://keys.openpgp.org/vks/v1/by-fingerprint/07948FFA64160A425BCD27EAC732B1D1C28F4E2F
Thank you
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEs7M6f/ZpXzXMAQR+Urj1rJei2oYFAmXTnbsACgkQUrj1rJei
2oYutQ/+NhjAXneUS+IhwMySmgeaU6JS1fhesCQPhPKQ2e4Wo4MuH7A0OLlHk+wK
Adrifi84Kv5UZIK7C2nFwXzIxFaKRNv+dvbe0wAMrX7MtjokEe8WZxDqjFKQHvP1
WHXltMFPv5KBrpyO3EBUxm0GjQmZx7UV3yfp3FV/Flne9vDSKbEYgq3zZlvXjXNI
d3hI00eyuxhPEiw/aT/W0NQSlRI4Tiv4l5ugq6VsdKjeKS09KYUCgO9qAz2iuyA1
rb9oHRwr1TBNc65Ln7pBTgHUWaXqecGIu7DH6VrNQmK+4QKtejp5NuE/1Z6H7XgS
BNsmJWLQUUmHcXdqjQ7tOPtT4hIQoMCKPDjVxSdyEVv6/LMY8BFhLPgtPhSlVsEn
1Rht66GIhIL/qkbnVlktg74TFSpAHjGnoPY6VGo6zZUU0P0GlCSL1S5qHQOF4Tnv
Auj7Hd65uuhBJn7mDukAYWonTAeezPu/MpTaMfy+Yjryc+7zW5CU2VXHFlgxbGuw
e+iZadGsi++ZYgblGBp7Xr/RvG1wux63Y15kASkXK0HvgkmS8g9lm90Ml4vaxHF0
wj48Bvy6VuLepPGZWiL7nWRDGMNH4gwoA8i8ZZAXTdG8pj+x/Xq5GGwmQT6rWR0j
IEic95a5roixoihs6z5f/COWDBTD1VoC+rbq2e6BQAuf4or7FVk=
=0Ffs
-----END PGP SIGNATURE-----
```Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41528Make Lox open invite endpoint only available to telegram bot2024-02-21T13:48:53ZonyinyangMake Lox open invite endpoint only available to telegram botWe have deployed Lox's distributor on `rdsys-frontend-01.torproject.org` and Lox client requests can be made to various Lox server endpoints at `rdsys-frontend-01.torproject.org/lox`. All except one of these requests requires a user to p...We have deployed Lox's distributor on `rdsys-frontend-01.torproject.org` and Lox client requests can be made to various Lox server endpoints at `rdsys-frontend-01.torproject.org/lox`. All except one of these requests requires a user to present a valid Lox credential(s) in order to get the desired response. We would like to limit access to the one endpoint that doesn't require any credentials, `rdsys-frontend-01.torproject.org/lox/invite` to our telegram bot that is running on `polyanthum.torproject.org`. In the future, we will likely use a token to limit access instead, but during the testing/alpha phase, limiting access to polyanthum is probably sufficient.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41527install python3-sqlparse in polyanthum2024-03-11T12:25:55Zmeskiomeskio@torproject.orginstall python3-sqlparse in polyanthumAfter the upgrade to bookworm two weeks ago onbasca stopped working in polyanthum. It looks like is missing a dependency: python3-sqlparse
Can you install it?
I wonder how was removed by the upgrade.After the upgrade to bookworm two weeks ago onbasca stopped working in polyanthum. It looks like is missing a dependency: python3-sqlparse
Can you install it?
I wonder how was removed by the upgrade.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41526Deploy onionperf files parser on metricsdb-012024-03-07T14:23:37ZHiroDeploy onionperf files parser on metricsdb-01We need to deploy https://gitlab.torproject.org/tpo/network-health/metrics/tor_fusion/ on metricsdb-01.
Basically this thing will run, download onionperf files from collector and parse them. This will just happen once a day around 1am UT...We need to deploy https://gitlab.torproject.org/tpo/network-health/metrics/tor_fusion/ on metricsdb-01.
Basically this thing will run, download onionperf files from collector and parse them. This will just happen once a day around 1am UTC as at midnight is when collector fetches the archives from the various onionperf clients.
It's a little rust app and was thinking to create a group and user like for the metrics-api. But maybe it's a bit overkill and I should just put it in the parser space?HiroHirohttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41525gitlab not reachable over ipv6 from (at least) UK ISP Andrews and Arnold2024-03-06T15:07:47ZIan Jacksoniwj@torproject.orggitlab not reachable over ipv6 from (at least) UK ISP Andrews and Arnold```
zealot:~> ping gitlab.torproject.org
PING gitlab.torproject.org(gitlab-02.torproject.org (2a01:4f8:fff0:4f:266:37ff:feb8:3489)) 56 data bytes
```
That's from 2001:8b0:bb7b:4008:c50a:b4d5:6fc1:31f2.
Confirmed by other folks on `#a&a...```
zealot:~> ping gitlab.torproject.org
PING gitlab.torproject.org(gitlab-02.torproject.org (2a01:4f8:fff0:4f:266:37ff:feb8:3489)) 56 data bytes
```
That's from 2001:8b0:bb7b:4008:c50a:b4d5:6fc1:31f2.
Confirmed by other folks on `#a&a` at `irc.aachat.net`. Reports there suggest there might be a problem with Hetzner?
It's reachable from my personal colo, which is with Jump in London.
I don't know if this is a problem at the TPO end, or at the AAISP end. (It's quite inconvenient since it makes git push not work.)Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41523document donate-review deployment process and project in general2024-02-14T21:09:13Zanarcatdocument donate-review deployment process and project in generalin tpo/tpa/team#41519, we have identified that donate-review lacks documentation. #41518 is a task for @lavamind to review that project, but this is for @kez to document it as much as they can.in tpo/tpa/team#41519, we have identified that donate-review lacks documentation. #41518 is a task for @lavamind to review that project, but this is for @kez to document it as much as they can.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41522TPA-RFC-62: migrate tor-passwords to password-store2024-02-21T19:49:00ZanarcatTPA-RFC-62: migrate tor-passwords to password-storeIn #29677, we have reviewed a bunch of password managers. Bitwarden seems to be emerging as a possible candidate for an organisation-wide password management service, but in the short term however, we do not want to make any major change...In #29677, we have reviewed a bunch of password managers. Bitwarden seems to be emerging as a possible candidate for an organisation-wide password management service, but in the short term however, we do not want to make any major changes to our workflow. There's also an argument to be made that TPA should *not* be using a global password manager and is best protecting those secrets with a a different mechanism.
In any case, during a recent offboarding process (tpo/tpa/team#41519), it became very clear that our *current* password manager (pwstore) has major flaws:
1. key management: in this case, @hiro's key was expired and had to be manually removed from the user's list. this would be similar in pass, except that the keyid file is easier to manage, as its signature is managed automatically by `pass init`, provided that the `PASSWORD_STORE_SIGNING_KEY` variable is set
2. password rotation: because multiple passwords are stored in the same file, it's hard or impossible to actually see the last rotation on a single password
3. conflicts: because multiple passwords are stored in the same file, we frequently get conflicts when making changes, which is particularly painful if we need to distribute the "rotation" work
4. abandonware: a [pull request to fix Debian bookworm / Ruby 3.1 support](https://github.com/weaselp/pwstore/pull/8) has been ignored for more than a year at this point
5. counter-intuitive interface: there's no command to extract a password, you're presumably supposed to use `gpg -d` to read the password files, yet you can't use other tools to directly manipulate the password files because the target encryption keys are specified in a meta file (that latter issue is shared with pass, to be fair)
6. not packaged: pwstore is not in Debian, flatpak, or anything else
The main downside to pass is the .gpg-id system is less secure than pwstore: its signature is not enforced unless the environment variable is set, which is a bit brittle. It's also relying on the global GPG key store although in theory it should be possible to rely on another keyring by passing different options to GnuPG.
Finally, by splitting secrets into different files, we disclose **which** accounts we have access to, but I consider this a reasonable tradeoff for the benefits it brings.
Update: the above was put in an actual proposal, see https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-62-tpa-password-manageranarcatanarcat