TPA issueshttps://gitlab.torproject.org/groups/tpo/tpa/-/issues2024-03-28T01:14:48Zhttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41539Create an operations email list2024-03-28T01:14:48Zal smithCreate an operations email listThe operations team needs an email list to coordinate its work. (This will help with our grants@torproject.org email issues, as we'll be able to reduce the number of people using that alias once the operations list is established.)
**Re...The operations team needs an email list to coordinate its work. (This will help with our grants@torproject.org email issues, as we'll be able to reduce the number of people using that alias once the operations list is established.)
**Requirements**
1. Does **not** require a moderation queue
2. Allows people who are not subscribed to the list to send email to the list **without friction**
3. Is not archived (for anyone, including members of the list)
4. Is not displayed on lists.torproject.org
Is that something a list can do?
If so, we request `tor-operations@` to be created. :smile:
Note: It's possible that an operations list exits already, per this ticket from 8 years ago, but I don't think so based on my quick test. Just adding for due diligence since I noticed it: https://gitlab.torproject.org/tpo/tpa/team/-/issues/15992Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2024-03-31https://gitlab.torproject.org/tpo/tpa/team/-/issues/41473consider enforcing 2FA across gitlab2024-02-07T15:55:26Zanarcatconsider enforcing 2FA across gitlabIn #41470, we investigated the impact of an [authentication bypass in GitLab](https://about.gitlab.com/releases/2024/01/11/critical-security-release-gitlab-16-7-2-released/#account-takeover-via-password-reset-without-user-interactions) (...In #41470, we investigated the impact of an [authentication bypass in GitLab](https://about.gitlab.com/releases/2024/01/11/critical-security-release-gitlab-16-7-2-released/#account-takeover-via-password-reset-without-user-interactions) ([CVE-2023-7028](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-7028)). One of the key takeaways is that 2FA renders the attack mostly moot. This makes this group (tpo/tpa) immune to it, but not all users benefit from this.
We should consider enforcing 2FA more broadly here. One likely first target would be tpo/web, which has only a handful of users without 2FA (one of which was @gitolite-merge-bot, which was removed access in #41469). But we could broaden this to all of tpo.
Thoughts, @gaba, @lavamind ?Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2024-02-05https://gitlab.torproject.org/tpo/tpa/team/-/issues/41213lock down legacy git infrastructure2024-03-26T20:49:37Zanarcatlock down legacy git infrastructureAs part of the Gitolite retirement procedure (TPA-RFC-36, #41180), lock Gitolite repositories without any changes in the last
two years, preventing any further change.As part of the Gitolite retirement procedure (TPA-RFC-36, #41180), lock Gitolite repositories without any changes in the last
two years, preventing any further change.legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcat2024-01-31https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252bookworm upgrades, second batch2024-02-06T18:22:35Zanarcatbookworm upgrades, second batchupgrade all those servers to Debian bookworm
- [x] bacula-director-01.torproject.org (@lavamind)
- [x] btcpayserver-02.torproject.org (@anarcat)
- [x] bungei.torproject.org (@lavamind, `crypttab` had a typo, had to boot rescue to recove...upgrade all those servers to Debian bookworm
- [x] bacula-director-01.torproject.org (@lavamind)
- [x] btcpayserver-02.torproject.org (@anarcat)
- [x] bungei.torproject.org (@lavamind, `crypttab` had a typo, had to boot rescue to recover, grub KVM console seems inaccessible)
- [x] carinatum.torproject.org (@kez, had issues during upgrade, followup in https://gitlab.torproject.org/tpo/network-health/doctor/-/issues/40034)
- [x] check-01.torproject.org (@kez, had a prolonged outage: https://gitlab.torproject.org/tpo/network-health/metrics/tor-check/-/issues/40017, mod_qos broken, followup in https://gitlab.torproject.org/tpo/tpa/team/-/issues/41509)
- [x] colchicifolium.torproject.org (@anarcat)
- [x] collector-02.torproject.org (@anarcat)
- [x] crm-ext-01.torproject.org (@anarcat, PHP 8 compatibility issue, followup in #41511)
- [x] crm-int-01.torproject.org (@anarcat)
- [x] dangerzone-01.torproject.org (@kez)
- [x] donate-review-01.torproject.org (@kez)
- [x] gayi.torproject.org (@anarcat)
- [x] gitlab-02.torproject.org (@anarcat, migrated to standalone postgresql following upgrade issue #41426)
- [x] henryi.torproject.org (@kez)
- [x] majus.torproject.org (@anarcat, obsolete transifex-client package left around)
- [x] materculae.torproject.org (@lavamind, noticed extra load on the server, filed https://gitlab.torproject.org/tpo/tpa/team/-/issues/41507)
- [x] meronense.torproject.org (@lavamind, possible OOM regression see #41515)
- [x] metrics-store-01.torproject.org (@anarcat)
- [x] nevii.torproject.org (@anarcat, handful of issues with paths moved from `/usr/sbin` to `/usr/bin`)
- [x] onionbalance-02.torproject.org (@anarcat)
- [x] onionoo-backend-01.torproject.org (@anarcat)
- [x] onionoo-backend-02.torproject.org (@anarcat)
- [x] onionoo-frontend-01.torproject.org (@anarcat)
- [x] onionoo-frontend-02.torproject.org (@anarcat)
- [x] polyanthum.torproject.org (@lavamind)
- [x] probetelemetry-01.torproject.org (@anarcat)
- [x] rdsys-frontend-01.torproject.org (@anarcat)
- [x] rude.torproject.org (@lavamind)
- [x] survey-01.torproject.org (@lavamind)
- [x] telegram-bot-01.torproject.org (@anarcat)
- [x] weather-01.torproject.org (@anarcat, catastrophic data loss, see #41388)
31 machines
like the first batch, due date is approximate here, used to ping the team to organise this before the actual week planned in TPA-RFC-57, which is "last week of october".
an announcement need to be sent to remind people of this upcoming batch when the first due date is hit.Debian 12 bookworm upgradeanarcatanarcat2024-01-29https://gitlab.torproject.org/tpo/tpa/team/-/issues/41214review gitolite retirement progress and send a reminder2024-03-20T18:56:12Zanarcatreview gitolite retirement progress and send a reminderAs part of the Gitolite retirement procedure (TPA-RFC-36, #41180), review the progress of the migration and send a reminder:
- [x] how many repositories are left to migrate, populating #41215 with the result
- [x] did any repository get...As part of the Gitolite retirement procedure (TPA-RFC-36, #41180), review the progress of the migration and send a reminder:
- [x] how many repositories are left to migrate, populating #41215 with the result
- [x] did any repository get changes since the deprecation notice on 2023-06-08
- [x] send a reminder, similar to #41212legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcat2024-01-24https://gitlab.torproject.org/tpo/tpa/team/-/issues/41460oo.nc.torproject.net CAA record prevents issuance2024-01-10T13:13:41Zmicahmicah@torproject.orgoo.nc.torproject.net CAA record prevents issuanceThe CAA record that exists for torproject.org makes it not possible for `oo.nc.torproject.net` certificates to be generated on riseup's side.
The certificate is set to expire on Wed, 24 Jan 2024 05:54:25 GMT.The CAA record that exists for torproject.org makes it not possible for `oo.nc.torproject.net` certificates to be generated on riseup's side.
The certificate is set to expire on Wed, 24 Jan 2024 05:54:25 GMT.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2024-01-19https://gitlab.torproject.org/tpo/tpa/team/-/issues/41459Upgrade GitLab to 16.72024-01-08T15:22:28ZJérôme Charaouilavamind@torproject.orgUpgrade GitLab to 16.7GitLab 16.7 has been released yesterday, but the team's going on holiday and we fear regression could hit while we're away.
The last two such upgrades brought with them issues that threathened to saturate our storage...
After the holid...GitLab 16.7 has been released yesterday, but the team's going on holiday and we fear regression could hit while we're away.
The last two such upgrades brought with them issues that threathened to saturate our storage...
After the holidays, remove the hold on `gitlab-ce`.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2024-01-16https://gitlab.torproject.org/tpo/tpa/team/-/issues/41462CAA record prevents renewing certs for snowflake.torproject.net, 02.snowflake...2024-01-11T18:33:24ZDavid Fifielddcf@torproject.orgCAA record prevents renewing certs for snowflake.torproject.net, 02.snowflake.torproject.net, and snowflake-broker.torproject.net[Since 2017](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/18654#note_2591192),
we've used Let's Encrypt (via the [acme/autocert](https://pkg.go.dev/golang.org/x/crypto@v0.17.0/acme/autocert) p...[Since 2017](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/18654#note_2591192),
we've used Let's Encrypt (via the [acme/autocert](https://pkg.go.dev/golang.org/x/crypto@v0.17.0/acme/autocert) package)
to issue TLS certificates for the Snowflake bridges and broker.
The [new CAA record](tpo/tpa/team#41386) for torproject.net
has made that [stop working](tpo/anti-censorship/pluggable-transports/snowflake#40319).
These are the expiration dates
of the certificates we're not able to renew
using the Let's Encrypt accounts we have used up to this point:
|Domain|NotAfter|
|--:|---|
|02.snowflake.torproject.net|2024-01-13 14:03:53|
|snowflake.torproject.net|2024-01-20 11:23:43|
|snowflake-broker.torproject.net|2024-02-22 04:15:23|
The earliest expiration is less than a week from now.
How should we proceed?
Related: tpo/tpa/team#41455, tpo/tpa/team#41460Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2024-01-12https://gitlab.torproject.org/tpo/tpa/team/-/issues/41402gitlab-02 running out of disk space in gitlab-shared2023-12-11T22:43:56Zanarcatgitlab-02 running out of disk space in gitlab-sharedGitLab is running out of disk space again.
something happened in the last month that made it eat 100GB in about 3 weeks:
![image](/uploads/a00f17a4ece44c5d6c981619608228fe/image.png)
https://grafana.torproject.org/d/zbCoGRjnz/disk-usag...GitLab is running out of disk space again.
something happened in the last month that made it eat 100GB in about 3 weeks:
![image](/uploads/a00f17a4ece44c5d6c981619608228fe/image.png)
https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&var-class=All&var-instance=gitlab-02.torproject.org&from=now-90d&to=now
even more visible in the yearly:
![image](/uploads/473b9c39b11623631dcaad89f77d932d/image.png)
https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&var-class=All&var-instance=gitlab-02.torproject.org&from=now-1y&to=now
our daily rate is 6.65GB which means we have slightly more than 48h before running out of disk completely (15GB remaining)
battle plan:
- [x] @lavamind figure out a short term fix
- [x] erase artifacts in projects under `tpo/web/l10n` + disable "Keep latest artifacts"
- [x] erase artifacts in `gabi-205/arti` (22GB) + disable "Keep latest artifacts"
- [ ] deal with storage used by `tpo/core/debian/tor` (39GB)
- [x] @anarcat make a ticket to move artifacts to minio, possibly growing minio in the process (#41403)
- [ ] emergency procedures
- [ ] vacuum pipelines
- [ ] donate backup drive to shared
- [ ] move to MinIO (#41403)Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2023-12-18https://gitlab.torproject.org/tpo/tpa/team/-/issues/41420Pipeline housekeeping for tpo/core/debian/tor project2023-12-12T15:29:52ZJérôme Charaouilavamind@torproject.orgPipeline housekeeping for tpo/core/debian/tor projectAs a follow-up for https://gitlab.torproject.org/tpo/tpa/team/-/issues/41402#note_2970105 I would like to propose that daily-scheduled CI pipelines that build supported branches in `tpo/core/debian/tor` be automatically deleted after 2 w...As a follow-up for https://gitlab.torproject.org/tpo/tpa/team/-/issues/41402#note_2970105 I would like to propose that daily-scheduled CI pipelines that build supported branches in `tpo/core/debian/tor` be automatically deleted after 2 weeks.
There are currently three such schedules, and they each run daily, adding some ~50MB of logs. When CI artifacts cleanup is broken, they also each add ~250MB of artifacts.
Implementing this automatic deletion would:
- Significantly reduce the storage requirements for job logs (currently 13.80GB of job logs for these pipelines specifically)
- Help limit the impact of any GitLab regresssion causing CI build artifacts to accumulate
Artifacts and build logs for [CI pipelines that run on *tagged* commits](https://gitlab.torproject.org/tpo/core/debian/tor/-/pipelines?scope=tags) would not be affected, and still be kept indefinitely.
This can be implemented very easily using either a cron job or service/timer pair, to run the following shell command:
gitlab-rails runner "Ci::Pipeline.where(project_id: 1218, source: 'schedule', locked: 'unlocked').where('finished_at < ?', 2.weeks.ago).delete
Unless further discussion is needed, I will implement this on Monday, December 11.
/cc @anarcat @weaselJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2023-12-12https://gitlab.torproject.org/tpo/tpa/team/-/issues/41393Civi-automated thank-you messages include outdated tweet content2023-11-08T20:29:06Zal smithCivi-automated thank-you messages include outdated tweet contentHi!
I believe one or more of the automated thank-you emails that Civi handles is sending out a) outdated copy that encourages folks to post on social with the following (_"Let's resist the surveillance pandemic. Use a mask, use Tor. Joi...Hi!
I believe one or more of the automated thank-you emails that Civi handles is sending out a) outdated copy that encourages folks to post on social with the following (_"Let's resist the surveillance pandemic. Use a mask, use Tor. Join me and donate to @torproject: https://donate.torproject.org #UseAMaskUseTor"_) or b) [this link](https://twitter.com/intent/tweet?text=Let%27s%20resist%20the%20surveillance%20pandemic.%20Use%20a%20mask%2C%20use%20Tor.%20Join%20me%20and%20donate%20to%20%40torproject%3A%20https%3A//donate.torproject.org%20%23UseAMaskUseTor) or c) both.
You can see the following examples of folks who have tweeted this content:
- https://x.com/rommeltoledo/status/1714480706741527006
- https://twitter.com/search?q=%20Let%27s%20resist%20the%20surveillance%20pandemic.%20Use%20a%20mask%2C%20use%20Tor.%20Join%20me%20and%20donate%20to%20%40torproject%3A%20https%3A%2F%2Fdonate.torproject.org%20%23UseAMaskUseTor%20&src=typed_query&f=live
Because these are the emails that are sent by Civi in the weird alt setup (e.g. tpo/web/civicrm#104, tpo/web/civicrm#69), I cannot confirm which emails they are.
This is a tagline from the campaign 3 years ago, so I'd like to prioritize a fix to stop folks from sharing it.al smithal smith2023-10-31https://gitlab.torproject.org/tpo/tpa/team/-/issues/41367retire tb-build-04 and tb-build-052023-11-01T19:41:51Zanarcatretire tb-build-04 and tb-build-05following disk space issues on tb-build-04 and tb-build-05 (#40964), we've built two new shiny servers for the apps team (#41304), which they seem happy with.
now they should have moved over, make sure they did and then proceed with the...following disk space issues on tb-build-04 and tb-build-05 (#40964), we've built two new shiny servers for the apps team (#41304), which they seem happy with.
now they should have moved over, make sure they did and then proceed with the server retirement procedure, checklist:
1. [x] announcement
2. [x] nagios
3. [x] retire the host in fabric
4. [x] remove from LDAP with `ldapvi`
5. [x] power-grep
6. [x] remove from tor-passwords
7. [x] remove from DNSwl
8. [x] remove from docs
9. [x] remove from racks
10. [x] remove from reverse DNS
@richard have you folks moved out of the old build servers, can we retire those now? as a reminder, our procedure is:
1. retirement day: server shutdown
2. 7 days later: server destruction
3. 30 days after the shutdown: backups get destroyed
Can we start with the "retirement day" some time this week? In #40964 I believe we had today set for this, but let's see if you're ready first. :)
/cc @tpo/applicationsanarcatanarcat2023-10-31https://gitlab.torproject.org/tpo/tpa/team/-/issues/40696upgrade or rebuild pauli / puppet2024-01-17T17:34:26Zanarcatupgrade or rebuild pauli / puppetPuppet AKA pauli is going to be a particularly tricky bullseye upgrade, so it's not part of the large bullseye upgrade batches (#40690 or #40692).
We need to decide how we handle the Puppet 5 EOL situation. We could use upstream package...Puppet AKA pauli is going to be a particularly tricky bullseye upgrade, so it's not part of the large bullseye upgrade batches (#40690 or #40692).
We need to decide how we handle the Puppet 5 EOL situation. We could use upstream packages, split up the service, there's many possibilities (#33588). But this ticket is solely about upgrading the box, which could happen regardless, as bullseye was (bizarrely) with Puppet 5 (but not puppetdb!).
This also implies the decision on puppet-agent itself is done elsewhere (and that is probably #33588).Debian 11 bullseye upgradeJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2023-10-25https://gitlab.torproject.org/tpo/tpa/team/-/issues/40964tb-build-04 and tb-build-05 running out of disk space2023-10-23T14:46:32Zanarcattb-build-04 and tb-build-05 running out of disk space@richard is really nice and doesn't want to bother us, but they are running out of disk on the builders.
- tb-build-04 is https://www.hetzner.com/dedicated-rootserver/ax51-nvme/configurator#/
- tb-build-05 is https://www.hetzner.com/ded...@richard is really nice and doesn't want to bother us, but they are running out of disk on the builders.
- tb-build-04 is https://www.hetzner.com/dedicated-rootserver/ax51-nvme/configurator#/
- tb-build-05 is https://www.hetzner.com/dedicated-rootserver/ax161/configurator#/
and this is what it looks like:
![image](/uploads/ffd584eb5f7c68f7c61bf576a93d52e6/image.png)
https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&var-class=All&var-instance=tb-build-04.torproject.org&var-instance=tb-build-05.torproject.org&from=now-6M&to=now
ouch! :pita: (there is not pita emoji, wtf is the unicode comittee up to)
we should give them more disk space, which requires a budget approval. @richard or @micah or someone will take care of the red tape and tpa will handle hetzner.
update: plan is to setup two new boxes to replace the two at hetzner
- [x] figure out quote for the servers (@lavamind @anarcat, done, see https://gitlab.torproject.org/tpo/tpa/team/-/issues/40964#note_2901313, still valid as of 2023-07-06, 350$/mth per server amortized over 5 years)
- [x] formalize approval (@micah)
- [x] order the servers (@anarcat)
- [x] quintex racks the server (@anarcat keeps track)
- [x] TPA sets it up
- [ ] apps team moves over (@richard)
- [ ] retire the other two server at hetzneranarcatanarcat2023-10-23https://gitlab.torproject.org/tpo/tpa/team/-/issues/40763automate BTCpayserver upgrades2023-09-13T17:23:26Zanarcatautomate BTCpayserver upgradesin #33750 we switched the BTCpayserver to be hosted in TPA infra, which solves a lot of problems: user accounts, backups, monitoring, etc is covered by our normal policies.
but the app itself isn't upgraded automatically. figure out how...in #33750 we switched the BTCpayserver to be hosted in TPA infra, which solves a lot of problems: user accounts, backups, monitoring, etc is covered by our normal policies.
but the app itself isn't upgraded automatically. figure out how we'll take care of that, either through a docker-compose cron job, or manually.anarcatanarcat2023-10-13https://gitlab.torproject.org/tpo/tpa/team/-/issues/41346Migrate pauli to gnt-dal2023-10-11T12:35:03ZJérôme Charaouilavamind@torproject.orgMigrate pauli to gnt-dalWhile working on tpo/tpa/team#41341 we figured out that the latency between gnt-dal and gnt-fsn is likely the cause of additional delays when running a Puppet agent run, when `pauli` is configured to use the `puppetdb-01` PuppetDB backen...While working on tpo/tpa/team#41341 we figured out that the latency between gnt-dal and gnt-fsn is likely the cause of additional delays when running a Puppet agent run, when `pauli` is configured to use the `puppetdb-01` PuppetDB backend.
We decide to attempt to migrate `pauli` from gnt-fsn to gnt-dal in the hope that this will remove the extraneous delays.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2023-10-12https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/23Confidential issues and comments on them are sent out in the clear2024-03-04T15:13:46ZGeorg KoppenConfidential issues and comments on them are sent out in the clearI just tested out confidential issues a bit as that's one of the Gitlab
features I was looking forward to.
It turns out that notifications containing the creation of confidential
issues and comments to them are sent out in the clear, un...I just tested out confidential issues a bit as that's one of the Gitlab
features I was looking forward to.
It turns out that notifications containing the creation of confidential
issues and comments to them are sent out in the clear, unencrypted,
which is problematic.
I am not even sure who exactly gets those notifications. But that's a
different thing to investigate (and a bit hard for me to test as I filed
a confidential test issue in a project I am owner of and I don't want to
spam other projects).
So, Bugzilla solves this part very nicely actually: by default folks
watching a confidential issue just get a notice by email that that issue
got updated, so there is nothing confidential leaking out here, but one
has to log in to actually see what changed/got added. Alternatively, if
one adds an OpenPGP key to the user profile, one gets an encrypted mail
for issue updates with actual meat one can read locally after decrypting.
I am not sure what exactly is possible in Gitlab but it should *not*
send out notifications for confidential issue updates/creations leaking
sensitive information.anarcatanarcat2023-10-04https://gitlab.torproject.org/tpo/tpa/team/-/issues/41251bookworm upgrades, first batch2023-09-27T14:04:30Zanarcatbookworm upgrades, first batchupgrade all those servers to Debian bookworm
- [x] archive-01.torproject.org (@lavamind)
- [x] cdn-backend-sunet-02.torproject.org (@lavamind)
- [x] ci-runner-x86-03.torproject.org (@lavamind)
- [x] chives.torproject.org (@lavamind)
- [...upgrade all those servers to Debian bookworm
- [x] archive-01.torproject.org (@lavamind)
- [x] cdn-backend-sunet-02.torproject.org (@lavamind)
- [x] ci-runner-x86-03.torproject.org (@lavamind)
- [x] chives.torproject.org (@lavamind)
- [x] ~~ci-runner-x86-01.torproject.org~~ (retired)
- [x] dal-rescue-01.torproject.org (@lavamind)
- [x] dal-rescue-02.torproject.org (@lavamind)
- [x] hetzner-hel1-02.torproject.org (@lavamind)
- [x] hetzner-hel1-03.torproject.org (@lavamind)
- [x] hetzner-nbg1-01.torproject.org (@anarcat )
- [x] hetzner-nbg1-02.torproject.org (@anarcat)
- [x] loghost01.torproject.org (@lavamind)
- [x] mandos-01.torproject.org (@lavamind)
- [x] media-01.torproject.org (@lavamind)
- [x] neriniflorum.torproject.org (@lavamind)
- [x] ns3.torproject.org (@lavamind)
- [x] ns5.torproject.org (@anarcat)
- [x] palmeri.torproject.org (@lavamind)
- [x] perdulce.torproject.org (@anarcat)
- [x] relay-01.torproject.org (@anarcat)
- [x] static-gitlab-shim.torproject.org (@anarcat)
- [x] static-master-fsn.torproject.org (@anarcat)
- [x] staticiforme.torproject.org (@anarcat)
- [x] submit-01.torproject.org (@anarcat)
- [x] ~~tb-build-01.torproject.org~~ (retired)
- [x] tb-build-04.torproject.org (@anarcat)
- [x] tb-build-05.torproject.org (@anarcat)
- [x] tb-pkgstage-01.torproject.org (@anarcat)
- [x] tb-tester-01.torproject.org (@anarcat)
- [x] tbb-nightlies-master.torproject.org (@anarcat)
- [x] web-dal-07.torproject.org (@lavamind)
- [x] web-dal-08.torproject.org (@anarcat)
- [x] web-fsn-01.torproject.org (@anarcat)
- [x] web-fsn-02.torproject.org (@anarcat)
33 machines
due date is set as an alarm to organise this for the second or third week of September, with a 2 week advance notice. the actual due date should be more like 2023-09-22 or something like that.
an announcement need to be sent to remind people of this upcoming batch when the first due date is hit.Debian 12 bookworm upgradeanarcatanarcat2023-09-25https://gitlab.torproject.org/tpo/tpa/team/-/issues/41258materculae out of disk space2023-09-21T01:51:41ZKezmaterculae out of disk spaceprevious ticket: #40826
it's been a year, and nagios is complaining about materculae's /srv partition
```
# df -h /srv
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_materculae-srv 147G 135G 4.3G 97%...previous ticket: #40826
it's been a year, and nagios is complaining about materculae's /srv partition
```
# df -h /srv
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_materculae-srv 147G 135G 4.3G 97% /srv
```
in the previous ticket (#40826) @anarcat changed the warning threshold, which is why this warning popped up now.
according to grafana, the usage has only been about 15G in the past year, and the growth is linear. we could add another 20G and revisit in a year, or throw 40G or 60G at it to push things further down the road.
![image](/uploads/e8ddf8b69703273f73d891586f7fc137/image.png)anarcatanarcat2023-09-22https://gitlab.torproject.org/tpo/tpa/team/-/issues/41297A staging server for rdsys2023-12-12T18:59:16Zmeskiomeskio@torproject.orgA staging server for rdsysWe want to set up a staging server of rdsys, that will be automatically deployed on each commit from the CI. We'll need a new VM for it. Many things will be similar to polyanthium, but we might not need separation of services by users (i...We want to set up a staging server of rdsys, that will be automatically deployed on each commit from the CI. We'll need a new VM for it. Many things will be similar to polyanthium, but we might not need separation of services by users (it might be easier to deploy if we have everything in one user).
We will not need much disk space, CPU or RAM, whatever are the defaults you use now a days will be enough for us.
What we need there:
* [ ] ~~an account that we can ssh automatically from the CI to setup everything.~~ let's postpone this
* [x] We'll also need everybody from anti-censorship to be able to sudo into that account.
* [x] an email account that can send and receive emails over imap and smtp. Maybe gettor-tst@torproject.org?
* [x] a web server with:
* [x] https://bridges-tst.torproject.org proxing to http://localhost:7200
* [x] https://bridges-tst.torproject.org/moat proxing to http://localhost:7500/moat
* [x] https://bridges-tst.torproject.org/status proxing to http://localhost:7100/status
* [x] prometheus exporters to be exposed, I don't think we'll connect them to the prometheus server, but will be useful to be able to reach them for tests:
* [x] backend bridges-tst.torproject.org:7100/metrics
* [x] telegram bridges-tst.torproject.org:7600/metrics
* [x] gettor-distributor bridges-tst.torproject.org:7700/metrics
* [x] gettor-updater bridges-tst.torproject.org:7800/metrics
I don't have a strong opinion for port numbers, the domain names and the email address, I'm just putting some proposals here but I'm happy to adapt to what you think makes sense.anarcatanarcat2023-09-17