TPA issueshttps://gitlab.torproject.org/groups/tpo/tpa/-/issues2021-01-30T00:53:33Zhttps://gitlab.torproject.org/tpo/tpa/team/-/issues/40138Create a static HTML page at: status.torproject.org2021-01-30T00:53:33ZDavid Gouletdgoulet@torproject.orgCreate a static HTML page at: status.torproject.orgThe idea here is for TPO to have a `status` page where we can post status problem we know about so the world can learn if problems are known to the Tor team.
A great example would have been the v3 outtage recently where we could have pu...The idea here is for TPO to have a `status` page where we can post status problem we know about so the world can learn if problems are known to the Tor team.
A great example would have been the v3 outtage recently where we could have put out a small FAQ right away (go static HTML!) and then update the world as we figure out the problem but also expected return to normal.
I propose a static website at the moment in order to have this page going very soon and easily editable by many of us namely likely the network health team and we could also have the network team to help out.
We can then over time think of better automated way to get status on that page like automatic reports of sensors/metrics we have and so on. But for the moment, just a DNS + static HTML would be grand.
Thanks!
Update, next steps:
- [x] brainstorm/discuss requirements (see [goals](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/status#goals))
- [x] prototype cstate (@anarcat, people seem happy with it)
- [x] basic theming (thanks @duncan!)
- [x] rush deployment on static site network (@anarcat)
- [x] basic documentation (see [service page](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/status))
- [x] grant access to other teams
- [x] setup a mirrored repo in gitolite (allow access to the same members as here!)
- [x] setup CI in Jenkins @hiro
- [x] figure out which categories we want? (update as needed)
- [x] document the new git workflow for updates (@hiro)HiroHirohttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41083TPA-RFC-53: consider propagating 2FA everywhere, maybe at the April Tor Meeting2023-05-23T16:06:02ZanarcatTPA-RFC-53: consider propagating 2FA everywhere, maybe at the April Tor MeetingI've been meaning to do a training or work session or "EVERYONE GETS A YUBIKEY PARTY" thing for a while now. I don't know how it will look like. I don't know quite the requirements yet. But it feels like an opportunity.
/cc @shelikhoo @...I've been meaning to do a training or work session or "EVERYONE GETS A YUBIKEY PARTY" thing for a while now. I don't know how it will look like. I don't know quite the requirements yet. But it feels like an opportunity.
/cc @shelikhoo @linusanarcatanarcat2023-04-06https://gitlab.torproject.org/tpo/tpa/team/-/issues/41175re-evaluate our certificate authorities and pinning at Mozilla / Google Chrome2023-05-16T19:08:56Zanarcatre-evaluate our certificate authorities and pinning at Mozilla / Google ChromeIn two years from now, look at which certificate authorities and how that affects the pins we have in Mozilla Firefox and Google Chrome, see #41154 for background and the previous instance of this.In two years from now, look at which certificate authorities and how that affects the pins we have in Mozilla Firefox and Google Chrome, see #41154 for background and the previous instance of this.2025-05-16https://gitlab.torproject.org/tpo/tpa/nextcloud/-/issues/15Disable Nextcloud Talk app2022-11-03T13:02:39ZJérôme Charaouilavamind@torproject.orgDisable Nextcloud Talk appI'd like to propose that we disable the Talk app in Nextcloud.
@anarcat and I tried to use it for voice chat yesterday, and it failed with:
> Could not establish a connection with at least one participant. A TURN server might be needed...I'd like to propose that we disable the Talk app in Nextcloud.
@anarcat and I tried to use it for voice chat yesterday, and it failed with:
> Could not establish a connection with at least one participant. A TURN server might be needed for your scenario. Please ask your administrator to set one up following this [documentation ↗](https://nextcloud-talk.readthedocs.io/en/latest/TURN/)
So currently, due to incomplete configuration, it only potentially serves as a chat app, overlapping our use of IRC and Signal.
I think it might be worth evaluating it sometime down the road, if we decide to look at BBB alternatives, but now isn't the time.
@anarcat @gaba What do you think?Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/40838Change LDAP username from hacim -> micah2022-09-27T19:25:19Zmicahmicah@torproject.orgChange LDAP username from hacim -> micahI've talked with @micahflee about my username causing confusion, and he has gracefully agreed to use @micahflee instead. He has already changed his username here on gitlab, and I was able to successfully claim @micah.
When it comes to ...I've talked with @micahflee about my username causing confusion, and he has gracefully agreed to use @micahflee instead. He has already changed his username here on gitlab, and I was able to successfully claim @micah.
When it comes to LDAP, this is something I'll need help from the nice TPA folks to make happen.
Besides LDAP, are there other infrastructure where this would need to be changed?anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/40625Clean up lektor-staging.tpo off of static mirrors2022-02-15T14:47:35ZJérôme Charaouilavamind@torproject.orgClean up lektor-staging.tpo off of static mirrors@emmapeel confirmed we can get rid of it, as it isn't in use anymore.@emmapeel confirmed we can get rid of it, as it isn't in use anymore.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/status-site/-/issues/16Mark "V2 Onion Services deprecation" resolved as of 08-11-20212022-02-04T17:50:36ZdonutsMark "V2 Onion Services deprecation" resolved as of 08-11-2021Let's take down the V2 deprecation notice by marking it as resolved, and backdating its resolution to 08-11-2021 (Tor Browser 11's release date).Let's take down the V2 deprecation notice by marking it as resolved, and backdating its resolution to 08-11-2021 (Tor Browser 11's release date).https://gitlab.torproject.org/tpo/tpa/team/-/issues/40425improve the spam filter on the RT frontdesk2023-07-04T16:02:16Zchampionquizzerchampionquizzer@torproject.orgimprove the spam filter on the RT frontdeskOn frontdesk, we are consistently receiving nearly 1200-1300 spam tickets per month. Recently (since 17th September, 2021), we have started blocking certain recurring spammers on a user level, yet the statistics don't seem so promising. ...On frontdesk, we are consistently receiving nearly 1200-1300 spam tickets per month. Recently (since 17th September, 2021), we have started blocking certain recurring spammers on a user level, yet the statistics don't seem so promising. Since 18th September 00:00 UTC we have received 270 tickets in spam (and there's much more in the frontdesk queue which we haven't yet manually queued as spam). Would it be possible to do something on your end? :)
Thanks!improve mail servicesJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/40214Have prometheus send alert emails in plaintext rather than HTML2021-04-19T20:24:38ZCecylia BocovichHave prometheus send alert emails in plaintext rather than HTMLI have a general desire for fewer HTML-based emails in my life :)
The prometheus documentation for configuring email alerts suggests that we can choose between text and HTML: https://prometheus.io/docs/alerting/latest/configuration/#ema...I have a general desire for fewer HTML-based emails in my life :)
The prometheus documentation for configuring email alerts suggests that we can choose between text and HTML: https://prometheus.io/docs/alerting/latest/configuration/#email_config
Currently the alertmanager is configured to use the [default email HTML template](https://github.com/prometheus/alertmanager/blob/master/template/default.tmpl#L81), which is very ugly. There is no default text-based email template so we will have to [write our own](https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/). To do this, we add a custom template file that looks something like:
```
{{ define "email.custom.txt" }}
...
{{ end }}
```
and then modify our alertmanager config to reference this template:
```
receivers:
- name: 'email'
email_configs:
- to: 'receiver_mail_id@gmail.com'
from: 'mail_id@gmail.com'
text: '{{ template "email.custom.txt" . }}'
```https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/72monitor anonymous user account2020-09-30T19:05:30ZMatthew Finkelmonitor anonymous user accountI noticed user `writecode` is now a replacement for the trac `cypherpunks` account with the password in the account's name. I'm not opposed to having an anonymous account, but I know there was concern about creating one when we migrated ...I noticed user `writecode` is now a replacement for the trac `cypherpunks` account with the password in the account's name. I'm not opposed to having an anonymous account, but I know there was concern about creating one when we migrated to gitlab. @gaba suggested we open a ticket, monitor the account's activity, and discuss it at the next gitlab meeting.https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/58Delete or archive tpo/core projects not maintained by the tpo/core team2020-09-29T17:29:11ZNick MathewsonDelete or archive tpo/core projects not maintained by the tpo/core teamThere are a few projects under tpo/core that we should probably archive, delete, or something.
https://gitlab.torproject.org/tpo/core/pytorctl is superseded by stem. We don't maintain it any more; only obsolete tools like the old torfl...There are a few projects under tpo/core that we should probably archive, delete, or something.
https://gitlab.torproject.org/tpo/core/pytorctl is superseded by stem. We don't maintain it any more; only obsolete tools like the old torflow use it. No commits to that repository since 2015.
https://gitweb.torproject.org/tordnsel.git/ is written in Haskell, and in an old version too. We don't understand it; if we were going to maintain this, we would probably start by just rewriting it. No commits to that repository since 2016.
https://gitlab.torproject.org/tpo/core/rpm-package is not something we maintain, and has no corresponding it repository. There is only one ticket there, and it is about retiring the trac component "rpm packaging"
Not urgent, and let's not actually make these changes till other @tpo/core people can weigh in.Gabagaba@torproject.orgGabagaba@torproject.orghttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/46Gitlab should show text files in the browser2022-03-24T23:28:17ZGeorg KoppenGitlab should show text files in the browserRight now if I want to look at some .md or .txt file on Gitlab I need to
download and open it with an external application. However, that should
not be necessary. The browser should be sufficient for this task.Right now if I want to look at some .md or .txt file on Gitlab I need to
download and open it with an external application. However, that should
not be necessary. The browser should be sufficient for this task.https://gitlab.torproject.org/tpo/tpa/team/-/issues/41557order and setup new backup server (bungei 2 AKA bacula-storage-02)2024-03-28T12:44:38Zanarcatorder and setup new backup server (bungei 2 AKA bacula-storage-02)The new backup server (#41364) has been approved.
@lavamind can you settle the specs, and order the box? i'm happy to review and approve the hardware if you don't feel fully confident, of course...
i also put the server setup in this i...The new backup server (#41364) has been approved.
@lavamind can you settle the specs, and order the box? i'm happy to review and approve the hardware if you don't feel fully confident, of course...
i also put the server setup in this issue, but we can also spin that out in a separate one if we want to... one thing that's for sure is we want to move to barman for the psql backups (#40950), but one ... uh... problem with that approach is that we actually want cross-backups, so, technically, we *can't* actually deploy *that* on the new server ... oops?
so maybe we need to move other psql databases and we'll have to resize the partition anyways? urghl.
thanks!Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41555failed disk on fsn-node-022024-03-12T19:23:34ZJérôme Charaouilavamind@torproject.orgfailed disk on fsn-node-02One of the 10GB HDDs on fsn-node-02 has failed over the weekend. The raid-1 volume below `vg_ganeti_hdd` is thus degraded but otherwise healthy.One of the 10GB HDDs on fsn-node-02 has failed over the weekend. The raid-1 volume below `vg_ganeti_hdd` is thus degraded but otherwise healthy.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41553write a blog post about the static mirror system2024-03-11T17:06:03Zanarcatwrite a blog post about the static mirror systemI found [this post](https://alexcabal.com/posts/standard-ebooks-and-classic-web-tech) to be pretty interesting. I wish I could write about some fancy new high-tech system we've built in TPA that's the cutting edge of technology, but the ...I found [this post](https://alexcabal.com/posts/standard-ebooks-and-classic-web-tech) to be pretty interesting. I wish I could write about some fancy new high-tech system we've built in TPA that's the cutting edge of technology, but the reality is that we're a hodgepodge collection of legacy systems we're keeping alive by a wise combination of "if it ain't broken don't fix it" and "okay, this is too horrible, let's fix that tiny piece", migrating one system at a time toward modernity.
The static mirror system is an excellent example of this. When I arrived, it was mostly built from shell servers and... Jenkins, which was hard to use and generally disliked. We migrated to GitLab and built a shim to avoid having to replace the entire system. That handful of servers is pumping out gigabits per second, it's easy to deploy and scale out (although *that* could be made easier).
This is mostly summarizing and glorifying the docs I've already written in the [service docs](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/static-component/).
This would be, therefore, an interesting blog post on its own, but I think it could also serve as great advertisement for the job posting (tpo/tpa/team#41542).anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41522TPA-RFC-62: migrate tor-passwords to password-store2024-02-21T19:49:00ZanarcatTPA-RFC-62: migrate tor-passwords to password-storeIn #29677, we have reviewed a bunch of password managers. Bitwarden seems to be emerging as a possible candidate for an organisation-wide password management service, but in the short term however, we do not want to make any major change...In #29677, we have reviewed a bunch of password managers. Bitwarden seems to be emerging as a possible candidate for an organisation-wide password management service, but in the short term however, we do not want to make any major changes to our workflow. There's also an argument to be made that TPA should *not* be using a global password manager and is best protecting those secrets with a a different mechanism.
In any case, during a recent offboarding process (tpo/tpa/team#41519), it became very clear that our *current* password manager (pwstore) has major flaws:
1. key management: in this case, @hiro's key was expired and had to be manually removed from the user's list. this would be similar in pass, except that the keyid file is easier to manage, as its signature is managed automatically by `pass init`, provided that the `PASSWORD_STORE_SIGNING_KEY` variable is set
2. password rotation: because multiple passwords are stored in the same file, it's hard or impossible to actually see the last rotation on a single password
3. conflicts: because multiple passwords are stored in the same file, we frequently get conflicts when making changes, which is particularly painful if we need to distribute the "rotation" work
4. abandonware: a [pull request to fix Debian bookworm / Ruby 3.1 support](https://github.com/weaselp/pwstore/pull/8) has been ignored for more than a year at this point
5. counter-intuitive interface: there's no command to extract a password, you're presumably supposed to use `gpg -d` to read the password files, yet you can't use other tools to directly manipulate the password files because the target encryption keys are specified in a meta file (that latter issue is shared with pass, to be fair)
6. not packaged: pwstore is not in Debian, flatpak, or anything else
The main downside to pass is the .gpg-id system is less secure than pwstore: its signature is not enforced unless the environment variable is set, which is a bit brittle. It's also relying on the global GPG key store although in theory it should be possible to rely on another keyring by passing different options to GnuPG.
Finally, by splitting secrets into different files, we disclose **which** accounts we have access to, but I consider this a reasonable tradeoff for the benefits it brings.
Update: the above was put in an actual proposal, see https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-62-tpa-password-manageranarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41486Tor theme for LimeSurvey2024-02-01T22:42:52ZsajolidaTor theme for LimeSurveyAs part of tpo/ux/research#130, I created a Tor theme for LimeSurvey that has a Tor color scheme and prints better on paper.
You can see the theme in action on https://survey.potager.org/index.php/457772.
I stored it in a Git repositor...As part of tpo/ux/research#130, I created a Tor theme for LimeSurvey that has a Tor color scheme and prints better on paper.
You can see the theme in action on https://survey.potager.org/index.php/457772.
I stored it in a Git repository for now: https://gitlab.com/sajolida/fruity-twentythree-tor.
I'm wondering what's the best way of deploying it, also keeping future maintenance in mind.
LimeSurvey has a fancy theme editor that allows editing the CSS from a web interface:
![image](/uploads/d60aee1163b46772c6477e8d78e5104d/image.png)
This should be enough for future tweaks.
But I think that the first deployment needs a full clone of the Git repo (or copy of its files) because I'm also uploading fonts, adding more CSS files (`theme_tor.css`), and editing the config.yml, which is not available from the theme editor.
Also, right now my LimeSurvey account doesn't seem to have access to the theme editor. I don't really mind because I already have a prototyping setup to continue working on the theme, but then sysadmins would have to synchronize your LimeSurvey with my Git repo from time to time. This would also prevent breaking live surveys when doing experiments on the theme.
On the long term, maybe Tor could host this Git repo and automatically sync LimeSurvey in production whenever it's main branch is updated.
Cc: @donutsJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41473consider enforcing 2FA across gitlab2024-02-07T15:55:26Zanarcatconsider enforcing 2FA across gitlabIn #41470, we investigated the impact of an [authentication bypass in GitLab](https://about.gitlab.com/releases/2024/01/11/critical-security-release-gitlab-16-7-2-released/#account-takeover-via-password-reset-without-user-interactions) (...In #41470, we investigated the impact of an [authentication bypass in GitLab](https://about.gitlab.com/releases/2024/01/11/critical-security-release-gitlab-16-7-2-released/#account-takeover-via-password-reset-without-user-interactions) ([CVE-2023-7028](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-7028)). One of the key takeaways is that 2FA renders the attack mostly moot. This makes this group (tpo/tpa) immune to it, but not all users benefit from this.
We should consider enforcing 2FA more broadly here. One likely first target would be tpo/web, which has only a handful of users without 2FA (one of which was @gitolite-merge-bot, which was removed access in #41469). But we could broaden this to all of tpo.
Thoughts, @gaba, @lavamind ?Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2024-02-05https://gitlab.torproject.org/tpo/tpa/team/-/issues/41456monitor technical debt and legacy2024-03-27T15:29:42Zanarcatmonitor technical debt and legacyI often say that we have a huge technical debt in TPA, and that we keep needing to close things down and document and so on.
But we do not have hard data on this. After reading [Managing Technical Debt](https://jacobian.org/2023/dec/20/...I often say that we have a huge technical debt in TPA, and that we keep needing to close things down and document and so on.
But we do not have hard data on this. After reading [Managing Technical Debt](https://jacobian.org/2023/dec/20/tech-debt/), I realized we should at least keep track of metrics about this. What's interesting about that article is it says we shouldn't necessarily set *targets*, but keeping track of metrics would be a good start.
He specifically [suggests DORA metrics](https://jacobian.org/2022/jun/17/dora-metrics/), but I'm not sure it's the best match for us. Here's what I think we should monitor:
* tickets
* "lead time" (time between when a ticket enters backlog/next/doing and closing)
* start using the ~"Technical Debt" ticket and measure ticket counts
* general per-queue ticket counts (already done in monthly reports, put in prometheus, see https://gitlab.torproject.org/tpo/tpa/team/-/issues/40591)
* incidents:
* "lead time" is specially important here: how long do tickets get opened in incidents? might also be a measure of MTTR (mean time to recovery)
* "change failure rate": measure how many incidents are caused by deployment failures
* documentation: systematically measure how many services we have and how well they are documented (this is partially done, by hand, in the `service.md` wiki page, but could be somehow automated)
* untracked package counts: use anarcat's [puppet-package-check](https://gitlab.com/anarcat/scripts/-/blob/main/puppet-package-check?ref_type=heads) to generate metrics on how many packages are *not* managed by puppet, per host, as a rough estimate of the "puppetization ratio"
* unit test coverage: across all our software projects (or maybe per project?), what is the coverage of unit tests? (requires CI and extraction of those numbers in an exporter)
* out of date systems: how long does it take to update the fleet, and how long do we live on LTS? (at least partly tracked in Prometheus now, but not retained long enough to have good metrics, see also #40330)
The end result here is a small set of metrics that describe the current state of affairs, and its evolution over time. It will allow us to more easily realize when we're in trouble (e.g. https://gitlab.torproject.org/tpo/tpa/team/-/issues/41411) and evaluate how much effort we should put into this.
It might be more effective to have those metrics beyond the "one year" mark. Ticket counts, for example, are kept forever in the minutes, and that's a good thing, so we should consider expanding the storage retention here (#40330).
One thing Kaplan-Moss advises is to set time apart to deal with technical debt, he advises 10%. He also says we shouldn't set "sprints" to deal with technical debt, but I disagree with that: I have found that Debian upgrades are working well with sprints and wonder to what else we could extend the practice. On the other hand, the docs hack week wasn't a clear success for us, so maybe he's at least partly right in some aspects.cleanup and publish the sysadmin codebaseanarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41420Pipeline housekeeping for tpo/core/debian/tor project2023-12-12T15:29:52ZJérôme Charaouilavamind@torproject.orgPipeline housekeeping for tpo/core/debian/tor projectAs a follow-up for https://gitlab.torproject.org/tpo/tpa/team/-/issues/41402#note_2970105 I would like to propose that daily-scheduled CI pipelines that build supported branches in `tpo/core/debian/tor` be automatically deleted after 2 w...As a follow-up for https://gitlab.torproject.org/tpo/tpa/team/-/issues/41402#note_2970105 I would like to propose that daily-scheduled CI pipelines that build supported branches in `tpo/core/debian/tor` be automatically deleted after 2 weeks.
There are currently three such schedules, and they each run daily, adding some ~50MB of logs. When CI artifacts cleanup is broken, they also each add ~250MB of artifacts.
Implementing this automatic deletion would:
- Significantly reduce the storage requirements for job logs (currently 13.80GB of job logs for these pipelines specifically)
- Help limit the impact of any GitLab regresssion causing CI build artifacts to accumulate
Artifacts and build logs for [CI pipelines that run on *tagged* commits](https://gitlab.torproject.org/tpo/core/debian/tor/-/pipelines?scope=tags) would not be affected, and still be kept indefinitely.
This can be implemented very easily using either a cron job or service/timer pair, to run the following shell command:
gitlab-rails runner "Ci::Pipeline.where(project_id: 1218, source: 'schedule', locked: 'unlocked').where('finished_at < ?', 2.weeks.ago).delete
Unless further discussion is needed, I will implement this on Monday, December 11.
/cc @anarcat @weaselJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2023-12-12