TPA issueshttps://gitlab.torproject.org/groups/tpo/tpa/-/issues2021-04-26T14:01:05Zhttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/75change the default Git branch to "main" in GitLab2021-04-26T14:01:05Zanarcatchange the default Git branch to "main" in GitLabWe currently do not have any special configuration in GitLab regarding the branch name, which, in Git, defaults to `master`. Following recent events there has been a far and wide ranging conversation about changing that default to someth...We currently do not have any special configuration in GitLab regarding the branch name, which, in Git, defaults to `master`. Following recent events there has been a far and wide ranging conversation about changing that default to something else. Everywhere people are trying to get rid of that wording:
* GitHub is [changing to `main` on October 1st 2020](https://github.blog/changelog/2020-08-26-set-the-default-branch-for-newly-created-repositories/).
* Puppet has been [progressively removing mention of "master"](https://puppet.com/blog/removing-harmful-terminology-from-our-products/) and "blacklist"
* Git has [introduced a setting](https://lore.kernel.org/git/CAP8UFD1KfEps4hS8eadBK-E4e5WyWSh93XivRabZAVhiCuQimQ@mail.gmail.com/) to make the default changeable
* there's a proposal to [do the change in Wordpress as well](https://wptavern.com/proposal-to-rename-the-master-branch-from-wordpress-owned-git-repositories)
.. and that list is not exhaustive.
We should follow suit. There's already a [proposal to make the change in tor/core](https://gitlab.torproject.org/tpo/core/team/-/issues/2) but we should also change the default in GitLab as a whole. That's done in this admin section:
https://gitlab.torproject.org/admin/application_settings/repository
That affects only new repositories, for existing repositories, there are [various recipes](https://dev.to/damcosset/replacing-master-in-git-2jim) to make the change, but it's less trivial. Also note that this [does not affect wiki branch names](https://gitlab.com/gitlab-org/gitlab/-/issues/221159).
For now I propose we only change the default. GitHub has been working on [procedures to rename branches](https://github.com/github/renaming) which could prove interesting.
Update: @anarcat wrote this program to rename branches locally and on GitHub in batch: https://gitlab.com/anarcat/scripts/-/blob/main/git-branch-rename-remoteanarcatanarcat2020-10-24https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/11Announce move to Gitlab to tor-project@ and blog2020-11-20T16:55:59ZAlexander Færøyahf@torproject.orgAnnounce move to Gitlab to tor-project@ and blogWe need to announce to the community as well that we have moved.
* [x] announcement in tor-project@
* [ ] send mail to tor-project@ about trac shutting down
* [ ] blogpost
Blogpost should have:
- [ ] why and how we migrated
- [ ] how...We need to announce to the community as well that we have moved.
* [x] announcement in tor-project@
* [ ] send mail to tor-project@ about trac shutting down
* [ ] blogpost
Blogpost should have:
- [ ] why and how we migrated
- [ ] how to have a gitlab account and how to contribute
@gaba and I can work on this.Gabagaba@torproject.orgGabagaba@torproject.org2020-11-23https://gitlab.torproject.org/tpo/tpa/team/-/issues/34373retire trac and redirect trac.torproject.org to gitlab.torproject.org on dec ...2020-12-29T21:26:45Zanarcatretire trac and redirect trac.torproject.org to gitlab.torproject.org on dec 9th 2020Once trac is shutdown, redirections from trac.torproject.org needs to be perform to gitlab.torproject.org.
This happens 6 months after trac is put readonly (which is on june 12th) so around december 9th 2020.Once trac is shutdown, redirections from trac.torproject.org needs to be perform to gitlab.torproject.org.
This happens 6 months after trac is put readonly (which is on june 12th) so around december 9th 2020.anarcatanarcat2020-12-09https://gitlab.torproject.org/tpo/tpa/team/-/issues/40105establish 2021 roadmap2021-06-16T20:17:57Zanarcatestablish 2021 roadmapWe should make a roadmap for 2021. This could be either milestones in GitLab, or a roadmap page in the wiki, something like:
https://trac.torproject.org/projects/tor/wiki/org/teams/SysadminTeam
or the original, non-trac roadmap from th...We should make a roadmap for 2021. This could be either milestones in GitLab, or a roadmap page in the wiki, something like:
https://trac.torproject.org/projects/tor/wiki/org/teams/SysadminTeam
or the original, non-trac roadmap from the wiki:
https://gitlab.torproject.org/tpo/tpa/wiki-replica/-/blob/e5e84236b5072cb28f630b1ab62bcc32f8163903/tsa/roadmap/2020.mdwn
It's unclear to me how we'll make a roadmap for next year. Month-per-month long-term planning did not seem to work, or at least we definitely stopped tracking that after summer (although that could be due to the GitLab migration). I still think it would be wise to setup larger goals in the roadmap and not get tangled up in smaller details.
We should use the survey data (#40106) to prioritize the work.
I hope to complete this in January, with the rest of the team of course (but I volunteer to bottomline this).anarcatanarcat2021-01-15https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/95Occasionally clean-up Gitlab CI storage2021-12-06T21:38:25ZAlexander Færøyahf@torproject.orgOccasionally clean-up Gitlab CI storageToday it looks like we ran out of disk space on the CI runner. See this tpo/core/arti build from @asn: https://gitlab.torproject.org/asn/arti/-/jobs/11272
I think the one-shot solution is to run `docker system prune -a`, but this only s...Today it looks like we ran out of disk space on the CI runner. See this tpo/core/arti build from @asn: https://gitlab.torproject.org/asn/arti/-/jobs/11272
I think the one-shot solution is to run `docker system prune -a`, but this only solve the problem here and now, and not next time we run into it.
Since Gitlab CI themselves seems to use docker+machine, I think that is less of an issue for them, but others are reporting similar issues and many possible solutions seems suggested in their ticket trackers.
There is a cron job script here: https://gitlab.com/snippets/1729438 which looks at modified time of the images instead of just wiping everything.
Some better solutions seems suggested in here: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2310 and it sounds like this is being merged for the next release.Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2021-03-15https://gitlab.torproject.org/tpo/tpa/team/-/issues/40198move tb-build-02 to gnt-chi cluster2021-04-19T17:56:32Zanarcatmove tb-build-02 to gnt-chi clusterthe tb-builder regularly trigger load warnings in the main ganeti cluster (gnt-fsn, see #40100). create a new tb-build-xx machine into the alternate cluster (gnt-chi) to free up resources in the main cluster. this also involves shutting ...the tb-builder regularly trigger load warnings in the main ganeti cluster (gnt-fsn, see #40100). create a new tb-build-xx machine into the alternate cluster (gnt-chi) to free up resources in the main cluster. this also involves shutting down one of the existing tb-build machines, obviously.
in any case, I'd like to hear more from the TB team how they think we should handle this problem. what would be involved would be basically a hostname and IP address change, as we'd rebuild the box from scratch in a new cluster.
/cc @gk @boklm @sysrqb
comments? suggestions?
the cluster's disks may be slower: they are backed by an iSCSI SAN with SAS disks (as opposed to local, possibly more modern SATA disks), but there's plenty of RAM and decent processors. the [hardware inventory](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/new-machine-cymru#hardware-inventory) is available for those of you who are curious.
the boxes were created by @hiro in #34122, and are hopefully all in puppet, `profile::torbrowser_build`.
update: move was approved, tb-build-03 was created, tb-build-02 retirement checklist:
1. [x] announced the change (here, last week, and before)
1. [x] remove host from nagios
1. [x] stop VM
1. [x] after delay, retire host from parent, backups, and puppet
1. [x] remove from LDAP
1. [x] power-grep to remove the IP
1. [x] remove from tor-passwords
1. [x] remove from DNSwl (N/A)
1. [x] remove from wiki, inventory (N/A)
1. [x] N/A
1. [x] remove from reverse DNSanarcatanarcat2021-04-14https://gitlab.torproject.org/tpo/tpa/team/-/issues/40167TPA-RFC-10: adopt jenkins retirement plan2022-03-28T19:15:32ZanarcatTPA-RFC-10: adopt jenkins retirement planI've been brainstorming here and there about how we could eventually replace Jenkins. Now that people are getting excited about GitLab CI and that service is taking up more and more of our time, it seems like a good time to start plannin...I've been brainstorming here and there about how we could eventually replace Jenkins. Now that people are getting excited about GitLab CI and that service is taking up more and more of our time, it seems like a good time to start planning for Jenkins retirement.
Requires jenkins service docs first (#40166).
update: followup of this proposal is in https://gitlab.torproject.org/groups/tpo/-/milestones/27Retire Jenkinsanarcatanarcat2021-04-15https://gitlab.torproject.org/tpo/tpa/team/-/issues/34123Provide secrets/passwords management for Tor Browser Nightly signing2022-03-03T21:46:59ZMatthew FinkelProvide secrets/passwords management for Tor Browser Nightly signingAs mentioned in legacy/trac#34121, the Tor Browser Nightly signing machine will host an OpenPGP key and an NSSDB private key. Both of these should be password-protected. Instead of hard-coding these passphrases in a file or script on the...As mentioned in legacy/trac#34121, the Tor Browser Nightly signing machine will host an OpenPGP key and an NSSDB private key. Both of these should be password-protected. Instead of hard-coding these passphrases in a file or script on the server, having a password management system from where the passwords can be retrieved would be very nice.anarcatanarcat2021-04-20https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81evaluate mitigation strategies to work around GitLab's attack surface for git...2022-06-16T16:10:24Zanarcatevaluate mitigation strategies to work around GitLab's attack surface for git hostingas part of the possible gitolite to GitLab migration work (discussed in #36), one of the issues that came up is the relative security of gitolite vs GitLab. to quote @nickm:
> I am concerned about the security here. I know that the git...as part of the possible gitolite to GitLab migration work (discussed in #36), one of the issues that came up is the relative security of gitolite vs GitLab. to quote @nickm:
> I am concerned about the security here. I know that the gitolite installation is old and crufty, but the gitlab code-base is HUGE. For gitolite, as I understand it, you have to be able to make an ssh connection to the server before you can change anything in a repository. But in gitlab, it seems that the gitlab server has write access to all the repositories. That makes a much much bigger attack surface, unless I grievously misunderstand the architectures here.
My response to this was:
> Of course, GitLab is larger, and if there's an *unauthenticated* attack against GitLab, that could compromise our respositories. And there has been [vulnerabilities in GitLab](https://www.cvedetails.com/vulnerability-list/vendor_id-13074/Gitlab.html) before, for sure. But looking at that list [sorted by priority](https://www.cvedetails.com/vulnerability-list.php?vendor_id=13074&product_id=&version_id=&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&month=0&cweid=0&order=3&trc=178&sha=fe38ca18c40b857201e9ae9283dea03b71b724f0), i cannot find an unauthenticated vulnerability after reviewing the top ten. And I do not remember such an issue being ever disclosed although it's theoretically possible...
>
> So my concern with GitLab is not that unauthenticated users might hack at it, but more what *authenticated*, *trusted* users could do. And there, of course, there is a plethora of privilege escalation vulnerabilities that have been disclosed (and fixed!). And that's where, when you compare to gitolite, there is a problem: **our current gitolite install is vulnerable** to [4 such issues at least](https://www.cvedetails.com/vulnerability-list/vendor_id-19332/product_id-50401/Gitolite-Gitolite.html), and [possibly more](https://security-tracker.debian.org/tracker/source-package/gitolite3), partly because it's so old, but also partly because gitolite is less well-maintained.
>
> We are actively maintaining gitlab, following upstream releases quite closely. And while I'm not super happy about the attack surface (nevermind the complexity of managing the thing underneath!), everything (and especially security) is made of tradeoffs. And in this case, my opinion is that we should go towards GitLab for code hosting, because it will **facilitate code review** workflows, which is a large part of securing a product...
Or to put it more succintly:
> If we are worried about trust in our supply chain, we need to look elsewhere, because gitolite cannot be trusted in its current state. I would encourage all teams to start **signing their commits** and **verifying cryptographic history when doing releases**. Having **reproducible builds** is also part of the equation, of course.
This issue is a space to discuss those possible solutions. It applies mostly to GitLab but I would argue it also applies to the trust we put in gitolite as well (which I consider to be misplaced).anarcatanarcat2021-04-21https://gitlab.torproject.org/tpo/tpa/team/-/issues/30009replace hkdf with trocla for secrets management in puppet2021-06-14T13:59:05Zanarcatreplace hkdf with trocla for secrets management in puppetsecrets generated by puppet currently use a custom hkdf function that is homegrown. the ad-hoc standard for this in the puppet community i'm usually working with is [trocla](https://github.com/duritong/trocla) which is [well integrated w...secrets generated by puppet currently use a custom hkdf function that is homegrown. the ad-hoc standard for this in the puppet community i'm usually working with is [trocla](https://github.com/duritong/trocla) which is [well integrated with puppet](https://github.com/duritong/puppet-trocla).
Trocla generates, on the fly, a strong random password for each key you ask it. It also supports various hashing mechanisms (bcrypt, pgsql, x509, etc) so that the Puppet client never actually sees the cleartext. It seems like a better approach than sending the cleartext like we currently do.
So I'd like to start using this for new code and possibly convert existing code to this, if that's acceptable.
next steps:
- [x] test trocla
- [x] replace `hkdf()` calls with trocla
- [x] remove `hkdf()` source code
- [x] remove the puppet secret used by `hkdf()` (`/etc/puppet/secret`)anarcatanarcat2021-06-12https://gitlab.torproject.org/tpo/tpa/team/-/issues/32824Upgrade tpo onions to v3.2021-04-20T20:54:20ZcypherpunksUpgrade tpo onions to v3.https://onion.torproject.org/
Please upgrade to v3 and notify us by writing a blog. Thankshttps://onion.torproject.org/
Please upgrade to v3 and notify us by writing a blog. Thanksweasel (Peter Palfrader)weasel (Peter Palfrader)2021-06-15https://gitlab.torproject.org/tpo/tpa/team/-/issues/40287provision new OSUOSL ARM servers2021-06-17T20:21:21Zanarcatprovision new OSUOSL ARM serverswe have 32GB of RAM and 14 CPUs of ARM virtual machines to provision in their openstack cluster (thanks!!!). this should probably be used for gitlab runners to replace the retired jenkins / build-arm-10 builder (tpo/tpa/team#40284).
pre...we have 32GB of RAM and 14 CPUs of ARM virtual machines to provision in their openstack cluster (thanks!!!). this should probably be used for gitlab runners to replace the retired jenkins / build-arm-10 builder (tpo/tpa/team#40284).
pre-req checklist:
1. [x] partitions: setup /swapfile
2. [x] upgrades
3. [x] hostname fixed (had `.novalocal` suffix)
4. [ ] reverse DNS
5. [x] root password
6. [x] DNS works
main procedure:
1. [x] nextcloud spreadsheet
2. [x] tsa-misc
3. [x] puppet bootstrap
4. [x] nagios
6. [x] DNSWL (n/a)
5. [x] rebootanarcatanarcat2021-06-30https://gitlab.torproject.org/tpo/tpa/team/-/issues/40284retire build-arm-102021-06-29T18:29:05Zanarcatretire build-arm-10Linaro is shutting down their "Developer cloud", which means we'll lose access to build-arm-10. Retire the box before that:
> Hi Linaro Developer Cloud Users,
>
> Thanks for your interest and work on Linaro Developer Cloud during the l...Linaro is shutting down their "Developer cloud", which means we'll lose access to build-arm-10. Retire the box before that:
> Hi Linaro Developer Cloud Users,
>
> Thanks for your interest and work on Linaro Developer Cloud during the last
> several years. Really appreciated your open source
> development/test/validation work which has been contributed a lot to Arm64
> ecosystem.
>
> I'm writing to tell you that *we plan to close Linaro Developer
> Cloud(uk.linaro.cloud) external access *due to the internal change*. We are
> really sorry to make this hard decision, and can not continue offering
> resources after Jun 30th.*
>
> After Jun 30th, all the VM network access will be lost, and the data will
> not be kept. Please backup your data locally ASAP.
>
> If you have special requests to keep your resources, or if you have some CI
> jobs running in Linaro Developer Cloud, please let me know, we will try to
> help with migration to other platforms.
>
> Again, thanks a lot for your help during the last several years.
Note that this might impact tpo/core/tor#40347
automated retirement procedure:
1. [x] announcement: N/A, was already unavailable upstream
2. [x] nagios
3. [x] N/A: VM already stopped
4. [x] fabric retirement
5. [x] LDAP
6. [x] grep + DNS
7. [x] pwmanager
8. [x] DNSWL: N/A
9. [x] wiki/spreadsheet
10. [x] upstream
11. [x] reverse DNS: N/Aanarcatanarcat2021-06-30https://gitlab.torproject.org/tpo/tpa/team/-/issues/40294Prepare for Web Hosting on arti.torproject.org2023-08-28T15:41:49ZAlexander Færøyahf@torproject.orgPrepare for Web Hosting on arti.torproject.orgHello!
As part of the big transition towards Arti, we are going to create a place where current and hopefully upcoming contributors can find news and information about the Arti development project efforts. The network team and the comms...Hello!
As part of the big transition towards Arti, we are going to create a place where current and hopefully upcoming contributors can find news and information about the Arti development project efforts. The network team and the comms team would like to have a place where we can host entirely static web content about the Arti project.
I know we cannot do Gitlab CI pages with custom domains yet, so we may have to live with a slightly more complicated sync setup at first, but *ideally* at some point it would be awesome if it can be run via Gitlab CI and Gitlab Pages :-)
Is this something we can do? We don't have the page yet, this ticket is just to prepare for it and start the discussion.2021-07-01https://gitlab.torproject.org/tpo/tpa/team/-/issues/40337Email lists for Board committees2021-07-19T16:09:35ZIsabela FernandesEmail lists for Board committeesHello there, we have created different Board committees and would like to have an email lists for these committees to coordinate their work. Each list should have subscription moderated, all members can email the list, should have privat...Hello there, we have created different Board committees and would like to have an email lists for these committees to coordinate their work. Each list should have subscription moderated, all members can email the list, should have private archives too.
List: board-finance@lists.torproject.org
Admin: isabela@torproject.org, me@chelseakomlo.com
List: board-marketing@lists.torproject.org
Admin: isabela@torproject.org, me@chelseakomlo.com
List: board-legal@lists.torproject.org
Admin: isabela@torproject.org, me@chelseakomlo.com
List: board-executive@lists.torproject.org
Admin: isabela@torproject.org, me@chelseakomlo.comJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2021-07-15https://gitlab.torproject.org/tpo/tpa/team/-/issues/40309Install SSL cert on the donate.tpo onion2022-09-19T20:14:15ZdonutsInstall SSL cert on the donate.tpo onionWe're currently in the process of redesigning donate.torproject.org, with an aim to launch the new donate portal at the beginning of August.
As part of that work, we'd like to explore the possibility of enabling credit card payments on ...We're currently in the process of redesigning donate.torproject.org, with an aim to launch the new donate portal at the beginning of August.
As part of that work, we'd like to explore the possibility of enabling credit card payments on donate.tpo's onion site: http://yoaenchicimox2qdc47p36zm3cuclq7s7qxx6kvxqaxjodigfifljqqd.onion
At the moment the donation hangs on submission with the following console errors: https://gitlab.torproject.org/tpo/web/donate-static/-/issues/36
I'm going to do some digging to see if it's Stripe that's rejecting it, and for what reason – but the cert would be nice to have anyway, if possible. Thanks!Jérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.org2021-07-30https://gitlab.torproject.org/tpo/tpa/team/-/issues/40308use milestones to organise large projects from the roadmap2021-09-14T18:09:15Zanarcatuse milestones to organise large projects from the roadmapthe roadmap is quite verbose. we should consider using roadmaps to organise work, either for 2021 or (if we can't find the time), for 2022
i think those projects are particularly relevant for the roadmap thing:
- [ ] email delivery imp...the roadmap is quite verbose. we should consider using roadmaps to organise work, either for 2021 or (if we can't find the time), for 2022
i think those projects are particularly relevant for the roadmap thing:
- [ ] email delivery improvements
- [x] service retirements [jenkins](http://eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion/groups/tpo/-/milestones/27)
- [x] [blog migration](http://eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion/groups/tpo/-/milestones/26)
- [ ] improve communications and monitoring
- [ ] improve sysadmin code base
- [ ] bullseye upgrade (issues to be created)
- [ ] the *next* roadmap setup (#40307 and tickets not yet created)2021-09-07https://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/issues/9delete kez-bot user in nextcloud after test period2021-09-10T13:43:54Zanarcatdelete kez-bot user in nextcloud after test periodonce @kez is done with the `kez-bot` user, delete it from nextcloud.once @kez is done with the `kez-bot` user, delete it from nextcloud.anarcatanarcat2021-09-10https://gitlab.torproject.org/tpo/tpa/team/-/issues/19914DMARC causing trouble with Tor lists, From should be munged2022-12-10T03:21:37ZstarlightDMARC causing trouble with Tor lists, From should be mungedTor Project list server does not remove DKIM headers and reliably modifies the Subject: header in such a manner as to produce DKIM validation failures. Beyond adding a prefix which can be anticipated by conscientious senders, spaces are...Tor Project list server does not remove DKIM headers and reliably modifies the Subject: header in such a manner as to produce DKIM validation failures. Beyond adding a prefix which can be anticipated by conscientious senders, spaces are injected at unpredictable offsets. This badly degrades spam-scoring of messages by Google and other ESPs and frequently results in the sending of list messages to spam folders.
My recommendation is that the list server configuration be set to completely strip DKIM headers from forwarded messages.
Possibly if the list server software supports it, a DKIM-signed RFC-7001 Authentication-Results: header might be added though it seems to me the positive effect on spam scoring would be minor, where in comparison not stripping DKIM headers results in a substantial negative impact.anarcatanarcat2021-09-14https://gitlab.torproject.org/tpo/tpa/team/-/issues/40382TPA-RFC-12: document the triage process, office hours, and more2022-03-28T19:15:33ZanarcatTPA-RFC-12: document the triage process, office hours, and morewe noticed in the last meeting that the triage work is not clearly documented. it would help to have a checklist of things to go through and how we pass that work around.
while we're here, shake up the support policy a bit, see wiki-rep...we noticed in the last meeting that the triage work is not clearly documented. it would help to have a checklist of things to go through and how we pass that work around.
while we're here, shake up the support policy a bit, see wiki-replica!18.anarcatanarcat2021-09-21