TPA issueshttps://gitlab.torproject.org/groups/tpo/tpa/-/issues2023-09-26T20:43:33Zhttps://gitlab.torproject.org/tpo/tpa/team/-/issues/40723port puppet-managed configs to Debian bullseye2023-09-26T20:43:33Zanarcatport puppet-managed configs to Debian bullseyeas part of the %"Debian 11 bullseye upgrade", we have a bunch of (sometimes really old) configuration that needs to be ported to the new stuff that's in bullseye. I have identified, so far:
* [ ] `/etc/apt/apt.conf.d/50unattended-upgra...as part of the %"Debian 11 bullseye upgrade", we have a bunch of (sometimes really old) configuration that needs to be ported to the new stuff that's in bullseye. I have identified, so far:
* [ ] `/etc/apt/apt.conf.d/50unattended-upgrades`: to be investigated. probably mostly whitespace changes, but also possibly missing features. complicated by the fact that this is a third party Puppet module and would require significant work to catchup with the Debian package
* [ ] `/etc/unbound/unbound.conf`: switch to `include-toplevel` after the fleet is upgraded (does not work in buster)
* [x] `/etc/sudoers`: use `@include` instead of `#include`, former added only in bullseye and later. should be split out in a `sudoers.d` file to avoid future conflicts and, generally, split in snippets per service instead of this monolithic file
* [ ] `/etc/syslog-ng/syslog-ng.conf`: silly version number logic in the template, needs to be ported to newer config or replaced with rsyslog or journald
* [x] ~~`/etc/ferm/ferm.conf`: `web-cymru-01` had diffs pending from the previous upgrade (presumably?), might be worth catching up to *buster*, that is, unless we just ditch ferm completely (#40554)~~ the latter
* [ ] `/etc/lvm/lvm.conf`: same as above
* [x] ~~`/etc/bind/named.conf.options`: TBD, on fallax~~ fallax retired
if a file is added in the above list, do not forget to add it to the [conflicts resolution list](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bullseye#conflicts-resolution) in the upgrade procedure.
more such issues could come up, but for now that's what I got. for now the diff for those has been minimized as much as possible and the proposed version from the Debian package should generally be ignored.Debian 11 bullseye upgradehttps://gitlab.torproject.org/tpo/tpa/team/-/issues/40695upgrade or rebuild hetzner-hel1-01 (nagios/icinga)2023-11-21T22:45:00Zanarcatupgrade or rebuild hetzner-hel1-01 (nagios/icinga)Nagios is going to be a particularly tricky bullseye upgrade, so it's not part of the large bullseye upgrade batches (#40690 or #40692).
We need to decide whether we keep icinga around at all or replace it with Prometheus (https://gitla...Nagios is going to be a particularly tricky bullseye upgrade, so it's not part of the large bullseye upgrade batches (#40690 or #40692).
We need to decide whether we keep icinga around at all or replace it with Prometheus (https://gitlab.torproject.org/tpo/tpa/team/-/issues/29864). if we do keep icinga, we need to decide whether we keep the current "push to git to rebuild the config" model or "puppetize the setup" (https://gitlab.torproject.org/tpo/tpa/team/-/issues/32901).Debian 11 bullseye upgradehttps://gitlab.torproject.org/tpo/tpa/team/-/issues/40694upgrade eugeni to bullseye2023-11-15T21:33:42Zanarcatupgrade eugeni to bullseyeeugeni is going to be a tricky bullseye upgrade, so it's not part of the large bullseye upgrade batches (#40690 or #40692).
we might want to decide what to do with mailman (https://gitlab.torproject.org/tpo/tpa/team/-/issues/40471) and...eugeni is going to be a tricky bullseye upgrade, so it's not part of the large bullseye upgrade batches (#40690 or #40692).
we might want to decide what to do with mailman (https://gitlab.torproject.org/tpo/tpa/team/-/issues/40471) and schleuder (https://gitlab.torproject.org/tpo/tpa/team/-/issues/40564) *before* we do the upgrade. mailman 2, in particular, is EOL so we *will* need to upgrade or replace it.
we might also want to consider the impact of the %"improve mail services" roadmap here. it's possible we might want to completely rebuild eugeni in different components instead of upgrading it.Debian 11 bullseye upgradehttps://gitlab.torproject.org/tpo/tpa/team/-/issues/40568Better monitoring for webserver response times2022-07-26T17:42:40ZJérôme Charaouilavamind@torproject.orgBetter monitoring for webserver response timesIn the wake of tpo/tpa/team#40566, it was shown that our monitoring infrastructure isn't sufficiently sensitive with respect to web server response times. We had an ongoing DoS on the static mirror hosts for days and we only noticed when...In the wake of tpo/tpa/team#40566, it was shown that our monitoring infrastructure isn't sufficiently sensitive with respect to web server response times. We had an ongoing DoS on the static mirror hosts for days and we only noticed when the response times consistently surpassed 10 seconds.
We should probably modify the existing checks or add new ones that will monitor whether the static mirror host (or even any web host) is serving pages within an acceptable delay, say 1 second.Debian 11 bullseye upgradeJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/40471upgrade mailman to mailman 32024-02-20T17:06:19Zanarcatupgrade mailman to mailman 3Mailman 2 was removed from Debian bullseye, we need to either upgrade to Mailman 3 or get rid of it. This is part of the 2022-Q1/Q2 OKRs and the %"Debian 11 bullseye upgrade" milestone.
upgrade procedure: https://docs.mailman3.org/en/l...Mailman 2 was removed from Debian bullseye, we need to either upgrade to Mailman 3 or get rid of it. This is part of the 2022-Q1/Q2 OKRs and the %"Debian 11 bullseye upgrade" milestone.
upgrade procedure: https://docs.mailman3.org/en/latest/migration.htmlDebian 11 bullseye upgradehttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32901automate/puppetize Nagios installs2024-01-19T18:48:16Zanarcatautomate/puppetize Nagios installsone part of our install process is to configure Nagios, by hand, in the git repository. I usually do this by copy-pasting some similar blob of config from a possibly similar machine and hope for the best.
this is a manual step, and as p...one part of our install process is to configure Nagios, by hand, in the git repository. I usually do this by copy-pasting some similar blob of config from a possibly similar machine and hope for the best.
this is a manual step, and as part of the automation of the install process, it should be made automatic.
one way this could (and probably should) be done is by making Puppet automatically add its nodes into Nagios. this can be done using the [icinga2 module](https://github.com/Icinga/puppet-icinga2), for example. care should be taken to do a smooth transition, keeping existing configurations and just adding the Puppet ones on top, for new machines.
but this could (eventually) be retroactively added to all nodes, removing all manual configuration.
checklist:
1. [x] audit and import the module in our monorepo
1. [x] ~~enable on the nagios server, without writing any config (hopefully a noop)~~ not possible, config is overwritten by module, instead...
1. [ ] move the base configuration (`config/static`) from git into Puppet (mostly icinga.cfg and so on, because they are overwritten by the module)
1. [ ] enable a single config from puppet, as a test
1. [ ] add a new host check configuration
1. [ ] add a new service check configuration
1. [ ] add all *base* service checks for the new host (e.g. the services defined for the `computers` hostgroup, equivalent of pieces of `from-git/generated/auto-services.cfg`)
1. ~~[ ] convert legacy config into puppet (at this stage we only have the old hosts as legacy config)~~ done in third step
1. [ ] convert NRPE service definitions (`puppet:///modules/nagios/tor-nagios/generated/nrpe_tor.cfg`, generated from the git repo)
1. [ ] remove NRPE config sync from nagios to Puppet (the rsync to `pauli` in `config/Makefile`)
1. [ ] convert old hosts checks into puppet
1. [ ] convert old services checks into puppet
1. [ ] remove git hook receiver on nagios server (`/etc/ssh/userkeys/nagiosadm` key, which calls `/home/nagiosadm/bin/from-git-rw`)
It's a long way there, but getting to the state where *new* hosts are covered would already be a great improvement.Debian 11 bullseye upgradehttps://gitlab.torproject.org/tpo/tpa/team/-/issues/40755TPA-RFC-33: monitoring system upgrade or replacement2024-03-19T20:17:23ZanarcatTPA-RFC-33: monitoring system upgrade or replacementin #29864, we've gone pretty deep in comparisons between prometheus and icinga and how the first could replace the latter.
but now we're stuck at "i like this one better than the other" because we don't have a clear set of requirements....in #29864, we've gone pretty deep in comparisons between prometheus and icinga and how the first could replace the latter.
but now we're stuck at "i like this one better than the other" because we don't have a clear set of requirements.
the task here is to write a set of requirements for the new alerting system and, ultimately, make a proposal for the replacement of the deprecated Icinga 1 deployment we have now.
* [ ] establish requirements
* [ ] approve requirements
* if replacing icinga:
* [ ] review #29864 for ideas and tasks
* [ ] decide whether we keep the prometheus1/2 distinction
* [ ] deploy alert manager on prometheus1
* [ ] reimplement the Nagios alerting commands (optional?)
* [ ] send Nagios alerts through the alertmanager (optional?)
* [ ] rewrite (non-NRPE) commands (9) as Prometheus alerts
* [ ] scrape the NRPE metrics from Prometheus (optional)
* [ ] create a dashboard and/or alerts for the NRPE metrics (optional)
* [ ] review the NRPE commands (300+) to see which one to rewrite as Prometheus alerts
* [ ] turn off the Icinga server
* [ ] remove all traces of NRPE on all nodes
* if keeping icinga
* [ ] review work from @weasel done on DSA's Puppet/Icinga integration
* [ ] deploy that module or another inciga module inside puppet
* [ ] rewrite all the checks from the `nagios-master.cfg` file into puppet (300+)
* [ ] rebuild a new Icinga 2 server
* [ ] retire the old Icinga 1 serverold service retirement 2023anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32273archive private information from SVN2022-12-06T19:50:49Zanarcatarchive private information from SVNa common problem in the internal and corp SVN repository shutdown is "what do we do with all that stuff now". for example, the internal repository is shutdown now (#15949) but there is still information there that is valuable. or not. we...a common problem in the internal and corp SVN repository shutdown is "what do we do with all that stuff now". for example, the internal repository is shutdown now (#15949) but there is still information there that is valuable. or not. we're not sure. we think so, but maybe some of it should be destroyed.
so we need to answer the following questions:
1. which data should be kept and destroyed from the repositories?
2. where should it be kept?
so far, I went under the assertion that the answers were:
1. keep everything
2. in nextcloud
but it seems this might not be exactly right.old service retirement 2023https://gitlab.torproject.org/tpo/tpa/team/-/issues/32025Stop using corpsvn and disable it as a service2022-12-06T19:50:54ZRoger DingledineStop using corpsvn and disable it as a serviceIn legacy/trac#17202 we're going to decommission the server that runs our various svn services.
We have a plan for the public svn.tpo service: legacy/trac#15948
and we are making a plan for svninternal: legacy/trac#15949
That leaves c...In legacy/trac#17202 we're going to decommission the server that runs our various svn services.
We have a plan for the public svn.tpo service: legacy/trac#15948
and we are making a plan for svninternal: legacy/trac#15949
That leaves corpsvn, which I think is the most actively used still -- for example our accounting folks use it. This ticket is about making and finishing the plan for shutting down the corpsvn service.old service retirement 2023https://gitlab.torproject.org/tpo/tpa/team/-/issues/17202Shut down SVN and decomission the host (gayi)2022-12-06T19:51:28ZNick MathewsonShut down SVN and decomission the host (gayi)It is now 2015. Let us not have an SVN server running in 2016.
-- And is now 2020 and we are finally trying to shutdown this. Modifying this ticket to add the plan suggested by arma (with a few modifications by me).
(1) Freeze corps...It is now 2015. Let us not have an SVN server running in 2016.
-- And is now 2020 and we are finally trying to shutdown this. Modifying this ticket to add the plan suggested by arma (with a few modifications by me).
(1) Freeze corpsvn (i.e. make it read-only), and make a full checkout
of it somewhere, and have that accessible.
(2) Use Nextcloud for any other file people may need to save. *Not* move all the old files there, or at least not by default.
(3) Put together a strike team to look at the frozen corpsvn checkout,
plus the frozen internalsvn checkout. Build a list of categories (HR,
finance, grantwriting, grant manager, etc), and sort the files into
these categories, discarding as many files as possible. Figure out
where else people are storing these files currently (granthub? google
docs? their hard drive?). Make a comprehensive plan for how files of each category should be stored, and who should have read or write access per category. For example, there's no reason that HR documents should go into the same database, or even the same storage service, as grant proposals. Process started in https://bugs.torproject.org/32273
Update, 2021: there's a "forest" of tickets surrounding this, as the "tree" was lost in the Trac migration, i'll try to reconstruct related tickets:
* SVN/host shutdown (this ticket)
* [ ] #32273 - archive private information from SVN: determine what moves to where (presumably: "everything, to nextcloud")
* [x] #15949 - shutdown SVN internal (done, but the repository is still on gayi, and not archived anywhere else)
* [ ] #32025 - stop using corpsvn and disable it (still open, blocked mostly on @sue iirc)
* [x] #33537 - audit SVN accesses (led to the [access control document](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/svn/#access-control) and a private audit email with one minor remaining task, `Message-ID: 871rq02rvt.fsf@curie.anarc.at`, can probably be just closed)
* [x] #15948 - public SVN retirement (done, moved to the static site mirror system (#32031) and archive.org)
* [x] #31686 - textile retirement (done)
* [ ] #40260 - actual proposal (next step, blocker)
It seems the next step here is to write a policy proposal to make sure we're all on the same page ("let's move to Nextcloud") and schedule a call with Sue to make sure it works in her workflow.old service retirement 2023anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/40260TPA-RFC-11: SVN retirement2024-02-20T16:08:17ZanarcatTPA-RFC-11: SVN retirementDraft a proposal to retire SVN altogether. This was somewhat agreed on in #17202, and there are discussions on how exactly to do this in #32273 (how to archive it) and #32025 (how stop corpsvn specifically, the remaining live repo), but ...Draft a proposal to retire SVN altogether. This was somewhat agreed on in #17202, and there are discussions on how exactly to do this in #32273 (how to archive it) and #32025 (how stop corpsvn specifically, the remaining live repo), but this was all done before we came up with the TPA-RFC process.
Now that we have that process, it seems logical to go through with it explicitly so that all stakeholders can express their concerns about the change. I specifically plan on having a live call with sue about this, since she's the most impacted by this.
I create this ticket to track that proposal because #17202 has been sitting in the icebox forever and it's kind of hard to "grab" because it feels so big. Having "write a proposal" first seems more accessible.
draft proposal is in https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-11-svn-retirement
Next steps:
- [x] @susan draft requirements for the sensitive file storage service
- [x] @anarcat research whether we can restrict external sharing in Nextcloud
- [x] @anarcat schedule a meeting in late june to revise the aboveold service retirement 2023anarcatanarcat2024-01-19https://gitlab.torproject.org/tpo/tpa/team/-/issues/41511upgrade crm-ext-01 to php 8 or retire2024-02-20T20:04:45Zanarcatupgrade crm-ext-01 to php 8 or retirein https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252#note_2990644, @kez tested crm-ext-01 after I upgraded it and found the donate site completely broken by the PHP 8.2 upgrade. apparently, `implode` completely changed signature ...in https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252#note_2990644, @kez tested crm-ext-01 after I upgraded it and found the donate site completely broken by the PHP 8.2 upgrade. apparently, `implode` completely changed signature in PHP and the old signature was dropped in PHP 8, which breaks a *lot* of things.
exactly how much is unclear, @kez estimated just the work to estimate that work to be a few hours of work.
for now i rolled back to the php 7.4 package from bullseye, and added it to the sources.list file (although puppet might have killed the .list file already). we need to figure out a plan to go forward, either port the code, or retire the box, which is the ultimate goal once donate-neo goes to production.Redesign donate.torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41219migrate TPA's gitolite repositories to GitLab2024-03-28T20:24:38Zanarcatmigrate TPA's gitolite repositories to GitLabWe have decided to retire Gitolite in #41180, give the good example and migrate our repos to GitLab. This is the table established in TPA-RFC-36:
| Repository | data | Problem ...We have decided to retire Gitolite in #41180, give the good example and migrate our repos to GitLab. This is the table established in TPA-RFC-36:
| Repository | data | Problem | Fate |
|-------------------------------|--------------------------------------|-------------------------------------|----------------------------------|
| `account-keyring` | OpenPGP keyrings | hooks into the static mirror system | convert to GitLab CI |
| `buildbot-conf` | old buildbot config? | obsolete | archive |
| `dip` | GitLab ansible playbooks? | duplicate of `services/gitlab/dip`? | archive? |
| `dns/auto-dns` | DNS zones source used by LDAP server | security | check OpenPGP signatures |
| `dns/dns-helpers` | DNSSEC generator used on DNS master | security | check OpenPGP signatures |
| `dns/domains` | DNS zones source used by LDAP server | security | check OpenPGP signatures |
| `dns/mini-nag` | monitoring on DNS primary | security | check OpenPGP signatures |
| `letsencrypt-domains` | TLS certificates generation | security | move to Puppet? |
| `puppet/puppet-ganeti` | puppet-ganeti fork | misplaced | destroy |
| `services/gettor` | ansible playbook for gettor | obsolete | archive |
| `services/gitlab/dip-configs` | GitLab ansible playbooks? | obsolete | archive |
| `services/gitlab/dip` | GitLab ansible playbooks? | duplicate of `dip`? | archive? |
| `services/gitlab/ldapsync` | LDAP to GitLab script, unused | obsolete | archive |
| `static-builds` | Jenkins static sites build scripts | obsolete | archive |
| `tor-jenkins` | Jenkins build scripts | obsolete | archive |
| `tor-nagios` | Icinga configuration | confidentiality? | abolish? see also [TPA-RFC-33][] |
| `tor-passwords` | password manager | confidentiality | migrate? |
| `tor-virt` | libvirt VM configuration | obsolete | destroy |
| `trac/TracAccountManager` | Trac tools | obsolete | archive |
| `trac/trac-email` | Trac tools | obsolete | archive |
| `tsa-misc` | miscellaneous scripts | none | migrate |
| `userdir-ldap-cgi` | fork of DSA's repository | none | migrate |
| `userdir-ldap` | fork of DSA's repository | none | migrate |
Update: we don't have the free cycles to do the right thing here and we're instead going to move to GitLab only the repositories that do not require special handling, that is: repositories that are `archive` or `migrate`. Everything else will be moved to special servers while we figure out what to do with that legacy stuff.
- [x] `account-keyring` (destroy, only use the copy on `alberti`)
- [ ] `buildbot-conf` (archive)
- [ ] `dip` (archive?)
- [x] `dns/auto-dns` (migrate to `nevii`)
- [x] `dns/dns-helpers` (migrate to `nevii`)
- [x] `dns/domains` (migrate to `nevii`)
- [x] `dns/mini-nag` (migrate to `nevii`)
- [x] `letsencrypt-domains` (migrate to `nevii`)
- [x] `puppet/puppet-ganeti` (destroy)
- [ ] `services/gettor` (archive)
- [ ] `services/gitlab/dip-configs` (archive)
- [ ] `services/gitlab/dip` (archive?)
- [ ] `services/gitlab/ldapsync` (archive)
- [ ] `static-builds` (archive)
- [ ] `tor-jenkins` (archive)
- [x] `tor-nagios` (move to `nagios`, see also [TPA-RFC-33][], #40755)
- [x] `tor-passwords` (move to `pauli`)
- [x] `tor-virt` (destroy)
- [ ] `trac/TracAccountManager` (archive)
- [ ] `trac/trac-email` (archive)
- [x] `tsa-misc` (migrate, renamed to `fabric-tasks`)
- [x] `userdir-ldap-cgi` (migrate)
- [x] `userdir-ldap` (migrate)
[TPA-RFC-33]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-33-monitoring
The repositories that were migrated to pauli, nevii or nagios need special configuration to get notifications working again. it would also be pretty awesome if they could push to a mirror on GitLab. Finally, they need docs. So extras in the checklist for those repos:
- [ ] IRC notifications
- [ ] email notifications
- [ ] documentation updates (particularly howto/tls, howto/dns is barely documented...)
- [ ] GitLab mirror (optional)legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcat2024-01-17https://gitlab.torproject.org/tpo/tpa/team/-/issues/41217retire cupani2024-03-26T20:48:47Zanarcatretire cupaniOnce all lagacy Git repositories have been migrated to GitLab (#41215), retire cupani.Once all lagacy Git repositories have been migrated to GitLab (#41215), retire cupani.legacy Git infrastructure retirement (TPA-RFC-36)anarcatanarcat2024-04-24https://gitlab.torproject.org/tpo/tpa/team/-/issues/41218retire vineale2024-03-26T20:51:39Zanarcatretire vinealeOnce all lagacy Git repositories have been migrated to GitLab (#41215) and the redirections moved to the static mirror system (#41216), retire `vineale`.Once all lagacy Git repositories have been migrated to GitLab (#41215) and the redirections moved to the static mirror system (#41216), retire `vineale`.legacy Git infrastructure retirement (TPA-RFC-36)2024-06-08https://gitlab.torproject.org/tpo/tpa/team/-/issues/41216move legacy git redirections to the static mirror infrastructure2024-03-27T15:29:01Zanarcatmove legacy git redirections to the static mirror infrastructureOnce all lagacy Git repositories have been migrated to GitLab (#41215), the redirections can be moved from gitweb (`vineale`) to the static mirror system.Once all lagacy Git repositories have been migrated to GitLab (#41215), the redirections can be moved from gitweb (`vineale`) to the static mirror system.legacy Git infrastructure retirement (TPA-RFC-36)2024-03-08https://gitlab.torproject.org/tpo/tpa/team/-/issues/41215forcibly migrate remaining Gitolite repositories to GitLab2024-03-26T20:51:17Zanarcatforcibly migrate remaining Gitolite repositories to GitLabAs part of the Gitolite retirement procedure (TPA-RFC-36, #41180), migrate remaining repositories from Gitolite to GitLab.
The goal is to complete this from within 12 months of the announcement, or 3 months from the original due date of...As part of the Gitolite retirement procedure (TPA-RFC-36, #41180), migrate remaining repositories from Gitolite to GitLab.
The goal is to complete this from within 12 months of the announcement, or 3 months from the original due date of this ticket, that is, before 2024-06-08, at which point the servers are supposed to be retired.
The actual list of repositories to migrate should probably be added here and users warned about the impeding change one last time. An inventory will have previously been made in #41214.
In TPA-RFC-36, the following was established:
> ## Per-repository particularities
>
> This section documents the fate of some repositories we are aware
> of. If you can think of specific changes that need to happen to
> repositories that are unusual, please do report them to TPA so they
> can be included in this proposal.
>
> ### idle repositories
>
> Repositories that did not have any new commit in the last two years
> are considered "idled" and should be migrated or archived to GitLab by
> their owners. Failing that, TPA will *archive* the repositories in the
> GitLab `legacy/` namespace before final deadline.
>
> ### user repositories
>
> There are 358 repositories under the `user/` namespace, owned by 70
> distinct users.
>
> Those repositories must be migrated to their corresponding user on the
> GitLab side.
>
> If the Gitolite user does not have a matching user on GitLab, their
> repositories will be moved under the `legacy/gitolite/user/` namespace
> in GitLab, owned by the GitLab admin doing the migration.
>
> ### "mirror" and "extern" repositories
>
> Those repositories will be migrated to, and archived in, GitLab within
> a month of the adoption of this proposal.
>
> ### Applications team repositories
>
> [See tpo/tpa/team#41181.]
So, as a task list, per repos:
- [ ] 5 `mirror` or `extern` (migrate and archive)
- [ ] 12 TPA (see #41219)
- [x] 49 migrated
- [ ] 87 `Attic` (migrate and archive)
- [ ] 97 `Other` (migrate, archive "idle" repositories)
- [ ] 288 `user` (migrate and archive)
See https://gitlab.torproject.org/tpo/tpa/team/-/issues/41214#note_2983291 for how those numbers were established.legacy Git infrastructure retirement (TPA-RFC-36)2024-03-08https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/98Investigate push-signing and transparency logs as mitigation for repository a...2024-03-27T15:21:35ZNick MathewsonInvestigate push-signing and transparency logs as mitigation for repository attacksFor background see:
* https://people.kernel.org/monsieuricon/signed-git-pushes
* https://korg.docs.kernel.org/gitolite/transparency-log.html
The idea here would be that some repositories (eg tor.git) could require signed _pushes_. Th...For background see:
* https://people.kernel.org/monsieuricon/signed-git-pushes
* https://korg.docs.kernel.org/gitolite/transparency-log.html
The idea here would be that some repositories (eg tor.git) could require signed _pushes_. Then we could archive these signed pushes in an append-only log, for auditing.
There are subproblems that would need to be solved for this to work:
* Only allow signed pushes on certain repositories (would require a per-repository gitolite hook).
* Allow signed pushes (requires setting certain options in gitconfig, see `certNonceSeed`, `certNonceSlop`, and `advertisePushOptions`)
* Make an append-only log of these signed pushes (possibly using trillian, possibly using some simpler transparency-log tool).
* Make a tool to audit this log and make sure that it's consistent and that it generates the current state of the repository.
* Decide what to do about key management.
And possibly:
* Make a tool that can be used at pull time to check the latest branch against the log.
There are also some sub-sub problems:
* Can we disable the merge button on target repositories?legacy Git infrastructure retirement (TPA-RFC-36)https://gitlab.torproject.org/tpo/tpa/team/-/issues/41337port puppet-managed configuration files to Debian bookworm2024-02-08T16:28:16Zanarcatport puppet-managed configuration files to Debian bookwormDuring the first batch of bookworm upgrades (#41251), we found a few issues with the Puppet configs that should probably be tweaked before the next batch to remove noise.
We have some slight diffs in our Puppet-managed NTP configuration...During the first batch of bookworm upgrades (#41251), we found a few issues with the Puppet configs that should probably be tweaked before the next batch to remove noise.
We have some slight diffs in our Puppet-managed NTP configuration:
```
Notice: /Stage[main]/Ntp/File[/etc/ntpsec/ntp.conf]/content:
--- /etc/ntpsec/ntp.conf 2023-09-26 14:41:08.648258079 +0000
+++ /tmp/puppet-file20230926-35001-x7hntz 2023-09-26 14:47:56.547991158 +0000
@@ -4,13 +4,13 @@
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
-driftfile /var/lib/ntpsec/ntp.drift
+driftfile /var/lib/ntp/ntp.drift
# Leap seconds definition provided by tzdata
leapfile /usr/share/zoneinfo/leap-seconds.list
# Enable this if you want statistics to be logged.
-#statsdir /var/log/ntpsec/
+#statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
Notice: /Stage[main]/Ntp/File[/etc/ntpsec/ntp.conf]/content: content changed '{sha256}c5d627a596de1c67aa26dfbd472a4f07039f4664b1284cf799d4e1eb43c92c80' to '{sha256}18de87983c2f8491852390acc21c466611d6660083b0d0810bb6509470949be3'
Notice: /Stage[main]/Ntp/File[/etc/ntpsec/ntp.conf]/mode: mode changed '0644' to '0444'
Info: /Stage[main]/Ntp/File[/etc/ntpsec/ntp.conf]: Scheduling refresh of Exec[service ntpsec restart]
Info: /Stage[main]/Ntp/File[/etc/ntpsec/ntp.conf]: Scheduling refresh of Exec[service ntpsec restart]
Notice: /Stage[main]/Ntp/File[/etc/default/ntpsec]/content:
--- /etc/default/ntpsec 2023-07-29 20:51:53.000000000 +0000
+++ /tmp/puppet-file20230926-35001-d4tltp 2023-09-26 14:47:56.579990910 +0000
@@ -1,9 +1 @@
-NTPD_OPTS="-g -N"
-
-# Set to "yes" to ignore DHCP servers returned by DHCP.
-IGNORE_DHCP=""
-
-# If you use certbot to obtain a certificate for ntpd, provide its name here.
-# The ntpsec deploy hook for certbot will handle copying and permissioning the
-# certificate and key files.
-NTPSEC_CERTBOT_CERT_NAME=""
+NTPD_OPTS='-g'
Notice: /Stage[main]/Ntp/File[/etc/default/ntpsec]/content: content changed '{sha256}26bcfca8526178fc5e0df1412fbdff120a0d744cfbd023fef7b9369e0885f84b' to '{sha256}1bb4799991836109d4733e4aaa0e1754a1c0fee89df225598319efb83aa4f3b1'
Notice: /Stage[main]/Ntp/File[/etc/default/ntpsec]/mode: mode changed '0644' to '0444'
Info: /Stage[main]/Ntp/File[/etc/default/ntpsec]: Scheduling refresh of Exec[service ntpsec restart]
Info: /Stage[main]/Ntp/File[/etc/default/ntpsec]: Scheduling refresh of Exec[service ntpsec restart]
Notice: /Stage[main]/Ntp/Exec[service ntpsec restart]: Triggered 'refresh' from 4 events
```
Note that this is a "reverse diff", that is Puppet restoring the old
bullseye config, so we should apply the reverse of this in Puppet.
### sudo configuration lacks limits.conf?
Just notice this diff on all hosts:
```
--- /etc/pam.d/sudo 2021-12-14 19:59:20.613496091 +0000
+++ /etc/pam.d/sudo.dpkg-dist 2023-06-27 11:45:00.000000000 +0000
@@ -1,12 +1,8 @@
-##
-## THIS FILE IS UNDER PUPPET CONTROL. DON'T EDIT IT HERE.
-##
#%PAM-1.0
-# use the LDAP-derived password file for sudo access
-auth requisite pam_pwdfile.so pwdfile=/var/lib/misc/thishost/sudo-passwd
+# Set up user limits from /etc/security/limits.conf.
+session required pam_limits.so
-# disable /etc/password for sudo authentication, see #6367
-#@include common-auth
+@include common-auth
@include common-account
@include common-session-noninteractive
```
Why don't we have `pam_limits` setup? Historical oddity? To investigatte.Debian 12 bookworm upgradehttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41321bookworm upgrades, third batch: High complexity2024-02-01T04:09:39Zanarcatbookworm upgrades, third batch: High complexityupgrade the following servers, if they still exist, one by one, carefully.
- [x] `alberti.torproject.org` - ~~bullseye upgrade was #40693~~ done in one shot, oops!
- [ ] `eugeni.torproject.org` - bullseye upgrade was #40694
- [ ] `hetz...upgrade the following servers, if they still exist, one by one, carefully.
- [x] `alberti.torproject.org` - ~~bullseye upgrade was #40693~~ done in one shot, oops!
- [ ] `eugeni.torproject.org` - bullseye upgrade was #40694
- [ ] `hetzner-hel1-01.torproject.org` - bullseye upgrade was #40695
- [ ] `pauli.torproject.org` - bullseye upgrade was #40696, puppetdb upgraded separately in #41341
Note that some of those machines might not be running bullseye yet, see the related tickets for more information. An upgrade to bullseye MUST be performed first, even if we batch-upgrade them to bookworm, as there are significant issues with skipping the bullseye upgrade. Those were found while accidentally upgrading `alberti` from buster to bookworm, see #40693.
In [TPA-RFC-57](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-57-bookworm-upgrades#high-complexity-individually-done) this also included the ganeti cluster upgrades, but those have been split up in separate tickets (#41254 and #41253).Debian 12 bookworm upgrade2024-02-14