Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T17:02:28Zhttps://gitlab.torproject.org/legacy/trac/-/issues/34437migrate help.tpo into a gitlab wiki2020-06-13T17:02:28Zanarcatmigrate help.tpo into a gitlab wikiwe are currently using ikiwiki to host our documentation. that has served us well so far: it's available as a static site in the static mirror system and allows all sysadmins to have a static, offsite copy of the documentation when every...we are currently using ikiwiki to host our documentation. that has served us well so far: it's available as a static site in the static mirror system and allows all sysadmins to have a static, offsite copy of the documentation when everything is down.
but ikiwiki is showing its age. it's an old program written in Perl, difficult to theme and not very welcoming to new users. for example, it's impossible for a user unfamiliar with git to contribute to the documentation. it also has its own unique Markdown dialect that is not used anywhere else. and while Markdown itself is not standardized and has lots of such dialects, there is /some/ convergence around CommonMark and GFM (GitHub's markdown) as de-facto standards at least, which ikiwiki still has to catchup with. it also has powerful macros which are nice to make complex websites, but do not render in the offline documentation, making us dependent on the rendered copy (as opposed to setting up client-side tools to peruse the documentation).
gitlab wikis, in contrast, have a web interface to edit pages. it doesn't have the macros ikiwiki has, but that's nothing a few commandline hacks can't fix... or at least we should consider it. they don't have macros or any more powerful features that ikiwiki has, but maybe that's exactly what we want.
this is not blocking the trac to gitlab migration, but it would be nice to jump onboard with everyone, since we will be migrating the Trac wiki onto gitlab as well...anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/34436document the static mirror network and onionbalance system better2020-06-13T17:02:28Zanarcatdocument the static mirror network and onionbalance system betterwe have some documentation on the static mirroring system here:
https://help.torproject.org/tsa/howto/static-component/
it's mostly procedural and minimal: add a component, remove a component and that's it. it doesn't explain at all ho...we have some documentation on the static mirroring system here:
https://help.torproject.org/tsa/howto/static-component/
it's mostly procedural and minimal: add a component, remove a component and that's it. it doesn't explain at all how the system works, how to create or remove a new node in the network, how onion services interact with it, and how it actually works in puppet.
all this should be better documented. for example, I should be able to resolve #34396 without asking weasel. :)anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/34426document ud-ldap and its architecture better2020-06-13T17:02:27Zanarcatdocument ud-ldap and its architecture betterour LDAP documentation is minimal. go through the thing and document how the different components play with each other and common tasks (like creating a user and so on).our LDAP documentation is minimal. go through the thing and document how the different components play with each other and common tasks (like creating a user and so on).anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/34425document gitlab in our service docs2020-06-13T17:02:27Zanarcatdocument gitlab in our service docsin particular for backups (so linked to/from https://help.torproject.org/tsa/howto/backup/) but also our general policies and procedures.in particular for backups (so linked to/from https://help.torproject.org/tsa/howto/backup/) but also our general policies and procedures.anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/34424backport fabric to debian buster2020-06-13T17:02:27Zanarcatbackport fabric to debian busterWe rely on newer features of Fabric in our configuration that are not present in Debian buster. upload a backport to the official Debian backports.We rely on newer features of Fabric in our configuration that are not present in Debian buster. upload a backport to the official Debian backports.anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/34419move staticiforme/static-source off kvm52020-06-13T17:02:25Zweasel (Peter Palfrader)move staticiforme/static-source off kvm5This can be done at any convenient time, but it'll take a bit due to disk sizeThis can be done at any convenient time, but it'll take a bit due to disk sizeweasel (Peter Palfrader)weasel (Peter Palfrader)https://gitlab.torproject.org/legacy/trac/-/issues/34418move colchicifolium/collector off kvm52020-06-13T17:02:25Zweasel (Peter Palfrader)move colchicifolium/collector off kvm5We would like to move the VM to our new ganeti cluster.
Karsten, Given the big disks, this might take several hours.
Are there any constraints from your end or should we just do it at a convenient time?We would like to move the VM to our new ganeti cluster.
Karsten, Given the big disks, this might take several hours.
Are there any constraints from your end or should we just do it at a convenient time?weasel (Peter Palfrader)weasel (Peter Palfrader)https://gitlab.torproject.org/legacy/trac/-/issues/34417move henryi/consensus-health off kvm52020-06-13T17:02:25Zweasel (Peter Palfrader)move henryi/consensus-health off kvm5We would like to move the VM to our new ganeti cluster.
Are there any constraints from your end or should we just do it at a convenient time?We would like to move the VM to our new ganeti cluster.
Are there any constraints from your end or should we just do it at a convenient time?weasel (Peter Palfrader)weasel (Peter Palfrader)https://gitlab.torproject.org/legacy/trac/-/issues/34416move materculae/exonerator off kvm52020-06-13T17:02:24Zweasel (Peter Palfrader)move materculae/exonerator off kvm5We would like to move the VM to our new ganeti cluster.
Karsten, Given the big disks, this might take several hours.
Are there any constraints from your end or should we just do it at a convenient time?We would like to move the VM to our new ganeti cluster.
Karsten, Given the big disks, this might take several hours.
Are there any constraints from your end or should we just do it at a convenient time?weasel (Peter Palfrader)weasel (Peter Palfrader)https://gitlab.torproject.org/legacy/trac/-/issues/34415move carinatum/doctor off kvm52020-06-13T17:02:24Zweasel (Peter Palfrader)move carinatum/doctor off kvm5weasel (Peter Palfrader)weasel (Peter Palfrader)https://gitlab.torproject.org/legacy/trac/-/issues/34304new gnt-fsn node (fsn-node-07)2020-06-13T17:02:17Zanarcatnew gnt-fsn node (fsn-node-07)need to create one last ganeti node to replace kvm5 (#33084)need to create one last ganeti node to replace kvm5 (#33084)HiroHirohttps://gitlab.torproject.org/legacy/trac/-/issues/34185ganeti clusters don't like automatic upgrades2020-06-13T17:02:12Zanarcatganeti clusters don't like automatic upgradesthis weekend the ganeti cluster had a partial outage: nodes were reachable, but networking was broken on all instances.
this is, presumably, because of the Debian buster point release that occured on saturday (!). last time this happene...this weekend the ganeti cluster had a partial outage: nodes were reachable, but networking was broken on all instances.
this is, presumably, because of the Debian buster point release that occured on saturday (!). last time this happened, weasel identified openvswitch as the culprit, and hiro deployed a fix that would make it survive such situations. but either something else came up or the fix didn't work, because the problem happened again this weekend.
i fixed it by rebooting all nodes forcibly (without migrating first).HiroHirohttps://gitlab.torproject.org/legacy/trac/-/issues/34123Provide secrets/passwords management for Tor Browser Nightly signing2020-06-13T17:02:05ZMatthew FinkelProvide secrets/passwords management for Tor Browser Nightly signingAs mentioned in #34121, the Tor Browser Nightly signing machine will host an OpenPGP key and an NSSDB private key. Both of these should be password-protected. Instead of hard-coding these passphrases in a file or script on the server, ha...As mentioned in #34121, the Tor Browser Nightly signing machine will host an OpenPGP key and an NSSDB private key. Both of these should be password-protected. Instead of hard-coding these passphrases in a file or script on the server, having a password management system from where the passwords can be retrieved would be very nice.HiroHirohttps://gitlab.torproject.org/legacy/trac/-/issues/33921gitlab monitoring2020-06-13T17:01:44Zanarcatgitlab monitoringwe need to have gitlab metrics in grafana. we also need to make sure things work (nagios).
this implies monitoring the webserver (nginx i guess?) and it would also be nice to have internal gitlab metrics. there's a builtin prometheus ex...we need to have gitlab metrics in grafana. we also need to make sure things work (nagios).
this implies monitoring the webserver (nginx i guess?) and it would also be nice to have internal gitlab metrics. there's a builtin prometheus exporter in gitlab so we should be able to reuse that. see:
https://docs.gitlab.com/ee/administration/monitoring/prometheus/
https://docs.gitlab.com/ee/administration/monitoring/prometheus/gitlab_metrics.htmlHiroHirohttps://gitlab.torproject.org/legacy/trac/-/issues/33810ganeti monitoring2020-06-13T17:01:33Zanarcatganeti monitoringwe're migrating everything into ganeti, but maybe there's some extra monitoring we could think about, as ganeti is way more knowledgeable about its own internals than libvirt was. or at least that's the feeling I get.
some ideas:
* we...we're migrating everything into ganeti, but maybe there's some extra monitoring we could think about, as ganeti is way more knowledgeable about its own internals than libvirt was. or at least that's the feeling I get.
some ideas:
* we could have a nagios plugin that checks for N+1. riseup has something like this
* we could have a grafana dashboard that shows us the state of the cluster. we already have the main dashboard which we can set to show only the ganeti cluster
The current memory view goes about like this:
![snap-2020.04.03-17.33.23.png,700](uploads/snap-2020.04.03-17.33.23.png,700)
I'm not sure how we could improve this, but it seems to me having global (and/or per node?) memory, CPU, network and disk usage would be a great improvement as well.anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/33537audit SVN accesses2020-06-13T17:01:11Zanarcataudit SVN accessesone of things that came out of the SVN retirement discussion is that we don't exactly know how to tell who has access to what. we hope that access controls are correct and tight, but that institutional knowledge somewhat bitrotten over t...one of things that came out of the SVN retirement discussion is that we don't exactly know how to tell who has access to what. we hope that access controls are correct and tight, but that institutional knowledge somewhat bitrotten over the years, so we can't easily tell.
the task is to figure out exactly which set of users has access to which files.
this is in the context that the SVN service will be eventually retired, but we have hit a roadblock in the corpsvn retirement (#32025) where there are concerns about where to move the files to. so while we resolve those concerns, we will keep hosting corpsvn for a little longer and that means cleaning up the accesses.Gabagaba@torproject.orgGabagaba@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/33406automate reboots2020-06-13T17:00:59Zanarcatautomate rebootsin #31957 we have worked on automating upgrades, but that's only part of the problem. we also need to reboot in some situations.
we have various mechanisms to do so right now:
* `tsa-misc/reboot-host` - reboot script for kvm boxes, ki...in #31957 we have worked on automating upgrades, but that's only part of the problem. we also need to reboot in some situations.
we have various mechanisms to do so right now:
* `tsa-misc/reboot-host` - reboot script for kvm boxes, kind of a mess, to be removed when we finish the kvm-ganeti migration
* `tsa-misc/reboot-guest` - reboot a single host. kind of a hack, but useful to reboot a single machine
* `misc/multi-tool/torproject-reboot-simple` - iterate over all hosts with `rebootPolicy=justdoit` in LDAP and reboot them with `torproject-reboot-many`
* `misc/multi-tool/torproject-reboot-rotation` - iterate over all hosts with `rebootPolicy=rotation` in LDAP and reboot them with `torproject-reboot-many`, with a 30 minute delay between each host
* `ganeti-reboot-cluster` - a tool to reboot the ganeti cluster
There are various problems with all this:
* the `torproject-reboot-*` scripts do not take care of `rebootPolicy=manual` hosts
* the `ganeti-reboot-cluster` script has been known to fail if a cluster is unbalanced
* the `ganeti-reboot-cluster` script currently fails when hosts talk to each other over IPv6 somehow (see #33412)
* we have 5 different ways of performing reboots, we should have just one script that does it all
* reboot-{host,guest} do not check if hosts need reboot before rebooting (but the multi-tool does)
In short, this is kind of a mess, and we should refactor this. We should consider using needrestart, which knows how to reboot individual hosts.
I also added a [feature request to the needrestart puppet module](https://github.com/xneelo/hetzner-needrestart/issues/23) to expose its knowledge as a puppet fact, so we can use that information from PuppetDB instead of SSH'ing in each host and calling the dsa-* tools.anarcatanarcathttps://gitlab.torproject.org/legacy/trac/-/issues/33332move root passwords to trocla?2020-06-13T17:00:56Zanarcatmove root passwords to trocla?one manual step of our install process is to initialize the root password and set it in the password manager. that manual step could be completely skipped if we just set the root password in trocla.one manual step of our install process is to initialize the root password and set it in the password manager. that manual step could be completely skipped if we just set the root password in trocla.HiroHirohttps://gitlab.torproject.org/legacy/trac/-/issues/33288forrestii/fpcentral still has stretch packages (mongodb)2020-06-13T17:41:29Zanarcatforrestii/fpcentral still has stretch packages (mongodb)fpcentral requires mongodb to operate, but that package was removed from Debian stable in february 2019, mainly [because of license problems](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=916107). to quote wikipedia:
> MongoDB has ...fpcentral requires mongodb to operate, but that package was removed from Debian stable in february 2019, mainly [because of license problems](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=916107). to quote wikipedia:
> MongoDB has been dropped from the Debian, Fedora and Red Hat Enterprise Linux distributions due to the licensing change. Fedora determined that the SSPL version 1 is not a free software license because it is "intentionally crafted to be aggressively discriminatory" towards commercial users.
(there is also an [unpatched security issue](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=934783) against mongodb, by the way...)
we can't maintain this service like that in the long term. stretch will stop being supported this summer and mongodb isn't supported in its free form.
what's the plan for replacing mongodb?cypherpunkscypherpunkshttps://gitlab.torproject.org/legacy/trac/-/issues/33084decomission kvm5, 9 VMs to migrate2020-06-13T17:02:17Zanarcatdecomission kvm5, 9 VMs to migrate * [o] build-x86-08.torproject.org (buildbox; can be retired any time; but ideally keep while we have kvm5)
* [ ] #34415: carinatum.torproject.org (DocTor Host)
* [ ] #34418: colchicifolium.torproject.org (collector.torproject.org)
* ... * [o] build-x86-08.torproject.org (buildbox; can be retired any time; but ideally keep while we have kvm5)
* [ ] #34415: carinatum.torproject.org (DocTor Host)
* [ ] #34418: colchicifolium.torproject.org (collector.torproject.org)
* [x] gitlab-01.torproject.org (dip.torproject.org; retired, replaced with gitlab-02)
* [ ] #34417: henryi.torproject.org (consensus-health.torproject.org)
* [ ] #34416: materculae.torproject.org (exonerator.torproject.org)
* [x] palmeri.torproject.org (deb.tpo master)
* [x] perdulce.torproject.org (people.torproject.org)
* [ ] #34419: staticiforme.torproject.org (static-master.torproject.org)weasel (Peter Palfrader)weasel (Peter Palfrader)