The Tor Project issueshttps://gitlab.torproject.org/groups/tpo/-/issues2023-09-21T14:18:41Zhttps://gitlab.torproject.org/tpo/tpa/prometheus-alerts/-/issues/13Fix Tor DNS EL alert entry2023-09-21T14:18:41ZGeorg KoppenFix Tor DNS EL alert entryAfter https://gitlab.torproject.org/tpo/network-health/metrics/monitoring-and-alerting/-/merge_requests/19 got merged we need to update the respective alert entry.After https://gitlab.torproject.org/tpo/network-health/metrics/monitoring-and-alerting/-/merge_requests/19 got merged we need to update the respective alert entry.Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/tpo/core/tor/-/issues/40859Cleanup ci-driver.sh after 0.4.3 and 0.4.5 is a thing of the past2023-09-21T13:03:17ZAlexander Færøyahf@torproject.orgCleanup ci-driver.sh after 0.4.3 and 0.4.5 is a thing of the pastThis is a placeholder ticket for !586 which does some general clean-up of our CI driver script.This is a placeholder ticket for !586 which does some general clean-up of our CI driver script.Tor: 0.4.7.x-post-stableAlexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/tpo/network-health/team/-/issues/294Set bwscanner_cc back to 12023-09-21T12:08:11ZGeorg KoppenSet bwscanner_cc back to 1We tried to get our upload implementation properly in shape for the 1.6 release but failed so far: there are still a bunch of issues to debug (see: e.g. sbws#40152) and we are not risking at that point getting a proper 1.6 out by having ...We tried to get our upload implementation properly in shape for the 1.6 release but failed so far: there are still a bunch of issues to debug (see: e.g. sbws#40152) and we are not risking at that point getting a proper 1.6 out by having upload enabled by default. Rather, we'll ship the best upload version we currently have but it will be off by default due to `bwscanner_cc` being `1` in the consensus.Sponsor 61 - Making the Tor network faster & more reliable for users in Internet-repressive placesGeorg KoppenGeorg Koppenhttps://gitlab.torproject.org/tpo/network-health/onbasca/-/issues/154reuse tool fails after adding renovate.json2023-09-21T10:34:50Zjugareuse tool fails after adding renovate.json[The pipeline failing](https://gitlab.torproject.org/tpo/network-health/onbasca/-/pipelines/80543)
While this isn't probably important, i think we might run out again into this issue, which wasn't immediate to me how to solve it.[The pipeline failing](https://gitlab.torproject.org/tpo/network-health/onbasca/-/pipelines/80543)
While this isn't probably important, i think we might run out again into this issue, which wasn't immediate to me how to solve it.jugajugahttps://gitlab.torproject.org/tpo/network-health/onbasca/-/issues/153Implement clean command for onbrisca2023-09-21T10:34:50ZjugaImplement clean command for onbriscaSimilarly to the clean command for onbasca, to delete the logs in ~/.onbrisca, in the code path and the data in the DB.
Estimate: 3 * 1.5 (medium uncertainty)Similarly to the clean command for onbasca, to delete the logs in ~/.onbrisca, in the code path and the data in the DB.
Estimate: 3 * 1.5 (medium uncertainty)jugajugahttps://gitlab.torproject.org/tpo/network-health/onbasca/-/issues/160onbrisca dies with after getting/setting configurations via control port2023-09-21T10:34:50Zjugaonbrisca dies with after getting/setting configurations via control porttor doesn't seem to have a limit on the number of queries that can be done in a given time to the control port, so maybe stem has it.
@meskio: was the error `SocketClosed`? Looking at the possible errors using stem's `set_conf` and `get...tor doesn't seem to have a limit on the number of queries that can be done in a given time to the control port, so maybe stem has it.
@meskio: was the error `SocketClosed`? Looking at the possible errors using stem's `set_conf` and `get_conf` (https://gitlab.torproject.org/tpo/network-health/stem/-/blob/master/stem/control.py#L2204), it isn't one of them, but maybe is inherited or it raised somewhere else.
I think a way to solve this could be incrementing sleep time between queries.jugajugahttps://gitlab.torproject.org/tpo/network-health/onbasca/-/issues/144Set configuration option for the number of days or number of bridge measureme...2023-09-21T10:34:50ZjugaSet configuration option for the number of days or number of bridge measurements that should be keptAs commented in yesterday net meeting.As commented in yesterday net meeting.jugajugahttps://gitlab.torproject.org/tpo/network-health/onbasca/-/issues/145Add onbrisca dir to the linter configurations and CI2023-09-21T10:34:50ZjugaAdd onbrisca dir to the linter configurations and CII forgot to add it when i created.I forgot to add it when i created.jugajugahttps://gitlab.torproject.org/tpo/network-health/onbasca/-/issues/141Modify bridge scanner web API to include the consensus ratio threshold and th...2023-09-21T10:34:49ZjugaModify bridge scanner web API to include the consensus ratio threshold and the bridge ratio
After talking with @meskio, we have thought that for rdsys to accept both answers from bridgestrap and onbasca, we should modify the onbasca API to return 2 more fields in its JSON, one for the bridge ratio threshold (rdsys doesn't obta...
After talking with @meskio, we have thought that for rdsys to accept both answers from bridgestrap and onbasca, we should modify the onbasca API to return 2 more fields in its JSON, one for the bridge ratio threshold (rdsys doesn't obtain the consensus) and another for the ratio of each bridge.
So the onbasca JSON would be:
```
{
"bridge_results": {
"BRIDGE_LINE_1": {
"functional": BOOL,
"last_tested": "STRING",
"error": "STRING", (only present if "functional" is false)
"ratio": FLOAT
},
...
"BRIDGE_LINE_N": {
...
}
},
"error": "STRING", (only present if the entire test failed)
"time": FLOAT,
"bridge_ratio_threshold": FLOAT
}
```jugajugahttps://gitlab.torproject.org/tpo/network-health/onbasca/-/issues/142Ensure deployment instructions and install example include the steps for a no...2023-09-21T10:34:49ZjugaEnsure deployment instructions and install example include the steps for a non root userAfter meskio has explained me how things are deployed, we should facilitate the deployment for a non root user at https://gitlab.torproject.org/tpo/tpa/team/-/issues/41046.After meskio has explained me how things are deployed, we should facilitate the deployment for a non root user at https://gitlab.torproject.org/tpo/tpa/team/-/issues/41046.jugajugahttps://gitlab.torproject.org/tpo/network-health/onbasca/-/issues/130Use onbasca at bridge authority/bridgedb/rdsys2023-09-21T10:34:49ZMike PerryUse onbasca at bridge authority/bridgedb/rdsysWe could perform sbws-style measurement of bridges that bridgedb hands out. Here's a rough sketch of the idea:
Basically, the idea is to take the set of bridges at the bridge authority/bridgedb/rdsys, and perform sbws stream measurement...We could perform sbws-style measurement of bridges that bridgedb hands out. Here's a rough sketch of the idea:
Basically, the idea is to take the set of bridges at the bridge authority/bridgedb/rdsys, and perform sbws stream measurements on all of them, from a measurement authority. Stream bandwidths and ratios would be computed as normal, but instead of creating a consensus weight, the ratio would be used as a decision point for bridgedb to stop handing out the bridge. If the ratio is below 1.0 (as a tunable parameter -- we may want to use 0.75 or sth lower), then this means that that bridge is slower than average in terms of stream capacity, and likely overloaded or too slow to be handed out.
To do this, not only does onbasca need to know how to parse the bridge set from the bridge auth (or bridgedb/rdsys), it also needs to provide a document to the bridge authority/bridgedb/rdsys that tells it which bridges to stop handing out.
I believe for this, onbasca may need to support using obfs4 to connect to these bridges. I don't think obfs4 bridges allow their orport to be used as normal bridges, but I could be wrong.
Creating this ticket to track investigation and estimations. This may be quite the endeavor. It may also be the case that not enough people use obfs4 for this to be worthwhile, and we should focus load balancing work only on snowflake.
Cc: @meskio
Update: using this same ticket for the implementation and track its status:
- [X] Implement the HTTP endpoint to receive the bridges to measure
- [X] Receive a bridgeline via POST request
- [X] Reply with status code and json {fingerprint, valid} based on the data the scanner already have
- [X] Receive several bridgelines in the same request?
- ~~[ ] Reply with other format or with the ratio instead of a boolean?~~
- [X] Implement getting and parsing the bridges from the endpoint
- [X] Parse a single bridgeline from the POST request
- [X] Implement getting middle relays with some constrains ~~greater bandwidth than the bridge~~
- [X] select middle (and exits?) relays with FAST flag
- [X] select middle (and exits?) relays with ~~double bandwidth than the bridge or other flags?~~ STABLE flag, contact and uptime (greater than 30 days?)
- [X] Implement building 3 hops circuit
- [X] Adapt calculating ratios for relays
- [X] Implement creating the output
- [X] HTTPS?
- [ ] Tests
- [X] Implement basic unit tests for the models and the view (HTTP endpoint)
- [X] Fix them, currently not passing
- [X] Add more unit tests (eg. to test the calculated ratio)?
- [ ] Add integration test?
- [ ] Easy deployment
- ~~[X] Debian packages are probably not needed at this stage~~
- ~~[X] Create docker images?~~
- [X] Makefile or just documentation instructions
- [ ] Documentation
- [X] Add instructions on how to deploy it
- [ ] Add more docstrings?Sponsor 61 - Making the Tor network faster & more reliable for users in Internet-repressive placesjugajugahttps://gitlab.torproject.org/tpo/network-health/team/-/issues/329Archive doctor-modules repository2023-09-21T07:56:45ZGeorg KoppenArchive doctor-modules repositoryThe private doctor-modules repository contains two scripts for v2 onion attack detection which are not applicable anymore in the v3 world and one which is concerned with finding sybils per nickname. We can move that one into our regular ...The private doctor-modules repository contains two scripts for v2 onion attack detection which are not applicable anymore in the v3 world and one which is concerned with finding sybils per nickname. We can move that one into our regular `doctor` repository given that we have a `sybil_checker.py` script already there (which sends notifications to a public mailing list anyway).
We should make that private repo public then and archive it.Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/129test and possibly replace docker with podman in GitLab runners2023-09-21T02:26:46Zanarcattest and possibly replace docker with podman in GitLab runnersGitLab *finally* managed to port the GitLab runner infrastructure to be useable with Podman:
- https://docs.gitlab.com/runner/executors/docker.html#use-podman-to-run-docker-commands
- https://about.gitlab.com/releases/2022/08/22/gitlab-...GitLab *finally* managed to port the GitLab runner infrastructure to be useable with Podman:
- https://docs.gitlab.com/runner/executors/docker.html#use-podman-to-run-docker-commands
- https://about.gitlab.com/releases/2022/08/22/gitlab-15-3-released/#gitlab-runner-153
- https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27119
this could be tremendously useful for us, in many ways:
1. podman makes it much easier to run "rootless" containers, which could significantly improve the security of our runners
2. that, in turn, could make it easier to build container images in runners (see #123, #90, #89 for background on that work, and https://docs.gitlab.com/runner/executors/docker.html#using-podman-to-build-container-images-from-a-dockerfile for the upstream docs)
3. podman doesn't require a daemon, so runner jobs would could directly under systemd which, in turn, might make gitlab-runner upgrades less disruptive
4. podman is simpler than docker and therefore easier to package in Debian, which means the package may be more up to date (for example, upstream docker is at 22.06-beta0, but unstable has 20.10.17, and stable 20.10.5, while podman is at 4.2.0, which is already packaged in experimental, unstable has 3.4.7 and stable 3.0.1)
Unfortunately, GitLab and Podman has fixed their things in version 15.3 (which we run) and 4.2.0 (which we don't), respectively. So we're not quite ready to run this from the Debian side of things. First the 4.2.0 podman release would need to get into unstable, and there testing. *Then* we could see if we can either get a backport running, or setup a bookworm runner, which therefore might make this part of the %"Debian 12 bookworm upgrade" milestone.
See this page for progress on the podman packaging: https://tracker.debian.org/pkg/libpodDebian 12 bookworm upgradeJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41258materculae out of disk space2023-09-21T01:51:41ZKezmaterculae out of disk spaceprevious ticket: #40826
it's been a year, and nagios is complaining about materculae's /srv partition
```
# df -h /srv
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_materculae-srv 147G 135G 4.3G 97%...previous ticket: #40826
it's been a year, and nagios is complaining about materculae's /srv partition
```
# df -h /srv
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_materculae-srv 147G 135G 4.3G 97% /srv
```
in the previous ticket (#40826) @anarcat changed the warning threshold, which is why this warning popped up now.
according to grafana, the usage has only been about 15G in the past year, and the growth is linear. we could add another 20G and revisit in a year, or throw 40G or 60G at it to push things further down the road.
![image](/uploads/e8ddf8b69703273f73d891586f7fc137/image.png)anarcatanarcat2023-09-22https://gitlab.torproject.org/tpo/core/arti/-/issues/733Code to generate and sign onion descriptors2023-09-20T16:18:49ZNick MathewsonCode to generate and sign onion descriptorsArti: Onion service supportgabi-250gabi-250https://gitlab.torproject.org/tpo/network-health/doctor/-/issues/40033Add dannenberg back to consensus-health checks2023-09-20T09:00:00ZGeorg KoppenAdd dannenberg back to consensus-health checksAndreas asked us on 01/30/2021 to skip any dir-auth checks for dannenberg due to the overload/DDoS situation. This has calmed down and we should re-add the checks for it. While deploying the change we should make sure no personal mails a...Andreas asked us on 01/30/2021 to skip any dir-auth checks for dannenberg due to the overload/DDoS situation. This has calmed down and we should re-add the checks for it. While deploying the change we should make sure no personal mails are sent for that case: the ones delivered to the mailing list are sufficient.Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41108build a new VM to deploy donate-neo review environments2023-09-20T00:55:55ZKezbuild a new VM to deploy donate-neo review environmentscontext: tpo/web/donate-neo#6
donate-neo needs a place to deploy CI environments. we'll do this by creating a new server named donate-review-01, with 8G memory, 2 vCPU cores, 10G root, 2G swap, and 20G HDD. thw software stack for the re...context: tpo/web/donate-neo#6
donate-neo needs a place to deploy CI environments. we'll do this by creating a new server named donate-review-01, with 8G memory, 2 vCPU cores, 10G root, 2G swap, and 20G HDD. thw software stack for the review deployments will be apache, mod_wsgi, python 3.10, and sqlite.
as for the deployment procedure, i was thinking of having the CI pipeline build an installable package and scp it to the donate-review-01 box, SSH in, and call a script. this script will set up a virtual environment, install the required packages, and create a new apache config and restart apache.
@anarcat do those VM specs and deployment steps look good to you?https://gitlab.torproject.org/tpo/tpa/team/-/issues/41324Retire ci-runner-x86-012023-09-19T19:49:35ZJérôme Charaouilavamind@torproject.orgRetire ci-runner-x86-01Superseded by new podman-based runner `ci-runner-x86-02`.
1. ~~[ ] announcement~~ (N/A)
2. [x] nagios
3. [x] retire the host in fabric
4. [x] remove from LDAP with `ldapvi`
5. [x] power-grep
6. [x] remove from tor-passwords
7. ~~...Superseded by new podman-based runner `ci-runner-x86-02`.
1. ~~[ ] announcement~~ (N/A)
2. [x] nagios
3. [x] retire the host in fabric
4. [x] remove from LDAP with `ldapvi`
5. [x] power-grep
6. [x] remove from tor-passwords
7. ~~[ ] remove from DNSwl~~ (N/A)
8. [x] remove from docs
9. ~~[ ] remove from racks~~ (N/A)
10. [x] remove from reverse DNSJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/core/tor/-/issues/40791prop340: Implement packed and fragmented cells2023-09-19T18:12:28ZDavid Gouletdgoulet@torproject.orgprop340: Implement packed and fragmented cellsThis ticket is about implementing proposal 340 on the relay side of C-tor.
Important to note that we plan to only implement this support client side in arti.This ticket is about implementing proposal 340 on the relay side of C-tor.
Important to note that we plan to only implement this support client side in arti.Tor: 0.4.9.x-freezeDavid Gouletdgoulet@torproject.orgDavid Gouletdgoulet@torproject.orghttps://gitlab.torproject.org/tpo/network-health/metrics/descriptorParser/-/issues/49DescriptorParser is failing to create data consistently in VM and the DB2023-09-19T15:28:22ZHiroDescriptorParser is failing to create data consistently in VM and the DBThe parser is crashing and/or nooping somewhere.
- [ ] Find why this is happening and apply patch
- [ ] Reprocess the missed daysThe parser is crashing and/or nooping somewhere.
- [ ] Find why this is happening and apply patch
- [ ] Reprocess the missed daysHiroHiro