Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T14:39:24Zhttps://gitlab.torproject.org/legacy/trac/-/issues/13339Prop140: Complete Consensus diffs / Merge GSoC project2020-06-13T14:39:24ZmvdanProp140: Complete Consensus diffs / Merge GSoC projectGoogle Summer of Code finished over a month ago, and during this time I've been tidying up my code a bit and reading it for the merge. You will find it on github:
https://github.com/mvdan/tor
This ticket is for the sole purpose of foll...Google Summer of Code finished over a month ago, and during this time I've been tidying up my code a bit and reading it for the merge. You will find it on github:
https://github.com/mvdan/tor
This ticket is for the sole purpose of following the merge process and its progress. But as always I'm on IRC and mail if you want to contact me directly.
I just rebased against master this morning. Nick and Sebastian have been reviewing my code over the summer, but of course more sets of eyes are needed.
The test coverage for the diff generation and application is fine (see test_consdiff.c), but there aren't any tests for the stuff I wrote to wire it into serving and fetching consensus diffs. Not really sure how to go about that, can't really promise I'd have the time to dive into it.
And regarding commit messages and changelog entries, I pretty much went with my instinct. Chances are they can be improved - the commit messages for future reference and the changelog entries for future release changelogs - so criticism is welcome.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/20512Make a Tor 0.2.7.7 release, with the patch for #203842020-06-13T16:52:35ZRoger DingledineMake a Tor 0.2.7.7 release, with the patch for #20384We sent out a patch for #20384, and apparently some groups like opensuse picked it up and did a package update on their side, but now we're in the unfortunate position of having some relays running 0.2.7.6 because they're buggy and old, ...We sent out a patch for #20384, and apparently some groups like opensuse picked it up and did a package update on their side, but now we're in the unfortunate position of having some relays running 0.2.7.6 because they're buggy and old, and others running 0.2.7.6 because they updated to the patched package.
The smart thing on our side is probably to follow up with an actual release, with its own new version number and everything, which includes that patch plus the other things we've been wanting to backport (e.g. directory authority changes).
We have a release-0.2.7 branch which has a big pile of stuff merged into it from long ago. It is unlikely to now contain the set of things we would like in 0.2.7.7. But I bet it has some good suggestions.
Step one is to gather together the set of things we might want in 0.2.7.7 -- get the list from the current release-0.2.7 branch, and from trac tickets that have the backport-027 keyword, and... anywhere else we should be looking?
Step two is to pick the right subset of them for 0.2.7.7.
Then step three is to do the actual merging, and do up a changelog and release stanza, and put it out.
I can help with step two, but I think Sebastian and weasel can too (and I will be offline for a while starting in a few days, so best not to bottleneck on me anyway).
Who will get us through step one? :)
(And then once we've done an 0.2.7.7, we can think about an 0.2.6.11, and work our way back.)Tor: 0.2.7.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/21211Write and analyze proposals for compressing consensus (diff)s with better alg...2020-06-13T15:05:24ZNick MathewsonWrite and analyze proposals for compressing consensus (diff)s with better algorithms**The idea:** Consensus documents are compressed with zlib, but nobody has to compress any given consensus more than once. Therefore, we can safely use more CPU compressing them, and save bandwidth on consensus downloads by switching to...**The idea:** Consensus documents are compressed with zlib, but nobody has to compress any given consensus more than once. Therefore, we can safely use more CPU compressing them, and save bandwidth on consensus downloads by switching to something else instead of zlib for consensuses.
This same analysis also applies to consensus diffs.
For this ticket, we should look at the code complexity and potential bandwidth savings here, and decide whether they are worth it.Tor: 0.3.1.x-finalAlexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/21283Remove broken fallback directory mirrors2020-06-13T15:05:36ZteorRemove broken fallback directory mirrorsSome time around 0.3.0.?-rc, we should remove all non-functional fallback directories from the December 2016 list (or rebuild the entire list).
I'll create a draft branch here, but the script should be re-run just before the release (an...Some time around 0.3.0.?-rc, we should remove all non-functional fallback directories from the December 2016 list (or rebuild the entire list).
I'll create a draft branch here, but the script should be re-run just before the release (and ideally, operators should be contacted and given an opportunity to fix any errors).Tor: 0.3.2.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/21641Prop274: Rotate onion keys less frequently2020-06-13T15:06:55ZNick MathewsonProp274: Rotate onion keys less frequentlyLet's implement proposal 274: This proposal will save a fair amount of bandwidth now, and even more once we have consensus diffs.Let's implement proposal 274: This proposal will save a fair amount of bandwidth now, and even more once we have consensus diffs.Tor: 0.3.1.x-finalAlexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/21643Prop140: Extract, test, revise, and clean the diff code2020-06-13T15:06:58ZNick MathewsonProp140: Extract, test, revise, and clean the diff codeStep one of getting the consensus diff support of prop140 completed is to get the diff and patch code merged. Let's do this separately from the directory code, since that code is hairier, and may need more work.Step one of getting the consensus diff support of prop140 completed is to get the diff and patch code merged. Let's do this separately from the directory code, since that code is hairier, and may need more work.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21644Prop140: Fuzz the diff and patch code2020-06-13T15:06:58ZNick MathewsonProp140: Fuzz the diff and patch codeNow that we have fuzzing support, it would be ridiculous to start using diff/patch code without testing it through our fuzzers. So, let's do that.Now that we have fuzzing support, it would be ridiculous to start using diff/patch code without testing it through our fuzzers. So, let's do that.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21645prop140 / compression: Unified directory cache backend2020-06-13T15:06:59ZNick Mathewsonprop140 / compression: Unified directory cache backendWith prop140 and/or ahf's compression proposal, we'll need to cache more documents than previously, to include multiple consensuses, diffs, compressed consensuses, etc. We could do this in an ad-hoc way, but we'll probably be better off...With prop140 and/or ahf's compression proposal, we'll need to cache more documents than previously, to include multiple consensuses, diffs, compressed consensuses, etc. We could do this in an ad-hoc way, but we'll probably be better off with a real abstraction here.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21646prop140/compression: Refactor "directory request" code2020-06-13T15:06:59ZNick Mathewsonprop140/compression: Refactor "directory request" codeOur current notion of "what is a directory request" includes a bunch of fields that are strewn around directory connections and passed to different functions. It would be nice to instead have a "directory request" type that we created a...Our current notion of "what is a directory request" includes a bunch of fields that are strewn around directory connections and passed to different functions. It would be nice to instead have a "directory request" type that we created and passed around as appropriate. This would make it easier to test our request generation/parsing code.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21651prop140/compression: Refactor directory cache spooling code2020-06-13T15:07:03ZNick Mathewsonprop140/compression: Refactor directory cache spooling codeOur current logic for spooling things from a directory server is a bit loopy. Instead we should refactor it so that "things to be spooled" is a first-class object, with implementations depending on what we're spooling. This will make l...Our current logic for spooling things from a directory server is a bit loopy. Instead we should refactor it so that "things to be spooled" is a first-class object, with implementations depending on what we're spooling. This will make lots of our directory improvement stuff easier to implement.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21654Don't use fgets() when we might get EAGAIN2020-06-13T15:09:26ZNick MathewsonDon't use fgets() when we might get EAGAINOur tor_fgets() code, added in #20988, tries to make fgets() behave as we expect with nonblocking sockets. But really, fgets() is not such a great choice for nonblocking sockets in the first place! Let's use direct fd-level IO for thos...Our tor_fgets() code, added in #20988, tries to make fgets() behave as we expect with nonblocking sockets. But really, fgets() is not such a great choice for nonblocking sockets in the first place! Let's use direct fd-level IO for those instead.Tor: 0.3.1.x-finalAlexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/21662prop278: Add support for LZMA2 and/or Zstandard2020-06-13T15:07:06ZAlexander Færøyahf@torproject.orgprop278: Add support for LZMA2 and/or ZstandardAdd support for the compression schemes needed to implement prop#278.
See: http://facebook.github.io/zstd/ and http://7-zip.org/sdk.html for the respective libraries.Add support for the compression schemes needed to implement prop#278.
See: http://facebook.github.io/zstd/ and http://7-zip.org/sdk.html for the respective libraries.Tor: 0.3.1.x-finalAlexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/21663prop278: Refactor the torgzip module to support additional compression schemes2020-06-13T15:07:06ZAlexander Færøyahf@torproject.orgprop278: Refactor the torgzip module to support additional compression schemesThe current torgzip module should be refactored such that the new compression schemes needed for prop#278 can fit nicely into the code.
This is the tracking bug for this task.The current torgzip module should be refactored such that the new compression schemes needed for prop#278 can fit nicely into the code.
This is the tracking bug for this task.Tor: 0.3.1.x-finalAlexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/21664prop278: Make the current 'torgzip' module a submodule of a new 'compression'...2020-06-13T15:07:07ZAlexander Færøyahf@torproject.orgprop278: Make the current 'torgzip' module a submodule of a new 'compression' module- Create a new 'compression' module to handle all compression schemes.
- Refactor the current 'torgzip' module into a module that handles only GZip and deflate, but adheres to the 'compression' module API.
- Create modules for the new co...- Create a new 'compression' module to handle all compression schemes.
- Refactor the current 'torgzip' module into a module that handles only GZip and deflate, but adheres to the 'compression' module API.
- Create modules for the new compression schemes: LZMA2 and Zstd.Tor: 0.3.1.x-finalAlexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/21665Prop278: Establish an upper-bound for LZMA2 memory usage2020-06-13T15:07:07ZAlexander Færøyahf@torproject.orgProp278: Establish an upper-bound for LZMA2 memory usageOur initial analysis shows that LZMA2 can be quite a memory hog, which means we should establish some sort of upper-bound for its memory usage and how we can actually enforce it.Our initial analysis shows that LZMA2 can be quite a memory hog, which means we should establish some sort of upper-bound for its memory usage and how we can actually enforce it.Tor: 0.3.1.x-finalAlexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/21666Prop278: Code to decide whether we want to request and/or provide CPU-intensi...2020-06-13T15:07:08ZAlexander Færøyahf@torproject.orgProp278: Code to decide whether we want to request and/or provide CPU-intensive compression methodsWe need some code in Tor to decide if or when we want to use and provide CPU-intensive compression operations. The biggest concern in Prop278 is LZMA2.We need some code in Tor to decide if or when we want to use and provide CPU-intensive compression operations. The biggest concern in Prop278 is LZMA2.Tor: 0.3.1.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/21667Prop278: Handle new headers in directory.c2020-06-13T15:07:09ZAlexander Færøyahf@torproject.orgProp278: Handle new headers in directory.cHandle the newly defined headers and their new values from Prop#278 in the directory server/client code.Handle the newly defined headers and their new values from Prop#278 in the directory server/client code.Tor: 0.3.1.x-finalAlexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/21668Prop278: Update cached_dir_t and related types to no longer assume single com...2020-06-13T15:07:09ZAlexander Færøyahf@torproject.orgProp278: Update cached_dir_t and related types to no longer assume single compresison methodUpdate the data-types for cached_dir_t to handle multiple available compression schemes.
See also: #21651Update the data-types for cached_dir_t to handle multiple available compression schemes.
See also: #21651Tor: 0.3.1.x-finalAlexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/21677Prop140: analyze diff performance2020-06-13T15:07:11ZNick MathewsonProp140: analyze diff performanceTor: 0.3.1.x-finalNick MathewsonNick Mathewson