Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T15:07:11Zhttps://gitlab.torproject.org/legacy/trac/-/issues/21677Prop140: analyze diff performance2020-06-13T15:07:11ZNick MathewsonProp140: analyze diff performanceTor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21673prop140: Handle signatures correctly2020-06-13T15:07:10ZNick Mathewsonprop140: Handle signatures correctlyFor diffs to work properly, we need to check the input document and the output document in their entirety, including their signatures. Otherwise, the diffs won't apply correctly when they change the signatures!
But for *that* to work, ...For diffs to work properly, we need to check the input document and the output document in their entirety, including their signatures. Otherwise, the diffs won't apply correctly when they change the signatures!
But for *that* to work, we need to do what we can to minimize the odds that anybody has a consensus with different signatures, or with signatures organized differently.
As an alternative, we could change the diff format so that it always replaces all the old signatures with the new ones.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21651prop140/compression: Refactor directory cache spooling code2020-06-13T15:07:03ZNick Mathewsonprop140/compression: Refactor directory cache spooling codeOur current logic for spooling things from a directory server is a bit loopy. Instead we should refactor it so that "things to be spooled" is a first-class object, with implementations depending on what we're spooling. This will make l...Our current logic for spooling things from a directory server is a bit loopy. Instead we should refactor it so that "things to be spooled" is a first-class object, with implementations depending on what we're spooling. This will make lots of our directory improvement stuff easier to implement.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21650prop140: Clients request diffs and handle diffs in replies2020-06-13T15:07:02ZNick Mathewsonprop140: Clients request diffs and handle diffs in repliesFor the final piece of prop140, clients should ask for consensus diffs as appropriate, and handle them if they're received.
We may need proposal extensions here:
* Should clients begin doing this immediately, or should there be a tris...For the final piece of prop140, clients should ask for consensus diffs as appropriate, and handle them if they're received.
We may need proposal extensions here:
* Should clients begin doing this immediately, or should there be a tristate where "default" means "look at the consensus"?
* Should there be an option -- maybe for testing, maybe not -- that forces clients to look for directory guards that support consensus diffs?Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21649prop140: Caches serve diffs on request2020-06-13T15:07:02ZNick Mathewsonprop140: Caches serve diffs on requestWhen a client asks for it, caches should serve consensus diffs.
We'll want to integrate this with the new compression logic that ahf is proposing.When a client asks for it, caches should serve consensus diffs.
We'll want to integrate this with the new compression logic that ahf is proposing.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21648prop140: Caches generate diffs as appropriate2020-06-13T15:07:01ZNick Mathewsonprop140: Caches generate diffs as appropriateDirectory caches should pre-generate and pre-compress diffs as appropriate. It might be best to do this in a different thread or process. It might be best to pre-generate some and create others on demand.Directory caches should pre-generate and pre-compress diffs as appropriate. It might be best to do this in a different thread or process. It might be best to pre-generate some and create others on demand.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21647Prop140: directory caches cache multiple past diffs or consensuses2020-06-13T15:07:00ZNick MathewsonProp140: directory caches cache multiple past diffs or consensusesTo implement prop140, we need to store history about the consensus documents. This should be fairly simple if we have #21645 to work from.To implement prop140, we need to store history about the consensus documents. This should be fairly simple if we have #21645 to work from.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21646prop140/compression: Refactor "directory request" code2020-06-13T15:06:59ZNick Mathewsonprop140/compression: Refactor "directory request" codeOur current notion of "what is a directory request" includes a bunch of fields that are strewn around directory connections and passed to different functions. It would be nice to instead have a "directory request" type that we created a...Our current notion of "what is a directory request" includes a bunch of fields that are strewn around directory connections and passed to different functions. It would be nice to instead have a "directory request" type that we created and passed around as appropriate. This would make it easier to test our request generation/parsing code.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21645prop140 / compression: Unified directory cache backend2020-06-13T15:06:59ZNick Mathewsonprop140 / compression: Unified directory cache backendWith prop140 and/or ahf's compression proposal, we'll need to cache more documents than previously, to include multiple consensuses, diffs, compressed consensuses, etc. We could do this in an ad-hoc way, but we'll probably be better off...With prop140 and/or ahf's compression proposal, we'll need to cache more documents than previously, to include multiple consensuses, diffs, compressed consensuses, etc. We could do this in an ad-hoc way, but we'll probably be better off with a real abstraction here.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21644Prop140: Fuzz the diff and patch code2020-06-13T15:06:58ZNick MathewsonProp140: Fuzz the diff and patch codeNow that we have fuzzing support, it would be ridiculous to start using diff/patch code without testing it through our fuzzers. So, let's do that.Now that we have fuzzing support, it would be ridiculous to start using diff/patch code without testing it through our fuzzers. So, let's do that.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/21643Prop140: Extract, test, revise, and clean the diff code2020-06-13T15:06:58ZNick MathewsonProp140: Extract, test, revise, and clean the diff codeStep one of getting the consensus diff support of prop140 completed is to get the diff and patch code merged. Let's do this separately from the directory code, since that code is hairier, and may need more work.Step one of getting the consensus diff support of prop140 completed is to get the diff and patch code merged. Let's do this separately from the directory code, since that code is hairier, and may need more work.Tor: 0.3.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/13339Prop140: Complete Consensus diffs / Merge GSoC project2020-06-13T14:39:24ZmvdanProp140: Complete Consensus diffs / Merge GSoC projectGoogle Summer of Code finished over a month ago, and during this time I've been tidying up my code a bit and reading it for the merge. You will find it on github:
https://github.com/mvdan/tor
This ticket is for the sole purpose of foll...Google Summer of Code finished over a month ago, and during this time I've been tidying up my code a bit and reading it for the merge. You will find it on github:
https://github.com/mvdan/tor
This ticket is for the sole purpose of following the merge process and its progress. But as always I'm on IRC and mail if you want to contact me directly.
I just rebased against master this morning. Nick and Sebastian have been reviewing my code over the summer, but of course more sets of eyes are needed.
The test coverage for the diff generation and application is fine (see test_consdiff.c), but there aren't any tests for the stuff I wrote to wire it into serving and fetching consensus diffs. Not really sure how to go about that, can't really promise I'd have the time to dive into it.
And regarding commit messages and changelog entries, I pretty much went with my instinct. Chances are they can be improved - the commit messages for future reference and the changelog entries for future release changelogs - so criticism is welcome.Tor: 0.3.1.x-finalNick MathewsonNick Mathewson