Notes for implementing a directory protocol server

We had a discussion on irc about some things that one could get wrong while implemneting a directory server.

I've c&p these here. They should probably be incorporated in the spec as non-normative notes.

17:31 <+nickm> cve, diziet: there are a few things in the directory protocol 
               that create subtle difficulties you might want to know about
17:32 <+nickm> one of them is that there are numerous URLs of the format 
               A+B+C... or A-B-C... where A, B, C etc are all digests of 
               something.
17:32 <+nickm> For microdescriptors and descriptors, streaming compression is 
               essential for good bw usage: one desc compresses poorly, but 10 
               compress well.
17:33 <+nickm> It's important not to discard microdescriptors and descriptors 
               just because they're not listed in the consensus: They may be 
               listed in a future consensus, and we don't want to re-download 
               them
17:34 <+nickm> consensus diffs save a lot of BW, but they need to be 
               pre-computed, and generating them requires us to store multiple 
               past consensuses.
17:34 <+Diziet> These don't't sound like things that Rust web frameworks would 
                have much trouble with.  Rust web frameworks are not very 
                framework-y compared to many other languages'.  (My only other 
                real experience is Mason...)
17:34 <+nickm> consensus diffs and consensuses want to be pre-compressed: they 
               are big enough to benefit from stuff like lzma2, which is costly.
17:35 <+nickm> Consensuses and consensus diffs (and their compressed forms) 
               should be mmapped, not kept in RAM...
17:35 <+nickm> and _all_ of these documents should be spooled by reference, not 
               by copying all the requested documents into a huge queue
17:36 <+nickm> (that part should be much easier in rust)
17:38 <+opara> re frameworks: i've had good experience with axum, it's easy to 
               write middleware (like compression middleware), but it's 
               probably a bit too heavy (in terms of dependencies) for arti
17:41 <+nickm> (is there a good place to record all of the above? C tor found 
               them all through experience)
17:41 <+nickm> ((and I worry that maybe they didn't make it into the spec when 
               they don't affect the wire protocol))
17:46 <+Diziet> I think those special considerations about how to implement the 
                protocol for perf would make sense as non-normative comments in 
                the spec.
17:47 <+Diziet> The part about the A+B+C is just stated explicitly so doesn't 
                need another note :-).
17:48 <+Diziet> (A careful programmer who is thinking about availability and 
                attack resistance is going to come to some of these conclusions 
17:49 <+nickm> I don't have time to add these to the spec today, but is there a 
               pad i could drop them in?
17:55 <+Diziet> I'll transfer them into my personal todo container on gitlab 
                for now, and cve and I can decide where they might go.
17:55 <+Diziet> Or better, maybe just a ticket against torspec