David Goulet authored 7 years ago and Nick Mathewson committed 7 years ago

Fixes #21715

Signed-off-by: David Goulet <dgoulet@torproject.org>

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

This patch splits up `tor_compress_memory_level()` into static functions
in the individual compression backends, which allows us to tune the
values per compression backend rather than globally.

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

Use a switch-statement in `tor_compress()` and `tor_uncompress()` for
the given `compress_method_t` parameter. This allows us to have the
compiler detect if we forgot add a handler in these functions for a
newly added enumeration value.

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

This patch ensures that Tor checks if a given compression method is
supported before printing the version string when calling `tor
--library-versions`.

Additionally, we use the `tor_compress_supports_method()` to check if a
given version is supported for Tor's start-up version string, but here
we print "N/A" if a given compression method is unavailable.

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

This patch adds `tor_compress_version_str()` and
`tor_compress_header_version_str()` to get the version strings of the
different compression schema providers. Both functions returns `NULL` in
case a given `compress_method_t` is unknown or unsupported.

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

This patch adds support for checking if a given `compress_method_t` is
supported by the currently running Tor instance using
`tor_compress_supports_method()`.

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

This patch adds the `tor_compress_get_total_allocation()` which returns
an approximate number of bytes currently in use by all the different
compression backends.

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

This patch adds support for enabling support for Zstandard to our configure
script. By default, the --enable-zstd option is set to "auto" which means if
libzstd is available we'll build Tor with Zstandard support.

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

This patch adds support for enabling support for LZMA to our configure
script. By default, the --enable-lzma option is set to "auto" which
means if liblzma is available we'll build Tor with LZMA support.

See: https://bugs.torproject.org/21662

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

This patch changes the way `tor_compress_new()`,
`tor_compress_process()`, and `tor_compress_free()` handles different
compression methods. This should give us compiler warnings in case an
additional compression method is added, but the developer forgets to add
handlers in the three aforementioned functions.

See https://bugs.torproject.org/21663

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

See https://bugs.torproject.org/21663

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

See https://bugs.torproject.org/21663

Alexander Hansen Færøy authored 7 years ago and Nick Mathewson committed 7 years ago

This patch refactors the `torgzip` module to allow us to extend a common
compression API to support multiple compression backends.

Additionally we move the gzip/zlib code into its own module under the
name `compress_zlib`.

See https://bugs.torproject.org/21664

The item referred to the cdm_ht_set_status() case where the item was
not already in the hashtable.  But that already happens naturally
when we scan the directory on startup... and we already have a test
for that.

This brings us back up to ~94% coverage

Found by previous test.

This commit adds some helper functions to look up the diff from one
consensus and to make sure that applying it leads to another.  Then
we add them throughout the existing test cases.  Doing this turned
up a reference-leaking bug in consensus_diff_worker_replyfn.

Also, add a list of additional tests to write.

In several places in the old code, we had problems that only an
in-memory index of diff status could solve, including:
   * Remembering which diffs were in-progress, so that we didn't
     re-launch them.
   * Remembering which diffs had failed, so that we didn't try to
     recompute them over and over.
   * Having a fast way to look up the diff from a given consensus to
     the latest consensus of a given flavor.

This patch adds a hashtable mapping from (flavor, source diff), to
solve the problem.  It maps to a cache entry handle, rather than to
a cache entry directly, so that it doesn't affect the reference
counts of the cache entries, and so that we don't otherwise need to
worry about lifetime management.

I'll be using this a lot in the hashtable tweaks here.

This conscache flag tells conscache that it should munmap the
document as soon as reasonably possible, since its usage pattern is
expected to not have a lot of time-locality.

This will allow us to have weak references to cache entries.