We're calling mallocfn() and reallocfn() in the HT_GENERATE macro
with the result of a product.  But that makes any sane analyzer
worry about overflow.

This patch keeps HT_GENERATE having its old semantics, since we
aren't the only project using ht.h.  Instead, define a HT_GENERATE2
that takes a reallocarrayfn.

Specifically, count the age of the data queued in a linked directory
connection's buffers when counting a stream's age.

Another part of 11792.

Part of 11792.

(Uses the zlib-endorsed formula for memory needs for inflate/deflate
from "zconf.h".)

Fixes issue 10116

Fix part of bug 11972

Breaks compilation.

Used this coccinelle script:

@@
identifier c;
typedef circuit_t;
iterator name TOR_LIST_FOREACH;
iterator name SMARTLIST_FOREACH_BEGIN;
statement S;
@@
- circuit_t *c;
   ...
- TOR_LIST_FOREACH(c, \(&global_circuitlist\|circuit_get_global_list()\), head)
+ SMARTLIST_FOREACH_BEGIN(circuit_get_global_list(), circuit_t *, c)
  S
+ SMARTLIST_FOREACH_END(c);

Check for consistency between the queued destroy cells and the marked
circuit IDs.  Check for consistency in the count of queued destroy
cells in several ways.  Check to see whether any of the marked circuit
IDs have somehow been marked longer than the channel has existed.

The point of the "idle timeout" for connections is to kill the
connection a while after it has no more circuits.  But using "last
added a non-padding cell" as a proxy for that is wrong, since if the
last circuit is closed from the other side of the connection, we
will not have sent anything on that connection since well before the
last circuit closed.

This is part of fixing 6799.

When applied to 0.2.5, it is also a fix for 12023.

Since this is critical-path, let's tune the value we pass to
csiphash a little so it fits into one whole round.

Fixes ticket 11750.

dana koch authored Apr 29, 2014 and Nick Mathewson committed Apr 29, 2014

This was previously satisfied by using a temporary variable, but there
are three other instances in circuitlist.c that gcc is now bothered by,
so now introduce a CONST_TO_ORIGIN_CIRCUIT that takes a const
circuit_t instead.

If we fail in circuit_get_by_rend_token_and_purpose because the
circuit has no rend_info, don't try to reference fiends from its
rend_info when logging an error.  Bugfix on 8b9a2cb6, which is
going into Tor 0.2.5.4-alpha.

dana koch authored Apr 15, 2014 and Nick Mathewson committed Apr 15, 2014

A new set of unit test cases are provided, as well as introducing
an alternative paradigm and macros to support it. Primarily, each test
case is given its own namespace, in order to isolate tests from each
other. We do this by in the usual fashion, by appending module and
submodule names to our symbols. New macros assist by reducing friction
for this and other tasks, like overriding a function in the global
namespace with one in the current namespace, or declaring integer
variables to assist tracking how many times a mock has been called.

A set of tests for a small-scale module has been included in this
commit, in order to highlight how the paradigm can be used. This
suite gives 100% coverage to status.c in test execution.

Found by testing with chutney.  The old behavior was "fail an
assertion", which obviously isn't optimal.

Bugfix on 8b9a2cb6; bug not in any
released version.

(This wasn't supposed to get committed turned-on.)

In circuitlist_free_all, we free all the circuits, removing them from
the map as we go, but we weren't actually freeing the placeholder
entries that we use to indicate pending DESTROY cells.

Fix for bug 11278; bugfix on the 7912 code that was merged in
0.2.5.1-alpha

Also remove its helper function.

Doing this as part of the patch for #9683 to prevent possible bugs
down the line

This contains the obvious implementation using the circuitmux data
structure.  It also runs the old (slow) algorithm and compares
the results of the two to make sure that they're the same.

Needs review and testing.

In a couple of places, to implement the OOM-circuit-killer defense
against sniper attacks, we have counters to remember the age of
cells or data chunks.  These timers were based on wall clock time,
which can move backwards, thus giving roll-over results for our age
calculation.  This commit creates a low-budget monotonic time, based
on ratcheting gettimeofday(), so that even in the event of a time
rollback, we don't do anything _really_ stupid.

A future version of Tor should update this function to do something
even less stupid here, like employ clock_gettime() or its kin.

On busy servers, this function takes up something like 3-7% in
different profiles, and gets invoked every time we need to participate
as the midpoint in a hidden service.

So maybe walking through a linked list of all the circuits here wasn't
a good idea.

This gets us up to no interesting untested new or changed lines for
the 10169 code.

This patch splits out some of the functions in OOM handling so that
it's easier to check them without involving the rest of Tor or
requiring that the circuits be "wired up".

Otherwise freeing buffers won't help for a little while.

(Based on a suggestion by arma at #9777)

Right now this accounts for about 1% of circuits over all, but if you
pick a guard that's running 0.2.3, it will be about 6% of the circuits
running through that guard.

Making sure that every circuit has at least one ntor link means that
we're getting plausibly good forward secrecy on every circuit.

This implements ticket 9777,

Also, aggressively clear the buffers to try to make their bytes go
away fast rather than waiting for the close-marked-connection code
to get 'em.