The Tor Project issueshttps://gitlab.torproject.org/groups/tpo/-/issues2020-06-27T13:52:28Zhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27275Stop reporting appveyor on_success, because it's noisy2020-06-27T13:52:28ZteorStop reporting appveyor on_success, because it's noisyAppveyor currently reports:
```
on_success:
- cmd: ... success
on_failure:
- cmd: ... failure
```
which is really noisy.
Travis currently reports:
```
irc:
...
on_success: change
on_failure: change
```
which seems ok.
We ...Appveyor currently reports:
```
on_success:
- cmd: ... success
on_failure:
- cmd: ... failure
```
which is really noisy.
Travis currently reports:
```
irc:
...
on_success: change
on_failure: change
```
which seems ok.
We should make Appveyor notifications more like Travis.Tor: 0.3.5.x-finalhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27274ASan on OSX Travis is incompatible with Rust's santiziers2020-06-27T13:52:28ZTracASan on OSX Travis is incompatible with Rust's santiziersIn helping to debug https://trac.torproject.org/projects/tor/ticket/25386 I've found that an unfortunate case is being hit where `cargo test` is using rustc's copy of the asan runtime instead of the system's copy, which causes problems d...In helping to debug https://trac.torproject.org/projects/tor/ticket/25386 I've found that an unfortunate case is being hit where `cargo test` is using rustc's copy of the asan runtime instead of the system's copy, which causes problems due to presumably version mismatches between them.
The error looks like https://travis-ci.com/alexcrichton/tor/jobs/141409956 and only starts to show up after https://trac.torproject.org/projects/tor/ticket/27273 and https://trac.torproject.org/projects/tor/ticket/27272 are fixed.
AFAIK the only "fix" for this is to basically just delete the sanitizer runtimes in the Rust sysroot as they're not used anyway, but I'll try to keep thinking and see if there's a better solution!
**Trac**:
**Username**: alexcrichtonTor: 0.3.5.x-finalhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27273ASan fails to link on Travis due to rustc and linker arguments2020-06-27T13:52:28ZTracASan fails to link on Travis due to rustc and linker argumentsIn helping to debug https://trac.torproject.org/projects/tor/ticket/25386 I've found that a number of the link errors are attributable to the linker invocation on Travis. The recent `link_rust.sh` script is definitely necessary I think, ...In helping to debug https://trac.torproject.org/projects/tor/ticket/25386 I've found that a number of the link errors are attributable to the linker invocation on Travis. The recent `link_rust.sh` script is definitely necessary I think, except that I believe it needs a few tweaks:
* The `-lasan` argument should be replaced with `-fsanitizer=address` I believe to ensure that the C compiler links all of its relevant libraries (as they may have different names and different numbers of libs on some platforms I think).
* Second, the Rust compiler by default passes `-nodefaultlibs` to the linker which means that the linker script also passes this along. It looks, however, like this is incompatible with `-fsanitizer=address`, causing link errors. This argument should be safe to strip out though.
The first bullet was easy enough to fix locally and the second bullet was fixable by changing `link_rust.sh` to a `bash` script and then using something like:
```
linker_args="$@"
$CCLD @RUST_LINKER_OPTIONS@ ${linker_args//-nodefaultlibs/}
```
**Trac**:
**Username**: alexcrichtonTor: 0.3.5.x-finalhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27272ASan is incompatible with Rust's jemalloc on Travis2020-06-27T13:52:29ZTracASan is incompatible with Rust's jemalloc on TravisIn helping to debug https://trac.torproject.org/projects/tor/ticket/25386 I've found that many of the segfaults at runtime are attributable to Rust pulling in jemalloc by default for tests, which apparently doesn't play well with ASan wh...In helping to debug https://trac.torproject.org/projects/tor/ticket/25386 I've found that many of the segfaults at runtime are attributable to Rust pulling in jemalloc by default for tests, which apparently doesn't play well with ASan when linked in.
I've found that using code like:
```
#[global_allocator]
static ALLOCATOR: std::alloc::System = std::alloc::System;
```
is enough to solve the problem. This tells Rust that it should use the system allocator (e.g. the malloc/free symbols) instead of jemalloc. This was stabilized very recently in Rust, though, so using it may not be so trivial!
In some local testing I was able to get away with adding the above declaration to the `tor_allocate` crate for the most part, but crates like `crypto`, `external`, and `smartlist` don't already link to `tor_allocate` and needed the above declaration with a `#[cfg(test)]` as well. Once this was all added though I mostly no longer saw segfaults related to jemalloc and ASan
**Trac**:
**Username**: alexcrichtonTor: 0.3.5.x-finalhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27252Reduce the number of travis jobs2020-06-27T13:52:29ZteorReduce the number of travis jobsAfter legacy/trac#24629, we have some follow-up work.
In 0.2.9 and later:
* work out if we really need clang and gcc on Linux and macOS
In 0.3.2 and later:
* reduce the number of rust online travis jobs, to reduce travis network failuresAfter legacy/trac#24629, we have some follow-up work.
In 0.2.9 and later:
* work out if we really need clang and gcc on Linux and macOS
In 0.3.2 and later:
* reduce the number of rust online travis jobs, to reduce travis network failuresTor: 0.3.5.x-finalteorteorhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27251Add single-onion-v23-ipv6-md to make test-network-all2021-09-30T13:45:56ZteorAdd single-onion-v23-ipv6-md to make test-network-allWhen IPv6 single onion services work, we should add the single-onion-v23-ipv6-md to TEST_CHUTNEY_FLAVORS_IPV6in src/test/include.am.
We can also remove single-onion-ipv6-md, because it's redundant.When IPv6 single onion services work, we should add the single-onion-v23-ipv6-md to TEST_CHUTNEY_FLAVORS_IPV6in src/test/include.am.
We can also remove single-onion-ipv6-md, because it's redundant.Tor: 0.4.1.x-finalteorteorhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27250Duplicate line '#include "lib/net/address.h' in src/test/test_address.c2020-06-27T13:52:29ZNeel Chauhanneel@neelc.orgDuplicate line '#include "lib/net/address.h' in src/test/test_address.cWill submit a PR shortly.Will submit a PR shortly.Tor: 0.3.5.x-finalNeel Chauhanneel@neelc.orgNeel Chauhanneel@neelc.orghttps://gitlab.torproject.org/tpo/core/tor/-/issues/27248Can we make our node-related structures more efficient?2020-07-28T22:49:09ZNick MathewsonCan we make our node-related structures more efficient?Our profiles say that we're using a lot of RAM for our directory stuff. Some of that is for silliness like keeping text of directory stuff in RAM (yuck), but surely some of it is due to all our node_t/microdesc_t/routerstatus_t stuff. ...Our profiles say that we're using a lot of RAM for our directory stuff. Some of that is for silliness like keeping text of directory stuff in RAM (yuck), but surely some of it is due to all our node_t/microdesc_t/routerstatus_t stuff. Can we make that smaller?Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27247Clients are using RAM for cached_dir_t2020-06-27T13:52:29ZNick MathewsonClients are using RAM for cached_dir_tAccording to the profiles, clients are storing a cached_dir_t object for the consensus they have. This is ridiculous -- they don't need that, and they already have it somewhere else.
Directory caches probably don't need one of these ei...According to the profiles, clients are storing a cached_dir_t object for the consensus they have. This is ridiculous -- they don't need that, and they already have it somewhere else.
Directory caches probably don't need one of these either -- they could mmap it instead.Tor: 0.3.5.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27246Can we use less space for RSA onion keys on clients?2020-06-27T13:52:30ZNick MathewsonCan we use less space for RSA onion keys on clients?Our heap profiles say that we're using a HUGE amount of storage for RSA onion keys. That's pretty sad, considering that we only use them for legacy v2 hidden service stuff.
Maybe we can only parse them on-demand, when we need them?
Ma...Our heap profiles say that we're using a HUGE amount of storage for RSA onion keys. That's pretty sad, considering that we only use them for legacy v2 hidden service stuff.
Maybe we can only parse them on-demand, when we need them?
Maybe (if the openssl format is much more expensive than the raw asn1) we can store them in asn1, and only convert them to EVP_PKEY objects when we need them?Tor: 0.3.5.x-finalhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27245Don't store (micro)descriptor text on the heap so much.2020-07-28T22:48:57ZNick MathewsonDon't store (micro)descriptor text on the heap so much.We could use less RAM for our (micro)descriptor text if we kept the .new file mmapped, so that we didn't need to use the heap to hold them.We could use less RAM for our (micro)descriptor text if we kept the .new file mmapped, so that we didn't need to use the heap to hold them.Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27244Use mmap to hold our cached consensus, when we even need it stored in RAM at ...2020-06-27T13:52:30ZNick MathewsonUse mmap to hold our cached consensus, when we even need it stored in RAM at all.Tor: 0.4.0.x-finalhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27243Keep much less directory info in RAM on clients2020-07-28T22:48:27ZNick MathewsonKeep much less directory info in RAM on clientsAccording to the memory profiles we have, a huge amount of the memory usage on clients is spent on our directory structures.
This is a parent ticket.According to the memory profiles we have, a huge amount of the memory usage on clients is spent on our directory structures.
This is a parent ticket.Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27241Extract information from more kinds of wedged directory connections.2021-11-06T13:26:13ZNick MathewsonExtract information from more kinds of wedged directory connections.In main.c, we do:
```
/* This check is temporary; it's to let us know whether we should consider
* parsing partial serverdesc responses. */
if (conn->purpose == DIR_PURPOSE_FETCH_SERVERDESC &&
connection_get_inbuf_le...In main.c, we do:
```
/* This check is temporary; it's to let us know whether we should consider
* parsing partial serverdesc responses. */
if (conn->purpose == DIR_PURPOSE_FETCH_SERVERDESC &&
connection_get_inbuf_len(conn) >= 1024) {
log_info(LD_DIR,"Trying to extract information from wedged server desc "
"download.");
connection_dir_reached_eof(TO_DIR_CONN(conn));
```
We should also, at a minimum, do this for microdesc downloads.https://gitlab.torproject.org/tpo/core/tor/-/issues/27237When the network doesn't have any exits, use the mid weight as the exit weight2020-06-27T13:52:30ZteorWhen the network doesn't have any exits, use the mid weight as the exit weightIf we're building onion service paths, all our circuit ends will be mids.
```
/* if the consensus has no exits, treat the exit fraction as 100% */
if (router_have_consensus_path() != CONSENSUS_PATH_EXIT) {
f_exit = 1.0;
}
```
...If we're building onion service paths, all our circuit ends will be mids.
```
/* if the consensus has no exits, treat the exit fraction as 100% */
if (router_have_consensus_path() != CONSENSUS_PATH_EXIT) {
f_exit = 1.0;
}
```
Bug fix on 55ad54e0146 in 0.2.6.2-alpha.Tor: 0.3.4.x-finalteorteorhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27236When checking usable exit descriptors, always check exit policies2020-06-27T13:52:30ZteorWhen checking usable exit descriptors, always check exit policiesWe need to check exit policies whenever we're counting the usable exit descriptors. At the moment, we only check exit policies for exits in ExitNodes.
```
SMARTLIST_FOREACH_BEGIN(myexits_unflagged, const node_t *, node) {
if (n...We need to check exit policies whenever we're counting the usable exit descriptors. At the moment, we only check exit policies for exits in ExitNodes.
```
SMARTLIST_FOREACH_BEGIN(myexits_unflagged, const node_t *, node) {
if (node_has_preferred_descriptor(node, 0) &&
node_exit_policy_rejects_all(node)) {
SMARTLIST_DEL_CURRENT(myexits_unflagged, node);
/* this node is not actually an exit */
np--;
/* this node is unusable as an exit */
nu--;
}
} SMARTLIST_FOREACH_END(node);
```Tor: 0.3.4.x-finalteorteorhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27229Create fuzzing harness to compare C/Rust Functionality2022-06-17T13:42:06ZChelsea KomloCreate fuzzing harness to compare C/Rust FunctionalityIn porting over functionality to Rust, it can be useful to compare functionality between C/Rust. While ideally unit tests should catch most behavior, having a fuzzer to catch edge cases can be handy.
We should write a test harness that ...In porting over functionality to Rust, it can be useful to compare functionality between C/Rust. While ideally unit tests should catch most behavior, having a fuzzer to catch edge cases can be handy.
We should write a test harness that fuzzes C/Rust similar functions and compares their output. Ideally, a test would look something like this:
1. Setup C test case
2. Set up Rust test case
3. Provide both functions with the same generated arbitrary input
4. Compare results
It is worth noting that in most cases we will want to improve behavior when porting to Rust, but this tool can be useful for small cases where we want bitwise identical functions.
Alex Crichton recommended looking at https://github.com/alexcrichton/ctest as one option- it is worth looking at what a simple test harness should be and how to have code be reusable between tests.https://gitlab.torproject.org/tpo/core/tor/-/issues/27228pathbias_should_count(): Bug: Circuit X is now being counted2020-06-27T13:52:31Ztraumschulepathbias_should_count(): Bug: Circuit X is now being countedLooking through my tor.log I found 64 lines like this during the last day (out of 66 BUG lines):
```
Aug 20 03:09:58.000 [info] {BUG} pathbias_should_count(): Bug: Circuit 355 is now being counted despite being ignored in the past. Purpo...Looking through my tor.log I found 64 lines like this during the last day (out of 66 BUG lines):
```
Aug 20 03:09:58.000 [info] {BUG} pathbias_should_count(): Bug: Circuit 355 is now being counted despite being ignored in the past. Purpose is Measuring circuit timeout, path state is new (on Tor 0.3.5.0-alpha-dev )
```
The other BUGs were:
```
Aug 20 03:19:00.000 [info] {BUG} pathbias_count_build_success(): Bug: Succeeded circuit is in strange path state new. Circuit is a Measuring circuit timeout currently open. (on Tor 0.3.5.0-alpha-dev )
Aug 20 03:19:03.000 [info] {CIRC} extend_info_from_node(): Including Ed25519 ID for $name at $address
```
(Maybe relevant: legacy/trac#24966, legacy/trac#24903, legacy/trac#19535, legacy/trac#8196, legacy/trac#8081)Tor: unspecifiedtraumschuletraumschulehttps://gitlab.torproject.org/tpo/core/tor/-/issues/27226Crash in tortls/cert_matches_key with openssl 1.0.2p2020-06-27T13:52:31ZNick MathewsonCrash in tortls/cert_matches_key with openssl 1.0.2pOur unit test, `tortls/cert_matches_key`, does some questionable stuff that is not compatible with openssl 1.0.2p.
Namely, it calls `EVP_PKEY_asn1_new(999, 0, NULL, NULL)`, which now returns NULL.
Looking at the test, I'm not sure what...Our unit test, `tortls/cert_matches_key`, does some questionable stuff that is not compatible with openssl 1.0.2p.
Namely, it calls `EVP_PKEY_asn1_new(999, 0, NULL, NULL)`, which now returns NULL.
Looking at the test, I'm not sure what it's trying to do with this -- it's making a bogus public key method with a "compare" function that will always return "1". Later, it's using this thing to construct bogus PKEY objects.
This, like a lot of other tortls.c tests, is way too tightly coupled to openssl internals.Tor: 0.2.9.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/tpo/core/tor/-/issues/27225Perform fewer allocations in summarize_protocol_flags()2020-06-27T13:52:31ZNick MathewsonPerform fewer allocations in summarize_protocol_flags()According to our profiles, summarize_protocol_flags() does a huge number of allocations -- probably because it parses the same flags over and over.
Probably it would be simplest just to memoize the output.According to our profiles, summarize_protocol_flags() does a huge number of allocations -- probably because it parses the same flags over and over.
Probably it would be simplest just to memoize the output.Tor: 0.4.0.x-finalNick MathewsonNick Mathewson