The Tor Project issueshttps://gitlab.torproject.org/groups/tpo/-/issues2023-11-13T15:40:07Zhttps://gitlab.torproject.org/tpo/core/arti/-/issues/799Bootstrap event sender doesn't always report readiness correctly2023-11-13T15:40:07ZetaBootstrap event sender doesn't always report readiness correctlyIt's unclear whether this is working as intended or not, but it appears that loading a bootstrapped directory successfully from cache won't go and update the bootstrap status reported through `BootstrapEvents` as hitting 100%. The most I...It's unclear whether this is working as intended or not, but it appears that loading a bootstrapped directory successfully from cache won't go and update the bootstrap status reported through `BootstrapEvents` as hitting 100%. The most I managed to get it to was
```
2023-03-28T15:26:53.339788Z DEBUG arti_client::status: 45%: connecting successfully; directory is fetching authority certificates (8/8)
```
Since it never sets `ready_for_traffic` either, consumers depending on the bootstrap event sender might mistakenly think that bootstrap is still ongoing.
You can work around this in client code by calling `TorClient::bootstrap` manually and reporting completion when that function returns, but it'd be nice to have the event sender fixed;
- it seems weird that `TorClient::bootstrap` can return without `ready_for_traffic` also being set;
- and it seems weird that you will only see the 100% bootstrap event state when downloading a fresh directory.
(arti 1.1.2 installed via Cargo on Linux, but the problem also happens embedded on Android)Arti: Feature parity with the C implementationNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/tpo/core/arti/-/issues/783Failure to bootstrap due to cache directory problems results in a bad end-use...2023-11-07T12:14:30ZetaFailure to bootstrap due to cache directory problems results in a bad end-user experience### Summary
While working on onionmasq, I encountered a problem with the Arti client failing to bootstrap. After some investigation, this appeared to be related to an issue with the cache. Deleting the cache entirely and restarting fixe...### Summary
While working on onionmasq, I encountered a problem with the Arti client failing to bootstrap. After some investigation, this appeared to be related to an issue with the cache. Deleting the cache entirely and restarting fixed things up, but the end-user experience here was lacking, especially in the context of integrating Arti as a library.
### Steps to reproduce:
1. Something weird happens to the cache directory
2. The Arti client no longer works
### What is the current bug behavior?
I got the following error message on calling `bootstrap()` which was resoundingly unhelpful:
```
onion_tunnel: Failed to bootstrap: tor: cache access problem: Unable to bootstrap a working directory
```
I then realised (given insider knowledge) that I had to format out the causes, and got this:
```
onion_tunnel: Caused by: Unable to bootstrap a working directory
onion_tunnel: Caused by: Bad permissions in cache directory
onion_tunnel: Caused by: File or directory /data/user/0/org.torproject.artitoyvpn/cache/arti-cache/dir_blobs/con_microdesc_sha3-256-339e2da5279de87281cfd59f0b3d0f324a15045a8540e9a71a396d68c3e74d54 not found
```
There are a number of issues here (see below).
### What is the expected behavior?
- Bootstrapping should be able to gracefully deal with cache directory problems, without user intervention
- When embedding Arti, this just looks like the entire thing is irreparably broken, when in fact it's a transient issue that could be easily resolved by clearing the cache
- Users can't always do this themselves if Arti is embedded, nor would they necessarily receive indication that they need to.
- The embedding library can't do it without relying on API-unstable error munging tactics, since no machine-readable API is exposed to ascertain that the cache dir is at fault
- The error should be correctly categorized; this doesn't look like a permissions issue
- The top-level error could possibly be renamed, too: I initially read 'working directory' as 'some place to put scratch files' (i.e. the Git 'working directory'), which was confusing
- (perhaps "Unable to download a Tor network directory" or similar)
- (more opinionated, but) We should strongly consider rethinking the `Error::source` idea
- A 'normal' embedder (who didn't work on the Arti team for a year) would have no idea how to get more information out of the error, and would assume the nondescript `Unable to bootstrap a working directory` line was it
- There's no standard way to print an error's causes ([`std::error::Report`](https://doc.rust-lang.org/std/error/struct.Report.html) is still unstable)
- Even if there was, the user might be transforming the error into an `io::Error` or something, or passing it across an FFI boundary; having to thread the error-cause thing throughout the end-user codebase is just a pointless form of make-work
- This might make sense once the rest of the ecosystem has caught up, but right now it's a really obscure user experience
### Environment
- Version: arti-client 0.8.1 from crates.io, embedded inside onionmasq
- Operating system: Android 13Arti: Feature parity with the C implementationhttps://gitlab.torproject.org/tpo/core/arti/-/issues/711Use cargo-semver-checks or such to enforce stability in high-level APIs2023-05-31T18:41:09ZNick MathewsonUse cargo-semver-checks or such to enforce stability in high-level APIsThe `arti-client` crate depends on, and exposes, APIs from a bunch of other crates. As such, it is tricky to make sure that we haven't made any changes that **unintentionally** break compatibility with past versions of `arti-client`.
We...The `arti-client` crate depends on, and exposes, APIs from a bunch of other crates. As such, it is tricky to make sure that we haven't made any changes that **unintentionally** break compatibility with past versions of `arti-client`.
We should open other tickets (eg #710, #712) to limit the risk of breaking changes. But it would also be good to use what tooling we can to detect breaking changes, possibly as part of CI. We could, for example, enforce that there is a breaking semver change iff there is a BREAKING comment in semver.md.
The current front-runner seems to be [`cargo-semver-checks`](https://crates.io/crates/cargo-semver-checks), which claims to have no false positives (only false negatives). I haven't been able to figure out out to make it work, but hey, who knows.Arti: Feature parity with the C implementationhttps://gitlab.torproject.org/tpo/core/arti/-/issues/707have a way to get underlying file descriptors2023-01-18T14:07:23Ztrinity-1686ahave a way to get underlying file descriptorsCurrently the TCP/UDP types returned by a Runtime are opaque: there is no way to access the underlying file descriptor.
Access to that file descriptor is a requirement to implement transparent proxy (part of #72), and I suspect it woul...Currently the TCP/UDP types returned by a Runtime are opaque: there is no way to access the underlying file descriptor.
Access to that file descriptor is a requirement to implement transparent proxy (part of #72), and I suspect it would make onionmasq#23 easier (right now I think they'll have to implement their own TcpProvider, not wrapping arti default, to be able to get that).
It'll be kind of an issue with regard to crossplarformness: Windows has no FD, it has file handles which are mostly the same, but I guess different in some ways. Also some runtimes might not have backing fds for whatever reason, so returning a fd should probably be optionalArti: Feature parity with the C implementationhttps://gitlab.torproject.org/tpo/core/arti/-/issues/634Clarify and rationalise interaction between dormancy and have-we-been-bootstr...2023-11-15T19:05:52ZIan Jacksoniwj@torproject.orgClarify and rationalise interaction between dormancy and have-we-been-bootstrapped, and dormancy codeTorClient has two kinds of dormancy: 1. `Dormancy` 2. not-having-ever-had-`boostrap()`-called.
This is confusing, both at an API level, and internally. We should rationalise this.
* [ ] Define what the modes of a `TorClient` should ...TorClient has two kinds of dormancy: 1. `Dormancy` 2. not-having-ever-had-`boostrap()`-called.
This is confusing, both at an API level, and internally. We should rationalise this.
* [ ] Define what the modes of a `TorClient` should be
* [ ] Define sensible APIs for selecting those modes, and retcon appropriate semantics for existing APIs
* [ ] Rationalise the internals so that the mode we're in has a single source of truth and can be determined and tracked as appropriate
* [ ] Abolish the proliferating crate-level `Dormant` enums, and instead do something via `SleepProvider` or something https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/845#note_2853190Arti: Feature parity with the C implementationIan Jacksoniwj@torproject.orgIan Jacksoniwj@torproject.orghttps://gitlab.torproject.org/tpo/core/arti/-/issues/609Implement octal escapes in PT parsing (or change the spec)2023-11-15T19:04:25ZetaImplement octal escapes in PT parsing (or change the spec)The pluggable transport spec states that some strings passed from PT to host can be Tor CStrings:
```
The MESSAGE value is a human readable string formatted by the PT. The
<Message> contains the log message which can be a String o...The pluggable transport spec states that some strings passed from PT to host can be Tor CStrings:
```
The MESSAGE value is a human readable string formatted by the PT. The
<Message> contains the log message which can be a String or CString (see
section 2 in control-spec.txt).
```
According to control-spec.txt, a CString can include octal escapes:
```
in a CString, the escapes "\n", "\t", "\r", and the octal escapes
"\0" ... "\377" represent newline, tab, carriage return, and the
256 possible octet values respectively.
```
We currently don't implement this, but maybe we should? (Or, we should change the spec to clarify that you can't use these and simplify that way.)
---
The following discussion from !779 should be addressed:
- [ ] @nickm started a [discussion](https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/779#note_2845450):
> Let's open a ticket for this, and additionally add some kind of a "not implemented" error.
>
> That way if anybody actually uses these escapes, we'll learn about it instead of getting silent miscomputations.Arti: Feature parity with the C implementationhttps://gitlab.torproject.org/tpo/core/arti/-/issues/546Arti should provide a SOCKS proxy as a library2023-04-25T15:36:50ZIan Jacksoniwj@torproject.orgArti should provide a SOCKS proxy as a libraryCurrently we only provide our SOCKS functionality as unstable experimental APIs in the `arti` library crate.
We ought to provide this functionality as a proper API.Currently we only provide our SOCKS functionality as unstable experimental APIs in the `arti` library crate.
We ought to provide this functionality as a proper API.Arti: Feature parity with the C implementationIan Jacksoniwj@torproject.orgIan Jacksoniwj@torproject.orghttps://gitlab.torproject.org/tpo/core/arti/-/issues/1262Rethink descriptor publisher rate-limiting2024-02-01T18:10:55Zgabi-250Rethink descriptor publisher rate-limitingThe following discussion from !1951 should be addressed:
- [ ] @Diziet started a [discussion](https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/1951#note_2991914): (+1 comment)
> So, suppose it's 50s since we last uploa...The following discussion from !1951 should be addressed:
- [ ] @Diziet started a [discussion](https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/1951#note_2991914): (+1 comment)
> So, suppose it's 50s since we last uploaded. We reach this point and see that `duration_since_upload` is 50s, which is less than `UPLOAD_RATE_LIM_THRESHOLD` (60s).
>
> Then we call `start_rate_limit(60s)`. `start_rate_limit` calls `runtime.now()` and adds its argument, so scheduling a wakeup 60s from now.
>
> We will upload again 110s after the last upload. I think though, that we should do it 60s after.
>
> I think the root cause of this bug is the *storage* of a separate "we are rate limited" state in the reactor state, and using it to control the upload logic. Whether "we are rate limited" is really just "is the last upload more than `UPLOAD_RATE_LIM_THRESHOLD` ago" - ie, we could recalculate that on each loop iteration.
>
> In terms of `PublishStatus` (the status reporting output) I'm not sure "we are rate limited" is a particularly useful status to advertise. I think it's an entirely normal condition.
>
> Also we should perhaps randomise this?
>
> OTOH I don't think either of these questions are a blockers for this MR. The code here is a lot nicer, so thanks :-).Arti: Onion service supportgabi-250gabi-250https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/148Wrong domain of GitLab's mail server certificate2024-02-08T16:05:24ZMynacolWrong domain of GitLab's mail server certificateI wanted to reply to a GitLab issue by mail, but my mail server refused to send it, as the TLS certificate could not be verified. My mail server is configured to strictly verify the respective certificates.
The mail was headed to `[...]...I wanted to reply to a GitLab issue by mail, but my mail server refused to send it, as the TLS certificate could not be verified. My mail server is configured to strictly verify the respective certificates.
The mail was headed to `[...]@gitlab.torproject.org`. My mail server queried the MX record of gitlab.torproject.org, but only got a CNAME response, which leads to gitlab-02.torproject.org that points to the right IP addresses. Now my mail server expected a TLS certificate for gitlab.torproject.org, but your postfix provided a certificate for gitlab-02.torproject.org, which my mail server regarded as invalid.
The easiest way to fix this is to add a MX record to gitlab.torproject.org pointing at gitlab-02.torproject.org. That could even help with mail deliverability.
Alternatively, you can provide a certificate for gitlab.torproject.org from your mail server just like on the website.
Maybe the test page on [internet.nl](https://internet.nl/mail/gitlab.torproject.org/1127446/) helps you too.improve mail servicesJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/core/arti/-/issues/1252OnionServicePrefs2024-01-23T16:54:08Zgabi-250OnionServicePrefsThe following discussion from !1887 should be addressed:
- [ ] @nickm started a [discussion](https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/1887#note_2984621): (+5 comments)
> Here's a thought. What would you think ...The following discussion from !1887 should be addressed:
- [ ] @nickm started a [discussion](https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/1887#note_2984621): (+5 comments)
> Here's a thought. What would you think about adding a new `prefs` argument to OnionService here? It could, for example, include information about whether we should construct the keys if they are not already found.Arti: Onion service supporthttps://gitlab.torproject.org/tpo/core/arti/-/issues/1250Add CLI tests for `arti hss`2024-01-17T13:57:38Zgabi-250Add CLI tests for `arti hss`Arti: Onion service supporthttps://gitlab.torproject.org/tpo/core/tor/-/issues/40907Starting tor with --DormantOnFirstStartup doesn't set GETINFO dormant to 12024-01-24T19:08:12ZCrazyChaozStarting tor with --DormantOnFirstStartup doesn't set GETINFO dormant to 1### Summary
Starting tor with --DormantOnFirstStartup 1 on a new DataDirectory doesn't set GETINFO dormant to 1
### Steps to reproduce:
1. start tor with ```tor --DormantOnFirstStartup 1 --DataDirectory some-empty-dir```
2. do a ```GE...### Summary
Starting tor with --DormantOnFirstStartup 1 on a new DataDirectory doesn't set GETINFO dormant to 1
### Steps to reproduce:
1. start tor with ```tor --DormantOnFirstStartup 1 --DataDirectory some-empty-dir```
2. do a ```GETINFO dormant```
### What is the current bug behavior?
GETINFO dormant returns 0, as if it weren't in dormant mode
### What is the expected behavior?
GETINFO dormant returns 1, as it is in dormant mode
### Environment
- Which version of Tor are you using? Run `tor --version` to get the version if you are unsure.
- 0.4.6.10 and 0.4.5.10
- Which operating system are you using? For example: Debian GNU/Linux 10.1, Windows 10, Ubuntu Xenial, FreeBSD 12.2, etc.
- Pop!_OS 22.04 and Ubuntu 18.04.6
- Which installation method did you use? Distribution package (apt, pkg, homebrew), from source tarball, from Git, etc.
- apt and ?Tor: 0.4.9.x-freezehttps://gitlab.torproject.org/tpo/core/arti/-/issues/1220Implement the `arti hss` subcommand2024-02-21T17:04:15Zgabi-250Implement the `arti hss` subcommandWe need to implement roughly what is described in !1730We need to implement roughly what is described in !1730Arti: Onion service supportgabi-250gabi-250https://gitlab.torproject.org/tpo/core/arti/-/issues/1218Consider moving logging out of publisher reactor2024-01-30T17:34:35Zgabi-250Consider moving logging out of publisher reactor```rust
netidr_event = netdir_events.next().fuse() => {
// The consensus changed. Grab a new NetDir.
let netdir = match self.dir_provider.netdir(Timeliness::Timely) {
Ok(y) ...```rust
netidr_event = netdir_events.next().fuse() => {
// The consensus changed. Grab a new NetDir.
let netdir = match self.dir_provider.netdir(Timeliness::Timely) {
Ok(y) => y,
Err(e) => {
error_report!(e, "HS service {}: netdir unavailable. Retrying...", self.imm.nickname);
// Hopefully a netdir will appear in the future.
// in the meantime, suspend operations.
//
// TODO HSS there is a bug here: we stop reading on our inputs
// including eg publish_status_rx, but it is our job to log some of
// these things. While we are waiting for a netdir, all those messages
// are "stuck"; they'll appear later, with misleading timestamps.
//
// Probably this should be fixed by moving the logging
// out of the reactor, where it won't be blocked.
wait_for_netdir(self.dir_provider.as_ref(), Timeliness::Timely)
.await?
}
};
self.handle_consensus_change(netdir).await?;
}
```Arti: Onion service supportgabi-250gabi-250https://gitlab.torproject.org/tpo/core/arti/-/issues/1217Consider adding a keystore watcher2024-01-11T18:45:15Zgabi-250Consider adding a keystore watcherSometimes we want to react to keystore changes. For example, if the identity of the service is rotated through some external means (e.g. via the `arti hss` command), the descriptor publisher needs to create and publish a new descriptor (...Sometimes we want to react to keystore changes. For example, if the identity of the service is rotated through some external means (e.g. via the `arti hss` command), the descriptor publisher needs to create and publish a new descriptor (using the newly generated keys).
* do we want the ability to "watch" keystores? What would the implementation for this look like? Should the `Keystore` trait have a function that returns the receiving end of an e.g. `postage::watch` channel? Will the channel return a `()` indicating something changed (without specifying what), or should it return the list of affected `KeyPath`s (or the corresponding `KeyPathPattern`s)?
* implementing this for FS-based keystores is fairly easy, but what about the HSM-based ones?Arti: Onion service supporthttps://gitlab.torproject.org/tpo/core/arti/-/issues/1216Improve descriptor publisher documentation2024-02-01T13:08:27Zgabi-250Improve descriptor publisher documentationArti: Onion service supportgabi-250gabi-250https://gitlab.torproject.org/tpo/core/arti/-/issues/1213ipt manager should have more unit tests2024-01-23T15:59:20ZIan Jacksoniwj@torproject.orgipt manager should have more unit testsThere are some unit tests but they don't test all the exciting things that can happen. See the TODOs in the codebase tagged with this ticket.There are some unit tests but they don't test all the exciting things that can happen. See the TODOs in the codebase tagged with this ticket.Arti: Onion service supportIan Jacksoniwj@torproject.orgIan Jacksoniwj@torproject.orghttps://gitlab.torproject.org/tpo/core/arti/-/issues/1162Allow configuring other paths with respect to state_dir.2024-01-09T19:34:42ZNick MathewsonAllow configuring other paths with respect to state_dir.If you override `state_dir`, you might think that you've done enough to avoid using the default state directory in `${ARTI_LOCAL_DATA}` ... but you'd be wrong! The keystore location also defaults to `${ARTI_LOCAL_DATA}/keystore`, so you'...If you override `state_dir`, you might think that you've done enough to avoid using the default state directory in `${ARTI_LOCAL_DATA}` ... but you'd be wrong! The keystore location also defaults to `${ARTI_LOCAL_DATA}/keystore`, so you'd need to override that too.
It would be neat if we could define a directory as `${state_dir}/keystore`. But in order to do that, we would need to have the ability to let some `CfgPath`-like types get defined with respect to other `CfgPath`s. We would need some kind of a multi-step path resolution process.
To keep this simple, I'd suggest that we declare that there is only one level of dependencies here: that other paths may be defined in terms of `${state_dir}`, but that we don't allow every possible path to depend on every other possible path.
Is this worth doing?Arti: Onion service supporthttps://gitlab.torproject.org/tpo/ux/design/-/issues/66Create a new color system2024-01-29T22:18:17ZdonutsCreate a new color systemIn order to support the creation of the new style guide, we need to develop a new and more complex color system that's flexible enough to serve our brand needs and semantic use cases for web and app design. At minimum, we require scales ...In order to support the creation of the new style guide, we need to develop a new and more complex color system that's flexible enough to serve our brand needs and semantic use cases for web and app design. At minimum, we require scales based on the following brand colors:
- Purple
- Green
- Gray
- Blue
- Amber
- Read
And to support dark themes, an alternate background scale (e.g. dark gray, indigo, or dark purple).
Previous work in this area has included the explorations done by Ura Design in https://gitlab.torproject.org/tpo/web/styleguide/-/issues/19.design-dot MVPdonutsdonutshttps://gitlab.torproject.org/tpo/core/arti/-/issues/1152We need an API for unix sockets2024-02-28T12:49:52ZNick MathewsonWe need an API for unix socketsRight now we have a facility to listen on unix sockets for RPC, and we might want to have that for dns and socks as well. Additionally, we want the ability to have onion services (via tor-hsrproxy) *connect* to local unix sockets.
And ...Right now we have a facility to listen on unix sockets for RPC, and we might want to have that for dns and socks as well. Additionally, we want the ability to have onion services (via tor-hsrproxy) *connect* to local unix sockets.
And to complicate things further, we might eventually want to involve Windows named sockets, and whatever else exists in this area.
This won't be too hard to build but there are design issues to think about.
* Should this be a new API for tor-rtcompat?
* Should this be an optional `UnixSocketProvider` that is only present when building for `target_family = "unix"`?
* Should there be an `AbstractSocketProvider` that can take AF_UNIX addresses as well as `SocketAddr`s?
If I were building this without guidance, I think I would answer all three of those questions "yes". But what do you think? (cc @gabi-250 @Diziet)
See also #827 for nomenclature, which I am sure I am getting wrong here.Arti: Onion service supportNick MathewsonNick Mathewson