Namecoin support for onion sites in Tor Browser

added TorBrowserTeam201912R in Legacy / Trac component::applications/tor browser in Legacy / Trac owner::JeremyRand in Legacy / Trac priority::medium in Legacy / Trac resolution::fixed in Legacy / Trac reviewer::gk in Legacy / Trac severity::normal in Legacy / Trac status::closed in Legacy / Trac type::defect in Legacy / Trac labels

Note, even though this has legacy/trac#30029 (moved) as parent ticket, it is not related to our sponsor work. Thus, I am unparenting the ticket, to make that clear (i.e. legacy/trac#30029 (moved) is not blocking on this ticket).

Trac:
Cc: pili, GeKo, antonela, asn to pili, gk, antonela, asn
Parent: legacy/trac#30029 (moved) to N/A

HTTPS Everywhere: Aliases are under central control by the addon maintainer.

NO, HE supports different streams.

Trac:
Cc: pili, gk, antonela, asn to pili, gk, antonela, asn, boklm

Early branch: https://notabug.org/JeremyRand/tor-browser-build/src/namecoin (you want the namecoin branch).

The 3 most major shortcomings in its current state are:

ncprop279 isn't built locally, but instead is pulled from a binary release on namecoin.org. This is because ncprop279 currently needs a different version of the Go compiler than tor-browser-build uses, so it needs to be built in its own tree. The binary that's being pulled in is (in theory) reproducible with rbm (via the ncdns-repro repository), so it's not a security issue, but it makes the build workflow a lot more annoying than it should be. I'm in the process of getting a patch merged to Namecoin that will fix this issue; I expect it to be resolved in less than a month.
Electrum-NMC is pulled in via the Python source code tarball on namecoin.org. That tarball contains source code from various Python dependencies of Electrum. It would be a lot better to pull in those dependencies from their upstream source (either Git or tarballs), and then combine them with Electrum-NMC's source (from Git). I'm in the process of implementing this; I expect it to be resolved in less than 2 months.
This branch uses an official Electrum-NMC release rather than the Electrum-NMC branch I used in the live demo. The live demo branch has some patches that make initial syncup much faster (nearly instant), whereas the official release will probably take about 5 minutes to do initial syncup. Most of the patches for faster syncup are now undergoing review by upstream Electrum; this has already yielded much better code quality than the live demo branch (shocking, I know -- the Electrum devs know their own codebase better than I do!), but it does mean there's some delay in getting everything merged. I think it's likely that a lot of this code will be merged upstream in the next 2-3 months.

Anyway, while this isn't production-ready in any way, I figure it's a good idea to post it here for transparency purposes. If anyone wants to play around with it, build a nightly of Tor Browser (must be 64-bit Linux), and set the environment variable TOR_ENABLE_NAMECOIN=1 before you run Tor Browser. .bit and .bit.onion sites should "just work" (modulo the initial syncup time, see above). Right now .bit sites can point to IP address or onion services, and Namecoin TLS is not part of this patch. Prior to a release, I intend to disable IP addresses, so .bit can only point to a .onion, and we can evaluate how to do IP+TLS securely at a later date. Be sure to check out the awesome circuit display when you're viewing a Namecoin onion service! (Kudos to Arthur for the Torbutton patch that does this.)

Cheers!

ncprop279 isn't built locally, but instead is pulled from a binary release on namecoin.org.

Updated branch at https://notabug.org/JeremyRand/tor-browser-build/src/namecoin (commit hash accff2aff645c7dac487e9acd0cbd43fa8372b12). ncprop279 is now built locally.

The updated branch also disables --detach if Namecoin is enabled, which allows Namecoin to be used when double-clicking the Tor Browser launcher icon. (I noticed that this was broken in the previous branch during the Stockholm meeting.)

More updates should be coming fairly soon.

Electrum-NMC is pulled in via the Python source code tarball on namecoin.org. That tarball contains source code from various Python dependencies of Electrum. It would be a lot better to pull in those dependencies from their upstream source (either Git or tarballs), and then combine them with Electrum-NMC's source (from Git).

Updated branch at https://notabug.org/JeremyRand/tor-browser-build/src/namecoin (commit hash 4b23a0490f49e9c08c63ae5ab3c70d66be84115e). Electrum-NMC and its dependencies are now built locally from source, subject to a few exceptions:

PyQt isn't built at all. This doesn't matter for Tor purposes since Tor isn't going to use the PyQt GUI from Electrum-NMC.
Locales aren't built at all. Again, this doesn't matter for Tor purposes since the Electrum-NMC UI isn't going to be user-facing and therefore doesn't need to be translated.
Protobuf definitions aren't built from source, instead the pre-compiled Protobuf definitions that Electrum-NMC distributes are used. I'm fairly sure this isn't a problem since Protobuf is (AFAICT) only used in Electrum-NMC for the Payment Protocol, and that's not functionality that Tor will ever need to touch. In fact, I suspect that the Protobuf dependency can be completely ripped out of Electrum-NMC for Tor distribution, which will also reduce the final binary size noticeably.

Prior to a release, I intend to disable IP addresses, so .bit can only point to a .onion

This is implemented in 4b23a0490f49e9c08c63ae5ab3c70d66be84115e as well, though I haven't tested that change very thoroughly yet.

I'm hoping to have some additional updates within the next week or two.

Updated branch at https://notabug.org/JeremyRand/tor-browser-build/src/namecoin (commit hash 20c7d142948cad98193b50ec6ad75fc272421f1a). Here's the summary of the current status:

Namecoin is enabled only in Nightly.
Namecoin is enabled only if the environment variable TOR_ENABLE_NAMECOIN=1 is set when Tor Browser is executed. (At Georg's suggestion, I intend to transition this to using a Firefox pref in the future.)
Namecoin is enabled only on GNU/Linux targets. (I intend to add Windows, macOS, and Android/Linux support later.)
Namecoin will only work if Python 3.6 or higher is installed. Python 3.6 is pretty common nowadays, so this shouldn't be a huge barrier to testing it, although it's not ideal. (In the future, we could look at including a Python interpreter binary in the Tor Browser package, thus removing this requirement. Doing so would presumably be needed on Windows anyway.)
Stream isolation is supported in this patch, but is dependent on legacy/trac#19859 (moved). Until legacy/trac#19859 (moved) is merged, this patch will still connect without errors, but stream isolation will not be functional, which has two implications: degraded privacy and degraded performance. It should be possible to review both patches in parallel, and it should be easy to build this patch before legacy/trac#19859 (moved) is merged by manually setting the tor project's Git repo/commit to use the one provided in legacy/trac#19859 (moved). However, while I encourage review of this patch, I do not recommend merging this patch until legacy/trac#19859 (moved) is merged.
2 of the public ElectrumX Namecoin servers are currently down for maintenance. Since Namecoin connects to multiple servers simultaneously to improve performance, the performance of this patch will be degraded until those servers come back online. It still works fine and isn't particularly annoying, but there will be some higher-than-typical latency while we're waiting for those servers to come back online. (It would be awesome if the Tor community decides to set up some additional ElectrumX Namecoin servers.)
The 4 domain names that I demoed in Stockholm are http://federalistpapers.bit, http://onionshare.bit, http://riseuptools.bit, and http://submit.theintercept.bit. You can use either .bit or .bit.onion eTLD. (The difference between the two eTLD's was covered in the Namecoin session in Stockholm.)
Currently .bit domains can only point to a .onion domain. .bit domains that point to an A, AAAA, CNAME, or other DNS record will not resolve. Adding support for other record types can be done later.
Currently, the .bit eTLD is not considered a secure origin like .onion. So visiting a .bit domain will not be recognized by Firefox as secure, nor will it show the onion icon. Visiting a .bit.onion domain will work fine though. I will fix this later.
The circuit display panel will show the .onion domain that a Namecoin domain points to. Kudos to Arthur for that patch.
This patch includes an implementation of Prop279. My implementation differs from the spec by adding a "stream isolation ID" field to the RESOLVE command. If desired, I could submit a spec patch for Prop279 that makes it match this implementation. Let me know if you'd like me to do that.
I've probably forgotten some potentially relevant notes; if anything doesn't make sense or you otherwise have some questions for me, please don't hesitate to ask.
As far as I can tell, this patch is ready for review.

Cheers!

Trac:
Status: assigned to needs_review
Keywords: N/A deleted, TorBrowserTeam201911R added

Thanks for the patches! The construction with the ncdns-repro repo seems weird. I don't think this belongs into the top level of the tor-browser-build repo. Rather, looking at its contents this needs to get integrated into the projects. Or, rather, all the projects you need should get integrated into projects. But maybe I am missing something here. Could you elaborate?

FWIW: there is at least one (non-fatal) error I get during running the build that might be related to the above as you seem to try to remove a thing that cannot be removed:

git submodule update --init
Submodule 'ncdns-repro' (https://github.com/namecoin/ncdns-repro.git) registered for path 'ncdns-repro'
Cloning into '/home/gk/tor-browser-build/ncdns-repro'...
Submodule path 'ncdns-repro': checked out '1a81e71504466a518c0aa7562d393ce0e06fc22a'
make -C ncdns-repro submodule-update
make[1]: Entering directory '/home/gk/tor-browser-build/ncdns-repro'
./setup-submodule-symlinks
rm: cannot remove 'tor-browser-build': Is a directory
make[1]: Leaving directory '/home/gk/tor-browser-build/ncdns-repro'

Thanks for the patches! The construction with the ncdns-repro repo seems weird. I don't think this belongs into the top level of the tor-browser-build repo. Rather, looking at its contents this needs to get integrated into the projects. Or, rather, all the projects you need should get integrated into projects. But maybe I am missing something here. Could you elaborate?

My reasoning was that Namecoin already maintains rbm projects for most of the binaries we distribute, so it would reduce maintenance effort to simply reuse Namecoin's rbm projects folder (via a Git submodule) rather than require the Tor Browser devs to regularly copy Namecoin's projects folder into Tor Browser's projects folder. But, if you prefer to have Tor Browser keep its own copy of Namecoin's projects rather than reuse Namecoin's version as a Git submodule, that's fine with me and I can make that change. Let me know what you'd prefer.

FWIW: there is at least one (non-fatal) error I get during running the build that might be related to the above as you seem to try to remove a thing that cannot be removed:

Indeed, that error gets displayed the first time that the submodule is being set up. The error is harmless, but I can fix that pretty easily (though I'll wait for an answer to the above question so that I know whether the submodule is staying at all).

Replying to JeremyRand:

Thanks for the patches! The construction with the ncdns-repro repo seems weird. I don't think this belongs into the top level of the tor-browser-build repo. Rather, looking at its contents this needs to get integrated into the projects. Or, rather, all the projects you need should get integrated into projects. But maybe I am missing something here. Could you elaborate?

My reasoning was that Namecoin already maintains rbm projects for most of the binaries we distribute, so it would reduce maintenance effort to simply reuse Namecoin's rbm projects folder (via a Git submodule) rather than require the Tor Browser devs to regularly copy Namecoin's projects folder into Tor Browser's projects folder. But, if you prefer to have Tor Browser keep its own copy of Namecoin's projects rather than reuse Namecoin's version as a Git submodule, that's fine with me and I can make that change. Let me know what you'd prefer.

Hrm. I am not convinced yet that the maintenance effort is (so much) higher given that we want to review any changes anyway before applying them tor tor-browser-build and we might even want to cherry-pick some changes and not other which I think could be harder in the submodule setup. Thus, let's try it the way I proposed above and see it as part of the experiment we are doing.

I was not aware that Namecoin is using rbm as well, which is very nice! Keep it going. :)

FWIW: there is at least one (non-fatal) error I get during running the build that might be related to the above as you seem to try to remove a thing that cannot be removed:

Indeed, that error gets displayed the first time that the submodule is being set up. The error is harmless, but I can fix that pretty easily (though I'll wait for an answer to the above question so that I know whether the submodule is staying at all).

Thanks, yes, fixing this would be good I think.

Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam201911R deleted, N/A added

Hrm. I am not convinced yet that the maintenance effort is (so much) higher given that we want to review any changes anyway before applying them tor tor-browser-build and we might even want to cherry-pick some changes and not other which I think could be harder in the submodule setup. Thus, let's try it the way I proposed above and see it as part of the experiment we are doing.

Done. Updated code is at https://notabug.org/JeremyRand/tor-browser-build/src/namecoin (Git commit hash 88c59a9056a5c458f88e0beecd915e1cee30f2c0).

I was not aware that Namecoin is using rbm as well, which is very nice! Keep it going. :)

Yes, we're using rbm for all of our projects that aren't forks of Bitcoin codebases. We're also gently nudging upstream Electrum to move toward rbm as well (and we're using the integration of Electrum-NMC into Tor Browser as an opportunity to experiment with using rbm to build Electrum so that it's easier to justify this to upstream Electrum devs).

Thanks, yes, fixing this would be good I think.

No longer relevant, as that error message was related to the Git submodule, which is now removed.

Trac:
Keywords: N/A deleted, TorBrowserTesm201911R added
Status: needs_revision to needs_review

2 of the public ElectrumX Namecoin servers are currently down for maintenance. Since Namecoin connects to multiple servers simultaneously to improve performance, the performance of this patch will be degraded until those servers come back online. It still works fine and isn't particularly annoying, but there will be some higher-than-typical latency while we're waiting for those servers to come back online. (It would be awesome if the Tor community decides to set up some additional ElectrumX Namecoin servers.)

1 of those 2 servers has completed maintenance and is back online as of yesterday, so that should yield a performance bump for anyone testing these patches.

On another note: in the hypothetical event that the patches in this ticket pass review and are merged, my recommendation would be to time the merge to coincide with a blogpost, so that users of the Nightly builds are informed of what the changes are. Otherwise, there might be unnecessary confusion by users who observe a bunch of Namecoin code being merged to tor-browser-build (or who observe its presence in the Nightly binaries) and aren't sure what the scope of it is (e.g. it's important that users know that the code is disabled by default, so that they don't think they're being exposed to new attack surface without their opt-in consent). I'm happy to contribute to such a blogpost if/when we reach the point where it's relevant. Of course, if you strongly prefer to merge without a blogpost and worry about doing a blogpost later, you're welcome to do so.

Trac:
Keywords: TorBrowserTesm201911R deleted, TorBrowserTeam201911R added

Okay, let's start. Alas, I don't have time to review everything at once. My current plan is to go commit by commit and let you fix up things as we go. I hope this works for you.

1d46514c6be75dadcf9201961b1252edafcfa443:

+    "proxy": "socks5:127.0.0.1:9150::",

I guess that hard-coding is done by taking the Tor Browser default values? One thing to keep in mind is that not every user is using 9150 as the SOCKS port. I don't know how hard it is to make this more dynamic but it would be nice if it were possible (but it's not necessary for a nightly inclusion).

+    - patch

Are you sure this is needed? We are applying a lot of other patches without requiring patch being explicitly installed (because it seems to come at least in newer OSes installed by default).

+  #- name: electrum-nmc
+  #  project: electrum-nmc
+  #  enable: '[% c("var/namecoin") %]'
+  #- name: ncprop279
+  #  project: ncprop279
+  #  enable: '[% c("var/namecoin") %]'
+  #- name: stemns
+  #  project: stemns
+  #  enable: '[% c("var/namecoin") %]'

Please remove those comments here and just include those projects when it is time.

++        python3 TorBrowser/StemNS/poc.py &

I think we can't require users just having python3 available yet. Please add a check and somewhere so users get a notice when they need to install python3.

+      namecoin: '[% c("var/nightly") %]'

Please add a namecoin: 0 to all the other platforms given that you are not only checking for Namecoin support when dealing with Linux.

Some general remarks:

We try to make it possible to bisect issues in tor-browser-build in the sense that resulting builds are still running. It's hard sometimes and sometimes even not really possible, but that should be the goal anyways. With that in mind I think the commit I looked at above should be (to a large extend (that is without the changes in rbm.conf)) the last in your series of patches. Usually one is adding commits for all the projects and then in the final commit(s) one is "activating" those by getting them actually build once one sets the proper flags/or builds for the respective platforms/architectures. It would be nice if you could follow this general idea.
Please don't use the submodule approach you had in mind first and then later on discard it in favor of including all the needed projects directly. Just include those projects directly. This makes the patch set smaller, is easier to review, and is the right thing to do anyway.

Trac:
Keywords: TorBrowserTeam201911R deleted, TorBrowserTeam201911 added
Status: needs_review to needs_revision

Thanks for the feedback Georg. I've just pushed a revised set of patches (same branch as before; Git commit hash 26c5593a9230e22b52b03917ad274b8ea08a70b5) that I believe addresses your review so far.

Alas, I don't have time to review everything at once. My current plan is to go commit by commit and let you fix up things as we go. I hope this works for you.

Yes, in fact that's better than reviewing everything at once, since it lets me start addressing feedback sooner.

I guess that hard-coding is done by taking the Tor Browser default values? One thing to keep in mind is that not every user is using 9150 as the SOCKS port. I don't know how hard it is to make this more dynamic but it would be nice if it were possible (but it's not necessary for a nightly inclusion).

Doing this the "right way" will probably require patching Electrum-NMC; I'll put it on my to-do list. That said, can you elaborate on what types of users will be using a non-default SOCKS IP/port? The only cases I can think of are setups like Tails and Whonix, and those setups will need other changes to work with Namecoin because they use a control port filter that will interfere with StemNS. I definitely want to support those use cases (if nothing else because Whonix is my daily driver OS), but I'm curious if there's another set of use cases where this matters that I'm not aware of.

Are you sure this is needed? We are applying a lot of other patches without requiring patch being explicitly installed (because it seems to come at least in newer OSes installed by default).

Yes; removing that line causes the build to fail because patch isn't found. Maybe patch is only installed by default in newer versions of Debian than what's used here.

Please remove those comments here and just include those projects when it is time.

Done.

I think we can't require users just having python3 available yet. Please add a check and somewhere so users get a notice when they need to install python3.

Done. At the same time I also now check if their python3 is too old, and give them a notice for that case too.

Please add a namecoin: 0 to all the other platforms given that you are not only checking for Namecoin support when dealing with Linux.

Done.

We try to make it possible to bisect issues in tor-browser-build in the sense that resulting builds are still running. It's hard sometimes and sometimes even not really possible, but that should be the goal anyways. With that in mind I think the commit I looked at above should be (to a large extend (that is without the changes in rbm.conf)) the last in your series of patches. Usually one is adding commits for all the projects and then in the final commit(s) one is "activating" those by getting them actually build once one sets the proper flags/or builds for the respective platforms/architectures. It would be nice if you could follow this general idea.

Good point. I've rearranged the order and grouping of commits to hopefully bring it closer to what you describe. Let me know if this is an improvement, and whether there's anything remaining that I should do on this front.

Please don't use the submodule approach you had in mind first and then later on discard it in favor of including all the needed projects directly. Just include those projects directly. This makes the patch set smaller, is easier to review, and is the right thing to do anyway.

Done.

Thanks for the review!

Replying to JeremyRand:

Thanks for the feedback Georg. I've just pushed a revised set of patches (same branch as before; Git commit hash 26c5593a9230e22b52b03917ad274b8ea08a70b5) that I believe addresses your review so far.

Alas, I don't have time to review everything at once. My current plan is to go commit by commit and let you fix up things as we go. I hope this works for you.

Yes, in fact that's better than reviewing everything at once, since it lets me start addressing feedback sooner.

I guess that hard-coding is done by taking the Tor Browser default values? One thing to keep in mind is that not every user is using 9150 as the SOCKS port. I don't know how hard it is to make this more dynamic but it would be nice if it were possible (but it's not necessary for a nightly inclusion).

Doing this the "right way" will probably require patching Electrum-NMC; I'll put it on my to-do list. That said, can you elaborate on what types of users will be using a non-default SOCKS IP/port? The only cases I can think of are setups like Tails and Whonix, and those setups will need other changes to work with Namecoin because they use a control port filter that will interfere with StemNS. I definitely want to support those use cases (if nothing else because Whonix is my daily driver OS), but I'm curious if there's another set of use cases where this matters that I'm not aware of.

We have users that want to avoid using Tor Browser's tor and rather like to point to a system tor which they have running anyway. Granted, it's not the majority of users (by far) but it's an important use-case we support at least.

Trac:
Keywords: TorBrowserTeam201911 deleted, TorBrowserTeam201911R added

Namecoin support for onion sites in Tor Browser

Designs

Child items ...

Activity