Commit 5ff41f1a authored by Gaba's avatar Gaba 🦋
Browse files

Merge branch 'main' into 'Burnleydev-main-patch-61371'

# Conflicts:
#   README.md
parents f61dce6d 9cc4727b
# Tor Hackweek Project: Add support for UDP sockets over onion services, possibly with enabling support for WebRTC in Tor Browser
Summary:
- Goal: Present Hack Week results via Tor Browser on BBB
* Intermediate Tor steps:
- Add support for relaying UDP datagrams
- Add support for configuring UDP onion services
* Intermediate Tor Browser steps:
- Enable WebRTC
- ?
Skills Needed: C/C++, familiar with tor and/or Firefox and/or webrtc
# Team:
* sysrqb (UTC-5)
**Notes:** Possibly, ask mike and nick for the notes we took on how to make Tor support UDP, if you're interested in doing it that way? --nick
# Tor Hackweek Project: Circuit Padding Simulator
Summary: The Tor circuit padding framework is under active use by researchers to improve padding defenses against website traffic fingerprinting, and onion service circuit fingerprinting. The simulator is in need of update to the latest Tor release, as well as in need of performance improvements, and a more accurate way to deal with time-based
Skills Needed: Tor unit test framework, Tor C, maybe python
# Team:
* Mike (UTC-5)
# Tor Hackweek Project: Network Health tooling using Rust and arti
Summary: Network health has a series of tools in order to query the consensus,
search for patterns and correlate relays. This project is to rewrite some of
them into one unified tool written in Rust and using arti for any Tor related
interactions such as looking up the consensus or creating circuits for testing
connectivity.
Skills: Love, Passion and Legos
# Team:
* dgoulet
* juga?
# Feature ideas:
* Search relays with in the latest consensus by:
- Fingerprint (RSA and/or ED)
- Contact info (pattern, regex?)
- Bandwidth values
- AS
- Nickname (pattern, regex)
- Versions
- Flags
- ...
* Network stats
- Relay counting by flags or versions
- Extra info information such as "how many overload relays do we have"
- Count onion service bandwidth
* Bad relays config
- Generate bad relay dirauth configuration
* Network test
- Test Exit DNS
- Test ORPort/Dirport
- Test connectivity IPv4/IPv6
# Nick's Notes:
* For "network test", probably best to use tor-proto crate for low-level fine-grained network usage.
- Possibly, give tor-circmgr the ability to create circuits with specific paths?
- Possibly, consider `tracing` crate for fine-grained events during testing, to analyze timing?
- Look for opportunities to make these APIs simpler or better.
* For search and stats:
- For some of this info, will need to teach tor-dirmgr about fetching ns consensus and routerdesc documents. Also will need to teach tor-netdoc about fetching and parsing ns consensus documents.
- This may warrant refactoring tor-dirmgr.
- Consider sqlite or similar to build a relational database of decoded relay information.
- Consider tools to export the existing state of the network as json, xml, or something else so that arbitrary other scripting tools can handle it.
- Consider clap as a CLI-building tool.
- It might be convenient to teach tor-dirmgr to operate in a "cache-only" mode that just looks at the cache of another dirmgr, and doesn't fetch the directory. Ideally, this would be able to run concurrently with another process that is actually updating the directory.
- For future-proofing, consider making a variant routerdesc/consensus parser that retains recognized data and allows searching it.
# Tor Hackweek Project: Onion service v3 support for arti
Summary: arti currently does not do v3 onion services. Let's do some solid work towards making that possible.
Skills: Tor protocol, Onion services protocol, Rust, arti
# Team:
* ASN (UTC-3)
# Breakdown and suggested hacking order:
* tor-llcrypto crate: identify low-level operations needed for v3 onion services. (key blinding)
* tor-cell crate: suppport new relay message types. They should get their own module under relaycell::msg. We should have encode, decode, fuzz, and test vector support for them.
- Should we look for ways to make the cell parsing and encoding more generic so that other extensions can define their own cells? I think that would be a good ide eventually, but probably not for now.
* tor-netdoc crate: support v3 service descriptors. This should include outer and inner forms. [asn first target]
* tor-netdir crate: we need support for the hash index ring.
* tor-circmgr crate: **
- need support for building a circuit with a given exit
- need support for saying "nobody else can use this circuit!"
- need support for saying "this circuit is a valid open connection to (x.onion)", and for saying "i am building a connection to (x.onion), please wait.
* tor-dirmgr crate:
- Support for parallel v3 onion descriptor lookup. This involves adding a new ClientRequest type, and probably refactoring the ClientRequest trait so it can handle the things that make onion descriptor lookup different.
* tor-proto crate: we need support for taking a circuit and then doing an arbitrary send-this-cell-then-wait-for-that-cell handshake. **
* tor-proto crate: need support for extending a circuit with an end-to-end encryption hop, as used with onion services. **
* tor-proto crate: possibly, move some or all of crypto module into a different crate? Some might belong in tor-llcryppto, but not all.
* new tor-hsclient crate: given a cirmgr, a dirmgr, and an onion service name, it should be able to it get or construct a completed rendezvous circuit to that onion service. Its interface can be similar to circmgr. Use APIs in tor-retry for anything we want to attempt more than once.
- This is probably where any descriptor cache should live.
* tor-client crate: When told to connect to an onion address, get the circuit from tor-hsclient.
# Design notes:
* Don't do anything with v2 onions at all.
* Client-side support for onion services should be under an hs-client feature; server-side support should be under an hs-server feature. Common code should be under an "hs" feature and enabled by either.
* New functionality in tor-proto should probably NOT be hs-specific. That is, most of the hs-specific protocol handling stuff should be done in a different crate.
* Parsing shouldn't do crypto: decryption and signature-checking should happen in a post-parsing step.
- This means that for many message types, the parse operation will just put the body in a buffer, and the real parsing will happen after the decryption
- Use different types to prevent confusion here: See current use of the SelfSigned triat in tor-netdir to ensure that signatures are explicitly checked or explicitly ignored.
# Tor Hackweek Project: Onionshare download accelerator
Summary: Journalists are requesting ways to download large files from onion services faster. This is actually possible already with HTTP, without any Tor client modifications. By splitting up requests into 250k chunks using HTTP range requests, and using SOCKS username and password to allocate these requests properly onto different circuits, we can hit ~6 megabytes/sec per guard (so ~12 megabytes/sec with 2 gaurds), with a custom HTTP download acelerator. This download accelerator could be provided as part of onionshare. For more details, see [issues-1295][https://github.com/micahflee/onionshare/issues/1295].
Skills Needed: Python, HTTP/1.1, SOCKS
# Team:
* Mike (UTC-5)
* Saptak (UTC +5:30)
### Message mikeperry on OFTC IRC or email if there is interest in this.
# Tor Hackweek Project: Prototype Rust+Arti-based HTTP frontend cache for directory authorities
Summary: Directory authorities are under a lot of unnecessary load from excessive download requests. We have other projects in mind to reduce those requests, but one workaround is to add a frontend cache in front of one or more of the authorities' HTTP ports. With this project we'll write a Rust server use Arti's download and validation code to fetch directory information from an authority, and expose that information via a set of HTTP requests. With luck, our code will be reuseable in the future when relays or authorities are rewritten in Rust.
Goal: A proof-of-concept that can handle some HTTP requests, and relays the ones it can't understand to an authority. Ideally, more!
Skills Needed: Rust hacking experience helpful.
# Team:
* nickm (UTC-5)
# Breakdown for minimal version:
* tor-dirmgr crate: add support for retrieving text of particular documents
* new tor-dirservice crate:
- Add a set of responders that can handle http requests. These should be made to plug in to Hyper (a rust webserver).
- Teach these responders to understand tor's http requests.
- Teach them to answer requests for what we have by getting the answer from the tor-dirmgr crate.
- Have a fallback responder that forwards anything else to a configured HTTP address:port.
* Have an entry point (new crate) that launches the above as a webserver.
# Additional work to be more useful as a download cache:
* Teach tor-netdoc to handle votes
* Teach tor-netdoc to handle networkstatus consensus docs
* Teach tor-dirclient to handle more kinds of directory document (votes, ns consensuses, routerdescs, extrainfos)
* Teach tor-dirclient to be able to make unencrypted HTTP requests
* Teach tor-dirmgr how to download and store more kinds of directory documents
* Implement code to generate consensus diffs
* Teach tor-dirmgr (or similar) how to keep a good set of consensus diffs
* Teach tor-dirmgr (or similar) how to pre-compress large documents
# Additional work to be more useful as an upload frontend:
* New crate: support for storing and cross-validating incoming descriptors and extrainfos. Don't do this with the tor-dirmgr crate: use a separate implementation.
* Support for basic relay reachability testing on incoming descriptors
* Support for exclude lists and keypinning
* Support for uploading a validated descriptor to an authority
......@@ -77,7 +77,7 @@ To add yourself to a group use the group's pad please.
## 2 Prototype network-namespace-based torsocks
- Notes: [Prototype network-namespace-based torsocks](network-namespace-based-torsocks.md)
- Notes: [torsocks](torsocks.md)
- Contact: jnewsome and/or dgoulet
- Summary: Use network namespaces (or maybe something else?) to run target software in an environment where it can't talk to the real network; it can only talk to the tor socks port and/or some "shim" adapter. Might be able to remove or lessen dependence on LD_PRELOAD (which isn't available everywhere, can be "escaped", and can be a bit fragile). If we continued to use LD_PRELOAD could at least be used to prevent accidental connections to the real network.
- Skills: C, familiarity with network namespaces and/or LD_PRELOAD would be helpful. New code could potentially be written in Rust.
......
# Tor Hackweek Project: Vanguards doc updates, bugfixes, packages for onionshare and/or Tor Browser
Summary: The vanguards Tor control port addon provides defense against Guard discovery attacks, as well as other attacks against onion services, onion clients, and even Tor Browser exit traffic. Vanguards is in need of some doc updates, bugfixes, and we could even package it for one of onionshare or Tor Browser. Packaging it with Onionshare also helps improve the security properties of the above download accelerator item. See the bugtracker for specific items: [issues here](https://github.com/mikeperry-tor/vanguards/issues)
Skills Needed: Python, tech doc writing and/or review, packaging/build eng
# Team:
* Mike (UTC-5)
# Tor Hackweek Project: Visualize Tor metrics's data in ways that it can be useful for community
Summary: The goal is to have a dashboard with Tor usage per country in a way that is easy to see big changes happening. Right now we need to select each country to see the Tor usage. It would be good to have a way to see all the countries and the onews where usage is increasing (via bridge and direct connection).
Skills Needed: data visualization
# Team:
* gus (UTC-3)
* gaba (UTC-3)
* tara(?) (UTC +1)
* joydeep (UTC +5:30)
* djackson (UTC +1)
# Merging with the other project with similar goal:
- [hackweek-prometheus-alerts onion link](http://kfahv6wfkbezjyg4r6mlhpmieydbebr5vkok5r34ya464gqz6c44bnyd.onion/p/2021-hackweek-prometheus-alerts)
# PLAN THURSDAY MARCH 31ST
- get a domain for mb.torproject.org
- add shutdown data to metabase and add a dashboard - done
- import user relays data to metabase - done
- anomaly detection queries
- alert system [link here](https://www.metabase.com/docs/latest/users-guide/15-alerts.html)
# RESOURCES
## USE CASES (what do we want to do?):
- how tor is increasing
* show countries that have this cases:
- "big" decreasing user stats in the last 24 hours
- increase of uses of bridges in the last 24 hours
- which kind of transports are being increased/decreased
- anomalies in bridge usage by country
- ideas in [contribut to Tor metrics timeline](https://blog.torproject.org/contribute-to-tor-metrics-timeline)
- more in [exploring the Tor dataset with metabase](https://dustri.org/b/exploring-the-tor-dataset-with-metabase.html)
# DATA SETS:
* user stats per country: [userstats relay country](https://metrics.torproject.org/userstats-relay-country.csv) (the estimated number of directly-connecting clients)
* date: UTC date (YYYY-MM-DD) for which user numbers are estimated.
* country: Two-letter lower-case country code as found in a GeoIP database by resolving clients' IP addresses, or "??" if client IP addresses could not be resolved. If this column contains the empty string, all clients are included, regardless of their country code.
* users: Estimated number of clients.
* lower: Lower number of expected clients under the assumption that there has been no censorship event. If users < lower, a censorship-related event might have happened in this country on the given day. If this column contains the empty string, there are no expectations on the number of clients.
* upper: Upper number of expected clients under the assumption that there has been no release of censorship. If users > upper, a censorship-related event might have happened in this country on the given day. If this column contains the empty string, there are no expectations on the number of clients.
* frac: Fraction of relays in percent that the estimate is based on.
* bridge user per country / transport: [userstats-bridge-combined](https://metrics.torproject.org/userstats-bridge-combined.csv)
* date: UTC date (YYYY-MM-DD) for which user numbers are estimated.
* country: Two-letter lower-case country code as found in a GeoIP database by resolving clients' IP addresses, or "??" if client IP addresses could not be resolved.
* transport: Transport name used by clients to connect to the Tor network using bridges. Examples are "obfs4", "websocket" for Flash proxy/websocket, "fte" for FTE, "<??>" for unknown pluggable transport(s), or "<OR>" for the default OR protocol.
* high: Upper bound of estimated users from the given country and transport.
* low: Lower bound of estimated users from the given country and transport.
* frac: Fraction of bridges in percent that the estimate is based on.
## format for tor metrics data [userstats-relay-country](https://metrics.torproject.org/stats.html#userstats-relay-country)
* prometheus metrics that tor exports [issues-40063](https://gitlab.torproject.org/tpo/core/tor/-/issues/40063)
* Keepiton data on internet shutdowns
- link to download data: [keepiton-stop-data-2020](https://www.accessnow.org/keepiton-stop-data-2020)
- ID
- start_date
- end_date
- duration
- Info_source
- news_link
- continent
- country
- State/India
- geo_scope
- area_name
- ordered_by
- decision_maker
- shutdown_type_new
- affected_network
- full or service-based
- Facebook_affected
- Twitter_affected
- WhatsApp_affected
- Instagram_affected
- Telegram_affected
- other_service_details (specify)
- SMS_affected
- phone_call_affected
- telcos_involved
- gov_ack
- official_just
- other_just_details
- off_statement
- actual_cause
- other_cause_details
- election
- violence
- hr_abuse_reported
- users_notified
- users_affected/targetted
- legal_justif
- legal_method
- telco_resp
- telco_ ack
- econ_impact
- event
- an_link
- notes
# Todo:
- csv needs some cleaning
- country needs to be converted to country codes
* OONI data on censorship events
# TOOLS
- [dash-plo](https://dash.plot.ly)
- jupyter
- grafana: for the dashboard
- metabase: straight forward to use
- apache superset: to explore data
- csvkit
- https://pgloader.readthedocs.io/en/latest/ref/csv.html
- prometheus: for collecting data and alerts (alerts-manager)
- https://www.bamsoftware.com/git/tor-metrics-country.git/
- infolabe-anomalies: [link](http://lists.infolabe.net/lists/listinfo/infolabe-anomalies)
# Tasks:
- share the db (gus):
echo "Downloading userstats"
curl -O [userstat-relay-country](https://metrics.torproject.org/userstats-relay-country.csv)
echo "Removing the first 5 lines"
sed -i '1,5d' userstats-relay-country.csv
echo "Import db userstats-relay-country"
(echo .separator ,; echo .import userstats-relay-country.csv userstats) | sqlite3 userstats-relay-country.db
- a downloader to get data from metrics regularly to get into the db for the tool/s to use it.
- try grafana locally and gus will use his vps to install grafana
# How prometheus is working at TPI
[prometheus-monitored-services link](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/prometheus#monitored-services)
## monitoring:
* connectivity test (through blackbox exporter)
* rdsys
* bridgestrap
- prometheus is configured by puppet [prometheus-torproject](https://prometheus.torproject.org)
- grafana is configured by hand
#Tasks to work on:
- metrics data into prometheus
- install and configure component push gateway
- getting csv into promethus
- snowflake metrics data into prometheus
- grafana to visualize data
### [hackweek-metrics](https://gitlab.torproject.org/juga/hackweek_metrics/): to launch prometheus locally, parse a csv and send a silly metric to the pushgateway
# Tor Hackweek Project: Prototype network-namespace-based torsocks
Summary: Use network namespaces (or maybe something else?) to run target software in an environment where it can't talk to the real network; it can only talk to the tor socks port and/or some "shim" adapter. Might be able to remove or lessen dependence on LD_PRELOAD (which isn't available everywhere, can be "escaped", and can be a bit fragile). If we continued to use LD_PRELOAD could at least be used to prevent accidental connections to the real network.
Skills: C, familiarity with network namespaces and/or LD_PRELOAD would be helpful. New code could potentially be written in Rust.
# Team:
- jnewsome
- dgoulet
- boklm [boklm/torsocks-netns](https://gitlab.torproject.org/boklm/torsocks-netns)
### Where to meet: irc initially (jnewsome)
# Problems to address:
- Can't be completely confident that LD_PRELOAD intercepts all network requests.
- LD_PRELOAD can cause surprising breakages
- (Maybe) DNS leaks
# Solutions to explore:
- Network namespaces: Put target process(es) in a network namespace that can only talk to torsocks, getting ?higher confidence that there are no leaks. Stretch: create a synthetic network adapter that talks to tor, so we don't need to use LD_PRELOAD at all.
- SECCOMP: Create a seccomp filter that prevents network-related syscalls that don't originate from the LD_PRELOADd shim, getting higher confidence that there are no leaks. Stretch: use ptrace to rewrite network-related syscalls instead of using LD_PRELOAD.
# Alternative/fallback:
- Go through torsocks issues and increment/fix what we can through incremental improvements and cut a release. [link to issue](https://gitlab.torproject.org/tpo/core/torsocks/-/issues)
# Relevant issues:
- FR to disable network: [issues-26889](https://gitlab.torproject.org/tpo/core/torsocks/-/issues/26889). This mentions that `firejail` could be used for this. Looking at the man page for firejail, `--protocol=unix` seems like it'd do what we want.
- irssi: [issues-11727](https://gitlab.torproject.org/tpo/core/torsocks/-/issues/11727). I'd thought there was a DNS leak, but maybe I misremembered. It looks like the issue is each process getting its own onion-address-resolution-table + irssi using multiple processes.
- torsocks support for unix sockets: [issues-14132](https://gitlab.torproject.org/tpo/core/torsocks/-/issues/14132). (This would be let us disable net access completely)
# Notes:
- tor notes for how to set up a transparent proxy: [TransparentProxy](https://gitlab.torproject.org/legacy/trac/-/wikis/doc/TransparentProxy)
- Maybe we could write a wrapper script to set up a network namespace, configure all traffic in that ns to go through the transparent proxy, and then run the target program(s) in the namespace?
# Results:
- torsocks-netns: [torsocks-netns](https://gitlab.torproject.org/boklm/torsocks-netns:)
- Wrapper for torsocks that protects against inadvertent leaks.
- Puts torsocks + application in an empty network namespace and "smuggles" out to the tor socks port over a unix-pipe tunnel.
- No root or priveleges required!
- Proofs of concept without torsocks/LD_PRELOAD:
- Network namespace with ip tables rules + proxy (implemented via redsocks) to funnel everything to either tor's socks port or tor's DNS port. (Root/privelege required?)
- Tried using tor's transparent proxy functionality + iptables rules in a network namespace, but couldn't get it working. Creating a bridged adapter requires ~root. (Could use a priveleged binary like firejail if installed. firejail's default config doesn't allow it, though)
- Proof of concept using tor's shared onion pool (to address [issues-11727](https://gitlab.torproject.org/tpo/core/torsocks/-/issues/11727), fixing torsocks+irssi)
- torsocks handles resolving onion addresses using "onion cookies"
- irssi uses different processes to resolve vs connect, so doesn't work
- tor now *natively* supports onion cookies
- Modified torsocks to use tor's native onion cookie support and successfully connected to an onion irc server
# Questions/follow-ups from demo:
- Jeremy: how does it compare to [orjail](https://github.com/orjail/orjail)
- Jeremy: stream isolation in these modes? (-i with torsocks)
- Matt: (torsocks.conf supports 'AllowOutboundLocalhost 0|1|2')
- Nick: Tor always returns its local addresses from a given range specified in VirtualAddrNetworkIPv[46] , so you could in theory detect them like that.
defaults are 127.192.0.0/10 and [FE80::]/10
# PROJECT: Prototype network-namespace-based torsocks
## Summary: Reduce or remove torsocks's dependence on LD_PRELOAD
## Problems to address:
- Can't be completely confident that `LD_PRELOAD` intercepts all network requests.
- `LD_PRELOAD` can cause surprising breakages
- (Maybe) DNS leaks
## Solutions to explore:
- Network namespaces: Put target process(es) in a network namespace that can only talk to torsocks, getting higher confidence that there are no leaks. Stretch: create a synthetic network adapter that talks to tor, so we don't need to use `LD_PRELOAD` at all.
- SECCOMP: Create a seccomp filter that prevents network-related syscalls that don't originate from the `LD_PRELOAD`d shim, getting higher confidence that there are no leaks. Stretch: use ptrace to rewrite network-related syscalls instead of using `LD_PRELOAD`.
Alternative/fallback:
- Go through torsocks issues and increment/fix what we can through incremental improvements and cut a release. https://gitlab.torproject.org/tpo/core/torsocks/-/issues
## Skills Needed:
C, familiarity with network namespaces and/or `LD_PRELOAD` would be helpful. New code could potentially be written in Rust.
## Team:
jnewsome
~~dgoulet~~
boklm (https://gitlab.torproject.org/boklm/torsocks-netns)
## Relevant issues:
- FR to disable network: https://gitlab.torproject.org/tpo/core/torsocks/-/issues/26889. This mentions that `firejail` could be used for this. Looking at the man page for firejail, `--protocol=unix` seems like it'd do what we want.
- irssi: https://gitlab.torproject.org/tpo/core/torsocks/-/issues/11727. I'd thought there was a DNS leak, but maybe I misremembered. It looks like the issue is each process getting its own onion-address-resolution-table + irssi using multiple processes.
- torsocks support for unix sockets: https://gitlab.torproject.org/tpo/core/torsocks/-/issues/14132. (This would be let us disable net access completely)
## Notes:
- tor notes for how to set up a transparent proxy: https://gitlab.torproject.org/legacy/trac/-/wikis/doc/TransparentProxy
- Maybe we could write a wrapper script to set up a network namespace, configure all traffic in that ns to go through the transparent proxy, and then run the target program(s) in the namespace?
## Results:
- torsocks-netns: https://gitlab.torproject.org/boklm/torsocks-netns:
- Wrapper for torsocks that protects against inadvertent leaks.
- Puts torsocks + application in an empty network namespace and "smuggles" out to the tor socks port over a unix-pipe tunnel.
- No root or priveleges required!
- Proofs of concept without `torsocks/LD_PRELOAD`:
- Network namespace with ip tables rules + proxy (implemented via redsocks) to funnel everything to either tor's socks port or tor's DNS port. (Root/privelege required?)
- Tried using tor's transparent proxy functionality + iptables rules in a network namespace, but couldn't get it working. Creating a bridged adapter requires ~root. (Could use a priveleged binary like firejail if installed. firejail's default config doesn't allow it, though)
- Proof of concept using tor's shared onion pool (to address https://gitlab.torproject.org/tpo/core/torsocks/-/issues/11727, fixing torsocks+irssi)
- torsocks handles resolving onion addresses using "onion cookies"
- irssi uses different processes to resolve vs connect, so doesn't work
- tor now *natively* supports onion cookies
- Modified torsocks to use tor's native onion cookie support and successfully connected to an onion irc server
## Questions/follow-ups from demo:
- Jeremy: how does it compare to https://github.com/orjail/orjail
- Jeremy: stream isolation in these modes? (-i with torsocks)
- Matt: (torsocks.conf supports 'AllowOutboundLocalhost 0|1|2')
- Nick: Tor always returns its local addresses from a given range specified in VirtualAddrNetworkIPv[46] , so you could in theory detect them like that.
defaults are 127.192.0.0/10 and [FE80::]/10
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment