Sysadmin 101 workshop for new relay operators

added Next label

added Q2 label

assigned to @gus

This workshop will be facilitated by @kushal and @gman999.

When? June 4th, 2022, 1900 UTC.
Where? BBB - tor.meet.coop

Tasks:

Create a registration form + survey with problems they have, sysadmin skills.
Share the form on social media and comms channels.
Close the registration on June 4th and share the results with the facilitators.
Invite participants to attend the workshop.

added Doing label and removed Next label

adding @kushal in case not seeing the ticket...

Announce proposal for event:

So you want to contribute to the Tor network by running a relay or maybe a bridge?

The Tor network is the most important tool for evading surveillance and bypassing internet censorship. And Tor relays and bridges are vital to the health and integrity of the Tor network. Millions of users rely on relays and bridges to stay safe, and how you configure and maintain that relay or bridge is critical.

Your role is vital.

Running a relay or a bridge raises frequent questions.

should I run a relay or a bridge?
should I run a relay or a bridge from a residential/home internet connection?
which operating system should I run for my Tor node (hint: the one you are most comfortable with securing and maintaining)
more generally, what does it take to keep that relay or bridge operating safely, but both you and Tor users?

This event will start with a presentation seeking to approach some of the core issues that arise when running a Tor node. The session will move into an "ask me anything" discussion to approach other common and less common questions.

Geared towards current and prospective Tor bridge and relay operators, particularly those relatively new to running public internet services.

Seasoned Linux and BSD Tor operators will be attending the event ready to address the discussion.

A shorter version:

Join us June 4th at 1900 UTC for new and prospective Tor relay and bridge operators on the basic "sysadmin foo" required to contribute to the network.

A short introduction will approach some of the basics, like "Should I run a bridge or a relay?" and "Which operating system should I run?" (hint: the one you are most comfortable running). Feel free to come with your questions and to answer other people's questions.

Some edits inline based on https://forum.torproject.net/t/workshop-sysadmin-101-for-new-relay-operators/3400

Join us June 4th at 1900 UTC for new and prospective Tor relay and bridge operators on the basic “sysadmin foo” required to contribute to the network.

So you want to contribute to the Tor network by running a relay or maybe a bridge?

So you want to contribute to the open-source Tor network by running a relay or maybe a bridge?

The Tor network is the most important open-source tool for evading surveillance and bypassing internet censorship. Volunteer-operated Tor relays and bridges are vital to the health and integrity of the Tor network. Millions of users rely on relays and bridges to stay safe, and how you configure and maintain that relay or bridge is critical.

maybe drop "open-source" here then?

Your role is vital.

Volunteers aren't a nice enhancement. They are a core feature.

Running a relay or a bridge raises frequent questions.
Should I run a relay or a bridge?
Should I run a relay or a bridge from a residential/home internet connection?
Which operating system should I run for my Tor node (hint: the one you are most comfortable with securing and maintaining)
More generally, what does it take to keep that relay or bridge operating safely, but both you and Tor users?
This workshop will start with a presentation seeking to approach some of the core issues that arise when running a Tor node. The session will move into an “ask me anything” discussion to approach other common and less common questions.

This workshop will start with a presentation approaching some of the ...

Geared towards current and prospective Tor bridge and relay operators, particularly those relatively new to running public internet services.

The 90-minute event will be [g]eared towards..

Seasoned Linux and BSD Tor operators will be attending the event ready to address the discussion.

Done!

Here's the survey result (06-03-2022):

1. Do you run Tor relays or bridges?

59%: No, not yet
38%: Yes
3%: No response

2. What type of relay?

51%: I'm not running a relay, but I want to
23%: Guard/middle
15%: Bridge
15%: Exit
13%: I'm running a snowflake proxy
8%: No response

3. I'm running my relay(s) on a

55%: I'm not running any relays
27%: VPS, cloud or other virtualized system
17%: Dedicated hardware
7%: No response

4. Do you use a configuration management tool for your relays?

73%: Nope
12%: No response
9%: Yes, my own scripts
7%: Yes, I'm using Ansible (relayor) / Puppet / other tool

5. On which operating system(s) are you running your relay(s)?

40%: None
22%: Debian
16%: Ubuntu
11%: No response
8%: FreeBSD
8%: Other Linux
7%: Raspbian
5%: CentOS
3%: Fedora
2%: Arch Linux
2%: Windows
1%: OpenBSD
0%: NetBSD
0%: Other BSD

6. Where are you running your relay

48%: Not yet
31%: Commercial datacenter
17%: Home/residential connection
9%: No response
2%: Other
1%: University

7. How would you rate your experience about running Tor relays or bridge?

53%: I'm not running a bridge or relay, but I want to contribute
15%: I've been running my little relay for a few months now
13%: I'm running my relays for many years now
11%: I'm running more than one relay for some months
8%: No response

8. Do you have previous experiences running internet-facing services?

59%: Yes
34%: No
7%: No response

9. Have you ever attended to a Tor Relay Operator Meetup?

82%: No
12%: Yes
6%: No response

Sysadmin 101 notes - June 4th 2022 ~ 67 people in the workshop

Resources

Join the relay operator community: - IRC channel: #tor-relays on irc.oftc.net - Matrix channel: #tor-relays:matrix.org - Having issues to get in touch? Check this page: https://support.torproject.org/get-in-touch/irc-help/ - Mailing lists: - Tor-relays: https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays - Tor-announce: https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-announce

Tor Relay documentation:
- Documentation: https://community.torproject.org/relay/
- Support: https://support.torproject.org/relay-operators/
- Training: https://community.torproject.org/training/resources/tor-relay-workshop/
- Expectations: https://gitlab.torproject.org/tpo/community/team/-/wikis/Expectations-for-Relay-Operators
- Social contract and code of conduct: https://gitweb.torproject.org/community/policies.git/tree/ https://support.torproject.org https://community.torproject.org https://forum.torproject.org Next Relay Operator Meetup Online: June 25th 1900 UTC Relay operators meetup in EMF Camp in UK on June 5th: https://www.emfcamp.org/schedule/2022/425-tor-node-operators-meetup
Other Ressources
- slides: https://nycbug1.nycbug.org/
- survey stats #36 (comment 2810037)
- Running a relay isn't for everyone. If you're not comfortable running your own relay, consider running a Snowflake or Donating Here to one of the many non-profits that run exit relays: https://community.torproject.org/relay/community-resources/relay-associations/
- NSA "Tor stinks" url from the Guardian
https://commons.wikimedia.org/wiki/File:Tor_Stinks.pdf
- Metrics https://metrics.torproject.org/

Q/A

How many people signed up?
- 100+. With 60-70 attendees in practice.
Tor log: there have been x users in the last 6 hours... What's the algorithm for what a distinct tor user is? (torix)
- I believe bridges count it by IP address, rounded up to the next multiple of 8. Your bridge also publishes these stats plus more in its "extrainfo" descriptor, which you can find in https://collector.torproject.org/recent/bridge-descriptors/extra-infos/ and maybe also in the stats/ directory in your DataDirectory.
How much time (per week or month) and how many times, should you plan to invest?
- Depends on what you're doing and how you're doing it.
- "My eyeballs are the first line of defense." Watching the Tor logs, watching the system logs, can help you get more comfortable with how things are going (and what they look like when things are going fine).
What are the regular monitoring or upkeep activities we should be performing to not "set it and forget it"
- Log in regularly. Check for updates and if your box needs to be rebooted. (Set an alarm or calendar event to log in and check.) If using Debian/Ubuntu, enable UnattendedUpgrades.
- prometheus: https://forum.torproject.net/t/suggestion-a-summary-page-of-relay-bridge-install-guides-in-one-place/2425/4?u=gus
- george is into "agentless monitoring"
What are acceptable domains or communications approaches for listing in ContactInfo? E.g. what about a duck address.
- Use any domain that you use for normal communication. Don't use an address that you never check.
- Any contact info that you regularly check.
- DO NOT obfuscate your contact information! Maintainers already burn a lot of time trying to decipher obfuscated contact info!
- (Some people are concerned about spam, and that's why they try to obfuscate the address. But actually, spam isn't so bad these days; or if it is for you, consider using a separate email address for your contact info.)
Is a relay that allows exits to port 53 but routes those queries to a pihole considered a bad node that is tampering with traffic?
- Please no! Don't mess with exit traffic. Redirecting outgoing tcp port 53 connections to somewhere else is going to break things.
  - There have been cases where a DNS on a distinct machine increased performance
What is the best way to figure out if a bridge/IP got burned (i.e. blocked in certain countries)? What should be rotation intervals?
- At the beginning of 2022, we added a new feature where we're measuring reachability of bridges from inside Russia, and annotating relay-search with the results.
- Check metrics.torproject.org, there will be indicators if your bridge is blocked or not
- This "your bridge is blocked in Russia" feature is in-progress: the user experience at the end is not intended to be "you have to watch your metrics page and then go cycle your IP address manually". So don't worry too much about reacting to the relay-search page yet.
Can you release a Docker image instead? Why create OS-specific packages when you can Docker?
- Gus will share some images and scripts
- Bridge docker image: https://community.torproject.org/relay/setup/bridge/docker/
- #43
Can you talk about partial exit nodes? For example, I block all ports except for 8333 (Bitcoin), 9999 (DASH) & 53 (DNS/TCP)
- You need to allow exiting to ports 80 and 443 in order to get the Exit flag, and most clients skip relays that don't have the Exit flag.
- So, yes you can do just those tiny set of ports, but mostly clients will skip over your relay.
Can we run a relay on a 1 GB RAM Server? Yes, have an exit at Frantech with 1 GB https://metrics.torproject.org/rs.html#details/D00795330D77C75344C54FB8800531FAB3C40FBE
- Thanks, got it. Can you share how much memory footprint does running a relay have? Like 200-300 MBs?
This is traffic dependent, lowest is 512 MB (source) the more data being moved the more RAM needed till you reach other bottlenecks (Such as bandwidth limits/CPU)
Is it legal/okay to run a relay node on oracle cloud free tier servers?

-I run 2 guards and a bridge on their free ARM machines and have not had problems with them so far.
- Check your provider's TOS.
- Middle nodes rarely cause issues, exits cause abuse, so ask beforehand.
What is the "Sandbox" option in torrc? Would it be a good idea to run relays with it?
- Sandbox enables certain seccomp style rules to prevent your Tor relay from doing surprising activities (like unexpected syscalls). I would say "if your Tor package turns on Sandbox, then yes, use it", but if your package doesn't use it, it is fine to ignore for now.
Can you elaborate on deploying host-based and network-based intrusion detection for relays? What are best practices for not leaking sensitive data?
- One approach: do not store or let sensitive data touch your "unsecured" box in the first place.
NTP or SNTP for a Tor relay?

Since Debian Buster and Bullseye, timesyncd is enabled by default. Systemd-timesyncd is an SNTP client which is less accurate than NTP. I mean, Roger once said that the time accuracy in the Tor network of 1-2s is sufficient. You can also run chronyd and use NTS: https://anweshadas.in/how-to-use-network-time-security-on-your-system/ https://www.whonix.org/wiki/Network_Time_Synchronization https://tails.boum.org/contribute/design/Time_syncing/#index1h1

Lessons for next time: in the notification / reminder emails, suggest that people show up 5-15 minutes early so they have time to mess with their audio and make it work.+1+1
How about using singleboard computers like beagleboard, olimex for setting up bridges ?
- Whatever you're comfortable with. They can work. They might make your experience more exciting / more work. If you're excited about gaining that new experience, go for it.
- Make sure the hardware you pick can handle the throughput: fast relays don't easily fit on tiny hardware.
What is the advantage of ed25519 over RSA and should I change my keys? First and foremost ed25519 is more break-proof as compared to RSA. Second it also encrypts keys into a compatible OpenSSH format by default. Third if you must use RSA use only 4096 bits. Fourth YES, You should change keys asap.
- If you mean your Tor relay identity keys, Tor will do that for you automatically: for the past few years Tor will generate a new ED key for you automatically.
- But if you mean ssh key or gpg key or etc, listen to kushal
How accurate from "true" UTC does it matter?
- A few seconds is the limit, but NTP should get sub 1 second constantly, so should not be a issue
Instead of S/NTP, can we use a GPS receiver as a time source and synchronize your computer's clock to that?
is it ok to run Tor on a FreeBSD jail?
- Yes, many people do.
Why shouldn't I list my bridges in MyFamily?
- Gives off information about related nodes, for the anonymizing layer that may be bad
myfamily is bidirectional, and your bridge fingerprint is supposed to remain a secret, so if you list your bridge fingerprint in your relay's MyFamily, it's no longer secret.

There are future plans for changing Tor's design so you can list your bridges in your family: https://gitweb.torproject.org/torspec.git/tree/proposals/321-happy-families.md

...So in the future this will get more confusing because we will change to telling you to be sure to list your bridge in your MyFamily. :)
Does the ORPort number matter? In the beginning, the ORPort mattered a little bit, because some users are behind firewalls that only let them reach e.g. ports 80 and 443. that's why you see a bunch of relays using 443 for their ORPort. It matters a bit less these days I'd say, because more censorship is based on things other than destination port.
I set my ORPort to 22 and SSHD to 22222. Is this in any way a bad idea?
- One upside is picking 22 for your ORPort is: it helps users get around censorship, if they are allowed to use port 22, but they're not allowed to use other ports like 9001.
- One downside to picking 22 for your ORPort is: network admins around the internet look for connections to port 22 and freak out when they see them. So they see a user on their system connecting to your port 22, misunderstand and assume it's ssh, and send a warning mail or block you or something.
- DPI can distinguish ssh traffic from tor traffic, so maybe obfuscation through setting your ORPort on a known service port is not a good idea, as the service (ssh) has a distinct profile that is easily profiled and can be inspected for "baseline" behavior.
Should I use full disk encryption (FDE)? What are the tradeoffs? If you have a serial console, yes. But after a (re)boot, the password must be inserted. the torservers.net people actually recommend don't use full disk encryption for exit relays your own hardware, since that way if law enforcement steals your server it will be quickly clear that there is nothing on it. (Otherwise they stare at your encrypted disk and assume the worst.) For non exit relays, the tradeoff is more blurry. There have been cases where law enforcement shows up to steal guard relays, e.g. because they have some case where they think a user connected to your IP address. But also, protecting your relay identity key from external attack (offline masterkey) is useful. But also, you should be willing to throw away your key and start with a new one if anything suspicious happens.
Bridges should not have set myfamily, yet via metrics searching for the contact one can find relays and bridges operated with the same contact (therefore a family). why is that? [should merge this Q with the family Q above]
Does tor project support IRC block tor? https://support.torproject.org/abuse/tor-ban-irc/
there's been reports of large scale malicious relays in operation, how are we dealing with that problem? https://blog.torproject.org/malicious-relays-health-tor-network/
Is it ok to run multiple snowflake instances in a single network (for example an university network)? What is the recommended maximum of instances per network/IP-block/IP? (And why?)
- Yes it is ok.
- Almost all of our Snowflake volunteers are behind restricted NATs, which makes them less useful because they can't be matched with Snowflake users who are also behind restricted NAT. So if you have a choice, try to make your Snowflake not behind a restricted NAT.
If I reinstall my server, should I keep the keys from the earlier install, or should I generate new keys?
- If you reinstall servers, you should back up your key if it is not compromised or sign your "per box" key with your offline master key. Your per box key is your "reputation" if you don't have an offline master key.
Offline masterkey?

You sign your "per box key" with a key that never touches your physical hardware or VPS that runs the tor proxy, so the chances of it being compromised is a lot less than the key that resides on your hardware.

See -> https://support.torproject.org/relay-operators/upgrade-or-move/
Can you run firefox/chrome snowflake and a middle relay/obfs4 bridge from same home ip?
- Don't mix things, different systems is a good idea in a local LAN
- The reason to avoid having both kinds of bridges on one IP address is that if the IP address gets blocked because of one kind (e.g. Russia discovers your obfs4 bridge using bridges.torproject.org), then the other kinds (like your Snowflake) end up blocked too.

How useful is a middle relay these days with only 6 Mb/s upstream (because 10+ is required now)?

Not as useful as it was a decade ago. Consider running a bridge or a Snowflake on a connection like that. Thanks for contributing!

Notes:

- SECURITY IS IMPORTANT!
- DO NOT BE A MALICIOUS NODE OPERATOR! DO NOT PROXY!
- AUTOMATE PATCHING!!! (unattended-upgrades)
- Diversity is important: different tech stacks (network, OS, hardware)
- Automate OS updates
- Use selinux and built in security features. (see "SELinux for mortals")
- use ssh key authentication with ed25519 keys, dont use the default ssh port
- use ssh Postquantum crypto: KexAlgorithms sntrup761x25519-sha512@openssh.com
- disable root user and SSH password authentication (disable it).
- Use firewalls and fail2ban
- Use hardware kaystores (Yubikey, Solokey and Nitrokey)
  https://kushaldas.in/posts/ssh-authentication-using-fido-u2f-hardware-authenticators.html
 https://kushaldas.in/posts/using-your-openpgp-key-on-yubikey-for-ssh.html
- Use UTC on public facing servers; Makes correating events easier
- If possible: use NTS 
https://anweshadas.in/how-to-use-network-time-security-on-your-system/
- Optional but recommended: @daily emails and monitoring notifications
- Optional: Diversity in DNS servers (DO NOT MESS WITH DNS OR YOU HAVE A HIGH POSSIBILITY OF BEING IDENTIFIED AS A BAD RELAY)
- Recommended: If you run multiple tor nodes check "my family" option in torrc
- Communicate with the community, join irc, matrix, mailing list .... 
(Tor is a COMMUNITY effort!) 
IRC channels #tor #tor-relays on OFTC.
- Read change logs
- Optional but recommended: keep a changelog of tor versions on your client (there are tradeoffs when it comes to doing your adversary's job of enumartion though)
- Run a snowflake in your browser! It is a low effort, high impact contribution to the network!
You can run snowflake from a firefox extension: https://addons.mozilla.org/en-CA/firefox/addon/torproject-snowflake/

     - Other options here: https://snowflake.torproject.org/#extension

Next meetup ONLINE for relay operators: June 25, 1900 UTC <-- details will be announced in the tor-relay mailing list. https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Upcoming Hackweek, in late June: https://hackweek.onionize.space/hackweek/

Post-training survey sent!

Survey result: 11 answers.

Did you feel the format of the event (slides with presentation, with chat questions addressed during and then again after) seem appropriate?

    10 (91%): Yes, I enjoyed the format of the event
    1 (9%): I didn't attend the event
    0 (0%): No response
    0 (0%): No, I didn't enjoy the format of the event. 

If you didn't enjoy the format of the event, what you would like us to change next time?

    9 (82%): No response
    N/A
    I'd try to eliminate mic popping; also non-speaking presenters should mute their mics when not speaking; snuffling should be avoided.


Did you start running a new relay or bridge after the event?

    5 (45%): No, not yet
    3 (27%): No, and I don't have plans to do it
    2 (18%): No response
    1 (9%): Yes

If so, where is the node hosted?


    6 (55%): No response
    2 (18%): residential connection
    2 (18%): dedicated hardware in a data center
    1 (9%): vps or cloud provider

What operating system is it running?

    6 (55%): No response
    3 (27%): Debian
    1 (9%): Ubuntu
    1 (9%): Arch Linux

How was your experience using the video platform BigBlueButton (tor.meet.coop)?

    8 (73%): Good. It worked well!
    2 (18%): Ok, I had some issues connecting to BigBlueButton, but it's fine
    1 (9%): Bad. I had some issues with BigBlueButton and I couldn't attend
    0 (0%): No response
    0 (0%): I didn't attend / missed the workshop 

Were there questions you had that went unaddressed?

    9 (82%): No, all the questions I had were addressed
    1 (9%): No response
    1 (9%): Yes, some of my questions went unaddressed 

Were there other general topics we didn't address that maybe should have been included?

    8 (73%): No, the topics addressed all the questions that I had.
    2 (18%): Yes, there are other general topics that I would like you to address. (Please see the next question)
    1 (9%): No response 

If so, which topics went unaddressed?

    8 (73%): No response
    More specific workshops on nitty-gritty details.
    N/A
    Implementazione pratica di un nodo, sarebbe stato bello vedere proprio la creazione di un Relay

Overall, did you feel that the workshop was useful?

    3 (27%): No response
    Yes, I'm pretty confident about running a relay after the workshop
    It was fun but very technologically basic.
    Yes. There were many people who had experience with server hosting.
    Yes, it was clear at the end on who should and should not be running a relay/node.
    Assolutamente si
    Yes
    mostly
    Yes

How confident do you feel in running a relay/bridge?

    4 (36%): 3 - Very confident
    3 (27%): 1 - Not at all
    3 (27%): 2 - Confident
    1 (9%): No response 

Please list topics that you would like us to cover in the next workshop

    8 (73%): No response
    Overoptimisation. Local DNS server for example.
    Diversity: FreeBSD, HardenedBSD, openSolaris (illumos.org)
    POWER9 + Coreboot (raptorcs.com)  RISC-V
    Watchdog Timeout error while running a tor relay in arch linux 

Are you attending the next Tor relay operator meetup (June 25th, 1900 UTC)?

    8 (73%): Yes! Count me in!
    3 (27%): No
    0 (0%): No response

Notes for the next workshop:

Have a demo or hands on session
Check with meet.coop if there's a participation cap (max 70 pax?)
Run an audio test: test our mics before the workshop

closed

mentioned in issue #63

Sysadmin 101 workshop for new relay operators

Relay Operator - Sysadmin 101

Child items ...

Activity

Sysadmin 101 notes - June 4th 2022 ~ 67 people in the workshop

Resources

Q/A