- Tutorial
- Howto
- Using the ZNC IRC bouncer
- Changing your ZNC password
- Connecting to ZNC from an IRC client
- OFTC groups
- Using the Matrix bridge
- Adding channels to the Matrix bridge
- Pager playbook
- Disaster recovery
- Reference
- Installation
- Installation: ZNC
- Goals
- Necessary software
- Setup steps
- Obtain necessary software
- Create a special user
- Create a sudo password for yourself
- Choose a FQDN and get a TLS certificate
- Couple nice things
- Create initial ZNC config
- Create TLS cert that ZNC can read
- Create ZNC system service
- Access web interface
- Add TLS listener for ZNC
- Make ZNC reachable without tricks
- Adding a ZNC user
- SLA
- Design
- Issues
- Monitoring and testing
- Logs and metrics
- Backups
- Discussion
- Goals
- Must have
- Nice to have
- Non-Goals
- Approvals required
- Proposed Solution
- Cost
- Alternatives considered
- Discarded alternatives
IRC is the original Internet Relay Chat, one the first (1988) protocol created for "chatting" in real-time on the Internet, and the oldest one still in use. It is also one of the oldest protocols still active on the internet, predating the web by a few years.
This page is mostly a discussion of software that runs on top of IRC and operated by end users.
- Tutorial
- Howto
-
Reference
- Installation
-
Installation: ZNC
- Goals
- Necessary software
-
Setup steps
- Obtain necessary software
- Create a special user
- Create a sudo password for yourself
- Choose a FQDN and get a TLS certificate
- Couple nice things
- Create initial ZNC config
- Create TLS cert that ZNC can read
- Create ZNC system service
- Access web interface
- Add TLS listener for ZNC
- Make ZNC reachable without tricks
- Adding a ZNC user
- SLA
- Design
- Issues
- Monitoring and testing
- Logs and metrics
- Backups
- Discussion
Tutorial
Tor makes extensive use of IRC with multiple active channels on the OFTC network. Our user-visible documentation is at this wiki page.
Howto
We do not operate the OFTC network. The public support channel for
OFTC is #oftc
.
Using the ZNC IRC bouncer
The last time this section was updated (or that someone remembered to update the date her) is: 28 Feb 2020. The current ZNC admin is pastly. Find him on IRC or at pastly@torproject.org if you need help.
You need:
- your ZNC username. e.g.
jacob
. For simplicity, the ZNC admin should have made sure this is the same as your IRC nick - your existing ZNC password. e.g.
VTGdtSgsQYgJ
- a new password
Changing your ZNC password
If you know your existing one, you can do this yourself without the ZNC admin.
Given the assumptions baked into the rest of this document, the correct URL to visit in a browser is https://ircbouncer.torproject.org:2001/. There is also a hidden service at http://eibwzyiqgk6vgugg.onion/.
- log in with your ZNC username and password
- click Your Settings in the right column menu
- enter your password in the two boxes at the top of the page labeled Password and Confirm Password
- scroll all the way down and click Save
Done. You will now need to remember this new password instead of the old one.
Connecting to ZNC from an IRC client
Every IRC client is a little different. This section is going to tell you the information you need to know as opposed to exactly what you need to do with it.
- For a nick, use your desired nick. The assumption in this document is
jacob
. Leave alternate nicks blank, or if you must, add an increasing number of underscores to your desired nick for them:jacob_
,jacob__
... - For the server or hostname, the assumption in this document is
ircbouncer.torproject.org
. - Server port is 2001 based on the assumption blah blah blah
- Use SSL/TLS
- For a server password or simply password (not a nickserv password: that's
different and unnecessary) use
jacob/oftc:VTGdtSgsQYgJ
.
That should be everything you need to know. If you have trouble, ask your ZNC admin for help or find someone who knows IRC. The ZNC admin is probably the better first stop.
OFTC groups
There are many IRC groups managed by GroupServ
on the OFTC network:
@tor-chanops
@tor-ircmasters
@tor-ops
@tor-people
@tor-randoms
@tor-tpomember
@tor-vegas
People generally get access to things through one or many of the above groups. When someone leaves, you might want to revoke their access, for example with:
/msg GroupServ access @tor-ircmasters del OLDADMIN
Typically, you will need to add users to the @tor-tpomember
group,
so that they can access the internal channels
(e.g. #tor-internal
). This can be done by the "Group Masters", which
can be found by talking with GroupServ
:
/msg GroupServ info @tor-tpomember
Then you would add them to the group with:
/msg GroupServ access @tor-tpomember add $USER MEMBER
Using the Matrix bridge
Since mid-April 2021, many #tor-*
channels are bridged (or
"plumbed") between the OFTC IRC network and the
Matrix.org home server. This is still somewhat of an experimental
status, but so far it seems to be working well.
By default, you will appear on IRC as a user like "YourMatrixName [m]".
You can override this by sending !nick <yournick>
to "OFTC IRC Bridge
status". If your nick is registered, you will get a PM from NickServ stating
that you need to authenticate. Do so by responding with identify \<yourpassword\>
.
Also note that you can join any channel on bridged IRC networks (Freenode and
OFTC) through the Portal rooms functionality.
For example, #freenode_#channelname:matrix.org
corresponds to #channelname
on Freenode. This is the only supported way to access Tor's internal channels
through Matrix (you will need to set your nick first; see above). To join a portal room, send
!join #channel \<channel password\>
to "OFTC IRC Bridge status".
For more information see the general Matrix bridge documentation and the IRC bridge documentation.
Adding channels to the Matrix bridge
File a ticket in the TPA tracker, with the ~IRC label. Operators of the Matrix bridge need to add the channel, you can explicitly ping @gus and @ahf since they are the ones managing the bridge.
Pager playbook
Disaster recovery
Reference
We operate a virtual machine for people to run their IRC clients,
called chives
.
A volunteer (currently pastly) runs a ZNC bouncer for TPO people on their own infrastructure.
Some people connect to IRC intermittently.
Installation
The new IRC server has been setup with the roles::ircbox
by weasel
(see ticket #32281) in october 2019, to replace the older
machine. This role simply sets up the machine as a "shell server"
(roles::shell
) and installs irssi
.
Installation: ZNC
This section documents how pastly set up ZNC on TPA infra. It was originally written 20 Nov 2019 and the last time someone updated something and remembered to update the date is:
Last updated: 20 Nov 2019
Assumptions
- Your username is
pastly
. - The ZNC user is
ircbouncer
. - The host is
chives
.
Goals
- ZNC bouncer maintaing persistent connections to irc.oftc.net for "Tor people" (those with @torproject.org addresses is pastly's litmus test) and buffering messages for them when they are not online
- Insecure plaintext connections to ZNC not allowed
- Secure TLS connections with valid TLS certificate
- Secure Tor onion service connections
- ZNC runs as non-root, special-purpose, unprivileged user
At the end of this, we will have ZNC reachable in the following ways for both web-based configuration and IRC:
- Securely with a valid TLS certificate on port 2001 at ircbouncer.torproject.org
- Securely via a Tor onion service on port 80 and 2000 at some onion address
Necessary software
-
Debian 10 (Buster)
-
ZNC, tested with
pastly@chives:~$ znc --version ZNC 1.7.2+deb3 - https://znc.in IPv6: yes, SSL: yes, DNS: threads, charset: yes, i18n: no, build: autoconf
-
Tor (optional), tested with
pastly@chives:~$ tor --version Tor version 0.3.5.8.
Setup steps
Obtain necessary software
See previous section
Create a special user
Ask your friendly neighborhood Tor sysadmin to do this for you. It needs its
own home directory and you need to be able to sudo -u
to it. For example:
pastly@chives:~$ sudo -u ircbouncer whoami
[sudo] password for pastly on chives:
ircbouncer
But to do this you need ...
Create a sudo password for yourself
If you don't have one already.
-
Log in to https://db.torproject.org/login.html with the Update my info button. Use your LDAP password.
-
Use the interface to create a sudo password. It probably can be for just the necessary host (chives, for me), but I did it for all hosts. It will give you a gpg command to run that signs some text indicating you want this change. Email the resulting block of armored gpg output to changes@db.torproject.org.
-
After you get a response email indicating success, want 10 minutes and you should be able to run commands as the
ircbouncer
user.pastly@chives:~$ sudo -u ircbouncer whoami [sudo] password for pastly on chives: ircbouncer
Choose a FQDN and get a TLS certificate
Ask your friendly neighborhood Tor sysadmin to do this for you. It could be chives.torproject.org, but to make it easier for users, my Tor sysadmin chose ircbouncer.torproject.org. Have them make you a valid TLS certificate with the name of choice. If using something like Let's Encrypt, assume they are going to automatically regenerate it every ~90 days :)
They don't need to put the cert/keys anywhere special for you as long as the ircbouncer user can access them. See how in this ticket comment ...
root@chives:~# ls -al /etc/ssl/private/ircbouncer.torproject.org.* /etc/ssl/torproject/certs/ircbouncer.torproject.org.crt*
-r--r----- 1 root ssl-cert 7178 nov 18 20:42 /etc/ssl/private/ircbouncer.torproject.org.combined
-r--r----- 1 root ssl-cert 3244 nov 18 20:42 /etc/ssl/private/ircbouncer.torproject.org.key
-r--r--r-- 1 root root 2286 nov 18 20:42 /etc/ssl/torproject/certs/ircbouncer.torproject.org.crt
-r--r--r-- 1 root root 1649 nov 18 20:42 /etc/ssl/torproject/certs/ircbouncer.torproject.org.crt-chain
-r--r--r-- 1 root root 3934 nov 18 20:42 /etc/ssl/torproject/certs/ircbouncer.torproject.org.crt-chained
And the sysadmin made ircbouncer part of the ssl-cert group.
ircbouncer@chives:~$ id
uid=1579(ircbouncer) gid=1579(ircbouncer) groups=1579(ircbouncer),116(ssl-cert)
Couple nice things
-
Create a .bashrc for ircbouncer.
pastly@chives:~$ sudo -u ircbouncer cp /home/pastly/.bashrc /home/ircbouncer/.bashrc
-
Add proper
XDG_RUNTIME_DIR
to ircbouncer's .bashrc, only optional if you can remember to do this every time you interact with systemd in the futurepastly@chives:~$ sudo -u ircbouncer bash ircbouncer@chives:/home/pastly$ cd ircbouncer@chives:~$ echo export XDG_RUNTIME_DIR=/run/user/$(id -u) >> .bashrc ircbouncer@chives:~$ tail -n 1 .bashrc export XDG_RUNTIME_DIR=/run/user/1579 ircbouncer@chives:~$ id -u 1579
Create initial ZNC config
If you're rerunning this section for some reason, consider deleting everything and starting fresh to avoid any confusion. If this is your first time, then ignore this code block.
ircbouncer@chives:~$ pkill znc
ircbouncer@chives:~$ rm -r .znc
Now let ZNC guide you through generating an initial config. Important decisions:
-
What port should znc listen on initially? 2000
-
Should it listen on that port with SSL? no
-
Nick for the admin user? I chose pastly. It doesn't have to match your linux username; I just chose it for convenience.
-
Skip setting up a network at this time
-
Don't start ZNC now
ircbouncer@chives:~$ znc --makeconf [ .. ] Checking for list of available modules... [ ** ] [ ** ] -- Global settings -- [ ** ] [ ?? ] Listen on port (1025 to 65534): 2000 [ ?? ] Listen using SSL (yes/no) [no]: [ ?? ] Listen using both IPv4 and IPv6 (yes/no) [yes]: [ .. ] Verifying the listener... [ ** ] Unable to locate pem file: [/home/ircbouncer/.znc/znc.pem], creating it [ .. ] Writing Pem file [/home/ircbouncer/.znc/znc.pem]... [ ** ] Enabled global modules [webadmin] [ ** ] [ ** ] -- Admin user settings -- [ ** ] [ ?? ] Username (alphanumeric): pastly [ ?? ] Enter password: [ ?? ] Confirm password: [ ?? ] Nick [pastly]: [ ?? ] Alternate nick [pastly_]: [ ?? ] Ident [pastly]: [ ?? ] Real name (optional): [ ?? ] Bind host (optional): [ ** ] Enabled user modules [chansaver, controlpanel] [ ** ] [ ?? ] Set up a network? (yes/no) [yes]: no [ ** ] [ .. ] Writing config [/home/ircbouncer/.znc/configs/znc.conf]... [ ** ] [ ** ] To connect to this ZNC you need to connect to it as your IRC server [ ** ] using the port that you supplied. You have to supply your login info [ ** ] as the IRC server password like this: user/network:pass. [ ** ] [ ** ] Try something like this in your IRC client... [ ** ] /server \<znc_server_ip\> 2000 pastly:\<pass\> [ ** ] [ ** ] To manage settings, users and networks, point your web browser to [ ** ] http://\<znc_server_ip\>:2000/ [ ** ] [ ?? ] Launch ZNC now? (yes/no) [yes]: no
Create TLS cert that ZNC can read
There's probably a better way to do this or otherwise configure ZNC to read straight from /etc/ssl for the TLS cert/key. But this is what I figured out.
- Create helper script
Don't copy/paste blindly. Some things in this script might need to change for you.
ircbouncer@chives:~$ mkdir bin
ircbouncer@chives:~$ cat > bin/znc-ssl-copy.sh
#!/usr/bin/env bash
out=/home/ircbouncer/.znc/znc.pem
rm -f $out
cat /etc/ssl/private/ircbouncer.torproject.org.combined /etc/ssl/dhparam.pem > $out
chmod 400 $out
pkill -HUP znc
ircbouncer@chives:~$ chmod u+x bin/znc-ssl-copy.sh
- Run it once to verify it works
It should be many 10s of lines long. It should have more than 1 BEGIN [THING]
sections. The first should be a private key, then one or more certificates,
and finally DH params. If you need help with this, do not share the contents of
this file publicly: it contains private key material.
ircbouncer@chives:~$ ./bin/znc-ssl-copy.sh
ircbouncer@chives:~$ wc -l .znc/znc.pem
129 .znc/znc.pem
ircbouncer@chives:~$ grep -c BEGIN .znc/znc.pem
4
- Make it run periodically
Open ircbouncer's crontab with crontab -e
and add the following line
@weekly /home/ircbouncer/bin/znc-ssl-copy.sh
Create ZNC system service
This is our first systemd user service thing, so we have to create the
appropriate directory structure. Then we create a very simple znc.service
.
We enable
the service (start it automatically on boot) and use --now
to
also start it now. Finally we verify it is loaded and actively running.
ircbouncer@chives:~$ mkdir -pv .config/systemd/user
mkdir: created directory '.config/systemd'
mkdir: created directory '.config/systemd/user'
ircbouncer@chives:~$ cat > .config/systemd/user/znc.service
[Unit]
Description=ZNC IRC bouncer service
[Service]
Type=simple
ExecStart=/usr/bin/znc --foreground
[Install]
WantedBy=multi-user.target
ircbouncer@chives:~$ systemctl --user enable --now znc
Created symlink /home/ircbouncer/.config/systemd/user/multi-user.target.wants/znc.service → /home/ircbouncer/.config/systemd/user/znc.service.
ircbouncer@chives:~$ systemctl --user status znc
● znc.service - ZNC IRC bouncer service
Loaded: loaded (/home/ircbouncer/.config/systemd/user/znc.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-11-20 15:14:27 UTC; 5s ago
Main PID: 23814 (znc)
CGroup: /user.slice/user-1579.slice/user@1579.service/znc.service
└─23814 /usr/bin/znc --foreground
Access web interface
The sysadmin hasn't opened any ports for us yet and we haven't configured ZNC to use TLS yet. Luckily we can still access the web interface securely with a little SSH magic.
Running this command on my laptop (named cranium) creates an SSH connection
from my laptop to chives over which it will forward all traffic to
127.0.0.1:2000
on my laptop to 127.0.0.1:2000
on chives.
cranium:~ mtraudt$ ssh -L 2000:127.0.0.1:2000 chives.tpo
[... snip the message of the day ...]
pastly@chives:~$
So now I can visit in a browser on my laptop http://127.0.0.1:2000
and gain
access to ZNC's web interface securely.
Add TLS listener for ZNC
Log in to the web interface using the username and password you created during the initial ZNC config creation.
Visit Global Settings from the menu on the right side of the window.
For listen ports, add:
- Port 2001
- BindHost *
- All boxes (SSL, IPv4, ... HTTP) are checked
- URIPrefix /
Click Add and ZNC will open a TLS listener on 2001.
Make ZNC reachable without tricks
-
Ask your friendly neighborhood Tor sysadmin to allow inbound 2001 in the firewall.
I recommend you do not have 2000 open in the firewall because it would allow insecure web and IRC connections. All IRC clients worth using support TLS. If you're super tech savvy and you absolute must use your favorite IRC client that doesn't support TLS, then I think you're smart enough to make an SSH tunnel for your IRC client or use the onion service.
-
Ask your friendly neighborhood Tor sysadmin to configure an onion service.
I'm trying to convince mine to set the following options in the torrc
Log notice syslog # to use 3 hops instead of 6. not anonymous # can't do this if you want a SocksPort SocksPort 0 HiddenServiceSingleHopMode 1 HiddenServiceNonAnonymousMode 1 # actual interesting config HiddenServiceDir /var/lib/tor/onion/ircbouncer.torproject.org HiddenServiceVersion 3 HiddenServicePort 80 2000 HiddenServicePort 2000
This config allows someone to access the web interface simply with http://somelongonionaddress.onion. It also allows them to use somelongonionaddress.onion:2000 in their IRC client like they might expect.
Adding a ZNC user
The last time this section was updated (or that someone remembered to update the date her) is: 28 Feb 2020.
You need:
- the user's desired username (e.g.
jacob
). for simplicity, make this the same nick as their desired IRC nick even though this isn't technically required by ZNC. - the user's desired ZNC password, or a junk initial one for them (e.g.
VTGdtSgsQYgJ
). This does not have to be the same as their nickserv password, and arguably should not be the same for security reasons. - the user's nickserv password (e.g.
upRcjFmf
) if registered with nickserv. If you don't know if they are registered with nickserv, it's important to figure that out now. If yes, it's important to get the password from the user.
IMPORTANT: The user should NOT be logged in to IRC as this nick already. If they are, these instructions will not work out perfectly and someone is going to need to know a bit about IRC/nickserv/etc. to sort it out.
Additional assumptions:
- the user has not enabled fancy nickserv features such as certfp (identify with a TLS cert instead of a password) or connections from specific IPs only. I believe the former is technically possible with ZNC, but I am not going to document it at this time.
- the user wants to connect to OFTC
- the correct host/port for IRC-over-TLS at OFTC is irc.oftc.net:6697. Verify at https://oftc.net.
Have a ZNC admin ...
- log in to the web console, e.g. at
https://ircbouncer.torproject.org:2001
- visit Manage Users in the right column menu
- click Add in the table
- input the username and password into the boxes under Authentication
- leave everything in IRC Information as it is: blank except Realname is
ZNC - https://znc.in
and Quit Message is%znc%
- leave Modules as they are: left column entirely unchecked except chansaver and controlpanel
- under Channels increase buffer size to a larger number such as 1000
- leave Queries as they are: both boxes at 50
- leave Flags as they are: Auth Clear Chan Buffer, Multi Clients, Prepend Timestamps, and Auto Clear Query Buffer checked all other unchecked
- leave everything in ZNC Behavior as it is
- click Create and continue
The admin should be taken to basically the same page, but now more boxes are filled in and--if they were to look elsewhere to confirm--the user is created. Also The Networks section is available now.
The ZNC admin will ...
- click Add in the Networks table on this user's page
- for network name, input
oftc
. For - remove content from Nickname, Alt. Nicname, and Ident.
- for Servers on this IRC network, click Add
- input
irc.oftc.net
for hostname,6697
for port, ensureSSL
is checked, and password is left blank - if the user has a nickserv password, under Modules check nickserv and type the nickserv password into the box.
- click Add Network and return
The admin should be taken back to the user's page again. Under networks, OFTC
should exist now. If the Nick column is blank, wait a few seconds, refresh,
and repeat a few times until it is populated with the user's desired nick. If
what appears is guestXXXX
or is their desired nick and a slight modification
that you didn't intend (i.e. jacob-
instead of jacob
) then there is a
problem. It could be:
- the user is already connected to IRC, when the instructions stated at the beginning they shouldn't be.
- someone other than the user is already using that nick
- the user told you they do not have a nickserv account, but they actually do and it's configured to prevent people from using their nick without identifying
If there is no problem, the ZNC admin is done.
SLA
No specific SLA has been set for this service
Design
Just a regular Debian server with users from LDAP.
Issues
There is no issue tracker specifically for this project, File or search for issues in the team issue tracker with the ~IRC label.
Monitoring and testing
Logs and metrics
Backups
ZNC does not, as far as we know, require any special backup or restore procedures.
Discussion
This page was originally created to discuss the implementation of "bouncer" services for other staff. While many people run IRC clients on the server over an SSH connexion, this is inconvenient for people less familiar with the commandline.
It was therefore suggested we evaluate other systems to allow users to have more "persistence" online without having to overcome the "commandline" hurdle.
Goals
Must have
- user-friendly way to stay connected to IRC
Nice to have
- web interface?
- LDAP integration?
Non-Goals
- replacing IRC (let's not go there please)
Approvals required
Maybe checking with TPA before setting up a new service, if any.
Proposed Solution
Not decided yet. Possible options:
- status quo: "everyone for themselves" on the shell server, znc ran by pastly on their own infra
- services admin: pastly runs the znc service for tpo people inside tpo infra
- TPA runs znc bouncer
- alternative clients (weechat, lounge, kiwiirc)
- irccloud
Cost
Staff. Existing hardware resources can be reused.
Alternatives considered
- irssi in some terminal multiplexer like tmux screen or dtach
- weechat in the same or with another interface like web (Glowbear), Android or iOS
- lounge webchat (nodejs, not packaged in Debian)
- ZNC, a bouncer, currently ran by @pastly on their own infrastructure for some tpo people
- a Matrix gateway like Riot.IM
- KiwiIRC, both a service and a web app we could run
Discarded alternatives
Most other alternatives have been discarded because they do not work with IRC and we do not wish to move away from that platform just yet. Other projects (like qwebirc) were discarded because they do not offer persistence.
Free software projects:
- Briar - tor-based offline-first messenger
- Jabber/XMPP - just shutdown the service, never picked up
- Jitsi - audio, video, text chat
- Mattermost - opensource alternative to slack, not federated
- Retroshare - old, complex, not packaged
- Rocket.chat - not federated
- Scuttlebutt - not a great messaging experience
- Signal - in use at Tor, but poor group chat capabilities
- Telegram - doubts about security reliability
- Tox - DHT-based chat system
- Wire - not packaged in Debian
- Zulip - "team chat", not federated
Yes, that's an incredibly long list, and probably not exhaustive.
Commercial services:
- IRCCloud - bridges with IRC, somewhat decent privacy policy
- Slack - poor privacy policy
- Discord - voice and chat app, mostly for gaming
- Hangouts - Google service
- Whatsapp - tied to Facebook
- Skype - Microsoft
- Keybase - OpenPGP-encrypted chat, proprietary server-side
None of the commercial services interoperate with IRC unless otherwise noted.