conference.md

Documentation on video- or audo-conferencing software like Mumble,
Jitsi, or Big Blue Button.

> con·fer·ence | \ ˈkän-f(ə-)rən(t)s1a : a meeting ("an act or process
> of coming together") of two or more persons for discussing matters
> of common concern. [Merriam-Webster][]

[Merriam-Webster]: https://www.merriam-webster.com/dictionary/conference

While [howto/irc](howto/irc) can also be used to hold a meeting or conference, it's
considered out of scope here.

[[_TOC_]]

<!-- TODO add tutorial and how-to sections from template -->

# Reference

## Installation

TPI is currently using [Big Blue Button](https://bigbluebutton.org/) hosted by [meet.coop](https://meet.coop/)
at <https://tor.meet.coop/> for regular meetings.

## SLA

N/A. Meet.coop has a [disclaimer](https://www.meet.coop/disclaimer/) that serves as a terms of service.

## Design

N/A.

## Issues

There is no issue tracker specifically for this project, [File][] or
[search][] for issues in the [team issue tracker][search].

 [File]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/new
 [search]: https://gitlab.torproject.org/tpo/tpa/team/-/issues

Be warned that TPA does not manage this service and therefore is not
in a position to fix most issues related with this service. Big Blue
Button's [issue tracker is on GitHub](https://github.com/bigbluebutton/bigbluebutton/issues) and meet.coop has a
[forum](https://forum.meet.coop/).

### Known issues

Those are the issues with Big Blue Button we are aware of:

 * mute button makes a sound when pressed
 * breakout rooms do not re-enable audio in main room when completed
 * has no global remote keyboard control (e.g. Mumble has a way to set
   a global keyboard shortcut that works regardless of the application
   in focus, for example to mute/unmute while doing a demo)

## Monitoring and testing

TPA does not monitor this instance.

## Logs and metrics

N/A. Meet.coop has a [privacy policy](https://www.meet.coop/privacy/).

## Backups

N/A.

## Other documentation

 * [Meet.coop](https://www.meet.coop/):
   * [source code](https://git.coop/meet)
   * [forum](https://forum.meet.coop/)
 * [Big Blue Button](https://bigbluebutton.org/):
   * [source code](https://github.com/bigbluebutton/bigbluebutton/)
   * [issue tracker](https://github.com/bigbluebutton/bigbluebutton/issues)

# Discussion

## Overview

With the rise of the SARS-COV-2 pandemic, even Tor, which generally
works remotely, is affected because we were still having physical
meetings from time to time, and we'll have to find other ways to deal
with this. At the start of the COVID-19 pandemic -- or, more
precisely, when isolation measures became so severe that normal
in-person meetings became impossible -- Tor started looking into
deploying some sort of interactive, real-time, voice and ideally video
conferencing platform.

This was originally discussed in the context of internal team
operations, but actually became a requirement for a 3-year project in
Africa and Latin America. It's part of the 4th phase, to support for
partners online. Tor has been doing training in about 11 countries,
but has been trying to transition into partners on the ground, for
them to do the training. Then the pandemic started and orgs are moving
online for training. We reached out to partners to see how they're
doing it. Physical meetings are not going to happen. We have a year to
figure out what to do with the funder and partners. Two weeks ago gus
talked with trainers in brazil, tried jitsi which works well but
facing problems for trainings (cannot mute people, cannot share
presentations). They tried BBB and it's definitely better than Jitsi
for training as it's more like an online classroom.

Discussions surrounding this project started in [ticket 33700](https://bugs.torproject.org/33700) and
should continue there, with decisions and facts gathered in this wiki
page.

## Goals

### Must have

 * video/audio communication for a group of people of approx 2-10 people
 * specifically, work session for teams internal to TPI
 * also, training sessions for people *outside* of TPI
 * host partner organizations in a private area in our infrastructure
 * number of participants: 14 organisations with one training per month, max 4 per month, about 14-15 people per session, less than 20
 * chat fallback
 * have a mobile app 
 * allow people to call in by regular phone
 * a way for one person to mute themselves
 * long term maintenance costs covered

### Nice to have

 * Reliable video support. Video chat is nice, but most video chat
   systems usually require all participants to have video off
   otherwise the communication is sensibly lagged.
 * usable to host a Tor meeting, which means more load (because
   possibly > 20 people) and more tools (like slide sharing or
   whiteboarding)
 * multi-party lightning talks, with ways to "pass the mic" across
   different users (currently done with Streamyard and Youtube)
 * respecting our privacy, peer to peer encryption or at least
   encrypted with keys we control
 * free and open source software
 * tor support

### Non-goals

 * land a man on the moon

## Approvals required

 * grant approvers
 * TPI (vegas?)

The budget will be submitted for a grant proposal, which will be
approved by donors. But considering that it's unlikely such a platform
would stay unused within the team, the chosen tool should also be
approved by the TPI team as well. In fact, it would seem unreasonable
to deploy such a tool for external users without first testing it
ourselves.

## Timeline

 * april 2020: budget
 * early may 2020: proposal to funders
 * june 2020 - june 2021: fourth phase of the training project

## Proposed Solution

## Cost

Pessimistic estimates for the various platforms.

Each solution assumes it requires a dedicated server or virtual server
to be setup, included in the "initial setup". Virtual servers require
less work than physical servers to setup.

The actualy prices quoted from Hetzner but virtual servers would
probably be hosted in our infrastructure which might or might not
incur additional costs.

### Summary

| Platform        | One time  | Monthly        | Other                               |
| --------        | --------  | -------        | ----------------------------------- |
| Mumble          | 20 hours  | €13            | 2h/person + 100$/person for headset |
| Jitsi           | 74 hours  | €54 + 10 hours |                                     |
| Big Blue Button | 156 hours | €54 + 8 hours  |                                     |

### Caveats

 1. **Mumble is harder to use** and has proven to absolutely **require
    a headset** to function reliably
 2. it is assumed that **Jitsi and BBB** will have **similar
    hardware** requirements. this is based on the experience that BBB
    seems to scale better than Jitsi but since it has more features
    might require comparatively more resources
 3. BBB is marked as having a **lesser monthly cost** because their
    development cycle seems slower than Jitsi. that might be **too
    optimistic**: we do not actually know how reliable BBB will be in
    production. preliminary reports of BBB admins seem to say it's
    fairly stable and doesn't require much work after the complex
    install procedure
 4. BBB will take **much more time to setup**. it's more complex than
    Jitsi, but it also requires an Ubuntu, which we do not currently
    support in our infrastructure (and an old version too, so upgrade
    costs were counted in the setup)
 5. current TPA situation is that we will be **understaffed by 50%**
    starting on May 1st 2020, and by **75% for two months during the
    summary**. this project is **impossible to realize** if that
    situation is not fixed, and would still be **difficult to
    complete** with the previous staff availability.

A safe way to ensure funding for this project without threatening the
sability of the team would be to hire at least part time worker
especially for the project, which is 20 hours a month, indefinitely.

### Mumble

Assumed configuration

 * minimal Mumble server
 * no VoIP
 * no web configuration

One time costs:

 * initial setup: 4 hours
 * puppet programming: 6 hours
 * maintenance costs: near zero
 * **Total**: 10 hours doubled to 20 hours for safety

Recurring costs:

 * onboarding training: 2 hours per person
 * mandatory headset: 100$USD per person
 * [CPX31 virtual server](https://console.hetzner.cloud/): €13 per month

### Jitsi

Assumed configuration:

 * single server install on Debian
 * max 14 simultaneous users
 * dial-in capability

One time:

 * initial setup: 8 hours
 * Puppet one-time programming: 6 hours
 * Puppet Jigasi/VoIP integration: 6 hours
 * VoIP provider integration: 16 hours
 * **Total**: 36 hours, doubled to 72 hours for safety

Running costs:

 * Puppet maintenance: 1 hour per month
 * Jitsi maintenance: 4 hours per month
 * [AX51-NVMe physical server](https://www.hetzner.com/dedicated-rootserver/ax51-nvme): €54 per month
 * **Total**: 5 hours per month, doubled to 10 hours for safety, +€54
   per month

### Big Blue Button

Assumed configuration:

 * single server install on Ubuntu
 * max 30 simultaneous users
 * VoIP integration

One time fee:

 * initial setup: 30 hours
 * Ubuntu installer and auto-upgrade configuration: 8 hours
 * Puppet manifests Ubuntu port: 8 hours
 * VoIP provider integration: 8 hours
 * One month psychotherapy session for two sysadmins: 8 hours
 * Ubuntu 16 to 18 upgrade: 16 hours
 * **Total**: 78 hours, doubled to 156 hours for safety

Running costs:

 * BBB maintenance: 4 hours per month
 * [AX51-NVMe physical server](https://www.hetzner.com/dedicated-rootserver/ax51-nvme): €54 per month
 * **Total**: 4 hours per month, doubled to 8 hours for safety, +€54
   per month

## Alternatives considered

### mumble

#### features

 * audio-only
 * moderation
 * multiple rooms
 * native client for Linux, Windows, Mac, iOS, Android
 * [web interface](https://github.com/Johni0702/mumble-web) (usable only for "listening")
 * chat
 * [dial-in](https://github.com/slomkowski/mumsi), unmaintained, unstable

Lacks video. Possible alternatives for whiteboards and screensharing:

 * http://deadsimplewhiteboard.herokuapp.com/
 * https://awwapp.com/
 * https://www.webwhiteboard.com/
 * https://drawpile.net/
 * https://github.com/screego/server / https://app.screego.net/

#### installation

there are two different puppet modules to setup mumble:

 * https://github.com/voxpupuli/puppet-mumble
 * https://0xacab.org/riseup-puppet-recipes/mumble 

still need to be evaluated, but i'd be tempted to use the voxpupuli
module because they tend to be better tested and it's more recent

### jitsi

#### installation

ansible roles: https://code.immerda.ch/o/ansible-jitsi-meet/ https://github.com/UdelaRInterior/ansible-role-jitsi-meet

puppet module: https://gitlab.com/shared-puppet-modules-group/jitsimeet

there's also a docker container and (messy) debian packages

prometheus exporter: https://github.com/systemli/prometheus-jitsi-meet-exporter

### Nextcloud Talk

systemli is using this ansible role to install coturn: https://github.com/systemli/ansible-role-coturn

### BBB

#### features

 * audio, video conferencing support
 * [accesssible](http://docs.bigbluebutton.org/support/faq.html#accessibility) with live closed captionning and support for screen readers
 * whiteboarding and "slideshow" mode (to show PDF presentations)
 * moderation tools
 * chat box
 * embedded etherpad
 * dial-in support with Freeswitch
 * should scale better than jitsi and NC, at least [according to their FAQ](http://docs.bigbluebutton.org/support/faq.html#how-many-simultaneous-users-can-bigbluebutton-support): "As a rule of thumb, if your BigBlueButton server meets the minimum requirements, the server should be able to support 150 simultaneous users, such as 3 simultaneous sessions of 50 users, 6 x 25, etc. We recommend no single sessions exceed one hundred (100) users."

i tested an instance setup by a fellow sysadmin and we had trouble after a while, even with two people, doing a screenshare. it's unclear what the cause of the problem was: maybe the server was overloaded. more testing required.

#### installation

[based on unofficial Debian packages](https://docs.bigbluebutton.org/2.2/install.html), requires Freeswitch for
dialin, which doesn't behave well under virtualization (so would need
a bare metal server). Requires Ubuntu 16.04, [packages are closed
source](https://github.com/bigbluebutton/bigbluebutton/issues/8978) (!), [doesn't support Debian](https://github.com/bigbluebutton/bigbluebutton/issues/8861) or [other](https://github.com/bigbluebutton/bigbluebutton/issues/8876) [distros](https://github.com/bigbluebutton/bigbluebutton/issues/8956)

anadahz setup BBB using a [ansible role to install BBB](https://github.com/n0emis/ansible-role-bigbluebutton).

### Rejected alternatives

This list of alternatives come from the [excellent First Look
Media](https://tech.firstlook.media/how-to-pick-a-video-conferencing-platform) procedure:

 * [Apple Facetime](https://support.apple.com/HT204380) - requires Apple products, limited to 32 people
   and multiple parties only works with the very latest hardware, but E2EE
 * [Cisco Webex](https://www.webex.com/) - non-opensource, paid, cannot be self-hosted, but E2EE
 * [Google Duo](https://duo.google.com/about/) - requires iOS, Android, or web client, non-free,
   limited to 12 participants, but E2EE
 * [Google Hangouts](https://hangouts.google.com/), only 10 people, [Google Meet](https://meet.google.com/) supports 250
   people with a paid subscription, both proprietary
 * [Jami](https://jami.net/) - unstable but free software and E2EE
 * [Keybase](https://keybase.io/) - chat only
 * [Signal](https://signal.org/) - chat only
 * [Vidyo](https://www.vidyo.com/) - paid service
 * [Zoom](https://zoom.us/) - paid service, serious server and client-side security
   issues, not E2EE, but very popular and fairly reliable

### Other alternatives

Those alternatives have not been explicitely rejected but are somewhat
out of scope, or have come up after the evaluation was performed:

 * [bbb-scale](https://gitlab.com/bbb-scale/bbb-scale) - scale Big
   Blue Button to thousands of users
 * [Boltstream](https://github.com/benwilber/boltstream) - similar to Owncast, RTMP, HLS, WebVTT sync, VOD
 * [Galene](https://galene.org/) - single-binary, somewhat minimalist, breakout groups,
   recordings, screen sharing, chat, stream from disk, authentication,
   20-40 participants for meetings, 400+ participants for lectures, no
   simulcasting, no federation
 * [Lightspeed](https://github.com/GRVYDEV/Project-Lightspeed) - realtime streaming
 * [OpenCast](https://opencast.org/) - for hosting classes, editing, less interactive
 * [Owncast](https://github.com/owncast/owncast) - free software Twitch replacement: streaming with
   storage
 * [Venueless](https://venueless.org/) - BSL, specialized in hosting conferences
 * [Voctomix](https://github.com/voc/voctomix) and [vogol](https://salsa.debian.org/debconf-video-team/vogol) are used by the Debian video team to
   stream conferences online. requires hosting and managing our own
   services, although Carl Karsten @ https://nextdayvideo.com/ can
   provide that paid service.