nextcloud collaborative "pad" synchronization breaks down with multiple users

Today, in both the metrics and vegas meetings, we had problems using the "nextcloud pad", that collaborative, WYSIWYG, "markdown" text editor. We're trying to use this as a replacement for Storm's Etherpad installation and it worked the last meeting.

But now we were over 12 people in the pad and it didn't work so well. The main problem is synchronization, e.g. one user would "strikethrough" some text but others wouldn't see the style change. This is particularly critical in Vegas meetings as we use that style information to carry information about how the meeting should proceed.

There are also issues scrolling around the content using cursor keys (e.g. pgdown jumps to the end of the document instead of one page down), but that's probably unrelated.

added component::internal services/service - nextcloud owner::nextcloud-admin@torproject.org priority::medium severity::normal status::new type::defect labels

As one datapoint, I open devtools on the page and whenever I modified the page I saw there was a xhr POST request initiated to https://nc.torproject.net/apps/text/public/session/push. Interestingly, it seems like they all timed out (which could explain the lack of synchronization across browsers). I don't know if that was simply a local issue, or if that was experienced by everyone using the pad, as well.

Currently, when I edit the document I see the same request initiated but now there is a "412: Precondition Failed" response.

I think one problem is that we are using this app in nc.torproject.net right now:

https://apps.nextcloud.com/apps/text

and we should consider using a pad instead:

https://apps.nextcloud.com/apps/ownpad

It may be related to this bug in the text app:

https://github.com/nextcloud/text/issues/331

i just had a quick chat with micah about this, and here's a summary...

for the sake of clarity, I'll start by naming things, because it's hard and i think a source of confusion:

• onlyoffice: the Nextcloud collaborative editor with support for Word and Excel documents, kind of like Google docs (installed on nc.tpo)
• etherpad app, ownpad: the Nextcloud Etherpad app that bridges Nextcloud with an existing Etherpad lite instance (not installed on nc.tpo)
• markdown app, text app: the Nextcloud Text app that allows collaborative editing of text documents (installed on nc.tpo).

Let's take those one by one...

== The text app

The Text app we are having problems with in the meetings is built on top of Prosemirror, one of many collaborative editors I have found in a previous research last year, and tiptap, a rich text editor built on top of Prosemirror. This app is fairly recent (june 2019) and shipws with NC 17 by default.

Micah told me the problems we are seeing are known, and others have suffered the same problems. He suspects the NC people haven't yet made the app scale to tens of users like we've seen in the vegas and metrics meetings (or at least, more than 10). This would confirm sysrqb's observations that it would fall apart under load. It works fine for a couple people, with some quirks, like a small scratch pad, but not so well for larger groups.

There might be performance tweaks we could do in Nginx or elsewhere to fix the issue sysrqb observed.

== The etherpad app

This app is not yet installed on our instance, but could be. It was (or still is?) installed on nc.riseup.net but because Riseup's etherpad instance expires documents after a while, this led to data loss as the files disappeared from NC as well.

So while we could deploy that app in our instance, it might lead to the same problem as there's no obvious marker that a pad will be removed in the NC UI. There is, of course, a huge warning in the pad itself when you first open it, but people forget about this all the time.

If we use it only for ephemeral stuff like the vegas meeting notes, maybe that would be alright, but I would still be worried that people would use it for other things and lose data.

Maybe we could just use Riseup's pad service for our meetings? Do we absolutely need this to be integrated in Nextcloud? If so we would have three different collaborative editors in nextcloud, which doesn't seem very reasonable to me...

Alternatively, can't we just use meetbot to take minutes in meetings? :)

== The onlyoffice app

We could use the OnlyOffice app for meetings. We haven't tried that in a large scale yet, as far as I know. It might have exactly the same problems as the text app, maybe even worse because onlyoffice is much heavier. It would be worth a try, however.

== TL;DR:

Known issue. Needs debugging or we could just use pad.riseup.net or meetbot or onlyoffice for meetings.

also, thanks gaba and sysrqb for the diagnostics, that's super useful! :) in particular, thanks gaba for the detective work in the text app issue queue. it sure looks like there's a race condition in the code that can get triggered when multiple people edit at the same time. and since they found it, there's a good chance that specific issue gets fixed...

so maybe this is just a matter of "wait and see" (ie. use pad.riseup.net for a bit longer) while they fix their things...

Yes. I see the problem with using owncloud and may not be clear for people and we lose data that we do not want to lose.

I agree about moving any agenda/notes for meetings to pad.riseup.net and periodically archive them in nc.torproject.net if we want to save old notes. Once they fix the text app then we can consider if we move there or not. For now we can continue using pad.riseup.net as we use for network and anti-censorship team meetings.

So far it seems that people have reverted back to using Google Docs for writing letters and proposals which makes me both sad and scared.

Sad, because friends are working hard to make Nextcloud work for us and are wondering if they are wasting their time.

Scared, because we're putting our most sensitive data in the most sophisticated surveillance machine ever built by humanity.

We need to figure this out. I think one thing we should try is to elaborate on the various pain points we have with each platform, and whether they can be resonably resolved.

For example, we had a bad experience with the "text app" when collaborating with a large group. But we have made the same experience with the Only Office app. Maybe we could try that first?

I would also suggest that we try to write our documents in Nextcloud by default, and only revert to Google Docs when things start to fail.

But I understand that people take the easy way for now: after all, we have a job to do and that is not to serve as guinea pigs and beta-test every free software app out there. ;)

Trac:

And just to make sure the distinction is clear, i'll reiterate there are two collaborative editing apps in Nextcloud: the "Text app" and the "Onlyoffice app":

• Text app: Etherpad replacement. Simple, collaborative text editor. Has known problems with large (10+?) groups. Subjet of this bug report.
• Onlyoffice app: Google Docs replacement. More complex and slower. Large group collaborative capabilities unknown. Some known issues have been documented in #32390 (moved) and are being worked on upstream.

In the "create document" drop down:

The New text document option starts the Text app, but the Document, Spreadsheet and Presentation menus start the Onlyoffice app.

I figure it was worth clarifying that. I think Onlyoffice should be tested more, especially with large groups, but it should at least be used more extensively for collaborative documents made with smaller groups...

We are using nextcloud to archive documents, to save list of ideas for grants, for keeping track of broad goals for sponsors and I'm using it for calendar. We are not wasting anything setting up nextcloud. About grant writing it is ok if people use google doc until onlyoffice is usable.