wiki replacement considerations

assigned to @anarcat

changed the description

marked this issue as related to tpo/tpa/team#40909

added Next label

MkDocs experiment

As I related activity, I'm planning to investigate how Onion MkDocs could be a TPA-RFC-38 wiki replacement, under #13 (closed).

Pseudo code for a scheduled pipeline:

for wiki in team_wikis:
  git clone wiki docs/wiki
  apply_heuristics docs/wiki # like renaming home.md to README.md

mkdocs build
deploy somewhere

This way, we could see how a single website could host all the existing documentation, with minor changes and with a common, built-in search tool.

It can be an experiment related to this project. It may work, may not work or not match all the requisites, but seems worth a try.

Thanks for this Hackweek submission!

We will have an All Hands meeting on Nov 1st 2023 to present the projects for this Hackweek.

If you really plan to work on this proposal, please get ready to present it during the All Hands session.

You are going to have 5 min max to present the project.

hi @rhatto the robo...

i do intend to work on this at the hackweek, but i'm not sure how to introduce this nor how to hack on it in the first place.

but i'm not sure how to introduce this

Use the microphone !

nor how to hack on it in the first place.

This is a hard one, because we're kind of blocked.

Maybe start by just doing another session for brainstorming with more people, to discuss their needs etc.

Or start by reviewing the requirements. Example: if I'm not mistaken, Mediawiki was mentioned because it would make edits easier, but since GitLab improved it's editor, which may be as easier as using Mediawiki.

I don't think all Hackweek proposals must reach a conclusion, or have deliverables, or take the whole week. So maybe this is worth trying even if it help moving things only a little. And most importantly, if you want to do it. We may as well advance this by just doing some tinkering elsewhere.

That said, I'm not sure how much I can help on this next week. I'd like to try some stuff instead (like with the experiment described above).

@anarcat can you gather the requirements before the hackweek? Maybe a mail to tor-internal asking people to send you their needs about this? We had a retrospective where we talked about the wiki replacement. Maybe there are some requirements in the notes there (they may be in the all hands meeting notes)

For me 2 big requirements are searchable and editable without needing to be in a specific team. @ahf was also looking at this (gollum I think) for the hackweek and may have more insights.

that sounds like starting the hackweek before the hackweek. :) i'm swamped. i don't really have time to prepare much more than what i've already thrown in here. there are some requirements in the wiki page linked in the description here, that said... if you want to add to that, that would be great, otherwise i'll look into the all hands archives to see if there's anything missing during the hack week.

added Proposal label

changed the description

mentioned in issue policies#3 (closed)

It maybe possible to emulate an offline workflow for mediawiki with https://git.sr.ht/~enterprisey/syncwiki .

there's actually a git remote for mediawiki that is much more powerful than simply "upload/download a single page" which is what syncwiki seems to be doing:

https://github.com/Git-Mediawiki/Git-Mediawiki/

unfortunately, it's quite in a bad shape. i somehow ended up inheriting this project too, as i'm one of the GitHub maintainers because I sent a few patches. it's hard to keep up with the changing mediawiki API, and sometimes the scale of wikis is so massive that it's basically impossible to keep up.

but yeah, it's possible to sync locally. i wouldn't say it's easy or actually very practical, that said.

Yes, I think one the main trade-off of choosing a wiki is whether to have a web first wiki like media wiki or git/offline first wiki that is effectively a web page generator.

Web first wiki are easier to work with for less-tech users, as there will be no need to learn about git or other developer focused tools. On the other hand, Git/offline are more friendly to tech users as it is built on existing tools.

i'm going to start reviewing the proposal and requirements now. if you're interested in joining that work, ping me here or on IRC and i'll jump on a BBB call.

mentioned in commit tpo/tpa/wiki-replica@3ab65af9

we're in the bbb room now.

did a bunch of work in the document, documented in tpo/tpa/team#40909 (comment 2962894) and later.

could not progress on this issue, got taken over by a critical data loss in tpo/tpa/team#41388 (closed)

added Doing label

removed Next label

closed

removed Doing label

reopened

added Doing label

i worked on this quite a bit with @shelikhoo before being interrupted by tpo/tpa/team#41388 (closed).

@shelikhoo tried MkDocsEditor to act as an editor for mkdocs wikis for "normal users". It didn't work at all, lacked multi-user support and many vulnerabilities in its container. @shelikhoo also found the live edit plugin which was interesting because it's embedded inside existing pages. major caveat: it can't create pages.

@anarcat wrote a script to analyze the wikis in GitLab and found that, out of 1494 projects, 58 had "non-empty" (with at least one page) wikis, and 1053 had "empty" wikis (i.e. they didn't bother turning it off but didn't use it). "Excluding legacy/trac, more than half (63%) the wiki pages are in the tpo/team wiki. If we count only the first three wikis, that ratio goes up to 77% and if 85% of all pages live in the top 10 wikis. In other words, there's a very long tail of wikis (~40) that account for less than 15% of the page count. We should probably look at centralizing this, as it will make all further problems easier to solve, e.g. search and discovery."

@anarcat also tested the web IDE to see if it improved the workflow for users unfamiliar with code or MRs and it clearly did not. we also tested the search in the Web IDE, which did not work at all.

@anarcat also did an extensive review of TPA-RFC-38. reviewed the requirements, the personas, and other sections, not sure what the next step is

we identified a major concern with search. even in mkdocs, there's a serious accessibility issue with built-in search, as it is JS based and loads a 10MB file cached only for 10 minutes.

the bottomline is wikis are not searchable in GitLab free. You need "advanced search" for that, and even then, it doesn't seem it works that well. according to comments in https://gitlab.com/gitlab-org/gitlab/-/issues/15835, even entreprise users are complaining about search.

but still, using gitlab ultimate (tpo/team#202) might solve at least part of that problem for us.

otherwise, we might consider rendering the gitlab wikis as static sites and crawl those with a search engine, like solr, which is the search.tpo project (tpo/web/team#25)

or, as a bullet list:

@shelikhoo tried MkDocsEditor, not working, and whicthe live edit plugin which works but doesn't support creating pages (!), @anarcat tested the web IDE, doesn't work for novice users
@anarcat analyzed all wikis, found 58 with content, but 85% of all content is in 10 wikis, suggests centralizing
discussed search: mkdocs loads a 10MB file cached only for 10 minutes which is a major accessibility issues, unearthed the search.tpo project (tpo/web/team#25), mentioned that GitLab wikis are searchable in GitLab Ultimate (tpo/team#202)
@anarcat reviewed requirements, personas and other sections in the TPA-RFC-38 wiki proposal, not sure what the next step is

closed

removed Doing label

marked this issue as related to #13 (closed)

changed milestone to %Hackweek 2023

wiki replacement considerations

About the project

Participants

Summary

Skills

Links

Designs

Child items 0

Activity

MkDocs experiment