TPA-RFC-38 wiki replacement

added RFC label

changed the description

Thanks kez for working on this! My only comment is about "Users can edit wiki pages without logging in". I'm not sure about that. We also want to have a way to protect the wiki against spam.

agreed, i think the spec is more like "people with an account can edit any wiki", which is the main limitation in gitlab now. we don't necessarily want a "wiki" wiki where anyone can edit...

one of the goals i would like to have is offline support. i do a lot of editing in my own text editor and push changes over git, changing this workflow to a web interface would be extremely irritating for me. it would also be a liability for TPA because if the website is down, our docs are down.

so i guess a "nice to have" would be something like "offline support / git editing". but a must have is "high availability for TPA documentation", which "git" is only one of many implementations i guess...

another "must have", in my opinion, is a clear transition plan from the current wikis. we MUST be able to either convert the existing markup or use it as is (e.g. the wiki must have markdown support or there should be a converter). we've already got bitten by this in the Trac transition, and it was not fun.

i think we should also loop @ahf in here. he's done some tests setting up a static site with mkdocs, and so did i, with the TPA wiki: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/documentation#mkdocs

in general, this should be meshed with the service/documentation.md page, which has a lot of thoughts on this matter as well. maybe move the alternatives section into the RFC? there's a (redundant now?) goals section in there too:

https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/documentation#goals

... and the discussion section elaborates on the current problems with our documentation system...

so i guess a "nice to have" would be something like "offline support / git editing". but a must have is "high availability for TPA documentation", which "git" is only one of many implementations i guess...

funnily enough, all of the wikis that i've test-driven and liked were git (or mercurial/DARCS) based. i was concerned that a VCS-based wiki might not scale well, but it sounds like the availability is more important, so that solves that problem :p

in general, this should be meshed with the service/documentation.md page, which has a lot of thoughts on this matter as well. maybe move the alternatives section into the RFC? there's a (redundant now?) goals section in there too:

sounds good, thanks for pointing that out to me!

More granularity regarding permissions for editing the wiki is fine, but I am not sure I want a situation where anyone can just edit the wiki, logged in or not. In the past I had to worry about trac being edited by random people I don't want to be back to that.

As a side note, if you are looking into static site generators there is also jekyll which is quite good, although not in a stack that people like around here (ruby w sinatra).

As a side note, if you are looking into static site generators there is also jekyll which is quite good, although not in a stack that people like around here (ruby w sinatra).

I absolutely love jekyll, it's the SSG I used the most before lektor! I didn't realize you could integrate it with sinatra, that's really handy. I'll definitely add it as a possibility, but I worry that static generators might not scale well.

That sounds like a requirement that should be put in writing. What's "scale", how many pages, how fast, etc. :)

a.

...

On 2022-09-30 20:53:41, kezzle wrote:

I worry that static generators might not scale well.

-- Antoine Beaupré torproject.org system administration

I thought it might be a nice feature to have in case we ever want to enable it down the road (most wikis with anonymous editing let you turn it off) but it sounds like it's not a feature we'd ever really want to support. Which is good to me, it's one less requirement to worry about!

added Next label

@kez are you leading this effort or you want me to? no one is assigned this ticket (yet). i've put it into Next for now at least, please take the ticket or assign it to me according to how you want to run this.

I was planning on taking this, just forgot to assign myself

assigned to @kez

changed title from TPA-RFC-38 discussion ticket to TPA-RFC-38 wiki replacement

added Documentation label

Is this wiki for internal use by Tor Project or is this a public-facing resource?

public-facing, although it might be nice to be able to have private pages with more fine-grain permissions than "has an account or not"

Another requirement suggestion:

Support for page redirections:
- Pros: avoid dead links when reorganizing page structures.
- Cons: may leave dangling documents in the wiki tree.

OMG yes that, absolute must.

It's actually a hard requirement for the migration away from GitLab wiki so, as part of this design process, we should make a prototype that would show whether it's even possible at all to redirect the current wikis (or any GitLab URL, really) to some other site.

Otherwise any wiki migration is going to hellishly break all sorts of links everywhere.

...

On 2022-10-06 14:24:16, Silvio Rhatto (@rhatto) wrote:

Another requirement suggestion:

Support for page redirections:

Pros: avoid dead links when reorganizing page structures.

Cons: may leave dangling documents in the wiki tree.

agreed. fortunately most wiki systems seem to support that so i don't think it'll be a hard ask.

Currently is is done in GitLab wiki just by replacing page with link: https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Censorship-Events

Would it be possible to consider just unifying GitLab wikis under a single project? I think the main problem we have is the fragmentation across various projects and namespaces, making discovery and search much more difficult than it has to be (looking at you, GitLab Enterprise...). Because besides that issue, GitLab wikis do check a lot of boxes, like it or not.

Agreed, we should make it one of the alternatives evaluated. for me the biggest downside is it's not searchable... but then again, most of our sites aren't, that's a global problem we have thought of fixing before. see #33106 (closed) (SolR), tpo/web/manual#90 (closed) (www.tpo search), tpo/web/team#25 (search.tpo), etc...

...

On 2022-10-11 13:32:11, Jérôme Charaoui (@lavamind) wrote:

Would it be possible to consider just unifying GitLab wikis under a single project? I think the main problem we have is the fragmentation across various projects and namespaces, making discovery and search much more difficult than it has to be (looking at you, GitLab Enterprise...). Because besides that issue, GitLab wikis do check a lot of boxes, like it or not.

-- Antoine Beaupré torproject.org system administration

for me the biggest downside is it's not searchable.

In fact, wikis in GitLab are searchable. When you navigate to a project, say tpo/tpa/team and enter a search query in the top left search box, the results page has a "Wiki" tab that shows results for that query, but only in the context of that one project. So if we merge all our wikis under a single project, searching for things in that one project should surface more usefuls information, as opposed to the current setup where first one has to figure out in which wiki to submit a search query (TPA? Anti-censorship? Metrics? etc.)

Could we have ALL gitlab users to be included in that project? Or any other way for any gitlab.torproject.org user to edit/add anything in that wiki?

I don't think we can automatically give a user access to a repo. We'd need to hook in the user registration process with a bot or something, not very practical. We also do not have a group that is "everyone", as far as I know.

The "not everyone can edit the wiki" bug is an old GitLab bug:

gitlab#76

The upstream issue has been stalled for years, basically:

https://gitlab.com/gitlab-org/gitlab/-/issues/25177

A similar issue ("suggest edits") has also been stalled for basically ever:

https://gitlab.com/gitlab-org/gitlab/-/issues/42412

One thing to keep in mind here is that if we go with an external wiki, something that's not bound to GitLab user permissions, we're going to end up with a similar situation as this, in the sense that we'll need to grant access to new people (and revoke old accesses) on onboarding/offboarding.

Having a single wiki here would actually make things slightly easier because user/passwords will have already been created to the regular GitLab onboarding process. We'd just need to grant users access to the Wiki. We could also grant access to teams so that once a person joins a team, they automatically have wiki access as well...

...

On 2022-10-12 23:56:06, Gaba (@gaba) wrote:

Could we have ALL gitlab users to be included in that project? Or any other way for any gitlab.torproject.org user to edit/add anything in that wiki?

-- Antoine Beaupré torproject.org system administration

marked this issue as related to gitlab#76

added Backlog label and removed Next label

mentioned in issue tpo/web/team#42

marked this issue as related to tpo/web/team#42

Onion MkDocs

I'm starting to use this custom MkDocs configuration example (live rendered version) to build documentations like these:

I don't know if it has all the required features or if would work for an unified wiki, but it suits well in a per-project basis.

The search functionality is handy, but I wonder how it would behave for a bigger knowledge-base.

May also need some better plugin for automatically build page indexes with wildcard support.

Might be worth evaluating anyway.

nice. mkdocs is definitely one of my prime "git-based" workflows I would like to use. i managed to convert the TPA wiki to it and it wasn't completely crap, which is better than everything else i looked at so far. :p

...

-- Antoine Beaupré torproject.org system administration

looks great, i'll look into mkdocs a bit. thanks @rhatto!

added Icebox label and removed Backlog label

mentioned in issue #41119 (closed)

assigned to @anarcat and unassigned @kez

added Next label and removed Icebox label

we had a good conversation between @rhatto @lavamind and me today. we worked a bit on the TPA-RFC-38 document. conversation may continue this afternoon, see this pad:

https://pad.riseup.net/p/tormeet2023CR-WikiParty

see wiki-replica@26be7467 for the changes.

i moved the pad to the wiki, see https://gitlab.torproject.org/tpo/team/-/wikis/2023-Tor-Meeting-Costa-Rica-Wiki/wiki-work-party-notes

This issue has been waiting for information two weeks or more. It needs attention. Please take care of this before the end of 2023-05-26. ~"Needs Information" tickets will be moved to the Icebox after that point.

(Any ticket left in Needs Review, Needs Information, Next, or Doing without activity for 14 days gets such notifications. Make a comment describing the current state of this ticket and remove the Stale label to fix this.)

added Stale label

removed Stale label

no progress since the last update, sorry. i still intend on working on this, but this is probably not going to happen this week.

sounds good. Thanks for working on it!

i moved the pad to the wiki, see https://gitlab.torproject.org/tpo/team/-/wikis/2023-Tor-Meeting-Costa-Rica-Wiki/wiki-work-party-notes

one of the reasons i'm stalled on this is because I don't like the conclusion we reached when we talked about this. i didn't like it when we reached it either, but we were really converging towards setting up a mediawiki as a replacement for gitlab wikis.

but now i feel this is not the right direction. i can't articulate this clearly yet, so i don't want to go into details, but i feel the wiki is a kind of trap we keep walking into, and we should instead look at how we operate our documentation more globally.

i'll just dump those links that are one of the things that made me really question the conclusions we reached:

Engineering Documentation • Lorna Jane Mitchell • GOTO 2022 (Youtube video, 45min), slides, talk page
diataxis, the new site for https://documentation.divio.com/ which was a strong inspiration for the structure of the service template

The killer quote from the slides is "Use the same tools and workflows as for code", which implies:

source control
text-based markup and diagrams
continuous integration (includes checking links, previews, prose linting)

so maybe one problem we have here is scope: i wouldn't want TPA's wiki to end up in mediawiki, but maybe it's okay to have one wiki where we can dump things?

but then why not treat that wiki as a proper documentation space and do it correctly as well?

anyways, lots of questions at this point, and few answers, i'm pushing this back because i don't have time to think about it properly right now.

The other issue to consider is that documentation is not only curated by TPI but also is (or it was before Gitlab) a place where the Tor community (core contributors and volunteers in general) could contribute. We need a place that helps us to have good documentation and at the same time, people in the wider community can edit/add/improve. The big problem of the documentation right now is that is spread over teams and tools and siloed to only some people able to edit some parts of the documentation.

one of the reasons i'm stalled on this is because I don't like the conclusion we reached when we talked about this. i didn't like it when we reached it either

+1

i can't articulate this clearly yet

+1

We need a place that helps us to have good documentation and at the same time, people in the wider community can edit/add/improve. The big problem of the documentation right now is that is spread over teams and tools and siloed to only some people able to edit some parts of the documentation.

I think this is a good statement of the problem.

What was most interesting in the discussion at Costa Rica was that:

Part of the solution would involve a better way to include people in the edition process (by having better UX and user/permission management).
But part of the solution is about having a search engine indexing GitLab and all documentation sites (including the upcoming Development Portal).

It was also discussed that's hard to converge different projects/teams documentation needs into a single, centralized solution. There are software or teams with their own self-contained documentation, while there's common TPO/TPI documentation that could stay in a single "umbrella".

It maybe worth then to think more about the problem space and to have a glance on the existing documentation and which would be relevant to be brought to an "official" wiki system.

Another thing to consider is whether to have an official central wiki as well as a community driven one (but this also have trade offs).

added Backlog label and removed Next label

mentioned in issue tpo/team#115 (moved)

added Q4 Roadmap::Future labels

removed Roadmap::Future label

mentioned in issue tpo/community/hackweek#13 (closed)

mentioned in issue tpo/community/hackweek#26 (closed)

marked this issue as related to tpo/community/hackweek#26 (closed)

mentioned in commit wiki-replica@3ab65af9

I have tried https://github.com/MkDocsEditor/ and it didn't work at all. Its web interface couldn't create or save pages(delete works), and it does not have multi-user support. Additionally it have 67 vulnerabilities (1 low, 17 moderate, 37 high, 12 critical) for npm and 4 Critical 27 H 9M 1L 1U vulnerabilities in the container running it.

i was writing about the problems with wikis and realized we haven't talked about the wiki proliferation problem, so i went crazy and wrote another python program to do an inventory of wikis. i think i have done this before but can't find the code anymore.

i'm running the script now and will report back here when it's done, but we have the problem that we have hundreds of empty wikis, and dozens of wikis with only a handful of pages. part of the transition here should probably involve fixing that by converging into a smaller set of wikis, either One Big Wiki or One Wiki Per Team or something.

update: here's the summary of the script:

INFO: top 10 largest wikis: [('tpo/applications/team', 29), ('tpo/applications/tor-browser', 29), ('tpo/community/team', 30), ('tpo/operations/team', 32), ('tpo/anti-censorship/team', 35), ('tpo/core/team', 39), ('tpo/network-health/team', 56), ('tpo/tpa/team', 216), ('tpo/team', 1189), ('legacy/trac', 1619)]
INFO: 1494 projects
INFO: 383 without wikis
INFO: 1053 empty wikis
INFO: 58 non-empty wikis
INFO: 3516 pages total

Excluding legacy/trac, more than half (63%) the wiki pages are in the tpo/team wiki. If we count only the first three wikis, that ratio goes up to 77% and if 85% of all pages live in the top 10 wikis.

In other words, there's a very long tail of wikis (~40) that account for less than 15% of the page count. We should probably look at centralizing this, as it will make all further problems easier to solve, e.g. search and discovery.

i was writing about the problems with wikis and realized we haven't talked about the wiki proliferation problem, so i went crazy and wrote another python program to do an inventory of wikis. i think i have done this before but can't find the code anymore.

and i just found the damn code, here:

https://gitlab.com/anarcat/scripts/-/blob/a6dbda602d8d8e9c0ef810d308e35a09ff538836/gitlab-list-all-wikis

... which i wrote in 2021. ugh. it doesn't have anything on the script i ended up with, so i deleted it.

mentioned in commit gitlab-tools@d1fc6752

We have tested the Web IDE as a tool to allow users to more easily edit Git repositories, as a replacement for "a wiki", and... it's not great.. Here's me editing the sandbox.md markdown file:

You'll notice the preview, on the right, doesn't work. But even if it would, the interface is rather overwhelming and doesn't help the user very much getting through the day. Here, for example, is what happens when you try to "commit" your changes (assuming that you even know you need to do that):

Here the "normal", non-IDE editor:

... but then this one looks so good because I can commit directly to the main branch. If I take some random project (https://gitlab.torproject.org/anarcat-admin/test), then it quickly gets ugly:

This is because I already have a test project in my workspace. So total UX fail here. But even if you manage to create a fork, then you first make a commit, and then you are served with another UI element:

... the MERGE REQUEST (dum dum dum). At this point, most users unfamiliar with this workflow will have likely given up.

And that's assuming users could fork: most GitLab users have a low project limit and wouldn't be able to fork anyway.

So I think it's fair to assume this doesn't work as a replacement for a proper wiki workflow.

@rhatto @shelikhoo found this plugin which might be interesting for mkdocs https://pypi.org/project/mkdocs-live-edit-plugin/

serious caveat: no new page creation.

we also tested the Web IDE search, it just plain doesn't work. it doesn't even seem like it's issuing any network request at all. at least it can't find the string ldap in the TPA wiki, which is clearly wrong.

upstream also has trouble with wiki searches, even in the Ultimate edition, as entreprise customers are unhappy: https://gitlab.com/gitlab-org/gitlab/-/issues/15835

mentioned in commit wiki-replica@9d366488

mentioned in commit wiki-replica@7e3d2e9e

mentioned in commit wiki-replica@c810ca43

mentioned in commit wiki-replica@c0c1ba2b

i finished reviewing TPA-RFC-38, as a first pass. I reviewed the requirements, the personas, and other sections, and generally shook up the document a bit. i'm not sure what the next step is, we made good progrress in analyzing mkdocs as a possible solution in tpo/community/hackweek#13 (closed) but there are serious blockers there as well (search is a 10MB static asset downloaded every 10 minutes, no good edit workflow, compatibility issues).

marked this issue as related to tpo/team#202

TPA-RFC-38 wiki replacement

Designs

Child items ...

Activity

Onion MkDocs