make it possible to publish "same-day" posts

web IDE crashes the browser because it requires so many resources from the computer. It takes forever to load (15-45minutes) after many attempts. This is just to get to a place where you can upload an image, if an error occurs you have to start all over again.

That is really not great, I must say... but I'm not sure how to deal with this without removing GitLab from the equation... maybe we could use a local editor of some sort?

One thing Lektor has going for it is that it does have a local server you can run and actually edit things manually with. See for example the quickstart documentation... It does involve installing Lektor locally and starting it from the commandline, but after that is done, everything is done through a web interface that should be much lighter than GitLab.

This probably deserves its own bug report, in https://gitlab.torproject.org/tpo/web/lego/-/issues

takes ~15-30 minutes to upload a preview. normally the workflow involves a lot of back and forward with preview to fix markup mistakes. this 'editing and previewing' process ends up taking hours.

so this shouldn't be happening: build times of the preview should be very fast as we don't build a full copy of the blog, at least according to @lavamind.

could you clarify how to reproduce this? maybe show a merge request where the job took significantly longer than expected?

note that this issue is actually tracked in #40015 (closed). maybe it's better to continue that conversation there?

if something in the build process breaks and the preview doesn't populate, it is not possible to tell what is wrong and more time is lost in figuring what to fix

That shouldn't happen and is definitely a bug... Could you point at a specific instance of this happening? Maybe this should be tracked in a different issue as well.

doing anything more than adding a lead image (e.g., adding images inline, hyperlinking an inline image, resizing something) takes a mixture of markup and HTML, which requires a lot of trial and error (see second item)

Having a local preview should help a lot with this, hopefully.

we always depend on a person to finalize the process, you have to (a) ask a web person to merge the request (b) keep checking the website for an hour+ to ensure its there (c) often ask the person to do another step to actually make the post live because something was forgotten in the merge process. this adds up to the time to complete the task.

There were some discussions about access controls of the web repos recently, in team#37 (closed) (TPA-RFC-24). The change was basically to grant more permissions to members of the web team so that more people fcould do final merges, although I haven't re-read the proposal recently, I might get that wrong.

The point is: we can extend those permissions to more people. If people want to apply to be granted more access to the project, I think that would make perfect sense. Just ask. :)

currently is almost impossible to guarantee a post can be published on the "day-of", to do that we need to mobilize a lot of time to make sure the task can be done.

This seems to be the bottom line. So if you don't mind, I'm going to turn this issue into a meta-issue that tracks the goal of "publishing "same-day" posts" and we can track the underlying issues in separate issues.

Let me know if you want me to create the missing issues, but I will point out that I'll need more information to be able to act on those issues, so maybe it's best if the "victims" of those problems report them directly instead, if only to keep them in the loop better...

Thanks for documenting all this, and sorry for the wall of text!

web IDE crashes the browser because it requires so many resources from the computer. It takes forever to load (15-45minutes) after many attempts. This is just to get to a place where you can upload an image, if an error occurs you have to start all over again.

That is really not great, I must say... but I'm not sure how to deal with this without removing GitLab from the equation... maybe we could use a local editor of some sort?

One thing Lektor has going for it is that it does have a local server you can run and actually edit things manually with. See for example the quickstart documentation... It does involve installing Lektor locally and starting it from the commandline, but after that is done, everything is done through a web interface that should be much lighter than GitLab.

This probably deserves its own bug report, in https://gitlab.torproject.org/tpo/web/lego/-/issues

I made a ticket: lego#51 (closed)

takes ~15-30 minutes to upload a preview. normally the workflow involves a lot of back and forward with preview to fix markup mistakes. this 'editing and previewing' process ends up taking hours.

so this shouldn't be happening: build times of the preview should be very fast as we don't build a full copy of the blog, at least according to @lavamind.

could you clarify how to reproduce this? maybe show a merge request where the job took significantly longer than expected?

note that this issue is actually tracked in #40015 (closed). maybe it's better to continue that conversation there?

i just submitted a merge request for a test blog post (!110 (closed)) and building a preview took 10 minutes. so! maybe better than 15 or 30, but i will say that again, this adds up if i need to wait for the Web IDE to load, wait for the preview to build, check the preview, notice an error with markdown or typo, and restart the process all over again. i also left a comment in #40015 (closed).

added Needs Information label

changed title from Pain points while publishing a blog post to make it possible to publish "same-day" posts

added #40050 as child task

removed child task #40050

marked this issue as related to #40015 (closed)

changed the description

if something in the build process breaks and the preview doesn't populate, it is not possible to tell what is wrong and more time is lost in figuring what to fix

That shouldn't happen and is definitely a bug... Could you point at a specific instance of this happening? Maybe this should be tracked in a different issue as well.

idk if it should be tracked in a different ticket but it did happen in !96 (merged)

we always depend on a person to finalize the process, you have to (a) ask a web person to merge the request (b) keep checking the website for an hour+ to ensure its there (c) often ask the person to do another step to actually make the post live because something was forgotten in the merge process. this adds up to the time to complete the task.

There were some discussions about access controls of the web repos recently, in team#37 (closed) (TPA-RFC-24). The change was basically to grant more permissions to members of the web team so that more people fcould do final merges, although I haven't re-read the proposal recently, I might get that wrong.

The point is: we can extend those permissions to more people. If people want to apply to be granted more access to the project, I think that would make perfect sense. Just ask.

if i can make have power to merge a blog post, that'd be great.

i made #40051 (closed) for this!

...

On 2022-09-09 23:49:52, al smith wrote:

if i can make have power to merge a blog post, that'd be great.

-- Antoine Beaupré torproject.org system administration

marked this issue as related to lego#51 (closed)

mentioned in issue #40051 (closed)

marked this issue as related to #40051 (closed)

we just had an informal session between @smith, @lavamind, and me about this. we summarized the pain points and their possible solutions as this:

Pain point	Possible solutions
1. Web IDE slow	per-year folders
2. preview slow	new runners
3. broken builds don't preview
4. layout is hard	local IDE
5. blocked on people	more access (see #40051 (closed))

We expanded on each of those problems above, but to expand on those solutions:

per year folders

al mentioned the idea of lowering the number of folders that the Web IDE would have to render could improve performance. we could use, for example, per-year naming.

this would probably involve creating all new posts in a folder called (say) /blog/2022. Older articles would be stored in (say) /blog/archive and a rewrite rule would ensure /blog/foo would land in the archive.

To be tested, I guess?

new runners

We clarified that when TPA says that the build time was reduced to a couple of minutes, we really do just mean the build time, not the entire pipeline. So even if the build takes 3-4 minutes, that's just the build: another job needs to run to deploy the site, and more jobs then run to test the site, so that an entire pipeline will take more time, say 10-15 minutes on a regular basis.

Runner unavailable makes this problem worse, and there are plans to improve that situation by provisioning more runners. There is, for example, #40889, but we may want to make a new ticket specifically about this.

local IDE

we debated the idea of shifting the IDE to the client side. in other words, this means teaching people to use Git and Lektor locally. i think those are the steps people will need to learn:

git clone / fetch
git branching (yes, ouch, for merge requests it's kind of mandatory)
lektor installation
editing with the local lektor server
preview
git commit
git push
merge requests

So it's kind of a big hill to climb. We're going to see if we can sit down with @smith to see how feasible this would be at all. Specifically, @lavamind is going to look at the software options to see how this would be done on a mac beforehand, and then will try to see how that works with @smith.

more access

i opened #40051 (closed) to give more access to all. in that ticket i also expanded on the different ways to use merge requests, and i presented a version of that in person as well.

i think that's about it! will keep updating this ticket as we move this forward...

assigned to @lavamind

added Doing label

removed Needs Information label

In Ireland, @smith and I started exploring alternatives to the Web IDE to interact with the blog, specifically to allow building and previewing the blog locally, without GitLab CI.

We have been meeting on a weekly basis since then, and have seen success in installing Lektor and building the blog. We have also explored a git GUI interface called Gittyup, but found it complicated and confusing. One challenge that is that @lavamind doesn't have access to a Mac, so we need to screen-share and clickety-click our way during a BBB session.

The next step is to try another git GUI called SourceTree, which only free as in free beer and has Windows and Mac versions. I have access to a Windows computer so hopefully the versions on both platforms are close enough here so that I can learn it enough to avoid wasting too much of @smith's time.

mentioned in merge request tpo/tpa/ci-templates!31 (closed)

@smith and I have had success in setting up SourceTree and Lektor locally, allowing them to quickly build and preview a blog post. It took about an hour last Friday (including 45 min build time) to publish a new blog post. There still are some small quirks to work out and we plan to continue working on this this week.

could we get a summary of the current status here? what are the next steps?

i suspect this doesn't belong in Doing in any case, as there hasn't been any (documented) activity in the last two months.

@lavamind can you give an status update of this ticket please?

make it possible to publish "same-day" posts

Designs

Child items ...

Activity

per year folders

new runners

local IDE

more access