Lektor build is taking too long

As we discussed on IRC, the website build is taking too long because of the pagination on the press page. 1 hour to build our website is just a waste of resources and our time. Let's discuss possible solutions.

added Bug label

Here are three options

We could try and make it even more static, e.g. make all the sites html in templates/ (As emmapeel said on IRC)
We could use fixed length pagination (1, 2, 3.. and so on instead of years)
We could go back to what we had before

I have bumped the time allocated to tests, some builds were failing because they took longer than 1 hour (as https://gitlab.torproject.org/tpo/web/tpo/-/jobs/45198)

added Backlog label

We could use fixed length pagination (1, 2, 3.. and so on instead of years)

Could you run a test locally and see how much time it would take if we adopt this solution?

I have a Ryzen 9 5900X and an NVME drive.

Old time (3x lektor build):

real 11m11.601s
user 8m3.594s
sys 0m54.662s

New time (3x lektor build):

real 6m39.966s
user 5m15.306s
sys 0m47.254s

I can still see how the press takes longer compared to other pages.

Remember that when pages are edited, they take a little longer to build the first time.

Maybe the situation improves with the following builds.

I've merged @HackerNCoder fix with fixed length pagination.

mentioned in issue blog#40015 (closed)

The build is also not parallelized by Lektor build system. It's actually limited by 1 thread / 1 file at the time.

This is really a shame, since I think Lektor can improve the build quite a bit here.

added Roadmap::Future label and removed Backlog label

added Backlog Q3 labels and removed Roadmap::Future label

removed Backlog label

added Icebox label

added Roadmap::Future label and removed Icebox label

it would be helpful if we could clarify what is "too long" and what an acceptable time would be.

thanks!

Too long: between 40 minutes up to 1 hour to generate the staging website. Then, the same amount of time to push to production. So it's like 1h20 to 2 hours to get a simple modification like a TB release.

Before adding the pagination in /press, the website was building in ~20 minutes.

And in addition to this; when you have 8 cores/16 threads CPU with 32GB Ram.. I would expect parallel builds (multi-threading) to actually take this build to below 10 minutes. And otherwise I can compile the Linux kernel faster than this site.

You can also try to compare it to other lektor sites. For example this small demo site is building on GitHub Action (2 cores, low clock speed with limited memory and I/O) in 1m 6s: https://github.com/lektor/lektor-website/runs/6265527379?check_suite_focus=true.

This demo site build takes just a few seconds on my machine. Meaning your sites should take several minutes on my machine at most.

marked this issue as related to blog#40015 (closed)

This issue is very similar to blog#40015 (closed)

In short, there are some small things that can be done around the templates to help things, but the root of the problem is that Lektor is simply extremely inefficient when compiling sites that have lots of pages.

The problem and possible solutions are discussed here in upstream issue #180

AH I see. It's becomes exponentially worse with more pages.

That being said. In order to have a successful build. I may need to build this site 2 or 3 times, due to 'bugs'.

removed Q3 label

added TPA label

removed Roadmap::Future label

added Next label

changed due date to September 12, 2024

assigned to @gaba

assigned to @lavamind and unassigned @gaba

This issue has been waiting for information two weeks or more. It needs attention. Please take care of this before the end of 2024-09-11. ~"Needs Information" tickets will be moved to the Icebox after that point.

(Any ticket left in Needs Review, Needs Information, Next, or Doing without activity for 14 days gets such notifications. Make a comment describing the current state of this ticket and remove the Stale label to fix this.)

To make the bot ignore this ticket, add the bot-ignore label.

added Stale label

added Doing label and removed Next label

removed Stale label

In terms of low-hanging fruits, I think we should be able to reduce the number of builds from 3 to 2. Of course we wouldn't build more than once but it seems this is still required:

the build never needed to run three times in a row, it just needs to run twice. that's also because of i18n (and -scss, and community generator) but that's an unfixable problem with how lektor works

Given the amount of time kez spent looking at Lektor internals, I trust their assessment. But even shaving off this one extra build should yield a 30% improvement on building this website, so I'll push the necessary changes in the tpo/tpa/ci-templates project asap.

mentioned in commit tpo/tpa/ci-templates@f9b54e3a

marked this issue as related to lego#30 (closed)

marked this issue as related to #471

removed due date September 12, 2024

closed with commit 95b0ae8f

mentioned in commit tpo/web/l10n/tpo@95b0ae8f

Thanks to some updates and changes around the i18n and scss plugins, our Lektor websites now build in 1 pass instead of 2 or 3, so they should build roughly 66% faster in CI.

These changes will affect mainly www.torproject.org in addition to support.torproject.org and community.torproject.org. Other Lektor projects like newsletter.torproject.org were small enough that build times weren't a significant issue to start with.

The blog is unfortunately not affected by this improvement, because it never had to be built in mulitple passes. That issue is tracked in blog#40015 (closed)

\o/!!!!!!!!

Thanks @lavamind!!