Can shuffle be cached?

support
21

Here you go

git clone --single-branch -b hugo-forum-topic-54636 https://github.com/jmooring/hugo-testing hugo-forum-topic-54636
cd hugo-forum-topic-54636
hugo --logLevel info
Cloning into 'hugo-forum-topic-54636'...
remote: Enumerating objects: 10536, done.
remote: Counting objects: 100% (2481/2481), done.
remote: Compressing objects: 100% (2406/2406), done.
remote: Total 10536 (delta 108), reused 83 (delta 75), pack-reused 8055 (from 2)
Receiving objects: 100% (10536/10536), 14.75 MiB | 1.79 MiB/s, done.
Resolving deltas: 100% (345/345), done.
Start building sites … 
hugo v0.147.0-7d0039b86ddd6397816cc3383cb0cfa481b15f32+extended linux/amd64 BuildDate=2025-04-25T15:26:28Z VendorInfo=gohugoio

INFO  static: syncing static files to / duration 444.934µs
INFO  build:  step process substep collect files 10461 files_total 10461 pages_total 10461 resources_total 0 duration 410.364856ms
INFO  build:  step process duration 410.443843ms
INFO  build:  step assemble duration 316.310329ms
INFO  build:  step render substep pages site en outputFormat html duration 25.16427808s
INFO  build:  step render substep pages site en outputFormat rss duration 1.571929182s
INFO  build:  step render pages 11512 content 10463 duration 26.743412221s
INFO  build:  step render deferred count 0 duration 1.453µs
INFO  build:  step postProcess duration 14.311µs
INFO  timer:  name TestListRandomPagesMethodA count 10450 duration 4.655520497s average 445.504µs median 376.022µs
INFO  timer:  name TestListRandomPagesMethodB count 10450 duration 5.369023773s average 513.782µs median 416.176µs
INFO  timer:  name TestListRandomPagesMethodC count 10450 duration 16.532761861s average 1.582082ms median 1.049295ms
INFO  timer:  name TestGetPageCollection count 10450 duration 51.641570568s average 4.941777ms median 3.566887ms
INFO  build:  duration 27.481160439s

                   |  EN    
-------------------+--------
  Pages            | 10477  
  Paginator pages  |  1035  
  Non-page files   |     0  
  Static files     |     1  
  Processed images |     0  
  Aliases          |    12  
  Cleaned          |     0  

Total in 27509 ms
22

Excellent. I know where the problem is. Give me a minute or five.

23

I just revised the code in this comment:
https://discourse.gohugo.io/t/can-shuffle-be-cached/54636/17

It is much faster now.

In an attempt to be clever, I used the complement function to remove the current page from the page collection, but that operation is very expensive compared to a conditional check.

24

First run

$ hugo
Start building sites … 
hugo v0.147.0-7d0039b86ddd6397816cc3383cb0cfa481b15f32+extended linux/amd64 BuildDate=2025-04-25T15:26:28Z VendorInfo=gohugoio


                   |  EN    
-------------------+--------
  Pages            | 36938  
  Paginator pages  |     0  
  Non-page files   |     5  
  Static files     |     8  
  Processed images |    10  
  Aliases          |     0  
  Cleaned          |     0  

Total in 51939 ms

Second run

$ hugo
Start building sites … 
hugo v0.147.0-7d0039b86ddd6397816cc3383cb0cfa481b15f32+extended linux/amd64 BuildDate=2025-04-25T15:26:28Z VendorInfo=gohugoio


                   |  EN    
-------------------+--------
  Pages            | 36938  
  Paginator pages  |     0  
  Non-page files   |     5  
  Static files     |     8  
  Processed images |    10  
  Aliases          |     0  
  Cleaned          |     0  

Total in 51153 ms

So, it seems just about the same for me (I didn’t clear the public folder in all runs. Should I have?). Comp is low spec core-I5 3rd Gen with HDD and 8GB RAM, suitable for my writing blogs. Maybe the results are more visible on a higher spec setup?

25

Please do this again (the repo has changed) and post the console log:

git clone --single-branch -b hugo-forum-topic-54636 https://github.com/jmooring/hugo-testing hugo-forum-topic-54636
cd hugo-forum-topic-54636
hugo --logLevel info
26 
$ git clone --single-branch -b hugo-forum-topic-54636 https://github.com/jmooring/hugo-testing hugo-forum-topic-54636
cd hugo-forum-topic-54636
hugo --logLevel info
Cloning into 'hugo-forum-topic-54636'...
remote: Enumerating objects: 10544, done.
remote: Counting objects: 100% (2489/2489), done.
remote: Compressing objects: 100% (2411/2411), done.
remote: Total 10544 (delta 114), reused 88 (delta 78), pack-reused 8055 (from 2)
Receiving objects: 100% (10544/10544), 14.75 MiB | 3.19 MiB/s, done.
Resolving deltas: 100% (351/351), done.
Start building sites … 
hugo v0.147.0-7d0039b86ddd6397816cc3383cb0cfa481b15f32+extended linux/amd64 BuildDate=2025-04-25T15:26:28Z VendorInfo=gohugoio

INFO  static: syncing static files to / duration 465.955µs
INFO  build:  step process substep collect files 10461 files_total 10461 pages_total 10461 resources_total 0 duration 416.094788ms
INFO  build:  step process duration 416.181678ms
INFO  build:  step assemble duration 280.402941ms
INFO  build:  step render substep pages site en outputFormat html duration 11.672356485s
INFO  build:  step render substep pages site en outputFormat rss duration 1.649542467s
INFO  build:  step render pages 11512 content 10463 duration 13.328893533s
INFO  build:  step render deferred count 0 duration 1.637µs
INFO  build:  step postProcess duration 13.712µs
INFO  timer:  name TestGetPageCollection count 10450 duration 211.465384ms average 20.235µs median 11.403µs
INFO  timer:  name TestListRandomPagesMethodA count 10450 duration 4.432601804s average 424.172µs median 369.769µs
INFO  timer:  name TestListRandomPagesMethodB count 10450 duration 5.96531728s average 570.843µs median 484.683µs
INFO  timer:  name TestListRandomPagesMethodC count 10450 duration 15.720986093s average 1.5044ms median 1.097591ms
INFO  build:  duration 14.040138775s

                   |  EN    
-------------------+--------
  Pages            | 10477  
  Paginator pages  |  1035  
  Non-page files   |     0  
  Static files     |     1  
  Processed images |     0  
  Aliases          |    12  
  Cleaned          |     0  

Total in 14056 ms
27

When you built the test site earlier:

INFO  timer:  name TestListRandomPagesMethodA count 10450 duration 4.655520497s average 445.504µs median 376.022µs
INFO  timer:  name TestListRandomPagesMethodB count 10450 duration 5.369023773s average 513.782µs median 416.176µs
INFO  timer:  name TestListRandomPagesMethodC count 10450 duration 16.532761861s average 1.582082ms median 1.049295ms
INFO  timer:  name TestGetPageCollection count 10450 duration 51.641570568s average 4.941777ms median 3.566887ms
INFO  build:  duration 27.481160439s

When you built the test site after I fixed the page collection creation:

INFO  timer:  name TestGetPageCollection count 10450 duration 211.465384ms average 20.235µs median 11.403µs
INFO  timer:  name TestListRandomPagesMethodA count 10450 duration 4.432601804s average 424.172µs median 369.769µs
INFO  timer:  name TestListRandomPagesMethodB count 10450 duration 5.96531728s average 570.843µs median 484.683µs
INFO  timer:  name TestListRandomPagesMethodC count 10450 duration 15.720986093s average 1.5044ms median 1.097591ms
INFO  build:  duration 14.040138775s

So we know two things:

  1. The fix worked. The build time is now half of what it was.
  2. Method A (using random numbers) is about 3x faster than Method C (shuffle).

And we also know that your PC is ridiculuously slow. The build time on my exceedingly average PC is 5s (1/3 of your build time).

28

You should have seen when it was on Windows! This site I shared built in 10 minutes and 5 minutes after excluding hugo.exe in the inbuilt antivirus. In Linux, the fact it builds below a minute has made amazed me. The comp is old by modern standards. So the build speed being slow is to be expected. I chose to stick with it instead of running Termux in my tablet.

29

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.