Pagination performance

support
1

Hey Hugo Folks,

tl;dr: In building layouts should we prefer multiple, per-section, not DRY templates over a single, DRY template with conditional logic?

Two years ago we welcomed a kid and in that intervening time I’ve been using Hugo to document his life, recipes, and occasional blogging and it’s Just Worked. Awesome. It met my needs and I’ve loved the stability. Thanks to all.

With a tiny bit more time on my hands, I decided to update the themes and my Hugo version etc. Catching up my layouts to align to incremental updates to my theme (ananke forever!) was kinda painful. So I decided to refactor my template logic.

In the mists of time, I had multiple sections that had their own /section.html file. That’s been fine. But as I did a quick find(1), I had 8 of them, each with subtly different characteristics. So I pulled all the logic up into a single _default/section.html with several conditionals inside and used the respective $SECTION/_index.md frontmatter to provide the conditional signals needed to render each section appropriately.

I tested this and everything works. Hooray Hugo.

But OH MY GOD, my build times went into the toilet.

                   |  EN
-------------------+-------
  Pages            | 6239
  Paginator pages  |  399
  Non-page files   | 2615
  Static files     |   19
  Processed images |    0
  Aliases          |   21
  Cleaned          |    0

Total in 122994 ms

versus the “multi-file” approach:

                   |  EN
-------------------+-------
  Pages            | 6239
  Paginator pages  |   15
  Non-page files   | 2615
  Static files     |   19
  Processed images |    0
  Aliases          |   16
  Cleaned          |    0

Total in 3910 ms
hugo  13.20s user 3.52s system 416% cpu 4.016 total

So the big leap here was in the number of pagination files. Let’s crank up the pagerSize config parameter.

                   |  EN
-------------------+-------
  Pages            | 6239
  Paginator pages  |   95
  Non-page files   | 2615
  Static files     |   19
  Processed images |    0
  Aliases          |   21
  Cleaned          |    0

Total in 30298 ms
hugo  44.62s user 5.58s system 165% cpu 30.407 total

OK, so clearly playing with pagination has a pretty big effect. OK, for each of those _index.md files, let’s set pagination to off.

                   |  EN
-------------------+-------
  Pages            | 6239
  Paginator pages  |    0
  Non-page files   | 2615
  Static files     |   19
  Processed images |    0
  Aliases          |   20
  Cleaned          |    0

Total in 3892 ms

Aha. So, working hypothesis:

  1. Pagination is expensive
  2. It is more expensive when the lookup order (all the way up to _default/list.html) is “tall”
  3. It is more expensive when that file has conditional logic
  4. Thus: Tradeoff is difficulty of maintenance of layout VERSUS speed/ability to paginate robustly

I sense a bit of “Yeah, well duh, when you paginate things get slower because we have to track more state, etc. – What did you expect?” here. If this performance is within tolerances. Great. I’ll paginate in smaller batches. But if this performance is outside the norm, I wanted to pass the data along.

2

Are you calling Hugo’s embedded pagination template, or did you roll your own?

3

Hi @jmooring, I’m far too dim to roll my own pagination template :grin:

4

I would really, really like access to your site’s repository, privately if you wish. This smells like an O(n^2) problem.

Please let me know your Hugo version as well.

5 
Version: hugo v0.145.0+extended+withdeploy darwin/arm64
BuildDate=2025-02-26T15:41:25Z VendorInfo=brew

OK, so I did some debugging and I found a slip up in the template.

I had two variables for detecting whether we should do pagination: skip_pagination and .use_paginator (hm, sounds like that could create an O(n^2) fingerprint…).

After this change:

Total in 4407 ms
hugo -DF  13.18s user 3.85s system 376% cpu 4.523 total

versus:

Total in 267018 ms
hugo -DF  144.61s user 12.25s system 58% cpu 4:27.45 total


diff --git a/layouts/_default/list.html b/layouts/_default/list.html
index e669b20e..dd7753f9 100644
--- a/layouts/_default/list.html
+++ b/layouts/_default/list.html
@@ -69,10 +69,8 @@
       </section>
     {{ end }}
 
-    {{ if not .Params.skip_pagination }}
-      {{ if .Params.use_paginator | default true }}
-        {{- template "_internal/pagination.html" . -}}
-      {{ end }}
+    {{ if .Params.use_paginator | default true }}
+      {{- template "_internal/pagination.html" . -}}
     {{ end }}
   </article>
 {{ end }}

Thanks for helping me think through this process. I knew it wasn’t Hugo :slight_smile:

1 Like
6

That’s more like it; consistent with these results:

                   |  EN    
-------------------+--------
  Pages            | 10477  
  Paginator pages  |  1035  
  Non-page files   |     0  
  Static files     |     1  
  Processed images |     0  
  Aliases          |    12  
  Cleaned          |     0  

Built in 4809 ms

That’s v0.145.0 inside a VM with exceptionally average performance.

For anyone else stumbling across this, if you want to test pagination performance:

git clone --single-branch -b hugo-github-issue-8602 https://github.com/jmooring/hugo-testing hugo-github-issue-8602
cd hugo-github-issue-8602
hugo server

> 10,000 pages with 10 pages per pager.

And if you disable pagination in layouts/_default/list.html you will see almost no difference in build time.

1 Like