Performance expectations for large number of pages?

support
1

Hi,

I have a Hugo project with somewhere in the neighborhood of 15,000 markdown page files. I’m using a custom built theme, and the full build is pretty slow - around 1.5-2mins usually. I’m trying to optimize it, so what I did was create a super basic theme that basically only rendered single pages, and I disabled all custom shortcodes and there’s no lists or pagination or taxonomies. This “basic” theme I made is here: https://github.com/roytang/hugo-theme-basic

Using this theme, it still takes more than half a minute to run a build:

PS C:\repos\blog> hugo --templateMetrics
Building sites …
Template Metrics:

     cumulative       average       maximum
       duration      duration      duration  count  template
     ----------      --------      --------  -----  --------
     1.5827558s     103.077µs      1.9991ms  15355  _default/single.html
     614.2329ms     591.746µs     22.9866ms   1038  _internal/_default/rss.xml
     585.6646ms    585.6646ms    585.6646ms      1  _internal/_default/sitemap.xml
       1.9988ms       1.927µs       999.4µs   1037  _default/list.html
             0s            0s            0s      2  _internal/shortcodes/youtube.html
             0s            0s            0s      1  404.html
             0s            0s            0s      1  index.html


                   |  EN
+------------------+-------+
  Pages            | 16393
  Paginator pages  |     0
  Non-page files   |  3680
  Static files     |  1001
  Processed images |     0
  Aliases          |     1
  Sitemaps         |     1
  Cleaned          |     0

Total in 37765 ms

I’m wondering whether this is typically the minimum build time I should be expecting given the number of pages? I need to know what baseline to expect, or if there is any further optimization I can do (which I don’t see any at the moment). If it matters, I’m running on Windows 10 with 16GB of memory and an SSD. (Whether I’m using the SSD or my normal HDD doesnt seem to make a difference though) I also have a lot of images in the both in the static dir and in page bundles in the content dir, if that matters (this basic theme does not include those page resources in the generated HTML). I would guess those are the “non-page files” listed in the log above.

Looking through some older posts here, most of the performance analysis I’ve seen is based on the output of the stepAnalysis flag, but it seems that’s no longer available in my current build of hugo. (“hugo version” tells me that’s “Hugo Static Site Generator v0.58.2-253E5FDC windows/amd64 BuildDate: 2019-09-13T08:06:10Z”). I also saw an old post claiming someone was able to generate 10k posts in around 4s, which is a huge gap from what I have now, so I’m not sure if I’m doing something wrong!

Any advice would be greatly appreciated!

2

2ms per page isn’t slow! The limit can be the IO rate of your system. Any cache (not) activated?
You can try to set the ignoreFiles entry in the config to speed up a little.

1 Like
3

So, this performance is ok then? And I guess that as the number of posts scales up, I can expect a linear increase in build time?

Any cache (not) activated? -> How do I check this?

4

I saw from another thread (Page generation performance expectations?), that they recommend using a copy operation on the content folder as a benchmark to see how much time is lost due to I/O. I tried it out on my machine and it looks like I/O is indeed the bottleneck, I can’t do anything about that.

time cp -r content content.test

real    1m50.044s
user    0m0.359s
sys     0m28.250s

Anyway, in that case, I’ll consider the 2ms per page as a reasonable target to aim for while optimizing my theme.

Thanks!

5

some tipps here
https://www.disk-partition.com/windows-10/ssd-optimization-windows-10-4348.html

6

Your template metrics looks normal/fast. I would guess it’s the static/other files copying. Big files?