We’ve been having some challenges in getting Hugo to build a website with ~2,5 million pages. Up to a few hundreds of thousands of pages worked fine to build on my local (6-Core Intel Core i7 - 3,7 GHz, with 32GB of RAM) but growing beyond that the RAM usage gets out of control. Currently with the…

[image] Nizzlay: It seems to me that using this much RAM cannot be correct, seeing as the markdown files in the content folder are only ~7GB in total. Try running hugo with a few diagnostic flags and post their output in this topic so that people can have a look. First try: hugo --print-mem …

I had htop open when it was running, and I did notice that most of the time Hugo was using about 20% of 1 core, and then suddenly bursted to all 64 cores 100% for a short while. Memory usage increased steadily. Started slowly, and gained exponentially. Will do the test runs now, and post the resul…

hugo --print-mem is done. Output: Alloc = 81.4 GB TotalAlloc = 233.7 GB Sys = 129.2 GB NumGC = 34 Alloc = 65.3 GB TotalAlloc = 236.4 GB Sys = 129.2 GB NumGC = 34 Alloc = 67.9 GB TotalAlloc = 239.0 GB Sys = 129.2 GB NumGC = 34 Alloc = 70.5 GB TotalAlloc = 241.6 GB Sys = 129.2 GB NumGC = 34…

Results from hugo --templateMetrics --templateMetricsHints: hugo --templateMetrics --templateMetricsHints Start building sites … Template Metrics: cache cumulative average maximum potential duration duration duration count template ----- -------…

[image] Nizzlay: Error: Error building site: TOCSS: failed to transform "sass/main.scss" (text/x-scss). Check your Hugo installation; you need the extended version to build SCSS/SASS. This suggests your site is failing because you don’t have Hugo extended. What do you get running hugo version?…

Do you really have 2.5+ million distinct versions of your header, banner, menu, env, breadcrumbs, button, noscript, etc? The “cache potential” column suggests not. That’s got to contribute to your build time and memory usage, and careful use of partialCached should help quite a bit. I’ve never gone…

I know. Had some issues with that. Is resolved now: Building Hugo Extended for ARM64

Working on implementing some cache now. Will try a new run later today when that’s done.

The interesting thing is that after building the site, and then while running the deploy it will require 75-80GB of RAM again. [Screenshot 2021-03-04 at 15.15.50] It keeps hovering in the 75-80GB range for the RAM. Not increasing or decreasing beyond that range.

Hugo scalability?

support

frjo March 3, 2021, 2:29pm 5

I believe the reason Hugo is really really fast is also the reson you get problems with million of pages. There seems to be plans, see below, to fix this but that’s for the future.

Topic		Replies	Views
Generating a rather large website support	20	1856	July 1, 2019
Can't build website with 16GB of RAM (and it crashes my system) support performance	11	174	January 20, 2026
Slow build support	18	2094	June 8, 2021
Building a site with 50,000 pages taking a very long time support	66	2638	February 24, 2023
Resources for hugo to build big sites support	4	746	July 29, 2020

Hugo scalability?

Related topics