I’m somewhat at a loss presently as to how to proceed with this.
I have an internal Hugo built site (using a lightly modified Docsy theme) that did build successfully on current hardware a few weeks ago, but after some (what I thought were inconsequential) changes - like moving
/docs/content/en/, and disabling Lunr offline search - I now find that I can’t build the site anymore, and that after 50-80 mins of processing the Hugo process is killed (presumably by the OS).
I’ve talked about issues with memory before (What is --printMemoryUsage detailling about? - #6 by bep - and could really have used that output counter in gathering stats today Bep ) and thought I had this problem solved given the site was building a few weeks back.
I tried three runs on my machine using the latest 118.2 build, including after a clean restart and no user apps started, on my iMac 3.8 GHz 8-Core Intel Core i7 with 128 GB 2667 MHz DDR4 (macOS 13.5.1), and each one failed, with memory usage nearing 300GB. Latest example showed:
… and took around 50 minutes to run to failure. I ran most tests with the following options:
hugo -v --debug --logLevel debug --printPathWarnings --templateMetrics --templateMetricsHints --printMemoryUsage (and note that there’s not a lot of information comes out during the build even with all these turned on) but I didn’t get the final results as it ended with:
 15288 killed hugo --printPathWarnings --templateMetrics --templateMetricsHints
… as did all the other attempts. However, I wanted to try and understand a bit about where and when the memory was being used (hoping that the debug info might clue me in so that I could review the template for clues) and so I ended up graphing the memory outputs like so:
The X axis is a counter for each set of memory outputs to the terminal, and the Y axis is a log scale of values in MB. Unfortunately I don’t have any better timing metric than the number of memory statements to help understand what’s happening here. For about 60% of the time, memory usage is low - less than 100MB of system memory (I’ve come to ignore the TotalAlloc values and look at the Sys - which tends to reflect what Activity Monitor reports as Virtual Memory Size, and Alloc as Real Memory Size). I ran it again and graphed it again and got slightly different results, but a similar pattern:
This run lasted longer before it failed - I can’t explain why - but fail it did.
So I thought I’d had success previously with earlier releases, so I’ve attempted to run the same build on 116.0 and 115.4 as I recalled these were previously used on this system (Homebrew updates being intermittent here, I haven’t always caught every Hugo release, and Homebrew only installs the latest version each time).
116 came back with these outputs:
and a plot of:
Whilst 115.4 gave me:
and a plot of:
116 had the longest run time till failure - lasting almost a third longer time wise than 118; and 115.4 also lasted longer, but started to gobble up memory much earlier than either of the later versions.
In all cases, the jump in memory use from <1Gb to >100Gb happens very quickly, and from there onwards things get noticeably slower as the virtual memory kicks in hard and the memory faults rise rapidly to page info back in… but I don’t understand what’s happening in the build to explain this pattern, or what I can do with this knowledge, so I’m hoping someone here can advise please?
I haven’t had the time yet to revert the content (5.8GB of Markdown text with no images) back to the last known working build to test against these failures - but I’m not aware of any reason why this current content shouldn’t be able to build successfully on this hardware either.
Insight and suggestions welcome! Thank you for your time and efforts - I hope mine weren’t in vain, as this reporting has taken over 6 hours to do today.