I’m trying to set HUGO_MEMORYLIMIT while generating a website with a high number of pages (> 500k). It doesn’t take effect and I’m not 100% sure if it’s a bug or something I’m missing.
I defined HUGO_MEMORYLIMIT as environment variable. But still, hugo uses more and more memory until it gets killed for taking too much memory. See attached screenshot.
OS : Linux Hugo version : hugo v0.145.0-666444f0a52132f9fec9f71cf25b441cc6a4f355+extended linux/amd64 BuildDate=2025-02-26T15:41:25Z VendorInfo=gohugoio
I don’t think it’s fair to expect Hugo to build 500k pages with 1GB of memory. Or, that may be possible, but it really depends on the site in question.
You can get better indication about how the memory throttling behaves by running with hugo --logLevel info and look for log statements labeled dynacache.
Thought I’d ask for your general approach to understanding memory usage as it seems fitting here:
INFO build: step render substep pages site en outputFormat html duration 1m6.7023748s INFO dynacache: adjusted partitions' max size evicted 1150 numGC 34 limit 3.99 GB alloc 4.35 GB totalAlloc 16.12 GB
I am newer to Hugo so please forgive me here but could you help me understand what would be indicative of memory throttling based on these logs?
This is my interpretation so please correct me if I am wrong :
Below is the target memory usage configured from HUGO_MEMORYLIMIT?
3.99 GB alloc
Below is the actual memory allocated to this stage
alloc 4.35 GB
Below is the total memory available on the system?
totalAlloc 16.12 GB
As someone who wants to explore slower than expected build times should I use the actual memory usage versus target from “logLevel info” in combination with template metrics get an understanding of bottlenecks from partials and steps within the build process?
Thank you for your answer.
1GB was just for testing purposes, highlighting the issue.
I’m using 10GB while trying to build the 500k pages and was expecting it to be enough. I’ll try using logLevel to see if there is something I could optimize.
Ok, I finally managed to build the website by increasing the swap space. Performance wasn’t too bad, and the 400k pages were generated in about half an hour.
I set the MEMORYLIMIT to 5GB, but the total memory usage peaked at around 11-12GB. I’m not sure what totalAlloc refers to in the logs.
Also, the --renderSegments option is fantastic—it allows me to generate a few pages in just 1-2 minutes.
hugo --logLevel info
Start building sites …
hugo v0.145.0-666444f0a52132f9fec9f71cf25b441cc6a4f355+extended linux/amd64 BuildDate=2025-02-26T15:41:25Z VendorInfo=gohugoio
INFO static: syncing static files to / duration 6.355337ms
INFO build: step process substep collect files 175000 files_total 175000 pages_total 175000 resources_total 0 duration 3.010849974s
INFO build: step process substep collect files 104000 files_total 279000 pages_total 279000 resources_total 0 duration 3.066585141s
INFO build: step process substep collect files 14000 files_total 293000 pages_total 293000 resources_total 0 duration 3.008375648s
INFO build: step process substep collect files 59000 files_total 352000 pages_total 352000 resources_total 0 duration 3.007878646s
INFO build: step process substep collect files 46730 files_total 398730 pages_total 398723 resources_total 7 duration 2.241249054s
INFO build: step process duration 14.335230826s
INFO build: step assemble duration 1m38.074957441s
INFO postcss: use config file "website/postcss.config.js"
INFO exec: resolve "postcss" using node_modules/.bin
INFO dynacache: adjusted partitions' max size evicted 4898 numGC 31 limit 5.00 GB alloc 5.03 GB totalAlloc 9.14 GB
INFO dynacache: adjusted partitions' max size evicted 4897 numGC 31 limit 5.00 GB alloc 5.38 GB totalAlloc 9.50 GB
INFO dynacache: adjusted partitions' max size evicted 4286 numGC 31 limit 5.00 GB alloc 5.53 GB totalAlloc 9.64 GB
INFO dynacache: adjusted partitions' max size evicted 4278 numGC 32 limit 5.00 GB alloc 5.20 GB totalAlloc 10.95 GB
INFO dynacache: adjusted partitions' max size evicted 4532 numGC 33 limit 5.00 GB alloc 5.30 GB totalAlloc 14.58 GB
INFO dynacache: adjusted partitions' max size evicted 10352 numGC 35 limit 5.00 GB alloc 5.66 GB totalAlloc 23.63 GB
INFO dynacache: adjusted partitions' max size evicted 6980 numGC 36 limit 5.00 GB alloc 5.21 GB totalAlloc 26.96 GB
INFO dynacache: adjusted partitions' max size evicted 3790 numGC 37 limit 5.00 GB alloc 5.02 GB totalAlloc 30.74 GB
INFO dynacache: adjusted partitions' max size evicted 3826 numGC 38 limit 5.00 GB alloc 5.02 GB totalAlloc 34.94 GB
INFO dynacache: adjusted partitions' max size evicted 5004 numGC 39 limit 5.00 GB alloc 5.19 GB totalAlloc 39.44 GB
INFO dynacache: adjusted partitions' max size evicted 3908 numGC 40 limit 5.00 GB alloc 5.03 GB totalAlloc 43.62 GB
INFO dynacache: adjusted partitions' max size evicted 2258 numGC 41 limit 5.00 GB alloc 5.12 GB totalAlloc 47.98 GB
INFO build: step render substep pages site en outputFormat html duration 28m16.931595004s
INFO build: step render substep pages site en outputFormat rss duration 1m39.183463767s
INFO build: step render pages 407943 content 1178954 duration 29m56.496130599s
INFO build: step render deferred count 0 duration 4.709µs
INFO build: step postProcess duration 2.277599132s
INFO build: duration 31m55.523528146s
| EN
-------------------+---------
Pages | 407281
Paginator pages | 662
Non-page files | 7
Static files | 46
Processed images | 0
Aliases | 119
Cleaned | 0
: