We’ve been having some challenges in getting Hugo to build a website with ~2,5 million pages.
Up to a few hundreds of thousands of pages worked fine to build on my local (6-Core Intel Core i7 - 3,7 GHz, with 32GB of RAM) but growing beyond that the RAM usage gets out of control.
Currently with the ~2,5 million pages Hugo will try to request 200GB RAM from my PC (which I obviously don’t have), and will eventually crash after my disk runs out of space to facillitate the swap.
Tried setting this up on a server with 512GB of RAM to see if it would resolve the problem. In that system it does eventually finish building the site, but at it’s peak still consumes 180GB of RAM.
It seems to me that using this much RAM cannot be correct, seeing as the markdown files in the content folder are only ~7GB in total.
1 Like
Try running hugo with a few diagnostic flags and post their output in this topic so that people can have a look.
First try:hugo --print-mem to print the memory usage at regular intervals
Then run the combo: hugo --templateMetrics --templateMetricsHints to display metrics about template executions along with some calculated improvement hints.
I had htop open when it was running, and I did notice that most of the time Hugo was using about 20% of 1 core, and then suddenly bursted to all 64 cores 100% for a short while.
Memory usage increased steadily. Started slowly, and gained exponentially.
Will do the test runs now, and post the results.
hugo --print-mem is done.
Output:
Alloc = 81.4 GB
TotalAlloc = 233.7 GB
Sys = 129.2 GB
NumGC = 34
Alloc = 65.3 GB
TotalAlloc = 236.4 GB
Sys = 129.2 GB
NumGC = 34
Alloc = 67.9 GB
TotalAlloc = 239.0 GB
Sys = 129.2 GB
NumGC = 34
Alloc = 70.5 GB
TotalAlloc = 241.6 GB
Sys = 129.2 GB
NumGC = 34
Alloc = 73.1 GB
TotalAlloc = 244.2 GB
Sys = 129.2 GB
NumGC = 34
Alloc = 75.7 GB
TotalAlloc = 246.8 GB
Sys = 129.2 GB
NumGC = 34
Alloc = 78.3 GB
TotalAlloc = 249.4 GB
Sys = 129.2 GB
NumGC = 34
Alloc = 80.9 GB
TotalAlloc = 252.0 GB
Sys = 129.3 GB
NumGC = 34
Alloc = 83.5 GB
TotalAlloc = 254.6 GB
Sys = 129.3 GB
NumGC = 34
Alloc = 86.1 GB
TotalAlloc = 257.1 GB
Sys = 129.3 GB
NumGC = 34
Alloc = 88.7 GB
TotalAlloc = 259.7 GB
Sys = 129.3 GB
NumGC = 34
Alloc = 91.2 GB
TotalAlloc = 262.3 GB
Sys = 129.3 GB
NumGC = 34
Alloc = 93.8 GB
TotalAlloc = 264.9 GB
Sys = 129.3 GB
NumGC = 34
Alloc = 96.4 GB
TotalAlloc = 267.5 GB
Sys = 129.3 GB
NumGC = 34
Alloc = 99.0 GB
TotalAlloc = 270.1 GB
Sys = 129.3 GB
NumGC = 34
Alloc = 101.6 GB
TotalAlloc = 272.7 GB
Sys = 129.4 GB
NumGC = 34
Alloc = 104.2 GB
TotalAlloc = 275.2 GB
Sys = 129.4 GB
NumGC = 34
Alloc = 106.8 GB
TotalAlloc = 277.9 GB
Sys = 129.4 GB
NumGC = 34
Alloc = 109.4 GB
TotalAlloc = 280.4 GB
Sys = 129.4 GB
NumGC = 34
Alloc = 112.0 GB
TotalAlloc = 283.0 GB
Sys = 129.4 GB
NumGC = 34
Alloc = 114.6 GB
TotalAlloc = 285.6 GB
Sys = 129.4 GB
NumGC = 34
Alloc = 117.2 GB
TotalAlloc = 288.2 GB
Sys = 129.4 GB
NumGC = 34
Alloc = 97.3 GB
TotalAlloc = 290.8 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 68.4 GB
TotalAlloc = 293.3 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 67.3 GB
TotalAlloc = 296.0 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 69.8 GB
TotalAlloc = 298.6 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 72.4 GB
TotalAlloc = 301.2 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 75.0 GB
TotalAlloc = 303.7 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 77.5 GB
TotalAlloc = 306.3 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 80.1 GB
TotalAlloc = 308.8 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 82.7 GB
TotalAlloc = 311.4 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 85.3 GB
TotalAlloc = 314.0 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 87.8 GB
TotalAlloc = 316.6 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 90.4 GB
TotalAlloc = 319.1 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 92.9 GB
TotalAlloc = 321.6 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 95.4 GB
TotalAlloc = 324.2 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 97.9 GB
TotalAlloc = 326.7 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 100.5 GB
TotalAlloc = 329.2 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 103.1 GB
TotalAlloc = 331.8 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 105.6 GB
TotalAlloc = 334.3 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 108.2 GB
TotalAlloc = 336.9 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 110.7 GB
TotalAlloc = 339.4 GB
Sys = 130.1 GB
NumGC = 35
Alloc = 113.3 GB
TotalAlloc = 342.0 GB
Sys = 130.2 GB
NumGC = 35
Alloc = 115.9 GB
TotalAlloc = 344.6 GB
Sys = 130.2 GB
NumGC = 35
Alloc = 118.5 GB
TotalAlloc = 347.2 GB
Sys = 130.2 GB
NumGC = 35
Alloc = 85.8 GB
TotalAlloc = 349.7 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 65.7 GB
TotalAlloc = 352.3 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 68.2 GB
TotalAlloc = 354.9 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 70.8 GB
TotalAlloc = 357.5 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 73.4 GB
TotalAlloc = 360.1 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 76.0 GB
TotalAlloc = 362.7 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 78.6 GB
TotalAlloc = 365.2 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 81.1 GB
TotalAlloc = 367.8 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 83.7 GB
TotalAlloc = 370.4 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 86.3 GB
TotalAlloc = 373.0 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 88.8 GB
TotalAlloc = 375.5 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 91.4 GB
TotalAlloc = 378.1 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 94.0 GB
TotalAlloc = 380.7 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 96.6 GB
TotalAlloc = 383.3 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 99.2 GB
TotalAlloc = 385.8 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 101.7 GB
TotalAlloc = 388.4 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 104.3 GB
TotalAlloc = 391.0 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 106.9 GB
TotalAlloc = 393.6 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 109.5 GB
TotalAlloc = 396.2 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 112.1 GB
TotalAlloc = 398.7 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 114.6 GB
TotalAlloc = 401.3 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 117.2 GB
TotalAlloc = 403.8 GB
Sys = 130.8 GB
NumGC = 36
Alloc = 119.7 GB
TotalAlloc = 406.4 GB
Sys = 131.4 GB
NumGC = 36
Alloc = 77.6 GB
TotalAlloc = 408.9 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 66.6 GB
TotalAlloc = 411.5 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 69.2 GB
TotalAlloc = 414.1 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 71.7 GB
TotalAlloc = 416.6 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 74.2 GB
TotalAlloc = 419.2 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 76.9 GB
TotalAlloc = 421.8 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 79.6 GB
TotalAlloc = 424.5 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 82.2 GB
TotalAlloc = 427.1 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 84.7 GB
TotalAlloc = 429.6 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 87.3 GB
TotalAlloc = 432.2 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 89.9 GB
TotalAlloc = 434.8 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 92.4 GB
TotalAlloc = 437.4 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 95.0 GB
TotalAlloc = 439.9 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 97.6 GB
TotalAlloc = 442.5 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 100.2 GB
TotalAlloc = 445.1 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 102.7 GB
TotalAlloc = 447.6 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 105.3 GB
TotalAlloc = 450.2 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 107.9 GB
TotalAlloc = 452.8 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 110.5 GB
TotalAlloc = 455.4 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 113.1 GB
TotalAlloc = 458.0 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 115.6 GB
TotalAlloc = 460.5 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 118.2 GB
TotalAlloc = 463.1 GB
Sys = 131.5 GB
NumGC = 37
Alloc = 118.5 GB
TotalAlloc = 465.7 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 77.7 GB
TotalAlloc = 468.2 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 67.2 GB
TotalAlloc = 470.8 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 69.9 GB
TotalAlloc = 473.4 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 72.4 GB
TotalAlloc = 476.0 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 75.0 GB
TotalAlloc = 478.6 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 77.6 GB
TotalAlloc = 481.1 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 80.2 GB
TotalAlloc = 483.7 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 82.7 GB
TotalAlloc = 486.3 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 85.3 GB
TotalAlloc = 488.8 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 87.8 GB
TotalAlloc = 491.4 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 90.4 GB
TotalAlloc = 494.0 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 93.0 GB
TotalAlloc = 496.5 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 95.5 GB
TotalAlloc = 499.1 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 98.1 GB
TotalAlloc = 501.6 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 100.7 GB
TotalAlloc = 504.3 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 103.3 GB
TotalAlloc = 506.8 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 105.9 GB
TotalAlloc = 509.4 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 108.4 GB
TotalAlloc = 511.9 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 111.0 GB
TotalAlloc = 514.5 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 113.6 GB
TotalAlloc = 517.2 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 116.2 GB
TotalAlloc = 519.7 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 118.7 GB
TotalAlloc = 522.3 GB
Sys = 132.3 GB
NumGC = 38
Alloc = 121.3 GB
TotalAlloc = 524.8 GB
Sys = 133.1 GB
NumGC = 38
Alloc = 78.5 GB
TotalAlloc = 527.4 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 67.4 GB
TotalAlloc = 530.0 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 70.0 GB
TotalAlloc = 532.6 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 72.5 GB
TotalAlloc = 535.2 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 75.1 GB
TotalAlloc = 537.7 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 77.8 GB
TotalAlloc = 540.4 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 80.4 GB
TotalAlloc = 543.0 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 83.0 GB
TotalAlloc = 545.6 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 85.7 GB
TotalAlloc = 548.3 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 88.3 GB
TotalAlloc = 550.9 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 90.9 GB
TotalAlloc = 553.5 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 93.5 GB
TotalAlloc = 556.1 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 96.1 GB
TotalAlloc = 558.8 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 98.8 GB
TotalAlloc = 561.4 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 101.4 GB
TotalAlloc = 564.0 GB
Sys = 133.1 GB
NumGC = 39
Alloc = 108.9 GB
TotalAlloc = 571.5 GB
Sys = 133.7 GB
NumGC = 39
Alloc = 110.9 GB
TotalAlloc = 573.6 GB
Sys = 133.7 GB
NumGC = 39
Alloc = 114.4 GB
TotalAlloc = 577.0 GB
Sys = 134.2 GB
NumGC = 39
Alloc = 118.8 GB
TotalAlloc = 581.4 GB
Sys = 134.4 GB
NumGC = 39
Alloc = 119.4 GB
TotalAlloc = 582.0 GB
Sys = 134.6 GB
NumGC = 39
Alloc = 120.2 GB
TotalAlloc = 582.8 GB
Sys = 135.0 GB
NumGC = 39
Alloc = 120.7 GB
TotalAlloc = 583.3 GB
Sys = 135.0 GB
NumGC = 39
Alloc = 121.1 GB
TotalAlloc = 583.7 GB
Sys = 135.0 GB
NumGC = 39
Alloc = 78.1 GB
TotalAlloc = 584.8 GB
Sys = 135.8 GB
NumGC = 40
Alloc = 63.8 GB
TotalAlloc = 585.2 GB
Sys = 135.8 GB
NumGC = 40
Alloc = 64.3 GB
TotalAlloc = 585.6 GB
Sys = 135.8 GB
NumGC = 40
Alloc = 64.7 GB
TotalAlloc = 586.1 GB
Sys = 135.8 GB
NumGC = 40
Alloc = 65.1 GB
TotalAlloc = 586.5 GB
Sys = 135.8 GB
NumGC = 40
Alloc = 65.6 GB
TotalAlloc = 586.9 GB
Sys = 135.8 GB
NumGC = 40
Alloc = 66.0 GB
TotalAlloc = 587.4 GB
Sys = 135.8 GB
NumGC = 40
Alloc = 67.8 GB
TotalAlloc = 589.2 GB
Sys = 137.2 GB
NumGC = 40
Alloc = 68.3 GB
TotalAlloc = 589.6 GB
Sys = 137.2 GB
NumGC = 40
Alloc = 71.4 GB
TotalAlloc = 592.8 GB
Sys = 140.3 GB
NumGC = 40
Alloc = 71.4 GB
TotalAlloc = 592.8 GB
Sys = 140.3 GB
NumGC = 40
Alloc = 72.1 GB
TotalAlloc = 593.4 GB
Sys = 141.0 GB
NumGC = 40
Alloc = 77.6 GB
TotalAlloc = 599.0 GB
Sys = 141.0 GB
NumGC = 40
Total in 2287837 ms
Error: Error building site: TOCSS: failed to transform "sass/main.scss" (text/x-scss). Check your Hugo installation; you need the extended version to build SCSS/SASS.
Alloc = 79.2 GB
TotalAlloc = 600.6 GB
Sys = 141.0 GB
NumGC = 40
Output seems incomplete. This is not what it started with. It starts with only a few GB of memory consumption, keeps increasing constantly.
frjo
March 3, 2021, 2:29pm
5
I believe the reason Hugo is really really fast is also the reson you get problems with million of pages. There seems to be plans, see below, to fix this but that’s for the future.
2 million blog posts would be stretching it. We (I) have plans in this area (partial builds, partitioned builds), but that is still just on the drawing bord. If you really want this now, I would say that it should maybe be possible to do a build of the 2 million pages as a separate job, and let the 500 a day be the “online edition”.
1 Like
Results from hugo --templateMetrics --templateMetricsHints:
hugo --templateMetrics --templateMetricsHints
Start building sites …
Template Metrics:
cache cumulative average maximum
potential duration duration duration count template
----- ---------- -------- -------- ----- --------
0 7h58m58.51751133s 11.610452ms 7.588211707s 2475228 integrations/single.html
21 1h21m27.226070735s 1.972352ms 5.855533317s 2477866 partials/head.html
99 54m2.266119751s 1.308491ms 1.742517426s 2477866 partials/header.html
5 46m28.641773308s 1.125421ms 198.506875ms 2477864 partials/banner.html
99 35m49.740860003s 867.577µs 1.734687634s 2477866 partials/menu.html
0 22m8.167537219s 536.011µs 202.303614ms 2477873 partials/env.html
0 8m31.420139717s 195.198526ms 699.098998ms 2620 _default/section.sitemap.xml
59 8m29.083824624s 194.306803ms 697.189395ms 2620 partials/sitemaps/single.html
57 8m26.393166986s 193.132405ms 695.226255ms 2622 partials/sitemaps/list.html
92 8m23.027388279s 203.01µs 31.277803ms 2477841 partials/breadcrumbs.html
0 5m43.000081416s 131.266774ms 3.847800752s 2613 integrations/list.html
0 4m12.317310662s 96.230858ms 1m16.189323609s 2622 _internal/_default/rss.xml
49 3m42.808500415s 22.503µs 31.1513ms 9900945 partials/figure.html
99 3m24.2578996s 82.432µs 5.855092535s 2477866 partials/translations/links.html
50 3m20.852180216s 40.529µs 31.321931ms 4955707 partials/resourceloader.html
99 1m5.111885745s 26.277µs 31.059189ms 2477866 partials/translations/toggle.html
65 51.502923459s 20.771µs 3.716404ms 2479529 partials/sitemaps/item.html
94 50.852583157s 18.892µs 19.856292ms 2691642 partials/button.html
0 50.40569669s 10.081139338s 10.081386481s 5 news/single.html
65 26.374284782s 41.229µs 1.316323ms 639694 partials/integration/box.html
51 11.058987497s 50.019µs 3.183775ms 221092 partials/integration/table.html
0 7.583766074s 7.583766074s 7.583766074s 1 not-found/single.html
0 6.766337195s 6.766337195s 6.766337195s 1 news/list.html
99 2.646177127s 12.227µs 19.519451ms 216420 partials/sectionTitle.html
100 2.062717412s 832ns 30.891471ms 2477866 partials/vendor/google/noscript.html
9 1.09345787s 655.55µs 10.90604ms 1668 partials/sitemaps/index.html
0 67.39519ms 9.627884ms 23.836028ms 7 pages/single.html
99 38.548001ms 14.752µs 1.13052ms 2613 partials/search.html
56 25.919658ms 5.183931ms 12.952503ms 5 partials/footer.html
0 23.42425ms 5.856062ms 10.231924ms 4 pages/list.html
0 23.119386ms 11.559693ms 19.579037ms 2 index.html
11 9.913422ms 901.22µs 3.378858ms 11 partials/pages/content.html
0 8.167887ms 8.167887ms 8.167887ms 1 platform/overview/single.html
0 7.876634ms 7.876634ms 7.876634ms 1 platform/security/single.html
100 7.094793ms 3.547396ms 6.504894ms 2 partials/box.html
0 6.930391ms 3.465195ms 6.773265ms 2 _default/home.sitemap.xml
0 6.126372ms 3.063186ms 3.301082ms 2 _default/list.html
23 4.019581ms 200.979µs 1.204636ms 20 partials/imageInfo.html
0 3.256374ms 3.256374ms 3.256374ms 1 platform/features/single.html
0 2.954773ms 984.924µs 2.441ms 3 shortcodes/partial.html
100 2.212983ms 2.212983ms 2.212983ms 1 partials/staff.html
0 2.204713ms 1.102356ms 2.128119ms 2 _default/sitemap.xml
0 1.231798ms 68.433µs 202.303µs 18 shortcodes/column.html
33 871.718µs 58.114µs 153.467µs 15 partials/list.html
100 565.358µs 94.226µs 343.971µs 6 partials/vendor/youtube.html
100 407.035µs 58.147µs 361.086µs 7 partials/vendor/google/tagmanager.html
0 402.973µs 402.973µs 402.973µs 1 _internal/_default/sitemapindex.xml
0 364.267µs 20.237µs 117.529µs 18 shortcodes/block.html
0 357.525µs 44.69µs 171.682µs 8 shortcodes/title.html
100 288.209µs 144.104µs 253.666µs 2 partials/svg/integration.html
100 271.332µs 271.332µs 271.332µs 1 partials/social.html
0 247.8µs 123.9µs 227.074µs 2 _internal/alias.html
100 244.575µs 244.575µs 244.575µs 1 partials/pager.html
100 196.863µs 196.863µs 196.863µs 1 partials/vendor/form.html
100 188.527µs 94.263µs 187.206µs 2 partials/svg/clients/acr.svg
100 182.709µs 91.354µs 181.553µs 2 partials/svg/clients/martvisser.svg
100 177.155µs 177.155µs 177.155µs 1 partials/svg/staff/Irene.html
100 166.291µs 83.145µs 164.781µs 2 partials/svg/clients/freshcotton.svg
100 164.224µs 82.112µs 162.681µs 2 partials/svg/clients/buddhatobuddha.svg
0 162.828µs 81.414µs 147.263µs 2 shortcodes/button.html
100 160.473µs 160.473µs 160.473µs 1 partials/svg/staff/Jaap.html
100 156.954µs 156.954µs 156.954µs 1 partials/svg/staff/Leon.html
100 156.47µs 156.47µs 156.47µs 1 partials/svg/staff/Koendert.html
100 147.198µs 147.198µs 147.198µs 1 partials/svg/staff/linda.html
100 147.141µs 147.141µs 147.141µs 1 partials/svg/staff/Niels.html
100 145.672µs 145.672µs 145.672µs 1 partials/svg/staff/Darianna.html
100 138.492µs 138.492µs 138.492µs 1 partials/svg/staff/Jelle.html
100 120.359µs 60.179µs 119.481µs 2 partials/vendor/addapplicationform.html
100 96.705µs 19.341µs 92.274µs 5 partials/sprites.html
100 89.058µs 17.811µs 85.267µs 5 partials/vendor/chat.html
0 21.358µs 21.358µs 21.358µs 1 js/functions.js
0 1.772µs 1.772µs 1.772µs 1 js/variables.js
0 665ns 665ns 665ns 1 js/index.js
Total in 2355897 ms
Error: Error building site: TOCSS: failed to transform "sass/main.scss" (text/x-scss). Check your Hugo installation; you need the extended version to build SCSS/SASS.
This suggests your site is failing because you don’t have Hugo extended. What do you get running hugo version?
Do you really have 2.5+ million distinct versions of your header, banner, menu, env, breadcrumbs, button, noscript, etc? The “cache potential” column suggests not. That’s got to contribute to your build time and memory usage, and careful use of partialCached should help quite a bit.
I’ve never gone much above 150,000 entries in testing, with a much older version of Hugo, but small changes in the theme made huge differences in build time.
-j
I know. Had some issues with that. Is resolved now: Building Hugo Extended for ARM64
Working on implementing some cache now. Will try a new run later today when that’s done.
The interesting thing is that after building the site, and then while running the deploy it will require 75-80GB of RAM again.
It keeps hovering in the 75-80GB range for the RAM. Not increasing or decreasing beyond that range.
Nizzlay:
hugo --print-mem
There are too many partial files, and you might want to reduce the partials into lesser partial files by combing if possible. This will speed up more.
So you need to see what partial to be worked on based on most used partial.
1 Like
Here is another thing I implemented and worked well.
In the partial cache, you can mention which are the params that changes at the page level.
This has improved the build time from almost 40 minutes to less than 10 minutes. It’s for 130k pages build I’m talking about.
Reference -
The Full Partial Series Part 1: Caching! | Regis Philibert (read the section with the heading Stepping variants up a notch)
Credit to @regis for writing up such an amazing content
You’re welcome.
3 Likes
Regis, We need more cool posts like that.