I need help with 137,240 .md files site

support
1

i just built a site with 137,240 files, but I can’t build it locally or on Netlify because its too large

this is what I get on Netlify
10:32:40 PM: $ hugo --gc --minify 10:32:40 PM: Start building sites … 10:32:40 PM: hugo v0.133.1-47d00202e7e61769ce4d14691e43b27852c9cce4+extended linux/amd64 BuildDate=2024-08-26T13:58:46Z VendorInfo=gohugoio 10:33:47 PM: Failed during stage 'building site': Build script returned non-zero exit code: 2 (https://ntl.fyi/exit-code-2) 10:33:47 PM: ​ 10:33:47 PM: "build.command" failed 10:33:47 PM: ──────────────────────────────────────────────────────────────── 10:33:47 PM: ​ 10:33:47 PM: Error message 10:33:47 PM: Command was killed with SIGKILL (Forced termination): hugo --gc --minify 10:33:47 PM: ​ 10:33:47 PM: Error location 10:33:47 PM: In build.command from netlify.toml: 10:33:47 PM: hugo --gc --minify 10:33:47 PM: ​

and on my local, I get this

`` Start building sites …
hugo v0.133.1+extended darwin/arm64 BuildDate=2024-08-26T13:58:46Z VendorInfo=brew

Built in 2 ms
Error: error building site: failed to acquire a build lock: open /Users/Mrguy/Desktop/staticsites/projectname/.hugo_build.lock: too many open files``

2

You might not want this, but look into Netlify CLI. I build my site locally and then just run netlify deploy --prod --open to get the contents of public into my Netlify site. Added advantage: with the free account you don’t run into build minute issues because building times locally are not counted.

Regarding your 137k posts try finding out, why it runs long. Some points I would look into:

  • Image creation is time intense. Minimize “redoing” it by adding the resources folder to your Git repository. Check if you really need all the processing you are doing (for instance, do resizing and cropping simultaneously, not successive).
  • Run your server with
    You might not want this, but look into Netlify CLI. I build my site locally and then just run netlify deploy --prod --open to get the contents of public into my Netlify site. Added advantage: with the free account you don’t run into build minute issues because building times locally are not counted.

Regarding your 137k posts try finding out, why it runs long. Some points I would look into:

  • Image creation is time intense. Minimize “redoing” it by adding the resources folder to your Git repository. Check if you really need all the processing you are doing (for instance, do resizing and cropping simultaneously, not successive).
  • Run your server with --templateMetrics (display metrics about template executions) and --templateMetricsHints (calculate some improvement hints when combined with --templateMetrics) and look for partials that you can cache.
  • Get a FULL log and look for the timestamps. Where does it hang. With warnf you can for instance echo a note every time a template is run (keep the messages unique by adding a counter) and then find out in what post you experience longer times.
  • Maybe run hugo without --gc. that should do no harm in “normal” circumstances.
  • Remove netlify plugins you might use (image optimization, code checks etc) and see if it’s going faster.

Other than that… Post a full log file output (Netlify has a button on top for that) so we can see if there is anything BEFORE the script gets killed.

I would build the site locally and then upload it with the Netlify CLI.