109,503 Post Cannot generate static html from .md files

1.88 GB 109,503 Post as .md in post folder.
How to make recursively All files generate static html?

I can Make 5000 post and works fine.But I need all post make generate static html.

if use hugo commend PC going to restart. Already make 100 Folder/files as html.
Started building sites …

Need solution for this.

What is the exact error you are getting?

No error 390 HTML files create & then Just reboot my PC.

I suspect that with that amount of posts you need a very beefy PC with lots of memory.

I Agree with that.Need more powerful PC.
My PC configuration
2xIntel xeon 2.40 ghz
4x72GB SCSI 15K

But I think here is recursively option need so If it take 24 hours no problem but need slow & steady.

Thank you

Not sure what you mean by recursively, but the problem is:

Hugo reads all pages into memory to be able to create a site. This works fine for most use cases, but you’re having problems. What you want is a “streaming option”, but that is currently not an option.

My best guess is that your main problem is memory, so if you can test with a 16GB memory, that would be a start.

yes After log view its take 98% use from memory.here is must be option like slow & steady without large memory can handle big site.Okay I m using my Server XEON 1271 v3 with 32GB RAM 240GB SSD Let see What Happen.I will post again.

I think he means “incremental” rather than “recursive”

Does Hugo need to load all pages into memory on every rebuild?.. or only those which have changed?

If the latter, then it should be possible to build the site “incrementally” by adding, say, a couple of thousand markdown files at a time into the content directory and rebuilding each time.

If the former, then I presume the user needs to setup his computer with a humongous /swap disc, so it doesn’t crash and burn when the RAM is exhausted.

All that said, 100 000+ posts does seem a bit “enthusiastic”. Maybe some ‘pruning’ might help, too.

1 Like

I think you’re mixing the terms here.

On live-reload we’re only reloading the stuff you change (to make it a bit simplistic), but every page must currently be in memory for us to be able to handle all your clever range Site.Pages.

So incremental != streaming.

I’m not saying we’re not getting smarting about this in the future, but the thread starter is in the present.

1 Like