This is definitely the largest site I’ve ever heard of for Hugo. We had an earlier issue of someone triggering a bug with 10k posts, which at the time was the largest site I’d heard of for Hugo. He found a race condition that we fixed. He told us he did a POC with a bunch of other SSGs and Hugo was the only one that could even finish in under an hour (hugo took about 4 seconds IIRC). Of all the site generators he wrote, only one of the other generators even finished.
600k is a lot of posts. I’m curious what the total size of just the content files is on disk. I think you will likely be pushing the limits of the IO on the disk to even read that number of files in 60 seconds.
I’m also curious how long it takes for you to just copy the content directory (time cp -r content content.test). That will at least give us a minimum benchmark of what the lower end could be.
These numbers don’t seem unreasonable, but do look like a bit longer than I would think based on smaller tests I’ve run.
Hugo will run faster if step analysis and verbosity aren’t enabled.
There are also a few things we could do to make it even faster for this large use case. This is the first real case I’ve seen where doing partial renders may be worth it.
Thanks for considering Hugo for this. I don’t know if we are up for the challenge, but I’m personally excited to have a motivator for us to improve Hugo even further.