Hello,
Is there any possibility to have an incremental build for Hugo?
bep
July 25, 2022, 10:27am
2
What do you mean by “incremental build”?
Hello @bep ,
I was referring to a build that generates only the subset of HTML files that needed to be updated based on the changes made content folder (md/html files).
In my new project, I’m going to have 300k (this number going to increase) pages, and I need to update around 100K pages data every week, so I wanted to have an incremental build, which will only update these 100K pages without modifying other 200K pages.
So do we have any possibility of this?
bep
July 25, 2022, 10:59am
4
I have droddled a little around this over here:
opened 07:27AM - 19 Jul 22 UTC
Enhancement
I'm doing some ground work now making Hugo much more effective for big data/page… sets. One of the nice effects I see is that it takes Hugo much less time getting ready to render. A Hugo build basically looks like this:
1. Read and index data.
2. Render pages.
The flexibility of the Hugo API means that the second step always needs a full data set, even in a "partial render" situation. But the "render everything" comes at a cost (writing lots of file to disk etc.), which is especially true when we now may talk about "a million pages" and more.
There are a couple of situations where rendering only one or more segments of a site might come in handy:
1. When rendering everything in one go is too heavy for the server (but note that the upcoming Hugo should be much more efficient in this department).
2. When building at intervals and you have a segmented site with very different change frequency (e.g. "render the home page and the news section only").
You could argue that Hugo should be smart about "what's changed" and render just that (which is what we're doing in server mode), and you're right, but even then it's much simpler to determine _which segments have changed_.
I thought we could introduce a new concept of _named segments_ in the Site config (which we may also consider using for other things):
```toml
[segments]
[segments.hot]
[[segments.hot.matchers]]
# Need to also render the home page etc.
kind = '{home,term,taxonomy}'
[[segments.hot.matchers]]
path = '/news/**'
[segments.docs]
[[segments.docs.matchers]]
path = '/docs/**'
[segments.old]
[[segments.old.matchers]]
path = '/blog/**'
```
And then when building:
```
hugo --renderSegments hot,docs
```
I can add that the upcoming Hugo version will be much more effective with larger data sets (and with much I mean … much).
3 Likes
Hello @bep ,
That’s great, thank you so much for the quick reply.