How to minimize re-builds

Raymond_Camden · January 25, 2016, 10:53pm

Right now my Hugo site takes about 30 seconds to rebuild when I add/edit a post. I’m probably an extreme case as I’ve got over 5K blog entries. I can live with this but I was wondering if there is some way to minimize that time to make editing a bit quicker. So for example, my blog theme includes a count for categories and tags, and if I removed that, would Hugo then not bother updating existing entries? (I’ve got pagination disabled already.)

Any other tips?

bep · January 26, 2016, 12:46am

There is currently no partial rebuilds, so if you have turned off pagination, renders to memory (in server mode) – then your best bet is to make the templates do less. 30 seconds sounds on the long side for 5K posts, so I would watch out for page the loops, esp. in the single post template.

Raymond_Camden · January 26, 2016, 1:59am

What do you mean for pages with loops? I mean I get what a loop is obviously, but what about it would cause an issue?

bep · January 26, 2016, 8:25am

Some of this is optimized by Hugo (read cached), but if you for every page iterate over all the pages in the site to calculate something, that loop has a cost. It is pretty fast by itself, but times 5000 … And if you have many of these, maybe with some nested inner loops …

Raymond_Camden · January 26, 2016, 1:59pm

This makes sense, but would you be able to share a real example of this? Again - I know loops of course, but I’m just wondering if you can provide a practical example of what you mean. I certainly do not expect you to look at my code, but if you’re really bored (grin), I do have my site up on GitHub - https://github.com/cfjedimaster/raymondcamdendotcom. Again, I’m definitely not asking you to ‘fix’ my site. People ask me that all the time and it’s frustrating, but any real examples would be appreciated, and even if not, thank you for the help so far.

moorereason · January 26, 2016, 3:13pm

The first step in troubleshooting build times is to use the --stepAnalysis option. What does that give you?

Raymond_Camden · January 26, 2016, 3:48pm

Here is the raw data - and this is fascinating stuff:
`initialize & template prep:
3.6740017s (3.674588082s) 1596.40 MB 136328 Allocs
load data:
917.355µs (3.675620826s) 0.04 MB 691 Allocs
import pages:
1.033998991s (4.710207853s) 542.34 MB 2909795 Allocs
build taxonomies:
107.570974ms (4.818555082s) 10.54 MB 419375 Allocs
render and write aliases:
124µs (4.819480082s) 0.00 MB 0 Allocs
render and write taxonomies:
535.232749ms (5.355602628s) 243.89 MB 3726065 Allocs
render & write taxonomy lists:
22.655907ms (5.37922171s) 4.01 MB 85346 Allocs
render and write lists:
549.206467ms (5.929631906s) 150.17 MB 2076429 Allocs
render and write pages:
21.443298036s (27.374811241s) 12389.35 MB 257831415 Allocs
render and write homepage:
39.720038ms (27.416737626s) 7.19 MB 116478 Allocs
render and write Sitemap:
289.472078ms (27.708068543s) 53.59 MB 1086818 Allocs

bep · January 26, 2016, 5:26pm

I haven’t looked too closely, but if this

github.com

cfjedimaster/raymondcamdendotcom/blob/master/layouts/partials/article_list.html#L6


<section id="main">

{{ $paginator := .Paginate ((first 50 (where .Pages "Type" "post"))) }}

{{ range $paginator.Pages }}

<article class="article article-type-post" itemscope="" itemprop="blogPost">

    <div class="article-inner">

        {{ if and (isset .Params "banner") (not (eq .Params.banner "")) }}

        <a href="{{ .Permalink }}" itemprop="url">

            <img src="https://static.raymondcamden.com{{ .Params.banner }}" class="article-banner" title="{{ .Title }}"> {{ end }}

        </a>



        {{ partial "article_header" . }}

        <div class="article-entry" itemprop="articleBody">

            <p>

                {{ printf "%s" .Summary | markdownify }}

                <br>

            </p>

Is included in every page … Then I would bet on this being the consumer of time.

In the perfect world, this would be evaluated once and then reused, but it isn’t.

Raymond_Camden · January 26, 2016, 5:47pm

Isn’t it only used by the home page?

bep · January 26, 2016, 5:59pm

Sure, but you wanted an example.

I just tested your site.

Changing the theme to hyde-x reduced the build time from 90 s => 7 seconds.

So that is at least a work around for article writing. Find a more compact theme.

bep · January 26, 2016, 6:10pm

Have a look in the partials profile and sidebar, esp. the profile.

Raymond_Camden · January 26, 2016, 6:41pm

So dumb q here - are you suggesting I literally just use a different them when working locally, but build to the theme I use for production?

bep · January 26, 2016, 8:00pm

If you really like that theme I would use something like this for “writing articles”:

hugo server -t <some-other-theme>

Also, if you control the theme, and we had a “dev flag” (I use the buildDraft flag for this), you could put a conditional around the expensive parts: Don’t render when in dev mode.

But the constructs in the theme you use isn’t unusual, so someone should do something about it. I think there is a GitHub issue about it.

Raymond_Camden · January 26, 2016, 8:22pm

Oh, the dev flag sounds like a great idea actually. And hell, I’ve already got my own mods to the theme already so I wouldn’t be cluttering up the ‘real’ theme if that makes sense. I’ll give that a shot.

And I just did. By using a conditional in sidebar and profile, I’m now down to about 4.5 seconds locally. Awesome! Thanks for the tips!

Raymond_Camden · January 26, 2016, 8:28pm

Btw, is there already an ER for something like, “render this partial once and never rerun it - use the output from last time”? That would have solved my problem too, and seems like it would be helpful for a lot of folks.

bep · January 26, 2016, 9:19pm

You cannot do it as a general rule, but you’re not the first one to think about it:

github.com/gohugoio/hugo

Add PartialCached variant

opened 08:11AM - 21 Aug 15 UTC

closed 09:56PM - 11 Oct 16 UTC

bep

Enhancement Outdated

See http://discuss.gohugo.io/t/how-to-make-hugo-faster/1597/6 ``` {{ range $.S…ite.Pages.ByTitle }} {{ if and (in .Params.categories "Category") (in .Params.categories "sub-category") }} <li><a href="{{ .Permalink }}">{{ .Title }}</a> {{ if .Draft }} * {{ end }}</li> {{ end }} {{ end }} ``` The above is an extract of a partial used on every page. It is an ineffective construct (slightly better in 0.15, I guess) and it gets really bad when page count goes up. But it's static. And, although it is impossible to make a general assumption about how "cacheable" a given partial is (I think?), the user will know -- and should be able to say so. So, for a site with 2000 pages, the above should only be needed to be processed once. @tatsushid not sure how Ace include works in this area?

It shouldn’t be too hard to implement … Contributions welcome!

Raymond_Camden · January 26, 2016, 9:38pm

Heh, I’m barely scratching the surface of Hugo now - not sure I could add something like this.

Any way, thank you again for your, and @moorereason 's help here. I greatly appreciate it.

Topic		Replies	Views
[SOLVED] How to decrease build time? 25103 ms is a lot of time support	6	1519	January 5, 2018
My site build by Hugo have about 1600 posts, hugo server very slow support	6	830	February 25, 2020
Bug? Feature? Parameter I don't know yet? support	4	459	July 1, 2019
Build Performance support	5	936	May 26, 2021
Building a site with 50,000 pages taking a very long time support	66	1774	February 24, 2023

How to minimize re-builds

Related topics