Hello, I’m building a site with 50k pages, and Hugo is taking a very long time to build.
I’ve remove pagination from that section’s template, so it doesn’t have to create pagination pages for 50k pages.
I’m not sure what else to do or optimize. Please advise.
If you’re building your site within an IDE (e.g, Microsoft VS Code, etc.), the built-in file watcher can have am impact on build time.
If that’s the case, terminate your IDE and build the site from a terminal.
And if you haven’t done so arleady, look at the template metrics:
hugo --templateMetrics --templateMetricsHints
$ hugo -b=“https://www.psychedelicsdaily.com/” --minify --ignoreCache --printMemoryUsage --log
Start building sites …
hugo v0.110.0-e32a493b7826d02763c3b79623952e625402b168+extended linux/amd64 BuildDate=2023-01-17T12:16:09Z VendorInfo=snap:0.110.0
Soon as this build finishes, I’ll run template metrics as well. I don’t use IDEs. I use just a plain text editor and CLI.
| EN
-------------------+--------
Pages | 12403
Paginator pages | 11
Non-page files | 28
Static files | 2451
Processed images | 0
Aliases | 11929
Sitemaps | 1
Cleaned | 0
Total in 599511 ms
Took about 10 minutes. Seems a bit too long? Or is this normal?
I didn’t give it the full 50k articles, only a few thousand so far. 50K would take many times more. Probably an hour or more.
I’m also on BTRFS, a CoW/RoW file system, do you think that could be adding some time? I’ll setup a small XFS partition, move hugo data there and try to rebuild as well and test. I’ll post later.
600 s / 12,403 pages = 0.048 s/page
By comparison, I just built this dirt simple site, with pagination but no partial/shortcode calls:
Pages | 10461
Paginator pages | 1035
Non-page files | 0
Static files | 0
Processed images | 0
Aliases | 10
Sitemaps | 0
Cleaned | 0
Total in 6496 ms
That’s about 0.0006 s/page.
So, you’ll need to look at the template metrics to find the pain points.
$ hugo -b="https://www.psychedelicsdaily.com/" --minify --ignoreCache --templateMetrics --templateMetricsHints --verbose
Start building sites …
hugo v0.110.0-e32a493b7826d02763c3b79623952e625402b168+extended linux/amd64 BuildDate=2023-01-17T12:16:09Z VendorInfo=snap:0.110.0
INFO 2023/02/18 23:53:54 syncing static files to /
INFO 2023/02/19 00:11:35 postcss: use config file /home/stoned/websites/websites/psychedelicsdaily.com-hugo/postcss.config.js
INFO 2023/02/19 00:11:36 postcss: Browserslist: caniuse-lite is outdated. Please run:
npx browserslist@latest --update-db
Template Metrics:
cumulative average maximum cache percent cached total
duration duration duration potential cached count count template
---------- -------- -------- --------- ------- ------ ----- --------
4h39m40.333684011s 1.347276891s 1.662284908s 0 0 0 12455 _default/single.html
3h34m14.886141086s 1.002564821s 1.519628385s 60 0 0 12822 partials/sidebar.html
2h1m10.111010185s 567.002886ms 856.425911ms 0 0 0 12822 partials/widgets/related.html
10m23.531545993s 48.629819ms 647.716076ms 100 0 0 12822 partials/footer.html
9m51.071875508s 46.098258ms 190.26068ms 100 0 0 12822 partials/widgets/authors.html
9m1.499662887s 42.232074ms 164.908306ms 26 0 0 12822 partials/widgets/random.html
5m7.239408003s 24.667957ms 97.053703ms 32 0 0 12455 partials/related.html
4m36.713711062s 21.581166ms 105.59464ms 35 0 0 12822 partials/widgets/featured.html
4m35.412530006s 21.47801ms 147.273781ms 94 0 0 12823 partials/widgets/recent.html
3m41.957467871s 17.310674ms 85.310124ms 20 0 0 12822 partials/widgets/faq.html
1m3.977757364s 420.906298ms 544.379862ms 0 0 0 152 drugs/single.html
32.09807975s 324.223027ms 551.752298ms 0 0 0 99 section/community.html
29.176396708s 324.182185ms 403.438057ms 0 0 0 90 _default/list.html
21.110096465s 1.646268ms 151.550036ms 8 0 0 12823 partials/head.html
20.333405495s 1.585821ms 80.13156ms 100 0 0 12822 partials/widgets/tags.html
19.016503316s 1.484156ms 32.09668ms 100 0 0 12813 partials/tagcloud.html
18.33330752s 1.42972ms 151.906147ms 11 0 0 12823 partials/scripts.html
9.434472845s 735.803µs 25.084561ms 100 0 0 12822 partials/widgets/categories.html
5.014390871s 28.490857ms 150.521266ms 0 0 0 176 shortcodes/related.html
3.408320197s 265.797µs 19.295233ms 22 0 0 12823 partials/nav.html
3.371227925s 374.58088ms 422.322624ms 0 0 0 9 page/single.html
2.997104403s 237.733µs 31.404728ms 28 0 0 12607 partials/most_read.html
2.978666335s 330.962926ms 375.058725ms 0 0 0 9 section/psychedelics.html
2.064085183s 516.021295ms 545.921018ms 0 0 0 4 section/blog.html
1.424153206s 111.157µs 20.397728ms 98 0 0 12812 partials/author.html
1.085866521s 5.271196ms 240.888408ms 96 0 0 206 partials/carousel_featured.html
903.828556ms 4.635018ms 200.757266ms 0 0 0 195 _internal/_default/rss.xml
865.643929ms 57.346µs 6.484672ms 92 0 0 15095 partials/share.html
827.470349ms 64.53µs 12.642098ms 8 0 0 12823 partials/seo/schema.html
704.898971ms 352.449485ms 373.065391ms 0 0 0 2 section/drugs.html
616.94848ms 616.94848ms 616.94848ms 0 0 0 1 index.html
540.012297ms 77.144613ms 115.844288ms 0 0 0 7 shortcodes/list_tags.html
503.828495ms 39.291µs 14.46254ms 10 0 0 12823 partials/seo/twitter.html
417.736785ms 417.736785ms 417.736785ms 0 0 0 1 section/news.html
372.477048ms 29.047µs 6.03466ms 100 0 0 12823 partials/seo/google_analytics.html
230.850853ms 18.004µs 15.90247ms 100 0 0 12822 partials/widgets/search.html
199.613684ms 15.924µs 2.590154ms 0 0 0 12535 _internal/alias.html
177.263762ms 177.263762ms 177.263762ms 0 0 0 1 _default/index.json
161.150165ms 161.150165ms 161.150165ms 0 0 0 1 _internal/_default/sitemap.xml
149.053666ms 842.111µs 28.463226ms 0 0 0 177 shortcodes/signup.html
135.941623ms 10.603µs 928.116µs 52 0 0 12821 partials/breadcrumbs.html
120.664285ms 612.509µs 15.073213ms 100 0 0 197 partials/drugs.html
100.417936ms 7.831µs 3.42903ms 100 0 0 12822 partials/widgets/signup.html
83.801527ms 6.535µs 443.204µs 100 0 0 12822 partials/widgets/facebook.html
51.330466ms 51.330466ms 51.330466ms 100 0 0 1 partials/sections_list.html
46.139547ms 3.598µs 1.490435ms 100 0 0 12822 partials/widgets/phplist.html
44.677938ms 3.484µs 703.481µs 100 0 0 12822 partials/widgets/maps_news.html
33.631752ms 57.49µs 2.271085ms 15 0 0 585 partials/recent.base.html
33.509695ms 73.647µs 574.629µs 0 0 0 455 shortcodes/notice.html
22.625575ms 22.625575ms 22.625575ms 100 0 0 1 partials/recent_drugs.html
20.868221ms 20.868221ms 20.868221ms 100 0 0 1 partials/recent_faq.html
20.031493ms 1.562µs 142.032µs 100 0 0 12822 partials/site_social.html
18.351433ms 65.076µs 2.342322ms 0 0 0 282 shortcodes/imgcap.html
17.947747ms 17.947747ms 17.947747ms 100 0 0 1 partials/recent_psychedelics.html
17.822127ms 17.822127ms 17.822127ms 100 0 0 1 partials/recent_blog.html
4.221933ms 168.877µs 696.654µs 0 0 0 25 shortcodes/blockquote.html
1.768866ms 442.216µs 1.336458ms 0 0 0 4 shortcodes/radio.html
1.438877ms 1.438877ms 1.438877ms 0 0 0 1 404.html
555.11µs 555.11µs 555.11µs 100 0 0 1 partials/team.html
494.945µs 494.945µs 494.945µs 100 0 0 1 partials/testimonials.html
484.525µs 484.525µs 484.525µs 100 0 0 1 partials/features.html
432.656µs 432.656µs 432.656µs 100 0 0 1 partials/clients.html
263.813µs 263.813µs 263.813µs 0 0 0 1 _internal/shortcodes/youtube.html
249.676µs 249.676µs 249.676µs 100 0 0 1 partials/signup.html
237.652µs 1.342µs 101.034µs 100 0 0 177 partials/mailchimp.html
202.968µs 202.968µs 202.968µs 100 0 0 1 partials/welcome.html
197.717µs 197.717µs 197.717µs 100 0 0 1 partials/hero.html
168.994µs 168.994µs 168.994µs 0 0 0 1 shortcodes/aboutimg.html
161.077µs 161.077µs 161.077µs 0 0 0 1 shortcodes/calc.html
3.056µs 3.056µs 3.056µs 0 0 0 1 robots.txt
Error: Error building site: open /home/stoned/websites/websites/psychedelicsdaily.com-hugo/public/mission/index.html: no such file or directory
Total in 1063379 ms
So now I have to try to make sense out of this.
What exactly would I be looking to optimize in these files? Or how would I find out why they are costing so much time?
417.736785ms 417.736785ms 417.736785ms 0 0 0 1 section/news.html
This template should be used for the news section, but default/single.html is being used 12k times meaning it’s being used for all the news pages instead of the template I created.
I made a news.html in /layouts/section/news.html for that section, but doesn’t seem to be used?
For the items with a cache potential of 100, consider using partialCached.
partials/sidebar.html → 3h34m14s → ouch
<div id="sidebar">
{{ partial "widgets/search.html" . }}
<!--{{ partial "widgets/signup.html" . }}-->
<!--{{ partial "widgets/phplist.html" . }}-->
{{ if .Site.Home }}
{{ partial "widgets/faq.html" . }}
{{ partial "widgets/random.html" . }}
{{end}}
{{ partial "widgets/featured.html" . }}
{{ partial "widgets/authors.html" . }}
{{ partial "widgets/categories.html" . }}
{{ partial "widgets/tags.html" . }}
{{ partial "widgets/facebook.html" .}}
<!--{{ partial "widgets/maps_news.html" . }}-->
{{ partial "widgets/recent.html" . }} <!--Oldest-->
<!--{{ partial "widgets/related.html" . }}-->
</div>
Sidebar is just loading a few widgets, you can see on psychedelicsdaily.com shouldn’t cost that much, but why? I’m stumped.
I mean it does have to draw the entire sidebar on every page. I guess I don’t need it on the news page.
Which of those partial calls can you cache, either globally or per section? The partialCached function allows you to fine tune the caching strategy.
After you’ve done that, use bisection to zero-in on the pain.
Also, it seems like you’re generating a very large number of aliases. I have no idea what impact that has on performance, but it caught my attention.
Where are you seeing this?
Also, I just noticed in your sidebar that you tried to comment out the related widget.
<!--{{ partial "widgets/related.html" . }}-->
That doesn’t work.
https://gohugo.io/templates/introduction/#html-comments-containing-go-templates
HTML comments are by default stripped, but their content is still evaluated.
And the related widget …
2h1m10s --> partials/widgets/related.html --> ouch (again)
To comment it out:
{{/* {{ partial "foo.html" }} */}}
Ohh all my stuff I thought I commented out was still being processed, oh ok. I’ll use the /**/ comments instead. Thanks.
I also suspect that your “related” widget is inefficient. That’s irrelevant if you don’t want to display it, but I’d be interested in seeing the code.
{{ if .Site.Params.widgets.related }}
<div data-pagefind-ignore class="panel panel-default sidebar-menu shadow bg4">
<div class="panel-heading">
<p class="h4 panel-title"> Related Articles</p>
</div>
<div class="sidebar-recent panel-body">
{{ $page_link := .Permalink }}
{{ $tags := .Params.tags }}
{{ range .Site.RegularPages }}
{{ $page := . }}
{{ $has_common_tags := intersect $tags .Params.tags | len | lt 0 }}
{{ if and $has_common_tags (ne $page_link $page.Permalink) }}
<div class="item same-height-row clearfix">
<div class="name">
<h4><a href="{{ .RelPermalink }}">{{ .Title }}</a></h4>
</div>
<div class="image">
<a href="{{ .RelPermalink }}">
{{ if isset .Params "banner" }}
<img src="{{ .Site.BaseURL}}img/placeholder.png" data-src="{{ .Params.banner }}" class="img-responsive shadow" alt="{{.Title}}" width="255" height="192" title="{{.Title}}" loading="lazy"/>
{{ else }}
<img src="{{ .Site.BaseURL}}img/placeholder.png" class="img-responsive" alt="{{.Title}}" width="255" height="192" title="{{.Title}}" loading="lazy" />
{{ end }}
</a>
</div>
</div>
{{ end }}
{{ end }}
</div>
<hr class="hidden-md hidden-lg">
</div>
{{ end }}
Here’s the related widget. I’m trying to figure out what I did.
The related widget was commented out html style, so not intended to be displayed, but still costing time. Also, why is it ineffecient?
Yeah, that’s not good. At all.
Use this instead:
https://gohugo.io/content-management/related/
You’re ranging through every page, on every page, and using an intersect call which can’t be cheap.
<div data-pagefind-ignore class="related">
{{ $related := .Site.RegularPages.Related . | first 16 | shuffle }}
{{ with $related }}
<div><p class="h1">Related Articles</p></div>
<div class="row">
{{ range . }}
{{ if ne .Section "" }}
<!--<div class="col-md-2">
<div class="box-image-text">
<div class="top shadow">
<div class="image" style="overflow:hidden">
<a href="{{ .RelPermalink }}">
{{ if isset .Params "banner" }}
<img src="{{ .Site.BaseURL}}img/placeholder.png" data-src="{{ .Site.BaseURL}}{{ .Params.banner }}" class="img-responsive" alt="{{.Title}}" title="{{.Title}}" style="max-width: 100px; max-height: 50px;">
{{ else }}
<img src="{{ .Site.BaseURL}}img/placeholder.png" class="img-responsive" alt="{{.Title}}" width="255" height="192" title="{{.Title}}">
{{ end }}
</a>
</div>
<a href="{{ .RelPermalink }}"><h4>{{ .Title }}<h4></a>
</div>
</div>
</div>-->
<ul>
<li><a href="{{ .RelPermalink }}">{{ .Title }}</a></li>
</ul>
{{ end }}
{{ end }}
</div>
{{ end }}
</div>
Yes sir. I use the new method for related.
Here’s the new related widget I use. I never realized that HTML commenting doens’t stop Hugo from processing hugo code.
So far I’ve removed the related ineffecient widget that I wrote before Hugo had the related feature. So that’s off now.
Here’s the code for single.html, which I think was costing because of related. Let’s see.
{{ $baseurl := .Site.BaseURL }}
{{ $image := .Params.images}}
<!DOCTYPE html>
<html lang="en">
{{ partial "head.html" . }}
<body>
<div id="all">
<header>
{{ partialCached "nav.html" . }}
</header>
<div class="all-after-nav">
{{ partialCached "breadcrumbs.html" . }}
<article>
{{ if .Params.search_ignore }}
<div id="content" data-pagefind-ignore>
{{ else }}
<div id="content">
{{ end }}
<div class="container">
<div class="row">
<!-- *** LEFT COLUMN *** -->
<div class="col-md-8" id="blog-post">
<div class="alert-danger" style="padding: 5px 10px;">
<p>
<a href="/disclaimer">DISCLAIMER: Click here to read the disclaimer.</a>
</p>
<p>You agree to this disclaimer before using this website or any information contained within. If you do not agree with this disclaimer, you may not use this website.</p>
</div>
<div id="article-top"></div>
<div class="text-muted text-uppercase mb-small text-left">
<div class="article-meta article-meta-author">
{{ if .Params.author }} {{$author := index .Site.Params.authors (.Params.author)}}
<a class="btn btn-danger" href="{{.Site.BaseURL}}authors/{{ $author.name | urlize }}/">
<!---->
By {{$author.display_name}}
</a>
</i>
{{ end }}
<span class="btn btn-success">
{{ .Date.Format .Site.Params.date_format }}
</span> {{ range $index, $tag := .Params.tags }}
<a class="btn btn-warning" href="{{$baseurl}}tags/{{ $tag | urlize }}/">
<!---->#
{{ $tag }}
</a> {{ end }}
</div>
<div class="article-meta article-meta-share">
{{ partialCached "share.html" .}}
</div>
</div>
<div class="author-info">
</div>
<div id="post-content">
<div>
</div>
<div class="article-toc">
<!--<h2 class="text-primary">{{.Title}}</h2>-->
<!--<p>{{.Description}}</p>-->
<div id="ToC">
<p class="h1">Table of Contents</p>
{{.TableOfContents}}
</div>
</div>
<div class="post-content" style="margin-top: 60px;">
{{ with $.Params.images }}
<!--<img src="{{ index . 0 | absURL }}" alt="" class="img-responsive" style="max-height: 420px;" loading="lazy"/>-->
{{ end }}
{{ .Content }}
</div>
{{ partial "related.html" . }}
</div>
<div id="article-author-info" style="margin-top: 60px;">
{{ partialCached "author.html" .}}
</div>
<div style="margin-top: 60px;">
{{ partialCached "tagcloud.html" . }}
<div style="margin: 40px 0;">
{{ partialCachedCached "most_read.html" . }}
</div>
<!-- /#post-content -->
<div id="comments">
{{ if .Site.DisqusShortname }} {{ partial "disqus.html" . }} {{ end }} {{ if .Site.Params.comments.fb }} {{ partialCached "facebook_comments.html" . }} {{ end }}
</div>
</div>
</div>
<div class="col-md-4">
<!-- *** MENUS AND WIDGETS *** -->
{{ partial "sidebar.html" . }}
<!-- *** MENUS AND FILTERS END *** -->
</div>
</div>
</div>
</div>
</article>
{{ partialCached "footer.html" . }}
</div>
{{ partialCached "scripts.html" . }}
</body>
</html>
Yup, just realized that. I gotta go around changing all the HTML comments to hugo comments. Should help.