Building a site with 50,000 pages taking a very long time

Then the next most costly item is footer.html, I wonder why, I’ll look through that too but could use another pair of eyes:

<footer data-pagefind-ignore id="footer" class="footer bg4 shadow">
    <div class="container">
		<div class="row">
			{{ if isset .Site.Params "about_us" }}
			<div class="col-md-2 col-sm-6">
				<div class="h3">Psychedelics</div>
					<ul class="">
					{{ range .Site.Menus.footer }}
					<li class="" style="margin: 10px 0">
						<a href="{{ .URL }}">{{ .Pre }}{{ .Name }}</a>
					</li>
					{{ end }}
					</ul>
			</div>
			<!-- /.col-md-3 -->
			{{ end }}
			<div class="col-md-3 col-sm-6">
				{{ if isset .Site.Params "recent_posts" }}
					{{ if .Site.Params.recent_posts.enable }}
					<div class="h3">{{ i18n "recentPosts" }}</div>
					<div class="blog-entries">
						{{ range first 6 (where $.Site.RegularPages "Section" "!=" "") }}
						<div class="item same-height-row clearfix">
							<div class="image same-height-always">
								<a href="{{ .Permalink }}">
								{{ if isset .Params "banner" }}
									<img src="{{ .Site.BaseURL}}img/placeholder.png" data-src="{{.Site.BaseURL}}{{ .Params.banner }}" class="img-responsive" alt="{{.Title}}" title="{{.Title}}" width="255" height="192" loading="lazy" />
								{{ else }}
									<img src="{{ .Site.BaseURL}}img/placeholder.png" class="img-responsive" alt="{{.Title}}" title="{{.Title}}" width="255" height="192"  loading="lazy"/>
								{{ end }}
								</a>
							</div>
							<div class="name same-height-always">
								<p style=""><a href="{{ .Permalink }}">{{ .Title }}</a></p>
							</div>
						</div>
						{{ end }}
					</div>
					<hr class="hidden-md hidden-lg">
					{{ end }}
				{{ end }}
			</div>
			<!-- /.col-md-3 -->

			<div class="col-md-3 col-sm-6">
				<div class="h3">About Hash B.</div>
				<div style="">{{ .Site.Params.about_us | safeHTML }}</div>
				<hr class="hidden-md hidden-lg hidden-sm">
			</div>

			{{ if isset .Site.Params "address" }}
			<div class="col-md-3 col-sm-6">
				<div class="h3">{{ i18n "contactTitle" }}</div>
				<p> Message Hash Borgir:</p>
				<ul>
					<li>If you want to write for PsychedelicsDaily</li>
					<li>For any questions related to psychedelics & Microdosing</li>
					<li>To book Hash for lectures, talks, or Trip Sitting.</li>
					<li>Send your comments, questions, or concerns.</li>
					<li>Or if you just want to say hi :)</li>
				</ul>

<!-- 				{{ .Site.Params.address | safeHTML }} -->

				<a href="/contact" class="btn btn-small btn-template-main shadow">Contact Hash</a>
				<a class='read-more btn btn-small btn-template-main shadow' href='/about/'>About Us</a>
				{{ partial "site_social.html" . }}
				<hr class="hidden-md hidden-lg hidden-sm">
			</div>
			<!-- /.col-md-3 -->
			{{ end }}
		</div>
    </div>
    <!-- /.container -->
    <div style="padding: 12px;">
    <p class="" style="text-align: center;">I recently started to display some links to relevant products on amazon in order to try and cover server and hosting costs. I do sincerely apologize about the advertising and we hope that you understand.</p>
	<p class="center">Design, development, server administration, research and writing powered by potent psychedelics.</p>

</div>
</footer>
<!-- /#footer -->
<!-- *** FOOTER END *** -->

<!-- *** COPYRIGHT ***-->

<div id="copyright">
    <div class="container">
        <div class="col-md-12">
            {{ if isset .Site.Params "copyright" }}
            <p class="pull-left">{{ .Site.Params.copyright | safeHTML }}
            <a class="w3c" href="https://jigsaw.w3.org/css-validator/check/referer">
				<span class="w3">
					W3C
				</span>
				Valid CSS 3
			</a>
			<a class="w3c" href="https://validator.w3.org/nu/?doc=https%3A%2F%2Fwww.psychedelicsdaily.com%2F">
				<span class="w3">
					W3C
				</span>
				Valid HTML5
			</a>
            </p>
            {{ end }}
            <a class="subscribe-button icon-feed" href='/index.xml'>RSS Feed</a>
            <p class="pull-right">
              Design By: <a href="https://stoned.io">Hash C. Borgir</a>
            </p>

        </div>
    </div>
</div>
<!-- /#copyright -->
<!-- *** COPYRIGHT END *** -->

Look for other suspects where you might ranging through every page on every page.

50,000 x 50,000 = 2.5 Billion

You’re doing this 50,000 times:

{{ range first 6 (where $.Site.RegularPages "Section" "!=" "") }}

I’m not sure I follow the last bit. How would I display the last 6 recent articles in footer without costing it 50k times? Cache the footer? Or…?

Or give a slice of sections so it doesn’t have to do a check condition?

Yes, or just this portion of it. You should be looking for caching possibilities everywhere.

Think about building a house. Are you going to walk back out to the truck every time you need a nail?

I gotcha. How would I cache just this iteration in footer.html? I’ve partialCache’d the footer everywhere. Every dynamic module/wiget (not many) left alone and caching now every static object that doesn’t need to change form page to page.

Here’s the next most costly item: author.html

{{ if .Site.Params.widgets.authors }}
{{ $baseurl := .Site.BaseURL }}

<div data-pagefind-ignore class="panel sidebar-menu bg4">
    <div class="panel-heading">
      <p class="h4 panel-title"> {{ i18n "authors" }}</p>
    </div>

    <div class="panel-body">
        <ul class="authors-list nav nav-pills nav-stacked sidebar-authors">
            {{ range $name, $items := $.Site.Params.authors }}
                {{ range $key, $val := $items }}
                    {{ if eq $key "name" }}


                        <li>
                        <a href="{{ $baseurl }}authors/{{ $val | urlize | lower }}">

                          🧔 {{ $items.display_name }}

                        ({{ len (where $.Site.Pages ".Params.author" $name) }})
                        </a>
                        </li>

                    {{ end }}
                {{ end }}
            {{ end }}
        </ul>
    </div>
</div>
{{ end }}

And the one after that most costly, random.html

{{ if .Site.Params.widgets.featured }}
<div class="panel panel-default sidebar-menu bg4">

<div data-pagefind-ignore class="panel-heading">
            <p class="h4 panel-title"> You might also like</p>
</div>
            <div class="sidebar-featured">
                {{ range first 24 (where .Site.RegularPages "Section" "not in" (slice "faq" "community")) | shuffle }}
                <div class="item same-height-row clearfix">
                    {{ $slice := slice "🍄" "🌿" "🌱" "🌵" "🧘" "🙏" "🍂" "🌈" "❓" "✨" "🌟"}}
                    {{ $shuffled := shuffle $slice }}
                    {{ $random := index $shuffled 0 }}
                    <!--<div class="name same-height-always">
                        <h4><a href="{{ .Permalink }}">{{ .Title }}</a></h4>
                    </div>-->

                    <div class="image">
                        <a href="{{ .Permalink }}">
                          {{ if isset .Params "banner" }}
                            <!--<img src="{{ .Site.BaseURL}}img/placeholder.png" data-src="{{ .Site.BaseURL}}{{ .Params.banner }}" class="img-circle shadow" alt="{{.Title}}" title="{{.Title}}" width="40" height="40">-->{{$random}} {{ .Title }}
                          {{ else }}
                            <!--<img src="{{ .Site.BaseURL}}img/placeholder.png" class="img-circle" alt="{{.Title}}" title="{{.Title}}" width="40" height="40">-->🌿 {{ .Title }}
                          {{ end }}
                        </a>
                    </div>
                    <!--<div><h4 >{{.Description}}</h4></div>-->
                </div>

                {{ end }}
            </div>

            <hr class="hidden-md hidden-lg">
</div>
{{end}}

You’re looking at every page, on every page. Come on…

I honestly can’t even remember what I was doing years ago. Wow.

Same. That’s 2.5 billion comparisons.

How might I optimize that? Caching again? Which I just learned about tonight. Which is fantastic!

See my previous comment.

Ok I’ll get all the changes in and do another templte metrics test.

As for the goofy stuff I did, checking things on every page, seemed to never notice it on a few hundred pages, but yeah, those are problems. I gotta figure out what I was trying to do a few years ago and optimize/recode some stuff.

Thank you very much. Most helpful.

I remember what I was trying to do. Display the total # of articles by that author, next to the author name. What would be the most optimal way to do that?

            {{ range $name, $items := $.Site.Params.authors }}
                {{ range $key, $val := $items }}
                    {{ if eq $key "name" }}
                        <li>
                        <a href="{{ $baseurl }}authors/{{ $val | urlize | lower }}">
                          🧔 {{ $items.display_name }}
                        ({{ len (where $.Site.Pages ".Params.author" $name) }})
                        </a>
                        </li>
                    {{ end }}
                {{ end }}
            {{ end }}

Please advise. For the time being, it’s not necessary or vital to display this on the sidebar, so I’ll disable it, but I’d like to find out how to do this optimally in case I might need to. Thank you so much! You’re the best.

Pull that into a partial, call it with partialCached, using the author as as the cache key.

Or cache the whole thing, again, using the author as the cache key. Hard to say looking a bits and pieces of your code.

Template Metrics:

     cumulative       average       maximum      cache  percent  cached  total  
       duration      duration      duration  potential   cached   count  count  template
     ----------      --------      --------  ---------  -------  ------  -----  --------
  47m56.335089852s  177.354488ms  456.765463ms          0        0       0  16218  _default/single.html
  33m29.994986524s  121.193547ms  337.596316ms         63        0       0  16585  partials/sidebar.html
  16m24.853623451s    59.38219ms  254.237421ms         26        0       0  16585  partials/widgets/random.html
  11m59.087840608s   44.338872ms  237.344581ms         28        0       0  16218  partials/related.html
  7m48.157319884s   28.226053ms  138.492169ms         94        0       0  16586  partials/widgets/recent.html
  7m0.47508267s   25.352733ms  121.680465ms         19        0       0  16585  partials/widgets/faq.html
  49.061556268s     2.95801ms  149.760643ms          9        1     188  16586  partials/head.html
  32.337059797s  212.743814ms  290.734373ms          0        0       0    152  drugs/single.html
  14.183324094s      854.88µs   71.089107ms        100       99   16421  16591  partials/footer.html
  12.878318069s   130.08402ms  195.939541ms          0        0       0     99  section/community.html
  12.471682324s  138.574248ms  226.036178ms          0        0       0     90  _default/list.html
  11.362771499s     684.958µs   79.662173ms         12       99   16421  16589  partials/nav.html
  10.511584784s     557.377µs   92.615181ms         98       86   16227  18859  partials/share.html
   8.293333075s     499.869µs   67.060804ms         99      100   16584  16591  partials/widgets/featured.html
    8.03943046s     485.033µs   99.583655ms        100       99   16420  16575  partials/author.html
   7.452620441s   42.344434ms  100.862753ms          0        0       0    176  shortcodes/related.html
   7.416729476s     447.168µs   57.011919ms         10       99   16421  16586  partials/scripts.html
   4.847572701s     292.304µs      73.886ms         37       98   16321  16584  partials/breadcrumbs.html
   3.255965748s     196.426µs   65.461373ms        100       99   16421  16576  partials/tagcloud.html
    3.18862018s     192.259µs    67.09023ms        100      100   16584  16585  partials/widgets/categories.html
   2.349002122s     141.634µs   59.739518ms        100      100   16584  16585  partials/widgets/search.html
   2.262385987s   10.982456ms     85.9734ms         96        0       0    206  partials/carousel_featured.html
   1.870572301s  207.841366ms  217.782361ms          0        0       0      9  page/single.html
   1.697442792s     102.341µs   54.891913ms        100      100   16584  16586  partials/widgets/tags.html
   1.414451255s      86.405µs   54.237217ms         99       99   16217  16370  partials/most_read.html
   1.300384083s    6.668636ms  315.546508ms          0        0       0    195  _internal/_default/rss.xml
   1.143645666s   127.07174ms   163.62196ms          0        0       0      9  section/psychedelics.html
   1.080139314s      65.127µs   53.822834ms        100      100   16584  16585  partials/widgets/facebook.html
   753.373165ms      45.942µs   10.601996ms         10        0       0  16398  partials/seo/twitter.html
   671.651535ms  167.912883ms   219.22018ms          0        0       0      4  section/blog.html
   628.141178ms   89.734454ms  122.718859ms          0        0       0      7  shortcodes/list_tags.html
    449.91717ms   449.91717ms   449.91717ms          0        0       0      1  index.html
   397.724421ms   198.86221ms  215.712384ms          0        0       0      2  section/drugs.html
   336.631852ms     575.439µs   68.200926ms         71       66     384    585  partials/recent.base.html
   277.267742ms      17.012µs     237.514µs          0        0       0  16298  _internal/alias.html
   231.170778ms  231.170778ms  231.170778ms          0        0       0      1  _default/index.json
   194.270434ms  194.270434ms  194.270434ms          0        0       0      1  section/news.html
   175.347749ms  175.347749ms  175.347749ms          0        0       0      1  _internal/_default/sitemap.xml
   154.776519ms     874.443µs   15.815592ms          0        0       0    177  shortcodes/signup.html
   119.693133ms  119.693133ms  119.693133ms        100        0       0      1  partials/sections_list.html
    98.241437ms     498.687µs   21.706955ms        100       98     194    197  partials/drugs.html
    65.852407ms      144.73µs    10.24792ms          0        0       0    455  shortcodes/notice.html
    54.046377ms   54.046377ms   54.046377ms        100        0       0      1  partials/recent_psychedelics.html
    39.138739ms   39.138739ms   39.138739ms        100        0       0      1  partials/recent_faq.html
    34.188942ms   34.188942ms   34.188942ms        100        0       0      1  partials/recent_blog.html
    26.858911ms      95.244µs    4.792496ms          0        0       0    282  shortcodes/imgcap.html
    24.682914ms   24.682914ms   24.682914ms        100        0       0      1  partials/recent_drugs.html
    13.735335ms      83.244µs    1.656472ms         16        0       0    165  partials/seo/schema.html
     5.702445ms       34.56µs     297.099µs        100        0       0    165  partials/seo/google_analytics.html
     4.338365ms     173.534µs     455.703µs          0        0       0     25  shortcodes/blockquote.html
     1.131105ms    1.131105ms    1.131105ms          0        0       0      1  404.html
      560.212µs     560.212µs     560.212µs        100        0       0      1  partials/testimonials.html
      517.491µs     517.491µs     517.491µs        100        0       0      1  partials/features.html
      512.149µs       3.012µs      261.16µs        100        0       0    170  partials/site_social.html
      432.358µs     432.358µs     432.358µs        100        0       0      1  partials/clients.html
      398.483µs     398.483µs     398.483µs        100        0       0      1  partials/team.html
      268.485µs      67.121µs      151.65µs          0        0       0      4  shortcodes/radio.html
      241.252µs     241.252µs     241.252µs        100        0       0      1  partials/signup.html
      230.821µs     230.821µs     230.821µs          0        0       0      1  _internal/shortcodes/youtube.html
       208.75µs      208.75µs      208.75µs        100        0       0      1  partials/welcome.html
      172.431µs         974ns       41.73µs        100        0       0    177  partials/mailchimp.html
      168.633µs     168.633µs     168.633µs          0        0       0      1  shortcodes/aboutimg.html
      165.577µs     165.577µs     165.577µs          0        0       0      1  shortcodes/calc.html
      120.511µs     120.511µs     120.511µs        100        0       0      1  partials/hero.html
        3.407µs       3.407µs       3.407µs          0        0       0      1  robots.txt


                   |  EN    
-------------------+--------
  Pages            | 16772  
  Paginator pages  |    11  
  Non-page files   |    28  
  Static files     |  2451  
  Processed images |     0  
  Aliases          | 16298  
  Sitemaps         |     1  
  Cleaned          |     0  

Total in 193567 ms

Got it down to 3 minutes, that’s not bad. I’ll keep working at trying to optimize it and rethink what I need/want to display or not.

Thank you. If you have any other tips, that’d be helpful too.

If you’d like to see the source, please let me know your email and I can add you to the repository, I don’t want to post my code/link to repository publicly. Thank you.

I have to figure out why so many aliases are being generated. Where might I look for that?

AFAIK, only a few pages have aliases, rest should not.

+++
author = "hash"
authors = "hash-borgir"
banner = "https://news.psychedelicsdaily.com/cache/thumb/0ae76c673b9a854b04c783087597a8eb.jpg"
date = "2020-11-07T06:18:18-07:00"
description = ""
featured = false
images = ["https://patch.com/img/cdn20/users/57069/20201107/013927/styles/patch_image/public/marijuana-shutterstock-126159383___07133914296.jpg"]
reviewCount = "169814"
title = "5-states-pass-pot-ballot-measures-including-new-jersey"
tags = ["News", "cannabis"]
aliases = [""]
+++

Here is the front matter for the news pages. It seems like an empty “” alias tag is causing that? Could that be it?

find . -type f -name '*.md' -exec sed -i 's/aliases = \[""\]//g' {} + removed all empty alias front matter things and will try again.

Ok, that was it. Empty alias was cuasing empty alias pages to be generated.

Error: Error building site: open /home/stoned/websites/websites/psychedelicsdaily.com-hugo/public/about/index.html: no such file or directory

Also unsure why I get this error. All the markdown files are correctly in place.

But with Caching and giving up some few things, I was able to get

Start building sites … 
hugo v0.110.0-e32a493b7826d02763c3b79623952e625402b168+extended linux/amd64 BuildDate=2023-01-17T12:16:09Z VendorInfo=snap:0.110.0

                   |  EN    
-------------------+--------
  Pages            | 17999  
  Paginator pages  |   464  
  Non-page files   |    28  
  Static files     |  2451  
  Processed images |     0  
  Aliases          |   418  
  Sitemaps         |     1  
  Cleaned          |     0  

Total in 23505 ms


Just a tip. Throrougly check if adding partialCached to the post share and random partials breaks those partials. For example, the post share in all posts will only point to one post, and the random posts will not be ‘random’ per page since they will be the same.

Also, I’m not sure how this might help, but you can try to cache the images on your site. E.g.

[caches]
  [caches.images]
    dir = ':resourceDir/_gen'
    maxAge = "10h"

I also have one site with over 50k pages (no images), but since it is using pages from data, it takes over 3 minutes to build. I have cached a lot of the templates and resources, but even without caching, the build was just 10 minutes!

Yes, I figured as much so now I am having to carefully decide where I want randomness and where I don’t. Thanks for the tip.

Can you tell me about your dev workflow with so many files? hugo serve must be very slow.

I’m thinking after the initial build, don’t build them again, using some front matter.So the content directory with tens of thousands of pages doesn’t get regenerated evertytime I launch hugo serve

Start building sites … 
hugo v0.110.0-e32a493b7826d02763c3b79623952e625402b168+extended linux/amd64 BuildDate=2023-01-17T12:16:09Z VendorInfo=snap:0.110.0

                   |  EN    
-------------------+--------
  Pages            | 49286  
  Paginator pages  |  2438  
  Non-page files   |    28  
  Static files     |  2451  
  Processed images |     0  
  Aliases          |   419  
  Sitemaps         |     1  
  Cleaned          |     0  

Total in 443982 ms