GetTerms getting slows as the content grows

razon · June 18, 2024, 6:53am

Hi, I noticed that the GetTerms may get slow if there are a lot of taxonomies, am I doing something wrong? Is there any improvement suggestions?

Thanks.

Details

// layouts/_default/single.html
{{- $page := . }}
{{ $t := debug.Timer "page-taxonomies" }}
{{- range $kind := slice "tags" "categories" }}
  {{ $t1 := printf "page-taxonomies-%s" $kind | debug.Timer }}
  {{- with $page.GetTerms $kind }}
    {{- range . }}
      <span class="blog-post-taxonomy-meta">
        <a
          class="blog-post-taxonomy blog-post-taxonomy badge bg-secondary text-decoration-none fw-normal me-1"
          href="{{ .RelPermalink }}">
          {{- .Title -}}
        </a>
      </span>
    {{- end }}
  {{- end }}
  {{ $t1.Stop }}
{{- end -}}
{{ $t.Stop }}

page-taxonomies tooks time as follows.

Posts	`average`
`1k`	`703.042µs`
`3k`	`3.107381ms`
`5k`	`10.254508ms`
`7k`	`16.712375ms`
`10k`	`26.22568ms`

Full build log.

➜  hugo-testing git:(performance-page-get-terms) ✗ npm run build-1k

> hugo-testing@1.0.0 build-1k
> npm run clean && hugo --logLevel info -e prod-1k


> hugo-testing@1.0.0 clean
> rm -rf ./public

Start building sites … 
hugo v0.127.0+extended linux/amd64 BuildDate=unknown

INFO  build:  step process substep collect files 10000 files_total 10000 pages_total 10000 resources_total 0 duration 239.792269ms
INFO  build:  step process duration 239.888007ms
INFO  build:  step assemble duration 3.825292418s
WARN  found no layout file for "html" for kind "home": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "section": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "term": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
INFO  build:  step render substep pages site en outputFormat html duration 340.861272ms
INFO  build:  step render substep pages site en outputFormat rss duration 630.820993ms
INFO  build:  step render pages 14003 content 13989 duration 996.817434ms
INFO  build:  step postProcess duration 14.374µs
INFO  timer:  name page-taxonomies-categories count 1000 duration 165.943926ms average 165.943µs median 160.92µs
INFO  timer:  name page-taxonomies-tags count 1000 duration 517.265121ms average 517.265µs median 485.155µs
INFO  timer:  name page-taxonomies count 1000 duration 703.042707ms average 703.042µs median 662.903µs
INFO  build:  duration 5.06318678s

                   |  EN    
-------------------+--------
  Pages            | 14003  
  Paginator pages  |     0  
  Non-page files   |     0  
  Static files     |     0  
  Processed images |     0  
  Aliases          |     0  
  Cleaned          |     0  

Total in 5066 ms

===================================================================

➜  hugo-testing git:(performance-page-get-terms) ✗ npm run build-3k

> hugo-testing@1.0.0 build-3k
> npm run clean && hugo --logLevel info -e prod-3k


> hugo-testing@1.0.0 clean
> rm -rf ./public

Start building sites … 
hugo v0.127.0+extended linux/amd64 BuildDate=unknown

INFO  build:  step process substep collect files 10000 files_total 10000 pages_total 10000 resources_total 0 duration 241.428034ms
INFO  build:  step process duration 241.491891ms
INFO  build:  step assemble duration 27.219076548s
WARN  found no layout file for "html" for kind "section": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "home": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "term": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
INFO  build:  step render substep pages site en outputFormat html duration 1.398496477s
INFO  build:  step render substep pages site en outputFormat rss duration 2.348027684s
INFO  build:  step render pages 41894 content 43360 duration 3.816131487s
INFO  build:  step postProcess duration 10.663µs
INFO  timer:  name page-taxonomies-categories count 3000 duration 2.260937366s average 753.645µs median 658.334µs
INFO  timer:  name page-taxonomies-tags count 3000 duration 6.994667263s average 2.331555ms median 2.140747ms
INFO  timer:  name page-taxonomies count 3000 duration 9.322143788s average 3.107381ms median 2.851902ms
INFO  build:  duration 31.277857665s

                   |  EN    
-------------------+--------
  Pages            | 41894  
  Paginator pages  |     0  
  Non-page files   |     0  
  Static files     |     0  
  Processed images |     0  
  Aliases          |     0  
  Cleaned          |     0  

Total in 31282 ms

===================================================================

➜  hugo-testing git:(performance-page-get-terms) ✗ npm run build-5k

> hugo-testing@1.0.0 build-5k
> npm run clean && hugo --logLevel info -e prod-5k


> hugo-testing@1.0.0 clean
> rm -rf ./public

Start building sites … 
hugo v0.127.0+extended linux/amd64 BuildDate=unknown

INFO  build:  step process substep collect files 10000 files_total 10000 pages_total 10000 resources_total 0 duration 249.474582ms
INFO  build:  step process duration 249.528721ms
INFO  build:  step assemble duration 1m26.01390894s
WARN  found no layout file for "html" for kind "home": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "section": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "term": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
INFO  build:  step render substep pages site en outputFormat html duration 4.641431399s
INFO  build:  step render substep pages site en outputFormat rss duration 4.949556856s
INFO  build:  step render pages 69678 content 79525 duration 9.681625095s
INFO  build:  step postProcess duration 11.861µs
INFO  timer:  name page-taxonomies-categories count 5000 duration 11.366767241s average 2.273353ms median 2.121671ms
INFO  timer:  name page-taxonomies-tags count 5000 duration 39.723852567s average 7.94477ms median 7.846735ms
INFO  timer:  name page-taxonomies count 5000 duration 51.272544239s average 10.254508ms median 10.033733ms
INFO  build:  duration 1m35.947676955s

                   |  EN    
-------------------+--------
  Pages            | 69678  
  Paginator pages  |     0  
  Non-page files   |     0  
  Static files     |     0  
  Processed images |     0  
  Aliases          |     0  
  Cleaned          |     0  

Total in 95951 ms

===================================================================

➜  hugo-testing git:(performance-page-get-terms) ✗ npm run build-7k

> hugo-testing@1.0.0 build-7k
> npm run clean && hugo --logLevel info -e prod-7k


> hugo-testing@1.0.0 clean
> rm -rf ./public

Start building sites … 
hugo v0.127.0+extended linux/amd64 BuildDate=unknown

INFO  build:  step process substep collect files 10000 files_total 10000 pages_total 10000 resources_total 0 duration 230.010102ms
INFO  build:  step process duration 230.06412ms
        INFO  build:  step assemble duration 5m48.533652201s
WARN  found no layout file for "html" for kind "home": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "section": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "term": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
INFO  build:  step render substep pages site en outputFormat html duration 9.383299597s
INFO  build:  step render substep pages site en outputFormat rss duration 8.484321481s
INFO  build:  step render pages 97365 content 123424 duration 17.981488864s
INFO  build:  step postProcess duration 15.894µs
INFO  timer:  name page-taxonomies-categories count 7000 duration 25.645225301s average 3.663603ms median 3.633117ms
INFO  timer:  name page-taxonomies-tags count 7000 duration 1m31.066313772s average 13.009473ms median 13.163557ms
INFO  timer:  name page-taxonomies count 7000 duration 1m56.986626964s average 16.712375ms median 16.841175ms
INFO  build:  duration 6m6.748059213s

                   |  EN    
-------------------+--------
  Pages            | 97365  
  Paginator pages  |     0  
  Non-page files   |     0  
  Static files     |     0  
  Processed images |     0  
  Aliases          |     0  
  Cleaned          |     0  

Total in 366752 ms

===================================================================

➜  hugo-testing git:(performance-page-get-terms) ✗ npm run build   

> hugo-testing@1.0.0 build
> npm run clean && hugo --logLevel info


> hugo-testing@1.0.0 clean
> rm -rf ./public

Start building sites … 
hugo v0.127.0+extended linux/amd64 BuildDate=unknown

INFO  build:  step process substep collect files 10000 files_total 10000 pages_total 10000 resources_total 0 duration 248.939705ms
INFO  build:  step process duration 249.054085ms
INFO  build:  step assemble duration 8m24.48941548s
WARN  found no layout file for "html" for kind "home": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "section": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN  found no layout file for "html" for kind "term": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
INFO  build:  step render substep pages site en outputFormat html duration 19.399689575s
INFO  build:  step render substep pages site en outputFormat rss duration 15.801164224s
INFO  build:  step render pages 138690 content 181875 duration 35.343891795s
INFO  build:  step postProcess duration 11.233µs
INFO  timer:  name page-taxonomies-categories count 10000 duration 58.264440802s average 5.826444ms median 5.587527ms
INFO  timer:  name page-taxonomies-tags count 10000 duration 3m23.634542005s average 20.363454ms median 19.826354ms
INFO  timer:  name page-taxonomies count 10000 duration 4m22.256808904s average 26.22568ms median 25.357105ms
INFO  build:  duration 9m0.086308562s

                   |   EN    
-------------------+---------
  Pages            | 138690  
  Paginator pages  |      0  
  Non-page files   |      0  
  Static files     |      0  
  Processed images |      0  
  Aliases          |      0  
  Cleaned          |      0  

Total in 540090 ms

Reproduce Steps

I created a site from scratch without theme for reproducing.

// clone specified branch
git clone -b performance-page-get-terms https://github.com/razonyang/hugo-testing.git

cd hugo-testing

// create dummy content
npm ci
npm run create

// run builds
npm run build-1k
npm run build-3k
npm run build-5k
npm run build-7k
npm run build

jmooring · June 18, 2024, 2:15pm

I’m seeing a linear increase. Each page in the test has the same two terms, and average time is an average of 3 runs to normalize build time variations from run to run.

GetTerms average time (µs) vs. Number of posts

Relevant code:
https://github.com/gohugoio/hugo/blob/ad6d91cabd84aac1be6e83511a543643562cb1b2/hugolib/content_map_page.go#L448-L477

Also note that I ran these tests from the terminal with my IDE (vscode) closed to eliminate the effects of any file watching that might be occurring.

bep · June 19, 2024, 10:45am

Fuad_Efendi · June 20, 2024, 10:46pm

I tested today with about 270,000-pages site, built Hugo from source (latest checkout few hours ago) as @razon suggested; what was taking about 5 hours (with “category” taxonomy only), now is about 1 hour 20 minutes (with “category”, and with additionally “tags” taxonomies!)

hugo v0.128.0-DEV-d5542ed286746e89fb13a1f821d4955ace371773+extended darwin/arm64 BuildDate=2024-06-20T17:37:22Z

Thank you!

bep · June 21, 2024, 8:36am

That’s a beefy site

I assume you know about this relatively new segments feature:

Fuad_Efendi · June 21, 2024, 12:59pm

Yes I’ve heard about segments but didn’t try yet; I made mistake yesterday with double-taxonomy, my second taxonomy had cardinality 4 maybe that’s why it was so fast; after I changed it to another high-cardinality taxonomy, double-taxonomy calcs already running more than twelve hours, still cannot see when it may finish; it feels like “cartesian product” being calculated with double-taxonomy. Maybe 1000x10,000 (at least)…
But with just “1000” -cardinality taxonomy works 4x faster now, 1:20 vs. 5:00 hours

bep · June 21, 2024, 2:04pm

I’m pretty sure that high-cardinality taxonomy performance issues is not related to GetTerms, though.

razon · June 21, 2024, 3:28pm

I think the performance of GerTerms has been marked improved in next release, I’m gonna mark this topic as resolved.

Thank for @bep quick fix.

Check the average in the following log:

➜  hugo-testing git:(perf-terms) ✗ hugo version
hugo v0.127.0+extended linux/amd64 BuildDate=unknown

➜  hugo-testing git:(perf-terms) ✗ rm -rf content && ./create.sh 10000 1
➜  hugo-testing git:(perf-terms) ✗ hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 2.609336056s average 130.466µs median 174.297µs
➜  hugo-testing git:(perf-terms) ✗ hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 2.947163643s average 147.358µs median 185.33µs
➜  hugo-testing git:(perf-terms) ✗ hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 2.993108518s average 149.655µs median 182.244µs

➜  hugo-testing git:(perf-terms) ✗ rm -rf content && ./create.sh 10000 3              
➜  hugo-testing git:(perf-terms) ✗ hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 19.801299466s average 990.064µs median 1.179495ms
➜  hugo-testing git:(perf-terms) ✗ hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 23.854347475s average 1.192717ms median 1.425751ms
➜  hugo-testing git:(perf-terms) ✗ hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 18.94523462s average 947.261µs median 1.119509ms

➜  hugo-testing git:(perf-terms) ✗ rm -rf content && ./create.sh 10000 5              
➜  hugo-testing git:(perf-terms) ✗ hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 40.543056253s average 2.027152ms median 2.690726ms
➜  hugo-testing git:(perf-terms) ✗ hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 40.431768212s average 2.021588ms median 2.752857ms
➜  hugo-testing git:(perf-terms) ✗ hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 40.39149699s average 2.019574ms median 2.659197ms
➜  hugo-testing git:(perf-terms) ✗ 
➜  hugo-testing git:(perf-terms) ✗ 
➜  hugo-testing git:(perf-terms) ✗ 
➜  hugo-testing git:(perf-terms) ✗ ~/Workspaces/gohugoio/hugo/hugo version
hugo v0.128.0-DEV-478a9107a68aeb5dac9ea0cec0a347fadb708b64+extended linux/amd64 BuildDate=2024-06-19T13:18:43Z

➜  hugo-testing git:(perf-terms) ✗ rm -rf content && ./create.sh 10000 1
➜  hugo-testing git:(perf-terms) ✗ ~/Workspaces/gohugoio/hugo/hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 257.17568ms average 12.858µs median 7.685µs
➜  hugo-testing git:(perf-terms) ✗ ~/Workspaces/gohugoio/hugo/hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 258.909576ms average 12.945µs median 7.496µs
➜  hugo-testing git:(perf-terms) ✗ ~/Workspaces/gohugoio/hugo/hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 271.833406ms average 13.591µs median 7.702µs

➜  hugo-testing git:(perf-terms) ✗ rm -rf content && ./create.sh 10000 3                                         
➜  hugo-testing git:(perf-terms) ✗ ~/Workspaces/gohugoio/hugo/hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 525.15093ms average 26.257µs median 17.011µs
➜  hugo-testing git:(perf-terms) ✗ ~/Workspaces/gohugoio/hugo/hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 548.967573ms average 27.448µs median 16.909µs
➜  hugo-testing git:(perf-terms) ✗ ~/Workspaces/gohugoio/hugo/hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 568.951547ms average 28.447µs median 17.181µs

➜  hugo-testing git:(perf-terms) ✗ rm -rf content && ./create.sh 10000 5
➜  hugo-testing git:(perf-terms) ✗ ~/Workspaces/gohugoio/hugo/hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 792.032599ms average 39.601µs median 26.016µs
➜  hugo-testing git:(perf-terms) ✗ ~/Workspaces/gohugoio/hugo/hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 753.023446ms average 37.651µs median 25.887µs
➜  hugo-testing git:(perf-terms) ✗ ~/Workspaces/gohugoio/hugo/hugo --logLevel info | grep GetTerms | grep average
INFO  timer:  name GetTerms count 20000 duration 768.978456ms average 38.448µs median 25.979µs

system · June 23, 2024, 3:28pm

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
.GetTerms in depth understanding to debug support	7	1134	May 19, 2022
[Solved] Creating _index.md for taxonomyTerm pages tags, categories hangs Hugo	13	2673	March 21, 2018
Best pratices? Meta	6	1422	March 17, 2022
Special characters in taxonomy support	14	4214	January 21, 2017
How to paginate by tag? support	1	497	August 27, 2023

GetTerms getting slows as the content grows

Details

Reproduce Steps

Related Topics