.Page.Store is sometimes empty. Cannot load Mermaid script reliably

Hugo Version
hugo v0.114.0+extended darwin/amd64

Our repo

What I did
I followed this advice to set the state of (the existence of) mermaid diagrams in the .Page.Store.

I also used this strategy for other small scripts to reduce the page weight as my users have low data.

What I expected to happen

Given a page with mermaid diagrams in the .Content
When I render the mermaid block in .Content
Then the script is called in scripts.html

What actually happens

Sometimes the mermaid script is called when there’s nothing on the page. Sometimes all the other scripts are called but not mermaid. Sometimes no scripts are called. For example on this page

these scripts are called
scripts/youtube-player.min.js
scripts/tab-panels.min.js

but mermaid is not, though mermaid is present. On this page

No scripts are called (though tab panels are present).

I can’t tell why, or how it changes without changing the code. Is there something happening with caching? Can someone help me? Thanks for all your work on Hugo.

Key files
get store

set store

Update on this – as this is launching tomorrow and there’s no one around, I will have to push the scripts to every page and remove the page store gets.

Here’s the commit the above post referenced:

Testing this locally is not possible without me stripping out your remote data calls.

Can you reproduce the behavior described when running hugo, or only when running hugo server?

Yes it’s tricky isn’t it – the authentication is only to GitHub to avoid rate limiting, and not to private resources, but it’s a barrier. I don’t know if you can create a fine grained token without being in the org. I could make a branch without auth if that’s useful?

The behaviour is produced with hugo and hugo server. It’s also consistent on local and on production (netlify). As in, I can’t see what causes the page store to be nil on some pages and not on others, but the nils are the same on the same pages across local and prod.

I commented out all the partial calls to templates that get remote data:

issues.html
block/data.html
block/issue.html
byline.html

Then I ran this bash script to test if the mermaid script element is written to:

  • public/fundamentals/product/index.html
  • public/fundamentals/product/plan/index.html
#!/usr/bin/env bash
main() {

  declare file_to_test_1=public/fundamentals/product/index.html
  declare file_to_test_2=public/fundamentals/product/plan/index.html
  declare file_to_test_3=public/fundamentals/prep/index.html

  declare string_to_find=https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.esm.min.mjs
  declare counter=0

  while true; do
    (( counter++ )) || true
    echo "Iteration: " "${counter}"
    rm -rf public
    hugo --quiet
    if ! grep -q ${string_to_find} ${file_to_test_1}; then
      echo "Cannot find ${string_to_find} in ${file_to_test_1}"
      exit 1
    fi
    if ! grep -q ${string_to_find} ${file_to_test_2}; then
      echo "Cannot find ${string_to_find} in ${file_to_test_2}"
      exit 1
    fi
    if ! grep -q ${string_to_find} ${file_to_test_3}; then
      echo "Cannot find ${string_to_find} in ${file_to_test_3}"
      exit 1
    fi

  done

}

set -euo pipefail
main "$@"


Is has run 412 times without error.

Also, I searched your markdown for mermaid codeblocks, and only found three instances:

image

Thank you!

I’m not 100% sure what to do with this information. Can you explain more?

I’m saying that I cannot reproduce this locally

OK, I can consistently reproduce the problem below by replacing result of all the remote data calls with an empty map. Error emitted by test script above:

Cannot find https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.esm.min.mjs in public/fundamentals/prep/index.html

Where do the mermaid code blocks exist?

Path Status
content/fundamentals/product/plan/index.md :heavy_check_mark:
content/fundamentals/product/_index.md :heavy_check_mark:
content/blocks/backlog/index.md :x:

What’s different about content/blocks/backlog/index.md? It is headless (see front matter) and rendered with a partial. The rendering chain on your site is… complicated. It’s going to take me a bit to unravel it.

It is a bit complicated! It’s because it’s composing from many sources which are being developed independently by autonomous volunteer teams. The key partial is here

And we use this scratch map to store all our gubbins in to construct the lookups

At the bottom is the headers call, which is where the github auth is happening.

There are headless blocks setting states in the page store, but it is possible that mermaid is the only render hook operating on a headless local block doing this. However, there are other pages (ex linked in post 1) which don’t correctly set the page store for module tabs and youtube video.

Let’s just focus on the missing mermaid script element that should appear before the closing body tag. If we can figure out why this one is failing, that may help with the others.

So, we have a missing mermaid diagram only on /fundamentals/prep/. What’s the rendering chain?

Seq Action Path
1 CONTENT content/fundamentals/prep/index.md
2 LOADS TEMPLATE layouts/_default/baseof.html
3 LOADS TEMPLATE layouts/_default/prep.html
4 CALLS PARTIAL layouts/partials/block/block.html
5 CALLS PARTIAL layouts/partials/block/local.html
6 CALLS site.GetPage "content/blocks/backlog/index.md"
7 RENDERS CONTENT content/blocks/backlog/index.md
8 FIRES layouts/_default/_markup/render-codeblock-mermaid.html.html
9 CALLS .Page.Store.Set "hasMermaid" true

And content/blocks/backlog/index.md is where the markdown for the missing diagram resides.

Glad it’s relatively simple.

The page you are rendering in Step 1 is:
content/fundamentals/prep/index.md

But in Step 9 you are setting the .Store on:
content/blocks/backlog/index.md

Prove it to yourself by placing this at the end of layouts/_default/_markup/render-codeblock-mermaid.html.html:

<pre style="color: red">.Page.File.Path = {{ .Page.File.Path }}</pre>

We need to set the .Store on the top-level page loaded in Step 2, and we do that with the page function.

In your mermaid codeblock render hook, replace this:

{{- .Page.Store.Set "hasMermaid" true -}}

with this:

{{- page.Store.Set "hasMermaid" true -}}

Although I haven’t investigated the other uses of .Page.Store.Set in your templates, I suspect you have the same issue: incorrect context.

Wonderful! I understand!

Thank you so much for fixing this and explaining the problem to me so clearly.

…Oh but wait. There is something happening between server and build.

When I deployed this, the results on local and production are different.

The Mermaid shortcode in the headless block prints the same on both

Mermaid .Page.File.Path = blocks/backlog/index.md
Mermaid page.File.Path = fundamentals/sprints/3/prep/index.md
Mermaid page store = true

The scripts partial prints on local

Scripts page.File.Path = fundamentals/prep/index.md
Mermaid page store  =  true

And on production

Scripts page.File.Path = fundamentals/prep/index.md
Mermaid page store =

Can you guide me?

From your commits on the bugfix/page-store branch it looks a bit like you’re guessing.

Let’s go back to single change I made yesterday, a one line change to the mermaid render hook—no other changes.

My testing yesterday was running hugo server, which behaves differently than hugo. If I run hugo locally, sometimes the mermaid script element is written, sometimes it is not.

The inconsistent behavior points to a concurrency problem. Hugo renders things in parallel, and your rendering chain is convoluted (see above). We can test the concurrency theory by limiting Hugo to one thread:

HUGO_NUMWORKERMULTIPLIER=1 hugo

I ran this test script 175 times without failure.

#!/usr/bin/env bash
main() {

  declare file_to_test_1=public/fundamentals/product/index.html
  declare file_to_test_2=public/fundamentals/product/plan/index.html
  declare file_to_test_3=public/fundamentals/prep/index.html

  declare string_to_find=https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.esm.min.mjs
  declare counter=0

  while true; do
    (( counter++ )) || true
    echo "Iteration: " "${counter}"
    rm -rf public
    HUGO_NUMWORKERMULTIPLIER=1 hugo --quiet
    if ! grep -q ${string_to_find} ${file_to_test_1}; then
      echo "Cannot find ${string_to_find} in ${file_to_test_1}"
      exit 1
    fi
    if ! grep -q ${string_to_find} ${file_to_test_2}; then
      echo "Cannot find ${string_to_find} in ${file_to_test_2}"
      exit 1
    fi
    if ! grep -q ${string_to_find} ${file_to_test_3}; then
      echo "Cannot find ${string_to_find} in ${file_to_test_3}"
      exit 1
    fi

  done

}

set -euo pipefail
main "$@"

Limiting Hugo to one thread removes the symptom, but does not solve the problem with your code.

From your commits on the bugfix/page-store branch it looks a bit like you’re guessing.

:joy: I’m on holiday in Tuscany so I’m just fiddling around a bit in between swims! :swimming_woman:(I’m not really at work but we do have this class starting so all hands…)

These are the culprits:

https://github.com/CodeYourFuture/curriculum/blob/main/content/fundamentals/sprints/2/prep/index.md?plain=1#L17

https://github.com/CodeYourFuture/curriculum/blob/main/content/fundamentals/sprints/3/prep/index.md?plain=1#L13

If you comment those out, no problems. I can’t tell you why—I’m having a difficult time following the logic.


Not relevant, and it is currently not called, but this is wrong:
https://github.com/CodeYourFuture/curriculum/blob/main/layouts/_default/single.html#L8

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.