Race condition while computing backlinks : workaround appreciated!

Hello everyone,

I implemented my own version of backlinks (while waiting for an eventual support) as follows :

  1. I piggybacked on render-link.html to append interesting internal links (with source and destination url and title) to a global Scratch :
{{- if (and (not $isRemote) (eq .Page.Section "Apprenti-sage")) -}}
  {{- $graphrow := printf "%s|%s|%s|%s\n" ($link | safeURL) (.Page.GetPage (urls.Parse .Destination).Path).Title .Page.RelPermalink .Page.Title -}}
  {{- site.Home.Scratch.Add "links-as" $graphrow -}}
{{- end -}}
  1. Then I implemented a custom JSON layout that transform this Scratch into a JSON database in which each page lists its incoming and outgoing links :
{{- $raw := site.Home.Scratch.Get "links-as" -}}
{{- $lines := split $raw "\n" -}}
{{- $graph := dict -}}
{{- range sort $lines -}}
  {{- if ne . "" -}}
    {{- $fields := split . "|" -}}
    {{- $destLink := index $fields 0 -}}
    {{- $destTitle := index $fields 1 -}}
    {{- $sourceLink := index $fields 2 -}}
    {{- $sourceTitle := index $fields 3 -}}
    {{- $localgraph := dict (slice $sourceLink "frontlinks" $destLink) $destTitle (slice $destLink "backlinks" $sourceLink) $sourceTitle -}}
    {{- $graph = $graph | merge $localgraph -}}
  {{- end -}}
{{- end -}}
{{ $graph | jsonify (dict "indent" "  ") }}

Its working pretty great (as seen here : Résoudre le mystère • Hugues Le Gendre) but I am having a race condition because the links.json gets computed before all the links have been rendered… And therefore is incomplete (but not so much !).

While I am waiting for this new feature (that might help me) :

I am looking for a workaround…

Any idea on how to defer the execution of this one template ? I am thinking about a sleep equivalent, maybe a very expensive loop… I wanted to do

{{- $until := (add now.Unix 2) -}}
{{- while lt now.Unix $until -}}
  {{/* we just wait 2 seconds... */}}
{{- end -}}

but while is not a command of text/template

Any idea ?

Best regards !
Hugues

Currently I am considering setting-up a very lightweight server running along my hugo compilation that would wait a custom delay before answering. So when I use getJSON in my template, it will wait a bit before moving on…

(I continue to document my journey… :slight_smile:)

It’s not working, even with a very long (10s) delay, this template returns inconsistent data…

So

  • either the blocking call
{{- $dump := getJSON "http://localhost:3010/?delay=1000" -}}
{{- $raw := site.Home.Scratch.Get "links-as" -}}

is useless because the Scratch is retreived before (maybe at template execution as a static object…)

  • or some articles templates with links are still generated after this template, but I would have thought that Hugo had a pool of goroutines for generating content, so this should not happen…

@bep, sorry to call you out, but do you have any idea of what is happening ?

The backlinks “database” will not be complete until every page is rendered, but you cannot render a page until the backlinks “database” is complete.

:chicken::egg:

:egg::chicken:

If it were me, I would abandon your current approach and do this instead:

  1. In your render hook, add data attributes to the anchor element indicating source of link (data-permalink, data-title, etc.)
  2. Build the site.
  3. Use an external script to parse (using xmllint or similar) HTML files in the publish directory, and build a JSON database of backlinks.
  4. Build the site again.

Sure, but you can workaround that by adding something like this to the top of the JSON “database collector”:

{{ range site.Pages }}
{{ $tmp := .Content }}
{{ end }}

Or something.

Actually, I build the site twice and copy the resulting links.json to the data/ folder between builds. So there should not be any chicken-egg problem. I would very much like to avoid using an external link traversing tool is possible… but it might be the only solution…
thank you for your input.

Good thinking !
I was very hopeful but it is not working… Is the .Content getting cached ? If yes, is there a way to bust it ?

Just to let you know: in the end, I could not make it work so I coded a script with node (I already had other scripts to manage my site) that traverses the content folder and generates the links.json file before building.

Thanks for your help anyway!

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.