Race condition while computing backlinks : workaround appreciated!

support
1

Hello everyone,

I implemented my own version of backlinks (while waiting for an eventual support) as follows :

  1. I piggybacked on render-link.html to append interesting internal links (with source and destination url and title) to a global Scratch :
{{- if (and (not $isRemote) (eq .Page.Section "Apprenti-sage")) -}}
  {{- $graphrow := printf "%s|%s|%s|%s\n" ($link | safeURL) (.Page.GetPage (urls.Parse .Destination).Path).Title .Page.RelPermalink .Page.Title -}}
  {{- site.Home.Scratch.Add "links-as" $graphrow -}}
{{- end -}}
  1. Then I implemented a custom JSON layout that transform this Scratch into a JSON database in which each page lists its incoming and outgoing links :
{{- $raw := site.Home.Scratch.Get "links-as" -}}
{{- $lines := split $raw "\n" -}}
{{- $graph := dict -}}
{{- range sort $lines -}}
  {{- if ne . "" -}}
    {{- $fields := split . "|" -}}
    {{- $destLink := index $fields 0 -}}
    {{- $destTitle := index $fields 1 -}}
    {{- $sourceLink := index $fields 2 -}}
    {{- $sourceTitle := index $fields 3 -}}
    {{- $localgraph := dict (slice $sourceLink "frontlinks" $destLink) $destTitle (slice $destLink "backlinks" $sourceLink) $sourceTitle -}}
    {{- $graph = $graph | merge $localgraph -}}
  {{- end -}}
{{- end -}}
{{ $graph | jsonify (dict "indent" "  ") }}

Its working pretty great (as seen here : Résoudre le mystère • Hugues Le Gendre) but I am having a race condition because the links.json gets computed before all the links have been rendered… And therefore is incomplete (but not so much !).

While I am waiting for this new feature (that might help me) :

I am looking for a workaround…

Any idea on how to defer the execution of this one template ? I am thinking about a sleep equivalent, maybe a very expensive loop… I wanted to do

{{- $until := (add now.Unix 2) -}}
{{- while lt now.Unix $until -}}
  {{/* we just wait 2 seconds... */}}
{{- end -}}

but while is not a command of text/template…

Any idea ?

Best regards !
Hugues

2

Currently I am considering setting-up a very lightweight server running along my hugo compilation that would wait a custom delay before answering. So when I use getJSON in my template, it will wait a bit before moving on…

3

(I continue to document my journey… :slight_smile:)

It’s not working, even with a very long (10s) delay, this template returns inconsistent data…

So

  • either the blocking call
{{- $dump := getJSON "http://localhost:3010/?delay=1000" -}}
{{- $raw := site.Home.Scratch.Get "links-as" -}}

is useless because the Scratch is retreived before (maybe at template execution as a static object…)

  • or some articles templates with links are still generated after this template, but I would have thought that Hugo had a pool of goroutines for generating content, so this should not happen…

@bep, sorry to call you out, but do you have any idea of what is happening ?

4

The backlinks “database” will not be complete until every page is rendered, but you cannot render a page until the backlinks “database” is complete.

:chicken: → :egg:

:egg: ← :chicken:

If it were me, I would abandon your current approach and do this instead:

  1. In your render hook, add data attributes to the anchor element indicating source of link (data-permalink, data-title, etc.)
  2. Build the site.
  3. Use an external script to parse (using xmllint or similar) HTML files in the publish directory, and build a JSON database of backlinks.
  4. Build the site again.
5

Sure, but you can workaround that by adding something like this to the top of the JSON “database collector”:

{{ range site.Pages }}
{{ $tmp := .Content }}
{{ end }}

Or something.

6

Actually, I build the site twice and copy the resulting links.json to the data/ folder between builds. So there should not be any chicken-egg problem. I would very much like to avoid using an external link traversing tool is possible… but it might be the only solution…
thank you for your input.

7

Good thinking !
I was very hopeful but it is not working… Is the .Content getting cached ? If yes, is there a way to bust it ?

8

Just to let you know: in the end, I could not make it work so I coded a script with node (I already had other scripts to manage my site) that traverses the content folder and generates the links.json file before building.

Thanks for your help anyway!