How to ensure partial is executed just once?

[swamidass]

How can I ensure a partial is executed just once?

I wrote partial to aggregate views from a json file in the Scratch of each page, sensitive to aliases declared in the page to capture views from prior locations of each page. Because it aggregates views, this partial should only be executed one time.

To ensure it was only executed once, I made it a cached partial, but that doesn’t work. Instead, the partial is executed as many times as there are threads, creating some nasty race conditions.

So I had to set HUGO_NUMWORKERMULTIPLIER=1 in addition to using a cached partial. That’s an ugly workaround because it limits the whole site build to just one worker thread.

I’ve considered refactoring the code to not use page Scratch, but that turns out not to be trivial. Moreover, it would still waste cycles and memory on duplicate information. So that most likely isn’t the right solution.

Is there a better way to do this?

Ideally there would be a way to:

1. in a partial itself, to declare that it can only be executed once.
2. cache results of partials declared this way.
3. ensure in the hugo executable code that it is only executed once.
4. block return from the partial on subsequent calls till cached results are available.

I know multithreaded code can be difficult, but this seems to be a well-defined use case, and focused enough that it shouldn’t be too difficult to implement. Though, I don’t know enough of go to implement it for myself.

[bep]

@swamidass

To ensure it was only executed once, I made it a cached partial, but that doesn’t work. Instead, the partial is executed as many times as there are threads, creating some nasty race conditions.

The above statement isn’t true, but you’re right, there is not guarantee that it’s executed only once.

@swamidass

I know multithreaded code can be difficult, but this seems to be a well-defined use case, and focused enough that it shouldn’t be too difficult to implement.

I read your post and I did not understand what you want to do. Maybe we can start there.

[swamidass]

@bep

@swamidass

To ensure it was only executed once, I made it a cached partial, but that doesn’t work. Instead, the partial is executed as many times as there are threads, creating some nasty race conditions.

The above statement isn’t true, but you’re right, there is not guarantee that it’s executed only once.

I’m not sure exactly what isn’t true.

The race conditions are specific to cases in which it is assumed that the partial is executed only once. In most cases, this not a problem. But in this case it is.

@bep

I read your post and I did not understand what you want to do. Maybe we can start there.

Perhaps the code will make sense of this.

The partial in question here. And here is an example of usage:

      {{ $PAGES := partialCached "byviews" . }}

      ..,

      {{ with $.Scratch.Get "Views" }}
             ...print views
      {{ end }}

It is also used in this page to sort all pages by views (and also why I don’t want to merely calculate this per page).

      intersect $PAGES.ByViews  $ALL

You can see it in practice at this website:

Peaceful Science

---

So, is there a better way to do this?

[swamidass]

I should also add that many of my pages have aliases. In our migration to hugo most pages changed URLs, but we keep redirects to the old URLs using aliases. And that’s why aggregation is needed, and just a simple lookup won’t do.

The partial makes use of hugo’s URL resolution machinery (site.GetPage) to aggregate views across all URLs associated with a page.

[bep]

@swamidass

I’m not sure exactly what isn’t true.

• These runs in Goroutines which isn’t the same as threads.
• If you have 16 Goroutines running and all of them executes this partial at the same time then yes, it may be executed 16 times, but that does not sound like the common case.

I suggest you rewrite it to do something ala:

{{ $s := newScratch }}
{{ range .Site.Pages }}
  {{ $s.Set "Views" 0 }}
...
{{ end }}

[bep]

Evaluate moving the partialCache lock

[swamidass]

@bep

• These runs in Goroutines which isn’t the same as threads.

Got it.

@bep

I suggest you rewrite it to do something ala:

That doesn’t work because there is only one scratch for the whole site, not one for each page. It’s not clear to me how to efficiently make a mapping between pages and scratches. Perhaps a map would work, which uses the RelLink of each page as a key? But what would be an efficient way of creating that map?

Perhaps a scratch that maps to scratches is a better way? Is that even allowed?

[bep]

@swamidass

That doesn’t work because there is only one scratch for the whole site,

What I’m suggesting is that you get rid of all of that global scope.

If:

{{ $pagesAndStats := partialCached "byviews" . }}

If the aboe partial returned all you needed, it would not matter too much if it was executed 1 or 5 times.

[bep]

A related tip: There was a discussion earlier where one wanted to run something once after the build, which can be accomplished by defining an output format for the home page and put it last. You can do something similar by putting it first.

[swamidass]

@bep

What I’m suggesting is that you get rid of all of that global scope.

I understand this. But that doesn’t take away the need to have one scratch per page, not one for the whole site.

@bep

There was a discussion earlier where one wanted to run something once after the build, which can be accomplished by defining an output format for the home page and put it last. You can do something similar by putting it first.

This is interesting.

But won’t single page outputs be compiled concurrently with the home page? It seems that this only works in ensuring the ordering for a single page type (e.g. the index), but not ordering across the whole site. If so, that won’t work.

[jmooring]

@bep

There was a discussion earlier

https://discourse.gohugo.io/t/is-it-possible-to-have-a-content-type-build-last/34917/3