Ranging over collection of data files

support
1

Hello everyone,

I am trying to make a bibliography, by looping over a collection of .yaml files saved in a folder “data/articles”. Each of these files corresponds to one article and contains different variables such as “title” and “author”. For example, the file Lewis-2024.yaml looks like:

type: article
title: "One of the best articles in the world"
author: Lewis
year: 2024

When I use the following to make a list of available titles, everything works:

{{ range .Site.Data.articles }}
  <p>Title: {{ .title }}</p>
{{ end }}

However, when I try to filter this list, the following commands do not seem to work:

{{ range first 10 .Site.Data.articles }}
  <p>Title: {{ .title }}</p>
{{ end }}

{{ range where .Site.Data.articles "Params.author" "Lewis" }}
  <p>Title: {{ .title }}</p>
{{ end }}

This is really puzzling, as this looks like a trivial example and I am following the examples of the documentation (e.g., where | Hugo for the second one). I must be missing something very basic here… Any clue?

Thank you very much in advance for your help.

2

Could you please post the structure of the articles.yaml file.

3

Yes, I have edited my initial question with more details. “articles” is a folder that contains several .yaml files. I am providing a simplified example of such a file here. In my project each of these files contains a lot of information, that’s why I prefer keeping them as separate files instead of one single file. Thank you for your help.

4

What does hugo server -verbose say?

5

Running hugo server --verbose in the terminal provides the following error (for the example with first 10):

ERROR 2018/09/29 17:12:26 Error while rendering "section" in "bibliography/": template: bibliography/section.html:16:9: executing "main" at <first 10 $.Site.Data....>: error calling first: can't iterate over map[string]interface {}
WARN 2018/09/29 17:12:26 template: bibliography/section.html:16:9: executing "main" at <first 10 $.Site.Data....>: error calling first: can't iterate over map[string]interface {}

I guess that it works in the original example because range can loop over a map (which I seem to be dealing with here), while first and where can only work with an array.

Is this the explanation? (Slowly learning here, sounds counter-intuitive at first… if it is possible to loop over a map, why not also loop with conditions?) If yes, what would be a simple fix?

6

:face_with_monocle: … that needs some more debugging. range over .Site.Data.articles works, as it is a collection. But first should work too, as it iterates over the same collection. Try to print the content of this collection to the screen:

{{ printf "%#v" .Site.Data.articles }}
7

Now that I know where to look, the problem becomes more obvious. There have been many posts about this, see e.g. here.

What still puzzles me is that blog posts are usually saved as individual files in a content folder, and then the functions range can be used together with first and where to create appropriate lists. What is so different here? Why is a map produced, which cannot be processed by these two functions?

The {{ printf "%#v" .Site.Data.articles }} you suggested produces something like this:

map[string]interface {}{"one_article":map[string]interface {}{"type":article, "title":"Best article of the world", "author"="Lewis"}, "another_article":map[string]interface {}{"type":article, "title":"Second best article of the world", "author"="Lewis"}, "another_one":map[string]interface {}{"type":article, "title":"Yet another article", "author"="John"}}

8

@remek Can you possibly link us to the full source for the project? I have some ideas on how to help but it will be much easier to run it locally. If you’re worried about a private project, please feel free to DM as well. Thanks!

9

A good nights sleep helped to think about this.

range accepts maps, arrays and slices, first and where only accept an array or a slice.

The data in the data/ folder will be accessible as a map in the .Site.Data variable. A map by Golang definitions is an unordered container for data, the order within that container is quite random.
Posts in the content/ folder on the other side will be parsed into a slice of type Pages []*Page with a default order Default: Weight > Date > LinkTitle > FilePath, if not defined otherwise.

This makes sense in the context of first, as you want to reliably retrieve the first n elements of a slice. As the content of a map is rather random, first would do the same as any (if that function would exist), as you don’t know which elements come back. I don’t understand, why where would need a sorted data array, as a mixed map would suffice, but that is another topic.

And here comes sort to the rescue, which sorts maps, arrays, and slices and returns a sorted slice. Try sorting the data first before filtering:

{{ range first 2 (sort .Site.Data.articles) }}
    <p>Title: {{ .title }}</p>
{{ end }}

{{ $sorted := sort .Site.Data.articles }}
{{ range where $sorted "author" "Lewis" }}
    <p>Title: {{ .title }}</p>
{{ end }}

:raised_hands:

7 Likes
10

The above code example belongs in the Hugo Docs. Please do send a PR.

Also thank you for the thorough explanation. :+1:

1 Like
11

Thank you very much for the details and the thorough explanation! This completely addresses and solves my problem.

It would be great if this could go in the docs, as the problem is not that obvious, and it looks like a lot of people have been struggling with it.

@rdwatters: Thanks for your help! This is a private project so far, I have not posted online (yet). But the problem has now been solved.

2 Likes