Work-around for making pages from data?

Can you post a sample of your data? If I understand you correctly I think you want to make a MD file with some param that you can match to your data file and then load that particular records data to populate the page.

Does that sound right?

Hugo will look for Markdown files to generate pages. There is no way around that. (until #5074)

Now suppose you have an API endpoint per each page you need to generate. All you’d have to do is script a solution which will create an MD file for each pages from your data source and simply add one Front Matter parameter to it, the endpoint:

---
endpoint: /page/1564/index.json
---

Then from their layouts/single.html, you can use getJson on this endpoint to fetch your data. This is not ideal but pretty reliable for now.

Yea that is definitely similar to what I’d like to do. A great place to start for sure, thank you.

A related tip: If you use the Hugo Archetypes you can use getJSON and similar in that to script content.

4 Likes

I don’t actually have the data, I’m spitballing and thinking of a way to make a solution… you correctly understood my intent. I’ll have the .md file so the page generates, but no content. Just frontmatter. and in that frontmatter it’ll have some parameter (or as a previous comment suggested, using the filename as that parameter) to pull fields from the data file where the data unit matches the parameter or filename, and then populate the page with that content.

I did something similar for our team members page as seen here: https://perchsecurity.com/team/

I have a csv file with all our team members their bios linkedin etc. an entry in the csv looks like this:

Ben Marte,Nudist Developer,Sr. UI/UX Engineer,https://www.linkedin.com/in/benmarte/,"Ben Marte is a product of the dot com bubble era. He's worked with various companies in many industries throughout the years developing all sorts of products from web, print, video, 3D printing you name it he's probably done it. When he's not coding he's most likely playing video games in his boxer shorts all day and wishing he lived in Iceland."

I then use this for rendering a users bio on blog posts: https://perchsecurity.com/perch-news/what-were-gonna-do-right-here-is-go-back-wayback.../

Here’s what that looks like in my single template:

{{ if isset .Params "author" }}
        {{ $author := $.Params.author }}

        {{$team := getCSV "," "csv/team.csv"}}
        {{range $i, $r := $team}}
          {{if gt $i 0}}
            {{if eq $author (index $r 0)}}
              {{partial "author" (dict "name" (index $r 0) "title" (index $r 2) "linkedin" (index $r 3) "bio" (index $r 4))}}
            {{end}}
          {{end}}
        {{end}}
{{end}}

And this is what my author partial looks like:

<div class="flex-row around author">
  {{ if eq .type "contractor"}}
    <div class="flex-col">
        <p><b>{{.name}}</b> - {{.title}}</p>
        <p>{{.bio}}</p>
    </div>
  {{ else }}
    {{if .bio}}
        <p class="text-center">
            <img src="/images/team/{{.name | urlize}}.jpg" alt="{{.name}}" class="author-img">
        </p>
        <div class="flex-col">
            <p><b>{{.name}}</b> <br> {{.title}}</p>
            <p>{{.bio}}</p>
            {{ if .linkedin }}
                <p>
                    <a href="{{.linkedin}}">LinkedIn</a>
                </p>
            {{ end }}
        </div>
    {{else}}
        <div class="flex-col">
            <div class="author-no-bio">
                <p class="text-center">
                    <img src="/images/team/{{.name | urlize}}.jpg" alt="{{.name}}" class="author-img">
                </p>
                <p class="text-center">
                    <b>{{.name}}</b>
                    <br>
                    <span>{{.title}}</span>
                    <br>
                    {{ if .linkedin }}
                        <a href="{{.linkedin}}">LinkedIn</a>
                    {{ end }}
                </p>
            </div>
        </div>
    {{end}}
{{ end }}
</div>

So you can use the same approach just add an if statement where it matches your frontmatter param with a param in the data file and it should work.

Whats great about this approach is our content is version controlled and anyone in our company can edit the csv file in order to update team members, investors testimonials etc.

Hope that helps :slight_smile:

2 Likes

Really cool, thanks!

My goal is to make a sort of… product display. Not a store integration, but a product display which links to the product on an external website. Instead of making a content page for each item, i figured I could have all of that in a structure in the data file, and just reference it to build the page.

After looking at the archetypes docs… I think I know what you mean but not sure how that works exactly since when creating the content file based on the archetype, it wouldn’t know what values to search the data file with… or does it? And are you saying that’s a way that you use pull in data into the content file instead of using a template/partial to do it?

And something else I thought about… can taxonomies only be added to posts as long as they exist in that posts front matter at build time? For example, I couldn’t list the taxonomies I want for a particular item in that item’s data structure and have it be assigned the taxonomy, only unless that’s some function added to the archetype and you do the getJSON as you described, right? So, if I want to have my posts/products have various taxonomies, then those have to be in the frontmatter, full-stop, yea?

No way to add taxonomies to the post given your exact example though right? (I mean, unless you have your script generate those frontmatter taxonomy entries based on the data file?)

also, why JSON, why not toml or yaml?

You have to tell it. Basically 2 options:

  1. By convention (random element, use the day number, whatever)
  2. Passing what you want as a OS env, e.g. QUERY="name = 'foo'" hugo new "post/foo.md" and usegetenv` in your archetype template
1 Like

Ha! Thanks so much for that @bep :+1:

I wouldn’t have thought of using Hugo environment variables this way!


@gaetawoo

Here is a working example based on the code snippets given in this excellent topic: [Solved] CSV data key lookup

Contents of /data/abc.csv

id, description, color, number, cat, image
110, blue pearl, blue, 10, pearls, bpearl.jpg
111, green pearl, green, 12, pearls, gpearl.jpg
112, orange peel, orange, 19, fruit, opeel.jpg

Contents of archetype under /archetypes/post.md

{{- range $i, $r := getCSV "," "data/abc.csv" -}}
{{- $id := index $r 0 -}}
{{- if eq (getenv "HUGO_TITLE") $id }}
---
title: "{{ index $r 0 | title }}"
---
{{ end -}}
{{- end -}}

Basically withe the above we are telling Hugo to create a markdown file only if the environment variable value equals one of the values of the first column cells of the CSV.

And here is the way to create separate markdown files from a CSV with Hugo only:

Just paste the entire list of commands in the terminal and Hugo will magically create the separate files (you will only need to press ENTER for the last file:

env HUGO_TITLE="110" hugo new "post/110.md"
env HUGO_TITLE="111" hugo new "post/111.md"
env HUGO_TITLE="112" hugo new "post/112.md"

Enjoy! :tada:

P.S. Of course you could do something similar with a JSON.

6 Likes

Thanks! That shows a lot of power. I guess you could actually do the content that way too if one so chose. I’ll have to think through my options now that I have a few ways to approach.

I seem to recall @budparr writing a slick blog post on how to use csplit to create individual markdown files from data as well, but I can’t find it on The New Dynamic for some reason. Maybe he has some recommendations as well.

Mentioned here, btw:

The above technique will output various markdown files from a single CSV file when using the command hugo new. After that you will have a pretty conventional content directory.

Also you could use some automated RegEx in your favorite editor to generate the list of terminal commands. :wink:

That article is 404. The other technique I was using up until today involved the csplit command indeed and the article that documented it is over here:

But the above technique with Hugo Environment Variables and Archetypes, feels way more natural, since it involves less commands and less messing about with the CSV and it can be adapted for use with a JSON very easily.

Anyway @rdwatters Hugo has become quite powerful and we haven’t quite realized.

1 Like

I just want to say that I have used the above technique with a complex CSV to split it in various markdown files and it works flawlessly.

I did encounter a couple of caveats that are unrelated to Hugo but I managed to fix them so I’m posting these fixes here in case someone else encounters them:

  1. When exporting a CSV from Excel all UTF-8 characters turn into question marks. There are involved hacks circulating the web that seem like overkill.

Solution: Ditch Excel and use Sublime Text’s Advanced CSV Package

  1. However when working with Advanced CSV it stripped quotation marks whenever I justified the columns.

Solution: Disable auto_quotes in settings of Advanced CSV.

Tip 1: To use a comma in a description cell of your CSV you need to escape it by quoting the entire cell.

Tip 2: You can use Hugo functions outside the getCSV range to fetch whatever resources you may need and have them rendered as front matter parameters.

Tip 3: The Hugo new command can create markdown files in the specified PATH (Big time saver)

I am very happy with the above. It’s pretty fast and efficient and it shows the power of Hugo with Data.

Again my big thanks to @bep for suggesting Environment Variables.

1 Like

What exactly do you mean by this (other than what you’ve written in the code example above)?

For example if you have an Archetype called orange.md

Then you can do:

hugo new "/orange/varieties/belladonna.md"

And belladonna.md will be created in the above PATH.