An alternative to making page resources?

support
1

Short version:
Is there an alternative to putting resource files in the page bundle directories / having hugo copy the files on each build?

Longer version:
This is less of a “I can’t make it work” and more of a “what are the best practices here.”

I am building a site where there are lots of pages (right now there are page bundles) with big resources. Basically, each page represents a topic with a bunch of PDF files. The page describes the topic, and the templates build nice lists of all the PDF files that are resources.

Everything works great on my local machine. On the server, dealing with all of these large files is problematic. It might be the file system on the server I am using (it is AFS), but I get no control over that. Not only do site builds take a long time, they often fail (either with a “bus error” or “interrupted system call” - examples at the end of this note). Plus, it seems wasteful to do all the excess copying.

I would also prefer to have these files in a different place (the site is maintained using GIT, and managing these large binary files in GIT is not ideal for my desired workflow).

Hugo doesn’t do anything with the contents of these files other than copy them. I do make use of the list of page resources (for example, to generate a list of links to them). For each resource, I get the name, get a link to the item, …

Is there a good alternative to having page resources? Is there a way that I could add a “resource-like” file (or URL) to the resources list in the page front matter? Are there best practices for dealing with these kinds of large resource files?

Here is an example of what I am trying to achieve:
http://pages.cs.wisc.edu/~gleicher/HugoTest/talks/
It does work well - when the sever actually builds things (which it did this one time).

I appreciate any suggestions. I guess I could just be happy that I have gotten it to work, but I’d like to make it be happier with my server infrastructure.

Thanks!

Mike

Often, when I build, I get something like…

ERROR 2020/06/28 15:32:47 Failed to publish Resource for page "/u/g/l/gleicher/private/Hugo/HugoHome/content/talks/2004_11_gametech/index.md": read /u/g/l/gleicher/private/Hugo/HugoHome/content/talks/2004_11_gametech/Gdsem-notes.pdf: interrupted system call

or…

unexpected fault address 0x11a2fe9
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x11a2fe9 pc=0x11a2fe9]

goroutine 1 [running, locked to thread]:
runtime.throw(0x1d9d409, 0x5)
        /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc000db25e8 sp=0xc000db25b8 pc=0x4f1712
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:692 +0x443 fp=0xc000db2618 sp=0xc000db25e8 pc=0x508203
github.com/aws/aws-sdk-go/aws/endpoints.init()
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.27.1/aws/endpoints/defaults.go:3087 +0x88839 fp=0xc000dbfe98 sp=0xc000db2618 pc=0x11a2fe9
runtime.doInit(0x306b800)
        /usr/local/go/src/runtime/proc.go:5414 +0x8a fp=0xc000dbfec8 sp=0xc000dbfe98 pc=0x50097a
runtime.doInit(0x3070000)
        /usr/local/go/src/runtime/proc.go:5409 +0x57 fp=0xc000dbfef8 sp=0xc000dbfec8 pc=0x500947
runtime.doInit(0x3078040)
        /usr/local/go/src/runtime/proc.go:5409 +0x57 fp=0xc000dbff28 sp=0xc000dbfef8 pc=0x500947
runtime.doInit(0x307d340)
        /usr/local/go/src/runtime/proc.go:5409 +0x57 fp=0xc000dbff58 sp=0xc000dbff28 pc=0x500947
runtime.doInit(0x3064e80)
        /usr/local/go/src/runtime/proc.go:5409 +0x57 fp=0xc000dbff88 sp=0xc000dbff58 pc=0x500947
runtime.main()
        /usr/local/go/src/runtime/proc.go:190 +0x1ce fp=0xc000dbffe0 sp=0xc000dbff88 pc=0x4f3d4e
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000dbffe8 sp=0xc000dbffe0 pc=0x524331
2

May be something like https://www.netlify.com/products/large-media/ ?

3

Thank you. That solves the storage/management piece, but not the Hugo piece.
How do I attach a “foreign file” (or list of them) to a Hugo page so that I can easily iterate over the list and place them appropriately on the page?

4

Can you iterate over a data file to display the links to PDFs (or images) using:

Or include a slice of links in your frontmatter?

5

Thank you.

The data file thing is a good idea - but it breaks my organizational goal of keeping everything together. A page bundle allows everything about the object to be in one place (and news things can be added locally). As far as I understand, data templates have to go in the data directory. If there was a way to have a yaml file in the page bundle…

The frontmatter list was my last resort… but I did it. I put a list of URLs in frontmatter, and string processed it to get the name (which taxed my go skills). Not nearly as magical as just dropping files into the page bundle, but it does work.

        {{ with .Page.Resources.Match "*.pdf" }}
            {{- range . -}}
            <a href="{{ .RelPermalink }}" class="inlinebtn">{{ .Name }}</a>&nbsp; 
            {{- end -}}
        {{- end -}}
        {{ with .Page.Params.extpdfs }}
            {{- range . -}}
            {{ $name := split . "/" | last 1 }}
            <a href="{{ . }}" class="inlinebtn">{{ index $name 0 }}</a>&nbsp; 
            {{- end -}}
        {{- end -}}
6

I didn’t used it personally, but my understanding is that it is neutral vs your code/Git.

7

Directory structure:

content
├── post
│   └── test
│       ├── data.yml
│       └── index.md
└── _index.md

content/post/test/data.yml:

- base_url: "https://www.example.com/adobe-acrobat"
  ext: "pdf"
  filename: "file_number_one"
  link_text: "My Adobe Acrobat File"
- base_url: "https://www.example.com/ms-word"
  ext: "docx"
  filename: "file_number_two"
  link_text: "My Microsoft Word Document"
- base_url: "https://www.example.com/ms-excel"
  ext: "xlsx"
  filename: "file_number_three"
  link_text: "My Microsoft Excel Spreadsheet"

content/post/test/index.md:

{{< iterate-over-data-file >}}

layouts/shortcodes/iterate-over-data-file.html:

{{- $data := .Page.Resources.GetMatch "data.yml" | unmarshal -}}
<ul>
{{- range $data }}
  <li><a href="{{ printf `%s/%s.%s` .base_url .filename .ext }}">{{ .link_text }}</a>
{{- end }}
</ul>

Try it:

git clone --single-branch -b hugo-forum-topic-26569 https://github.com/jmooring/hugo-testing hugo-forum-topic-26569
cd hugo-forum-topic-26569
hugo server
1 Like
8

Oh, that’s awesome. Thank you. Not only because it gives a specific answer, but because I learned so much from reading your code!
I probably would never have found unmarshal - which looks incredibly handy!

9

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.