I have a use case to retrieve documentation files from a disparate set of remote repos and organize / insert them into my project. Based on a lot of reading in this forum (thank you!), my approach is to module-ize my site and use the mounts feature to pull down the content.
Trying to readFile the contents from the target points is not working as expected and I’m hoping someone can nudge me in the right direction. There was a similar question asked a while back here but no answers unfortunately.
The relevant config for my site looks something like:
module:
hugoVersion:
min: 0.79.0
extended: true
imports:
- path: ../../<theme> <-- i did not change themesDir yet
- path: <remote github.com repo>
mounts:
- source: "doc"
target: "static/gh_docs/"
If I run hugo serve --renderToDisk, I can see the files are being pulled into public/gh_docs as their original md files. (yay!)
However in my shortcode {{ $doc := .Get "file" | readFile }} to insert this file into a page I can only successfully read the file if:
I use renderToDisk
I call the shortcode with the file as “public/gh_docs/<file.md>”
Anything else seems to give a “file not found” error. I was not expecting to have to include “public” in the path as I would have thought Hugo would adjust for this being the root path of the site. Also, I have no idea how to access the files when running the dev server in memory. I tried putting the mount target in “content”, but this seemed to auto-convert my md files into a directory with the same name as the file with an index.html file underneath (not what I wanted). Targeting “assets” instead of “static” didn’t seem to make a difference either.
I expect I don’t fully understand how the virtual filesystem works with mounts such that I can figure out how to specify the file location to read properly. Any hints from someone who has something like this working ?
The file PATH under a Hugo project’s local file system, is different than the .Permalink or RelPermalink outputted by Hugo for a resource or a Page.
readFile works within the local file system not from a remote source.
Currently, as you may have noticed, fetching files from remote sources and having them outputted a pages, is not possible. We are all waiting for #5074 and #6310 to be resolved.
However with the above said, it appears that you have found some kind of a workaround for the current situation.
If --renderToDisk makes the remote sources available under /public/, then perhaps you can build some kind of a pipeline to copy these files in your project’s contentDir and then execute hugo again.
Thanks very much for the reply and suggestions. The distinction between URL references vs. file system is helpful to understand why only the literal local filesystem path worked (and that the file needs to be on disk as opposed to in memory).
My initial thought was indeed to create a custom pipeline to pull down the files and generate content pages from them, but it seemed like there was perhaps enough capability to do it all inside of Hugo.
If the “generate content from data” effort is the right answer for this use case, it may be simplest to just revert back to the custom program/pipeline for now until that becomes available.
I have not read this entire issue, but I suspect the answer may live in mounting the remote files below /assets somewhere, and then use resources.Match etc. to look for files.
Ultimately, yes, I would like to generate the pages from the data to make maintaining easier.
But I was first trying to just embed the remote files with a pipeline like readFile | markdownify in already generated content pages. The difficulty here was figuring out how to actually get hold of the files from the filesystem.
I did try to mount under assets per @bep 's suggestion, but was still trying to use readFile rather than .Match. Will give that a try – thanks again!
Here is another thing you can try -completely untested-.
Set the same folder as the contentDir and the assetDir.
Mount the remote sources to the respective content subdirectories or embed the .Content of those remote .Resources in existing pages with resources.Match by using a Shortcode (for example).
Hopefully once the Markdown files are pulled then Hugo will generate the respective Pages.
Thanks again for the suggestions. I did try a bit further, but couldn’t verify that remote mounts work at all when the target is in assets. e.g., renderToDisk does not show the files anywhere as it does when the target is in static and resources.Match seems to only return nil.
Am I understanding correctly that you want to embed imported repo contents into your pages? I.e. not generate pages from these?
What you can do is mount your repo under content and then .GetPage them, instead of readFile. Perhaps in a subdir, so you can add an _index.md and you can control _build options.
Interesting, thanks for the idea and references @pointyfar. I did try a bit to mount the remote docs under content but was struggling to figure out how to avoid the automatic rendering of the .md files to index.html files – I did not realize there are _build options to modify this behavior. Obviously have still not done enough reading :-). Thanks, I’ll try it again this way.
In the site I work on all of the content from external repos is vendored. We have reusable text files in subdirectories of the content directories. Each page has a gh_repo parameter. I also have site level parameters for each repo:
I use a shortcode with readFile to grab files and markdownify the reusable text files, it checks for the gh_repo page parameter, if it doesn’t find it or if the parameter matches the main github repo it grabs content from the main repo, otherwise it grabs content using the vendor file path.
It’s a bit kludgy, I don’t like adding the gh_repo parameter to each page, but it works.