Help with getJSON: unexpected end of JSON input

[aormsby]

I’m having basically the same issue where I want to use getJSON in a shortcode to reference a videos.json file at a relative URL. Here’s a short recreation.

Directory:

my-post
_ index.md
__ img
___ videos.json
___ other image files

I range through the img dir and explicitly pick out the videos.json file.

{{ range .Page.Resources.Match ( printf "%s%s%s" "*/" (.Get "src") "/*") }}
    {{ if eq (path.Base . ) "videos.json" }}
        {{ $videos := getJSON ( . ) }}

The same error:

Failed to get JSON resource "img/videos.json": unexpected end of JSON input

I really wish I understood why.

(From bep)
That said, I would recommend you put these types of JSON files into /data,

While that’s a good solution for many cases, I’m using this to populate an image gallery, and I want to keep the image and video content bundled with the post itself to maintain content portability and ease of use.

Is there anything you might recommend I try here? If I can just get the JSON data to read in properly, I should be fine moving forward.

EDIT: Shortly after I posted this I tried using .Permalink and that seems to work for the moment. However, I’m unsure if that’s an ideal solution. I’d still like some insight on the proper way to go about this. Also, I’m running all these tests on the local server, and I don’t know if that affects how something like .RelPermalink works or (currently) does not work in this environment.

EDIT 2: I really just want to be able to use the relative URL that seems to work fine under other circumstances like this.
{{ $videos := getJSON ( . ) }}
The . here would be the resource URL being used inside the range I used above, but it does not work with the getJSON function. Is it possible that only absolute site URLs are accepted with this function at this time? It would seem like a bit of an oversight if I couldn’t read in a JSON file using a relative URL.

EDIT 3: And now that I’ve realized I need set the json cache to 0 if I ever want it to refresh the data, I have a new problem where starting the local Hugo server actually fails because (I think) it can find neither the previously cached data nor the hosted file (since the host is not yet running). I gotta say, this is starting to look like a hopeless mess.

[pointyfar]

Hi,

I’ve moved your question into its own topic as the one you were replying to has been solved, is 2y old, and does not seem to be a similar issue beyond the use of getJSON.

It would be easier to get help here if you were to share your site code so that we can replicate your issue. Have a read here about Requesting Help .

[aormsby]

Well, it started out as the same issue, but it really snowballed the more I experimented with it. In the end, I did get things working eventually, and most of the problem came from misunderstanding some of the ways things worked (along with that somewhat strange caching situation). At the moment, I don’t have my notes in front of me - I may follow up with a working code sample later just to show a solution for others.

What confused me the most was probably how getJSON reacted differently to file paths than other functions (such as fileExists). They each required a different file path to get the expected results, which would imply they’re starting their search from different locations. While that’s not necessarily a bad thing, it’s also extremely unclear that it’s happening. While I do really enjoy working with Hugo, I do lose time to confusing things like that quite often.

[alexandros]

fileExists looks for files in the local file system

getJSON can fetch a data file from the local data folder or from an external URL and it takes variadic arguments.

So these two functions do have different syntax for a good reason.

Also you can always clear the cache by using the --ignoreCache CLI flag.

[aormsby]

fileExists looks for files in the local file system

That’s all well and good, and I’ve been using it to some extent to get information when using getJSON. Unfortunately, the results of the two functions are often unequal. For example, I can easily find a file in a relative position using fileExists, but then getJSON doesn’t work to retrieve it However, I don’t think this is fileExists fault.

Instead, I think the issue may be in this assumption –

getJSON can fetch a data file from the local data folder…

I’m not so sure it can. I have only been successful using getJSON with full URLs, so maybe there is a bug here somewhere. Passing in local directory data - either relative or absolute - does not retrieve the JSON file.

Doesn’t work → (from site root) /projects/armydillos/ArmyDillos-web/Build/ArmyDillos-web.json
Works → https://www.adamormsby.com/projects/armydillos/ArmyDillos-web/Build/ArmyDillos-web.json OR https://localhost:1313/projects/armydillos/ArmyDillos-web/Build/ArmyDillos-web.json

Oddly enough, getJSON functions run during site builds as well. Since I’m using full URLs and can’t use local path data with getJSON, I get errors like this one (this JSON file had not been pushed to the external site yet.)

Building sites … ERROR 2020/03/03 16:48:29 Failed to get JSON resource “https://www.adamormsby.com/projects/armydillos/ArmyDillos-web/Build/ArmyDillos-web.json”: Failed to retrieve remote file: Not Found

However, the published site with the new file works fine. If I could just use directory paths, I wouldn’t have this issue.

Is there just something wrong with getJSON?

[alexandros]

Please do share a repo as stated in the Requesting Help guidelines.

I am using getJSON on a daily basis to render data from JSON files that reside under the /data/ folder.

Something else is going on that is due to your setup. The function works as intended for me.

P.S. Is your JSON valid? There are online tools to check if it is.

[aormsby]

Okay, I think we’re not on the same page here.

Yes, my JSON is valid. However, I’m not using the /data/ folder as I mentioned in my initial post. However, I can see that was easier to understand before the post got moved, so I’ll restate the issue a bit clearer here.

I would like to access .json files inside my post bundles. Not the /data/ folder. My reason for this is that in some cases the post and its data are more easily organized and transferrable this way.

For example, I might have a directory structure like this.

posts/
_ my-post/
__ index.md
___ subfolder/
____ mydata.json

According to the getJSON docs, it should be possible to access a local json file (without using /data/). However, I can only access the file using a full URL, not a local path.

This is the issue. Something in the way getJSON is looks for the path doesn’t work. This is not a case where I want to use the /data/ folder.

Maybe I should just make a Github issue about it.

[alexandros]

@aormsby

I would like to access .json files inside my post bundles. Not the /data/ folder. My reason for this is that in some cases the post and its data are more easily organized and transferrable this way.

Then getJSON will not work as you’ve seen. You are making the call for a file residing in the contentDir

As per the Docs link you posted:

The local CSV files to be loaded using getCSV must be located outside of the data directory.

That is for CSV. Not about JSON.

@aormsby

Maybe I should just make a Github issue about it.

What you describe in this topic is a new feature request and not a bug, so do make sure to reflect that if you open an issue on GitHub.

The Docs are clear.

[pointyfar]

But why even use getJSON at all? Your json file is already in your page bundle, you could just do

{{ with .Page.Resources.GetMatch $jsonfile }}
  {{ .Content | transform.Unmarshal }}
{{ end }}

It would be easier to help if you also mention what you want to accomplish (more than just "I want to getJSON a json file-- which is how you are trying to do it.).

[aormsby]

I’m not sure you’ve read the docs closely enough. Here are the exact words that support my case being valid and explicitly states it should be functionally possible.

[aormsby]

@pointyfar

{{ with .Page.Resources.GetMatch $jsonfile }}
{{ .Content | transform.Unmarshal }}
{{ end }}

Because honestly, I had no idea I could do that. I see now that there’s a good page on this in the docs, but I never ran across it since I didn’t know that’s what I was looking for. I’ll give it a shot.

mention what you want to accomplish

Actually, working with the data has been super easy! I just use it to populate some HTML in my template. Just accessing it in the first place has been an odd challenge, so that’s really my goal at this point.

[pointyfar]

BTW,

{{ $json := getJSON "/content/some/folder/data.json" }}

works perfectly fine from within a shortcode. If it does not work for you, there is something wrong with how you are doing things. We can’t help if we can’t replicate your issue.

[aormsby]

Yes, that’s correct on a local build. Unfortunately, there is no /content/ directory in the /public/ folder of the site after you build it, so that path wouldn’t exist in the live site directories. So wouldn’t that only work on a local build then?

I’m setting up a test case, and I’ll post when available.

EDIT:
Unless I’ve read the docs header ‘Load Local Files’ and incorrectly misinterpreted it to mean ‘Load files in the relative directories without a full URL’. That would certainly change my perspective on things.

[aormsby]

Which statement is more accurate?

A) getJSON is meant to be used on my local file directory, not a built site.
B) getJSON can be used to access local files on a built, published site.

[pointyfar]

I don’t understand what your question is.

getJSON does its “magic” while your site is being built. Once your site is built, getJSON has nothing to do with it anymore. You will have your static pages with whatever information you pulled out of your json.

Do not think of it like jQuery getJSON, which does the get-ing after the site is built.

[gosho]

That’s a nice one.
However, given that content directory is configurable and not necessarily in hugo working directory, is there any function at all that can be used to fetch content that does to reside in hugo?

Some context: I have distinct repositories/direcotries for the hugo generator and the content. I do a pre-build step with nodejs and fetch commits jsons from github for each .md because the gitinfo object scenario is too limited for my case. The jsons are stored side-by-side with the mds in the content directory. Then I get them with GetJSON in several partials. And it was fine until the moment I decided to instead of copy the content in hugo to employ --contentDir flag and use it from where it resides.

Part of the problem is how do you dynamically supply the name of the content directory to such function if that is necessary for it to function properly? In the case of GetJSON I had to start my paths with content, but as soon as I considered using --contentDir referring to an external directory, it started to fail for obvious reasons. And there seems to be no way of telling the content directory dynamically, making the code quite dependable on env changes with hardcoded paths here and there.

I suppose that instead of storing the jsons i want to read side-by-side with the markdowns they correspond to, I could store them in the data folder, making them available to GetJSON on a well-known base path. But I was hoping for a less labor intensive solution coming directly from Hugo.

The constraints we discuss here if we use --contentDir referencing a directory outside hugo are not only on GetJSON. ReadDir for example is in the same boat. Perhaps the documentation could emphasize more on that?

[maiki]

@aormsby

Which statement is more accurate?

A: local file, not a built site.

I wish I could definitively say all the ways getJSON works, but I only use it to pull in data from remote, public URLs (I search for and show recent commits from hundreds of repos, and for related posts in some forums).

When I build the site–including hugo server–getJSON goes and grabs the resource at the URL and caches it for access. This of course makes it possible to link to a resource generated by the site itself, from it’s published form online, but I try to avoid those kinds of data loops, because one cache clearing screws everything up.

[maiki]

@gosho

I do a pre-build step with nodejs and fetch commits jsons from github for each .md because the gitinfo object scenario is too limited for my case.

Which scenario are you referring? I pull in commits for a few hundred repos from a self-hosted gitea instance, and of all my getJSON uses that has been the least difficult to get working.

[aormsby]

@pointyfar

getJSON does its “magic” while your site is being built. Once your site is built, getJSON has nothing to do with it anymore.

Ah okay, that’s the part I’ve misunderstood this whole time. I had definitely been thinking of it as something like jQuery’s function like you said. Knowing this, I could likely still use it for my purposes since I just want some static data to be dropped from the JSON files into a template.

However, you also mentioned transform.Unmarshal as an alternative (which does work fine). If they’re both only working during the build, what would be the benefits/drawbacks of choosing either one?

I suppose as maiki stated, the cache clearing issue might be avoided by using the resource matching with Unmarshal in this scenario.

@maiki

When I build the site–including hugo server – getJSON goes and grabs the resource at the URL and caches it for access. This of course makes it possible to link to a resource generated by the site itself, from it’s published form online, but I try to avoid those kinds of data loops, because one cache clearing screws everything up.

[pointyfar]

@gosho

However, given that content directory is configurable and not necessarily in hugo working directory, is there any function at all that can be used to fetch content that does to reside in hugo?

You are probably better off asking your question on a separate topic, with a sample repo attached.