Loading JSON from GitHub

support
1

I am sure this problem will have surfaced already, but I don’t know what exact terms to search for and my searches didn’t yield any results. I am trying to load a JSON file from Github. Those files are delivered as plain/text if I am right and I am receiving the error:

ERROR error calling resources.GetRemote: failed to resolve media type for remote resource "https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/robots.json"

The layout (just trying to get it and debug it for now:

{{ $data := dict }}
{{ $url := "https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/robots.json" }}
{{ with resources.GetRemote $url }}
  {{ with .Err }}
    {{ errorf "%s" . }}
  {{ else }}
    {{/* {{ $data = . | transform.Unmarshal }} */}}
    {{ warnf "%#v" $data }}
  {{ end }}
{{ else }}
  {{ errorf "Unable to get remote resource %q" $url }}
{{ end }}

I can see the section in the docs talking about content type issues but I fail to understand or apply it here. I am using the same code, but as I wrote above, I am pretty sure it’s plaintext, not octet-stream or similar. The error occurs with and without that commented transform.Unmarshal line.

What am I overlooking? Basically, how can I retrieve a remote plaintext and then transform it into JSON?

2

I think what you see can be fixed following the note in resources.GetRemote | Hugo

1 Like
3

That’s what I mean with the section about content type issues. I end up in the error section before that line:

{{ $url := "https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/robots.json" }}
{{ with resources.GetRemote $url }}
  {{ with .Err }}
    {{ warnf "blabla" .}}
    {{ errorf "%s" . }}
  {{ else }}
    {{ $data := .Content | transform.Unmarshal }}
    {{ warnf "%#v" $data }}
  {{ end }}
{{ else }}
  {{ errorf "Unable to get remote resource %q" $url }}
{{ end }}

CLI prints the blabla. I am not getting to the point where I can do something to the retrieved source.

There is probably a way to tell resource.GetRemote what content format to expect. Or the link to the raw files on Github is not supposed to be used.

4

This has been discussed, but there’s no option for this.

We trust that the server Content-Type is correct.

Or the link to the raw files on Github is not supposed to be used.

GitHub has a API that I’m sure reports correct mime types.

5

Changing the URL to this https://api.github.com/repos/ai-robots-txt/ai.robots.txt/contents/robots.json shows the correct mime type in the browsers dev tools. Maybe this works too for GetRemote

But it seems, the file content is wrapped inside the received JSON.

Taken from GitHubs documentation at REST API endpoints for repository contents - GitHub Docs

1 Like
6

There is an almost similar URL in the docs example for Base64Decode

7

@davidsneighbour This works fine for me on both Ubuntu and Windows:

{{ $url := "https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/robots.json" }}
{{ $r := resources.GetRemote $url }}

No errors. Can you test on another platform?

8

I am on Ubuntu, and it returns error calling resources.GetRemote: failed to resolve media type for remote resource "https://api.github.com/repos/ai-robots-txt/ai.robots.txt/contents/robots.json" if I don’t check for an error:

{{ $url := "https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/robots.json" }}
{{ with resources.GetRemote $url }}
{{ . }}
{{ end }}

The API way with $url being https://api.github.com/repos/ai-robots-txt/ai.robots.txt/contents/robots.json also results in the error.

I am getting more and more convinced that I have a weird setup issue going on. Loading all those URLs in a browser results in JSON and proper content types.

I’ll try my luck adding these as modules I guess.

9

Have you tried testing on a different platform, or in a virgin VM?

Have you defined or redefined any mediaTypes in your site configuration?

1 Like
10

That seems to be the reason. I am getting results when I am running the layout on an empty site without other modules. I’ll investigate and write up what I did wrong or what interfered.

2 Likes
11

My JSONFeed was the culprit.

I have a module that offers feeds of all kinds (atom, rss, json). The configuration of the jsonfeed looked like this:

[mediatypes]

[mediatypes.'application/feed+json']
suffixes = ['json']

[outputformats]
[outputformats.jsonfeed]
mediaType = "application/feed+json"
path = ""
baseName = "feed"
rel = "alternate"
protocol = ""
isPlainText = true
isHTML = false
noUgly = false
permalinkable = false

Apparently Hugo did not like that I took over the .json suffix and handled all JSON from here with application/feed+json? After commenting those lines out resources.GetRemote magically started working. I have now removed the new mediatype and moved to mediaType = "application/json" for the outputformat, using the internal one.

That works for now, but won’t fix the issue at hand: The feed needs to be delivered as application/feed+json AND with .json as extension and I have a feeling that once .json is taken it won’t be available for any other mediatype?

Hugo does not much in relation to delivering the file from the browser to the user so a quick solution is probably to just output it as whatever and then regulate the media type via server configuration.

I stumbled over more issues in my understanding of custom mediatypes. I guess I’ll re-read large parts of the docs.

Thanks for your help!

12

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.