Support for Link Header field in GetRemote (used for API pagination)

feature
1

Currently GetRemote returns a limit set of Response Headers.

Some REST API utilize Paginating support via Resonse Headre Link field

example for GitHub REST API

< Link:
<https://api.github.com/repositories/11180687/issues?page=1&per_page=2>; rel="prev",
<https://api.github.com/repositories/11180687/issues?page=3&per_page=2>; rel="next",
<https://api.github.com/repositories/11180687/issues?page=233&per_page=2>; rel="last",
<https://api.github.com/repositories/11180687/issues?page=1&per_page=2>; rel="first"

support for this would be nice in the GetRemote method to add the Link header field if it exists.
This would allow much easier collecting paginated content.

What do you think?

  • did I miss it somewhere and its possible
  • worth an issue?
  • workarounds: manual counting returned values, construct page links manually, uh
  • graphql - but this needs authentication even for public info
2

I don’t think that we have access to the headers on a GetRemote request, and yes, I think it’s worth a feature request in the repo. It’s not nice of the API in question though, to “hide” this information in the headers, that though is a completely different topic.

But I guess it’s important too for things like rate limiting (which in all my projects is done via header) and some exotic features like content type… yeah, let’s have headers in an array to range through if required.

+1 from me.

3

Yes.

I originally just added what I thought would be “useful” in the .Data, but we could certainly expand on that.

This is what we have today:

func responseToData(res *http.Response, readBody bool) map[string]any {
	var body []byte
	if readBody {
		body, _ = io.ReadAll(res.Body)
	}

	m := map[string]any{
		"StatusCode":       res.StatusCode,
		"Status":           res.Status,
		"TransferEncoding": res.TransferEncoding,
		"ContentLength":    res.ContentLength,
		"ContentType":      res.Header.Get("Content-Type"),
	}

	if readBody {
		m["Body"] = string(body)
	}

	return m
}
1 Like
4

created issue:

Maybe other want to opt in if they also need a new field in the response

5

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.