Paged Rest API with Hugo v0.143.0

tips & tricks
1

Paged Rest API with Hugo v0.143.0

There’s no automatic following paged results as with Powershell Invoke-RestMethod FollowRelLink. But we now have access to the Response Header fields incl. link. This enables us to implement a partial that returns all items from a paged REST API.

Here’s a quite bare PoC implementation of a getRemotePaged partial that will recursively get all paged results by following the Response Headers next link.

API header

Some APIs require special headers to make the server return valid data, target a specific API version or use authenticated requests.

For GitHub REST API it would be something like this:

{{ $headers := dict
   "Accept" "application/vnd.github+json"
   "X-GitHub-Api-Version" "2022-11-28"
}}

maybe you want to add token authentication to get higher rate limit by using your GitHub token via Environment variable

{{ with getenv "HUGO_GITHUB_TOKEN" }}
   $headers = merge $headers (dict "Authorization" (printf "Bearer %s" .))
{{ else }}
   {{ warnidf "NOTOKEN" "anonymous access only. To use a github token set ENV:HUGO_GITHUB_TOKEN. \
      Will help to overcome rate limits" }}
{{ end }}

Headers Link field

Let’s setup the Response Header so it will give us the Links back needed for paged results. We will also add our API headers from above.

The important field is the Response Headers attribute to the global options.

{{ $opts := dict
   "method" "get"
   "responseHeaders" (slice "Link")
   "headers" $headers
}}

If we now call something like
resources.GetRemote "https://api.github.com/repos/gohugoio/hugo/releases?per_page=50"
we will not only get the JSON back, which can be retrieved via .Content method.

Use the .Data.Headers method to get a list of all headers. You’ll see it includes our requested .Data.Headers.Link field.

{
   "ContentLength": -1,
   "ContentType": "application/json; charset=utf-8",
   "Headers": {
      "Link": [
         "\u003chttps://api.github.com/repositories/11180687/releases?per_page=50\u0026page=2\u003e; \
         rel=\"next\", \u003chttps://api.github.com/repositories/11180687/releases? \
         per_page=50\u0026page=7\u003e; rel=\"last\""
      ]
   },
   "Status": "200 OK",
   "StatusCode": 200,
   "TransferEncoding": null
}

As you can see the links are wrapped in an array with just one element. The value is a string directly from the response header from the server. It follows the specification for that field. (some resources are listed at the bottom - check out the github issue for more details)

Extract Links

We are only interested in the Link field and need to extract these. a structure like this will be nice:

{
   "next": "https://api.github.com/repositories/11180687/releases?per_page=50&page=2",
   "last": "https://api.github.com/repositories/11180687/releases?per_page=50&page=7"
}

We will use an inline partial within getremotePaged. With some range and split magic it will extract our links and return a map as shown before.

it’s called like this partial "getLinks" .Data.Headers

{{ define "partials/getLinks" }}
   {{ $links := dict }}
   {{ with index .Link 0 }}
      {{ range split . "," }}
         {{ with split .  ";" }}
            {{ $link := trim (index . 0) "<> " }}
            {{ $rel :=  index (split (index . 1) `"`) 1 }}
            {{ $links = merge $links (dict $rel $link) }}
         {{ end }}
      {{ end }}
   {{ else }}
      {{ warnf "NO LINK HERE %v" . }}
   {{ end }}
   {{ return $links }}
{{ end }}

Retrieve full results (recursive)

Now let’s just

  • get the first page
  • if there’s a first link
    • call ourself with this next link
    • add the result to all we had before
  • return the combined result
{{ $pages := slice }}
{{ with try (resources.GetRemote $url $opts) }}
   {{ with .Err }}
      {{ errorf ".Err %s" . }}
   {{ else with .Value }}
      {{ $links := partial "getLinks" .Data.Headers }}
      {{ $pages = .Content | transform.Unmarshal }}
      {{ with $links.next }}
         {{ $pages = $pages | append (partial "getRemotePaged" .) }}
      {{ end }}
   {{ else }}
      {{ warnf "Unable to get remote resource %q" $url }}
   {{ end }}
{{ end }}
{{ return $pages }}

Call it from a layout template

with that above as a base you can fetch all data from a paged API and use a standard loop over all objects.


<h2>Hugo Releases</h2>
{{ $url := "https://api.github.com/repos/gohugoio/hugo/releases?per_page=50" }}

{{ $items := partial "getRemotePaged" $url }}
<ul>
{{ range $items }}
   <li><a href="{{ .html_url }}">{{ or .name .tag_name }}</a></li>
   <p>{{ .body | truncate 100 | $.RenderString }}</p>
{{ end }}

Appendix

Working example on GitHUb

You’ll find a more elaborated working example based on the above code on my GitHub repository.

relevant files are

  • layouts_default\release.html
  • layouts\partials\getRemotePaged.html
git clone --depth=1 --single-branch -b getRemotePaged https://github.com/irkode/hugo-forum getRemotePaged
cd getRemotePaged
hugo server

References

2 Likes
2

You need to rename index.md => _index.md in your demo site. But other than that it works great.

3

thx for the hint and props. Fixed

and learned again: always retest with a clean public folder - broke it while simplifying the structure :frowning:

1 Like
4

@irkode Thanks for sharing this neat solution!