Image download unreliable: some of the images are broken

support
,
1

Hi, I am using content adapter to

  • make a call to an API to get a list of resources
  • for each resource, make another call and get some metadata
  • for each resource, construct an image path for a IIIF server, get the image, and store it as a resource for each page.

All works as expected for small collections (up to 10 items). For a call of 60, say, 5 or 10 of the images will be unreadable. Different ones each time. In server logs, there is no record of any problems. I am not sure what’s wrong with the images; they have the correct filesize and are named and placed appropriately. When I view that image via a IIIF api call, they render fine.

I clear the central hugo cache frequently.

I’ve spent several days on this and am really at a loss. Any help greatly appreciated.

I just upgraded to Hugo 0.138.0 but problem persists.

{{/* Get remote data. */}}
{{ $data := dict }}
{{ $docs := dict }}

{{ $api_collection := "https://datasite/viewer/api/v1/objects?collection=swkr&rows=30" }}
{{ $apiurlbooks := "https://datasite/viewer/api/v1/books/" }}
{{ $iiifpath := "https://iiifserver/api/image/books/" }}

{{ $opts := dict
  "method" "get"
  "headers" (dict
    "Content-Type" "application/x-www-form-urlencoded"
  )
  "body" `{"complete": true}`
}}

{{ with resources.GetRemote $api_collection $opts }}
  {{ warnf " URL for api_collection is: %s" $api_collection  }}
  {{ with .Err }}
    {{ errorf "Case 1: Unable to get remote resource %s: %s" $api_collection . }}
  {{ else }}
    {{ $data = .Content | transform.Unmarshal }}
    {{ $docs = index $data "response" "docs"  }}
  {{ end }}
{{ else }}
  {{ errorf "Case 2: Unable to get remote resource %s" $api_collection }}
{{ end }}

{{/* Add pages */}}
{{ range $docs }}
  {{ $uri := urls.JoinPath $apiurlbooks .identifier  }}
  {{/* Get remote data: a call for each book */}}
  {{ $opts := dict
    "method" "get"
    "headers" (dict
      "Content-Type" "application/x-www-form-urlencoded"
    )
    "body" `{"complete": true}`
  }}

  {{ with resources.GetRemote $uri $opts }}
    {{ with .Err }}
      {{ errorf "Case 1: Unable to get remote resource %s: %s" $uri . }}
    {{ else }}
      {{ warnf "Reading : %s" $uri  }}
      {{ $data := .Content | transform.Unmarshal }}
      {{ $title := index $data "title"   }}

      {{ $date := time.AsTime (index $data "metadata" "publication_date" "value") }}
      {{ $publication_date_text := index $data "metadata" "publication_date_text" "value" }}
      
      {{ $dates := dict "date" $date  "publication_date_text" $publication_date_text }}
      {{ warnf "date : %s" $date  }}
      {{ $subjects := index $data "metadata" "subject" "value" }}
      {{ $identifier := index $data "identifier" }}
      {{ $path_alias := path.Join "/books" $identifier }}
      {{ $params :=  dict 
        "dates" $dates
        "datebook" $date
        "ss_book_identifier" $identifier
	"identifier" $identifier
        "path_alias"  $path_alias
        "subjects"  $subjects
       }}
      
      {{ $page := dict
        "title" $title
        "dates" $dates
        "kind" "page"
        "path" $identifier
        "params" $params
      }}
       {{/* Add page. */}}
      {{ $.AddPage $page }}
    {{ end }}
  {{ end }}

  {{/* Retrieve and add cover images */}}
  {{ $item := . }}
  {{ with $thumbnailurl := urls.JoinPath $iiifpath $item.identifier "/1/full/,400/0/default.jpg" }}
	{{ $opts2 := dict
    "method" "get"
    "headers" (dict
      "Content-Type" "image/jpeg"
    )
    "body" `{"complete": true}`
  }}
    {{/* Important: Placing the image inside the book directory seems to be the trick that associates the Resource with the Page */}}
    {{ $newpath := printf "%s/%s.jpg" $item.identifier  $item.identifier }}
    {{ with resources.GetRemote $thumbnailurl $opts2  }}
      {{ warnf "thumbnailurl  %s" $thumbnailurl }}
      {{ with .Err }}
          {{ errorf "Case 1 content adaptor Unable to get remote resource %s: %s" $thumbnailurl . }}
      
      {{ else }}
        {{ $content := dict
          "mediaType" .MediaType.Type
          "value" .Content
        }}
        {{ $resource := dict
          "content" $content
          "path" $newpath
        }}
        {{ $.AddResource $resource }}
        {{ warnf "Resource added %s " $resource.path }}
      {{ end }}
    {{ else }}
        {{ errorf "Case 2 content adaptor Unable to get remote resource %s" $thumbnailurl }}
    {{ end }}
  {{ end }}
{{ end }}