Search using lunr.js does not return pages/urls when text exist in page sections

support
1

We are building a website using the bigspring theme and we added lunr.js to search the content. We noticed that when the search text is in a blog post, then the page (blog post url) is included in the search results. But when the text exist in a page with many sections then the text is not returned.

For example we have created a “services” section with an all services page “_index.md” and service1.md, service2.md

index.md

---
title: "All Services"
subtitle: "Success catalysts"
# meta description
description: "Company services "
draft: false
# content section
section: "services"
---

service1.md

date: "2021-07-12"
draft: false
title: "Service1"
description: "Contact us"
icon: "fas fa-laptop-house"  
layout: "layout-3"


######################### banner #####################
banner:
  title: "Service1"
  image: "images/products/02.jpg"
  content : "something"
  background_class: "bg-light"
  button:
    enable : true
    label : "Try for Free"
    link : "get-demo/"

      
######################### product_info #####################
product_info:
  enable : true
  title: "New experiments <br> Launch in minutes"
  content: "Some lorel ipsum"
  features:
  - image: "images/products/01.jpg"
    content : "##### For Football Teams

    Adipiscing elit Consequat tristique eget amet, tempus eu at consecttur. Leo facilisi nunc viverra tellus. Ac laoreet sit vel consquat. consectetur adipiscing elit. tempus eu at consecttur.

    <br><br>

    ##### For Basketball teams

    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Consequat tristique eget amet, tempus eu at consecttur. Leo facilisi nunc viverra tellus. Ac laoreet sit vel consquat.
    "

  - image: "images/products/02.jpg"
    content : "##### For Design Teams

    Adipiscing elit Consequat tristique eget amet, tempus eu at consecttur. Leo facilisi nunc viverra tellus. Ac laoreet sit vel consquat. consectetur adipiscing elit. tempus eu at consecttur...

The text eg. “adipiscing” that exists in service1.md is not returned in search results.

Configuration

In config.toml:

[outputs]
home = ["HTML", "RSS", "JSON", "WebAppManifest"]

Code for indexing index.html:

<script>
    window.store = {
        {{ range .Site.Pages  }}
        "{{ .Permalink }}": {
            "title": "{{ .Title  }}",
            "tags": [{{ range .Params.Tags }}"{{ . }}",{{ end }}],
            "content": {{ .Content | plainify }}, 
            "url": "{{ .Permalink }}"
        },
        {{ end }}
    }
</script>
<script src="/js/lunr.min.js"></script>
<script src="/js/search.js"></script>

The search.js

function displayResults (results, store) {
    const searchResults = document.getElementById('results')
    if (results.length) {
    let resultList = ''
    for (const n in results) {
        const item = store[results[n].ref]
        resultList += `
            <li>
                <h2>
                    <a href="${item.url}">${item.title}</a>
                </h2>'
                <p>${item.content.substring(0, 150)}...</p>
                <p>${item.tags}</p>
            </li>
        `;
    }
    searchResults.innerHTML = resultList
    } else {
    searchResults.innerHTML = 'No results found.'
}
}

const params = new URLSearchParams(window.location.search)
const query = params.get('query')

if (query) {
    document.getElementById('search-input').setAttribute('value', query)

    const idx = lunr(function () {
        this.ref('id')
        this.field('title', {
            // boost search to 15
            boost: 15
        })
        this.field('tags')
        this.field('content', {
            // boost search to 10
            boost: 10
        })

        for (const key in window.store) {
            this.add({
            id: key,
            title: window.store[key].title,
            tags: window.store[key].category,
            content: window.store[key].content
            })
        }
  })

  const results = idx.search(query)
  displayResults(results, window.store)

  document.getElementById('search-title').innerText = 'Search Results for ' + query
}

I have tried the .Pages and .Site.RegularPages options in index.html but didn’t work.

Do you have any suggestions?

Links:
https://demo.gethugothemes.com/bigspring-light/

2

Do all pages have “tags” in front matter, or just some pages?

3

Just some pages have “tags”

4

So, the first thing I noticed is that for pages with tags, you are generating invalid JSON due to a trailing comma. I don’t know if that is the problem, but it’s a place to start.

  "http://localhost:1313/posts/post-1/": {
  	"title": "Post 1",
  	"tags": ["a", "b", ],          <---- Trailing comm after last tag
  	"content": ,
  	"url": "http://localhost:1313/posts/post-1/"
  },
5

I have tried to remove comma
"tags": [{{ range .Params.Tags }}"{{ . }}" {{ end }}],

and also tested to remove this line so as not to use tags in search and still doesn’t work.

6

You also have a trailing comma at the end of your JSON. Again, this may have nothing to do with your problem, but you are generating invalid JSON.

To insert a comma between instead of after, use a construct like:

{{ range $k, $v := slice "a" "b" "c" }}
  {{ if $k }},{{ end }}
  {{ $v }}
{{ end }}
7

But this is less fragile way to generate valid JSON:

{{ $data := dict }}
{{ range $k, $_ := .Site.Pages }}
  {{ $m := dict .Permalink (dict
      "title" .Title
      "tags" .Params.tags
      "content" (.Content | plainify)
      "url" .Permalink
    )
  }}
  {{ $data = $data | merge $m }}
{{ end }}
{{ jsonify (dict "indent" "  ") $data }}

In your template do this:

{{ $data := dict }}
{{ range $k, $_ := .Site.Pages }}
  {{ $m := dict .Permalink (dict
      "title" .Title
      "tags" .Params.tags
      "content" (.Content | plainify)
      "url" .Permalink
    )
  }}
  {{ $data = $data | merge $m }}
{{ end }}

<script>
  window.store = {{ $data }}
</script>

Also, I noticed this:

But your example is “index.md” (no leading underscore).

1 Like
8

Hi @jmooring and thank you for spending time on my case,

I missed the underscore when typing. It is correct in the project.

I used your sugggested code but hasn’t changed anything, still not returning the rest articles.

9

Please post a link to the public repository for your project.

See Requesting Help.

Let us see your code

Include a link to the source code repository of your project, because we really need the context of seeing your templates and partials to be able to help you. It is trivial to do a quick git clone on your repo, then run hugo server in your project, to help you out. On the other hand, recreating your code from screenshots, or sort of guessing at it, is not.

If you can’t share your repository for whatever reason, consider creating a dummy repo that you can share, which reproduces the problem you’re experiencing.

10

Thank you. It will take time to prepate a public repository and post the link.

The pages with normal content are indexed as expected but pages that have section in front matter like the product_info in the service1.md (see the code above) are not indexed.

The bigsping template provides this index.json but seems that we have to include these sections

layouts_default\index.json

{{ $.Scratch.Add "index" slice }}
{{ range site.RegularPages }}

{{ $date:= time.Format ":date_long" .PublishDate }}

{{ with .Params.image }}
{{ if (fileExists (add `assets/` .)) }}
{{ $image:= resources.Get . }}
{{ $image:= $image.Resize "650x" }}
{{ $.Scratch.Set "image" $image.Permalink }}
{{ end }}
{{ end }}
{{ $image:= $.Scratch.Get "image" }}

{{ $.Scratch.Add "index" (dict "title" .Title "date" $date "image" $image "categories" .Section "contents" .Plain "permalink" .RelPermalink) }}
{{ end }}
{{ $.Scratch.Get "index" | jsonify }}

We tried {{ range .Site.Pages }} or {{ range .Pages }} but it does not include in index.json.

The section is
services:

  • _index.md
  • service1.md
  • service2.md
    …

And finally the produced index.json is empty:


{
"categories": "services","contents": "","date": "July 12, 2021","image": null,"permalink": "[/services/service-1/](http://localhost:1313/services/service-1/)","title": "Service 1"},
{"categories": "products","contents": "","date": "July 12, 2021","image": null,"permalink": "[/products/product-1/](http://localhost:1313/products/product-1/)","title": "Product 1"},
11

I had to configure each type and loop (using range) in the contents in order to create a correct window.store in index.html displayed above. In some cases I had to concatenate strings different properties and add them in content property so that all text is searchable.

Some helpful snippets:

 //For simple pages 
{{ range where .Site.RegularPages "Type" "page" }}
"{{ .Permalink }}": {
        "title": "{{ .Title  }}",
        "tags": [{{ range .Params.Tags }}"{{ . }}",{{ end }}],
        "content": {{ .Content | plainify }}, // Strip out HTML tags
        "url": "{{ .Permalink }}"
    },
    {{ end }}

    // Products
    {{ range where .Site.RegularPages "Section" "products"}}
        {{ $data := "" }}
        {{ range $k, $_ := .Params.product_info.features }}
            {{ $m := (trim .content "#*") | plainify }}
            {{ $data = printf "%s" $data | printf "%s%s" $m }}
        {{ end }}

        "{{ .Permalink }}": {
            "title": "{{ .Title  }}",
            "tags": [{{ range .Params.Tags }}"{{ . }}",{{ end }}],
            "content": {{ printf "%s" (.Params.cta.content | plainify) | printf "%s%s" (.Params.intro_video.content | plainify)
                        | printf "%s%s" $data | printf "%s%s" (.Params.product_info.content | plainify)
                        | printf "%s%s" (.Params.banner.content | plainify) | printf "%s"
            }},
            "url": "{{ .Permalink }}"
        },
    {{ end }}
12

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.