Search Index .json-file for Lunr.js

Hey guys, I’ve seen a lot of people asking for a tutorial on creating a search index .json-file. In a thread @bep recommended having a look at his commits for his blog. With the help of bep’s code it shouldn’t be rocket science.

I hope to provide a small tutorial of my own in the coming week, but if anyone is attacking this right now, please look at bep’s search index commit (updated link) and publish an example in this thread.

1 Like

OK, so I couldn’t wait. Assuming you’ve got Lunr.js working well (see here, or perhaps here), here’s a quick and dirty copy of @bep’s code:

Create /content/search-index.md, fill it with:

+++
date = "2017-03-05T21:10:52+01:00"
type = "json"
url = "index.json"

+++
```
(bep shows another way of writing above code in the link I posted at the top).

I keep all layout files in my themes folders so then i created a file with the path `themes/THEME/layouts/json/single.html`

Then filled it with
```
{{- $.Scratch.Add "index" slice -}}
{{- range where .Site.Pages "Type" "not in"  (slice "page" "json") -}}
{{- $.Scratch.Add "index" (dict "title" .Title "href" .Permalink "tags" .Params.tags "content" .Plain) -}}
{{- end -}}
{{- $.Scratch.Get "index" | jsonify -}}
```
Json file then reachable through `/index.json`. Hope it helps!

But note that this isn’t the best way of doing it in Hugo > .0.20. Hugo is a fast moving target. I would say it would be cleaner to create the JSON search index as an alternative output format to the home page, with its own template (which then would look similar to the one listed).

3 Likes

I have also implemented search pretty much like @Rick and I really need time to understand how the new output & alternative output formats work, but right now I can’t do it.

At least I hope you give us some kind of warning if things are about to break in a newer version.

I was actually just playing with this.

  1. Put this in config.toml
[outputs]
	home = [ "HTML", "JSON"]
	page = [ "HTML"]

That tells your site to make a JSON page for the home page.

  1. Make a file called index.json in the layouts folder of your preference (mine is in my theme for reasons)

  2. Use Rick’s example for the content of what he did as /json/single.html. Mine is different since I wanted to include categories and tags, and I’m using lunrjs prebuilt to look for uri instead of href:

{{- $.Scratch.Add "index" slice -}}
{{- range where .Site.Pages "Type" "not in"  (slice "page" "json") -}}
{{- $.Scratch.Add "index" (dict "uri" .Permalink "title" .Title "content" .Plain "tags" .Params.tags "categories" .Params.tags) -}}
{{- end -}}
{{- $.Scratch.Get "index" | jsonify -}}

I think that’s what bep was talking about. And I have to say, I like it a lot better than trying to use node (hugo-lunr or lunr-hugo) or Grunt, since I tend to write the docs on my iPad and auto-push the site updates when git syncs.

7 Likes

And is possible an solution multilingual?

Not sure about the Lunr.js side of it, but in Hugo it is pretty straight forward to create multiple language versions of the same output format, say a JSON index for the home page. You would just have to make sure to have content files for each language for the page kind you want (i…e home page) and set the JSON in either the front matter’s or the site config’s outputs.

Thank you for the clarifications!

Having custom (aka non-Google-based) search is the last stumbling block before fully migrating to Hugo, so I wonder if this is recommended way to do it?

The two popular Python-powered static-site-generators have plugin which use Tipue (jquery plugin for that so wonder if anyone has deployed it with Hugo and/or how does it compare with Lunr.js?

Lunr.js is a vanilla JS plugin and it’s blazing fast (although it can be slower on larger sites)

Tipue is a jQuery plugin so it’s not as fast as Lunr.js since jQuery is slower than vanilla JS

Would it be sufficient for sites with several hundreds blog posts?

Any other recommendation?

I haven’t used Lunr.js it with hundreds of pages yet. But I have found out a way to keep the JSON file size in check by having Lunr.js search only for titles and descriptions -not the whole content-.

If you need a full blown content search then look into Algolia. It’s a paid product but it has a free plan also. And people in this forum seem to love it.

I’ve used Lunr and Tipue on larger sites. A few hundred posts shouldn’t be a problem, particularly if you limit your fields, as @alexandros suggests. For larger sites they can be a slog and if search is important you’d probably want a server-side solution like Algolia.

How did you like Tipue on larger sites?

[quote]For larger sites they can be a slog and if search is important you’d probably want a server-side solution like Algolia.
[/quote]

Well, for my private/non-profit and small business needs, cashing $49/mo is a bit prices in regard to Algolia…my whole webhosting at Webfaction is $10/mo. :slight_smile:

@gour Tipue is fine. I wouldn’t use it now because I try not to use JQuery, but I think it performs pretty well. You can see it in action by hitting this page, and I think there or 2-3k pages on this site: http://www.ndbooks.com/books/?q=animal&page=1

One thing I didn’t like (and not sure if this is a general issue, or just the implementation I was using, but it reads from the Title meta tag, which means the results don’t look great.

Here’s an implementation with Lunr.js/vanillajs and the site has around 200 pages (though I’m querying a lot of text on this one) https://www.retroreport.org/search-results/?query=hillary

I hear you about price. I’m deciding right now between Algolia and building my own on a couple of new sites (actually, moving the two sites above to Hugo right now). I think it depends on how important search is to your users. If there’s no real ROI, then it’s certainly not worth it.

Here’s a presentation we had at our Meetup last year on some search options: https://www.thenewdynamic.org/event/2016/08/04/search-for-static-sites-by-aidan-feldman/

3 Likes

Thanks for this discussion, is there any guide how to apply Lunr.js to hugo generated sites? I am not Javascript expert… Thank you very much. I would be interested in some search field to be available on generated sites.

Use the forum search, and read about how to request help.

Hey @Ladislav_Jech,

Sorry for the late reply, feel I’m a main culprit in the Lunr discussions. @maiki is right, that’s how most of us have learned (while trying to ask only questions that haven’t been answered)

But as just might happen, I have the thread for you right here (check the links in my second post in the thread first).

I am also looking for a complete lunr.js setup guide.

I had stumbled across the thread you linked, and I got the index.json created by Hugo working. But that’s only small part of the whole setup. As I don’t know JS, I am lost at how to interface that .json file with lunr.js in the Hugo templates.

So I started collecting real-world examples where people use lunr.js + Hugo: How to add lunr.js to your site?

I haven’t yet found time to understand the sources linked in there, reverse engineer and extract only the lunr.js stuff into my site.

I have also opened an issue on HugoDocs requesting to add a start-to-finish Hugo + lunr.js setup guide.

Hey mate,
I utterly suck at js, you really only need to generate the searchindex and then make sure lunr.js points to it – there’s no “javascript coding” involved.

/Good luck