Live Hugo Site search with Lunr.js


#1

For those interested in setting up site search with the current version of Hugo (15), I’ve finally got the following to work using lunr.js.

A few quick things:

  • I’m running a gulp build concomitantly with hugo that includes uglify, concat, min, etc. This is for an internal-only company project, so PM me if you want my Gulpfile.js for any reason
  • This is a live search in a full-page overlay and isn’t currently set up for Google Analytics, although this is forthcoming
  • You’ll need a separate call to lunr.js before the following scripts in the cascade.
  • I refused to use any jQuery because I wanted to learn without using a library as a crutch.
  • I an keeping all my images in the folder respective to the content. I then copy/replace them to static as part of a bash script (see 5 below). This was necessary because my colleagues wanted drag and drop when creating .md files in GitHub. This almost means an additional JavaScript IIFE when you want your images to render on the client (see#6)
  • You will need to create content/json/dummy.md with yaml and some dummy content for this to work, since I think Hugo still needs at least one file in the json folder to create

1. The Search FORM HTML

layouts/partials/site_header/site_search_form.html

<!--layouts/partials/site_header/site_search_form.html-->
<!--BEGIN FORM-->
<form action="" id="site-search" class="search-form" role="search">
    <a href="#" id="close-search"><i class="icon-cancel"></i></a>
    <input type="search" id="search-input" aria-label="Search the {{ .Site.Title }} website..." class="site-search" autocomplete="off" placeholder="Search DST documentation..." required>
		<ul id="search-results">
	         <!-- The following <li> can be uncommented for styling purposes, but these reflect the same css classes included in the script. -->
		    <!-- <li class="search-result">
		        <a href="#"><h5>Sample Search Result Heading</h5></a>
		        <p>Here is a simple description area that should be kept relatively short, but just in case, let's make this a little bit longer.</p>
		        <div class="in-section">Found in: Digital Content Writers Guide</div>
		        <ul class="tags">
		            <li><i class="icon-tags"></i></li>
		            <li><a href="#" class="tag">sushi</a></li>
		            <li><a href="#" class="tag">japanese</a></li>
		        </ul>
		    </li> -->
		</ul>
    <div class="press-escape" title="Press Escape to Close the Site Search Overlay">Press <em>ESC</em> to close.</div>
</form>
<!--END FORM-->
<!--Search toggle button (place this wherever appropriate in your source-->
<a href="#" id="search-button" role="button"><span> Press <em>S</em> to search</span><i class="icon-search"></i></a>

2. The layout/template to create site-index.html

layouts/section/json.html

<!--layouts/section/json.html-->
<!--note that I include title, subtitle (optional), tags, and description in all my content .md yaml. Once you get this down, remove the extra white space because it will cut down your site-index.json file considerably.-->
[{{ range $index, $page := .Site.Pages }}
{{ if ne $page.Type "json" }}
{{ if and $index (gt $index 1) }},{{ end }}
{
    "url": "{{ $page.Permalink }}",
    "title": "{{ $page.Title }}",
    <!--Add content subtitle to index only if set, otherwise set to empty since subtitle is *still* added to lunr.js index.-->
    {{ if $page.Params.subtitle }}
		"subtitle": "{{ $page.Params.subtitle }}",
		{{ else }}
		"subtitle":"",
    {{ end }}
    "section": "{{ .Section }}",
    "tags": [{{ range $tindex, $tag := $page.Params.tags }}{{ if $tindex }}, {{ end }}"{{ $tag }}"{{ end }}],
    "description": "{{.Description}}",
    "content": "{{$page.PlainWords}}"
}{{ end }}{{ end }}]

###3 The lunr AJAX call for the index and toggle of the full-screen search overlay

assets/js/modules/search-lunr-ajax-call.js

//I will try and add more comments to this later...
/*Begin full-screen search overlay toggle*/
//Note that this requires the element.classList method, which is not supported in IE9. If you need IE9 support, use the classList.js polyfill (https://github.com/eligrey/classList.js/)
//Full-screen overlay opens via click/touch event or if the user hits "s' on the keyboard. When search is open, this is controlled for so that people can search words with "s" in them
var searchOverlay = document.querySelector('.search-form');
var searchButton = document.getElementById('search-button');
var searchInput = document.getElementById('search-input');
var closeSearch = document.getElementById('close-search');
closeSearch.onclick = function() {
    if (searchOverlay.classList.contains('open')) {
        searchOverlay.classList.remove('open');
    }
}
window.addEventListener('keyup', function(event) {
    var keyPressed = event.keyCode;
    if (keyPressed === 83 && searchOverlay.classList.contains('open')) {
        return;
    } else if (keyPressed === 83) {
        searchOverlay.classList.add('open');
        if (searchInput.value.length > 0) {
            searchInput.value = '';
        }
        searchInput.focus();
    } else if (keyPressed === 27 && searchOverlay.classList.contains('open')) {
        searchOverlay.classList.remove('open');
    }
}, true);
searchButton.addEventListener('click', function(event) {
    searchOverlay.classList.toggle('open');
    searchInput.focus();
}, true);
/*End search overlay toggle*/

/*Begin Lunr live search*/
//for more information on lunr.js, go to http://lunrjs.com/
var searchData;
var searchInput = document.getElementById('search-input');
searchInput.addEventListener('keyup', lunrSearch, true);
window.index = lunr(function() {
    this.field('id');
    this.field('url');
    this.field('title', { boost: 50 });
    this.field('subtitle');
    this.field('description');
    this.field('tags',{ boost: 30});
    this.field('content', { boost: 10 });
//boosting for relevancy is up to you.
});

var searchReq = new XMLHttpRequest();
searchReq.open('GET', '/site-index.json', true);
searchReq.onload = function() {
    if (this.status >= 200 && this.status < 400) {
        console.log("Got the site index");
        searchData = JSON.parse(this.response);
        searchData.forEach(function(obj, index) {
            obj['id'] = index;
            window.index.add(obj);
        });
    } else {
        console.log("Failed status for site-index.js. Check /static/site-index.json");
    }
}
searchReq.onerror = function() {
    console.log("Error when attempting to load site-index.json.");
}
searchReq.send();

function lunrSearch(event) {
    var query = document.querySelector("#search-input").value;
    var searchResults = document.querySelector('#search-results');
    if (query.length === 0) {
        searchResults.innerHTML = '';
    }
    if ((event.keyCode !== 9) && (query.length > 2)) {
        var matches = window.index.search(query);
        displayResults(matches);
    }
}

function displayResults(results) {
    var searchResults = document.querySelector('#search-results');
    var inputVal = document.querySelector('#search-input').value;
    if (results.length) {
        searchResults.innerHTML = '';
        results.forEach(function(result) {
            var item = window.searchData[result.ref];
            var section = item.section.split('-').map(s => s.charAt(0).toUpperCase() + s.slice(1)).join(' ');
            var appendString = '<li class=\"search-result\"><a href=\"' + item.url + '\"><h5>' + item.title + '</h5></a><p>' + item.description + '</p><div class=\"in-section\">In: ' + section + '</div><ul class=\"tags\"><li><i class=\"icon-tags\"></i></li>';
            // var tags = '';
            for (var i = 0; i < item.tags.length; i++) {
                appendString += '<li><a href=\"/tags/' + item.tags[i] + '\" class=\"tag\">' + item.tags[i] + '</a> ';
            }
            appendString += '</ul></li>';
            searchResults.innerHTML += appendString;
        })
    } else {
        searchResults.innerHTML = '<li class=\"search-result none\">No results found for <span class=\"input-value\">' + inputVal + '</span>.<br/>Please check spelling and spacing.</li>';
    }
}

4. The style that gives you the full-screen search experience

assets/sass_modules/search-overlay.scss

//NOTE that the variable names are my own. You will obviously need to adjust according to how you design your sass variables.scss file. Honestly, you will probably want to write your own styling, especially since the variables and mixins are my own, but I'm throwing this in for good measure. The icon-* classes reflect [fontello](http://fontello.com/), which I used to reduce the overhead of including larger libraries like Font Awesome.

#search-button {
    display: inline;
    color: $base-font-color;
    background-color: $top-bar-background-color;
    font-size: $top-bar-height/4;
    height: $top-bar-height;
    line-height: $top-bar-height;
    cursor: pointer;
    position: absolute;
    top: 0px;
    right: $top-bar-height/4;
    transition: all .3s ease-in-out;
    &:hover {
        color: $brand;
        span {
            color: $brand;
            em {
                color: $brand;
                border-color: $brand;
            }
        }
    }
    span {
        display: none;
        @include MQ(L) {
            display: inline;
            height: $top-bar-height/2;
            line-height: $top-bar-height/2;
        }
    }
    em {
        font-family: "Lucida Console", "Courier New", courier, monaco, monospace;
        font-style: normal;
        font-weight: 100;
        border: 1px solid lighten($base-font-color, 35%);
        padding: .25em;
        border-radius: 3px;
        color: $base-font-color;
    }
    i.icon-search {
        font-size: 2em;
        @include MQ(L){
            font-size:1.5em;
        }
    }
}

form.search-form {
    position: fixed;
    top: 0;
    left: 0;
    width: 100%!important;
    width: 100vw!important;
    height: 100%;
    height: 100vh;
    display: none;
    overflow-y: scroll;
    &.open {
        display: block;
        z-index: 999;
        background-color: white;
    }
    input#search-input {
        display: block;
        width: 90%;
        margin-left: 5%;
        margin-top: 1.5em;
        @include MQ(L) {
            width: 70%;
            max-width: 70%;
            margin-left: auto;
            margin-right: auto;
            margin-top: 2rem;
            font-size: 2rem;
        }
    }
}

div.press-escape {
    display: none;
    @include MQ(L) {
        display: block;
        position: absolute;
        margin-top: 20px;
        font-size: 16px;
        bottom: 25px;
        left: calc(50% - 100px);
        right: calc(50% - 100px);
        width: 200px;
        em {
            font-family: "Lucida Console", "Courier New", courier, monaco, monospace;
            font-style: normal;
            font-weight: 100;
            position: relative;
            font-size: .7em;
            border: 1px solid lighten($base-font-color, 35%);
            border-radius: 3px;
            color: $base-font-color;
            top: -.67em;
            padding: .25em .5em 1.5em;
        }
    }
}

#close-search {
    width: 18px;
    position: fixed;
    top: 20px;
    right: 30px;
    z-index: 999;
}

ul#search-results {
    display: block;
    width: 90%;
    margin-left: 5%;
    margin-top: 1.5em;
    padding-left: 0;
    @include MQ(L) {
        width: 70%;
        margin-left: auto;
        margin-right: auto;
        margin-top: 2rem;
    }
    li {
        list-style: none inside none;
        border: 1px solid $search-result-border-color;
        position: relative;
        display: block;
        height: 100%;
        margin-left: 0;
        margin-bottom: .75em;
        padding-left: 0;
        z-index: 999;
        margin-left: -10px;
        margin-right: -10px;
        a {
            &:hover {
                cursor: pointer!important;
            }
            &:focus {
                h5 {
                    background-color: $search-result-title-background-color-hover;
                    margin-left: -1px;
                    width: calc(100% + 2px);
                }
            }
        }
        h5 {
            transition: all .2s ease-in-out;
            background-color: $search-result-title-background-color;
            color: white;
            margin-top: -1px;
            padding-top: -1px;
            padding-left: 10px;
            padding-right: 10px;
            font-weight: bold;
            &:hover,
            &:focus,
            &:active {
                cursor: pointer!important;
                background-color: $search-result-title-background-color-hover;
                margin-left: -1px;
                width: calc(100% + 2px);
            }
        }
        div.in-section {
            display: block;
            width: 100%;
            padding-left: 10px;
            font-size: .75em;
            display: inline-block;
            color: black;
            font-weight: bold;
        }
        p {
            flex-grow: 1;
            padding-left: 10px;
            padding-right: 10px;
            margin-top: 5px;
            margin-bottom: 0px;
        }
        //see _icons.scss for span.tags-icon
        &.none {
            border: 3px solid $warning-color;
            padding-left: 10px;
        }
    }
    ul.tags {
        display: none;
        @include MQ(M) {
            display: flex;
            margin-bottom: .5rem;
            margin-top: .5rem;
            width: 100%;
            padding-left: 10px;
            justify-content: flex-end;
            li {
                display:none;
                @include MQ(M){
                    display:inline;
                }
                border: 0px;
                margin: 0 5px;
                padding: 0;
                //fontello icon
                .icon-tags {
                    color: $cap-black;
                }
                //only show mobile-plus size since tags are hidden in search overlay for mobile.
                a.tag {
                    font-size: $tag-font-size-tablet-plus;
                    background-color: $tag-background-color;
                    color: $tag-font-color;
                    &:hover {
                        background-color:$tag-background-color-hover;
                    }
                }
            }
        }
    }
}

i.icon-cancel {
    color: $white;
    font-size: 1.2rem;
    background-color: $base-font-color;
    border-radius: 50%;
}

span.input-value {
    color: $warning-color;
    font-weight: bold;
    font-style: italic;
}

###5. The Bash Script that puts it all together

hugo-build-index-and-serve.sh

# HUGO BUILD SITE AND JSON SITE INDEX
# Assign pwd to curdir variable
curdir=$(pwd)
# Build Hugo. It's okay if there are dupes in the publishdir ("public") because we are just looking for the newest site-index.html anyway
hugo
cp ${curdir}/public/json/index.html ${curdir}/static/site-index.json
cd $curdir
# Cleans out Build Directory "Public" if it exists
if [ -d "public" ]; then
	rm -rf public
fi
# "Image copy" = copies all images in "content" to /static/images/ before running the server. NOTE:
# 1. This requires additional client-side tweaks so that images still serve correctly to the browser.
# 2. This is based on the assumption that content md files reference images from relative images directory: eg, ![my alt text](images/my-image.jpg)
# Begin Image copy
find ${curdir}/content \( -iname '*.jpg' -o -iname '*.png' -o -iname '*.gif' \) -type f -exec cp -v -- {} ${curdir}/static/images/ \;
# End Image copy
# Open Google Chrome
open -a Google\ Chrome.app http://localhost:1313
hugo server

###6. Client-side <img src="*" tweak if you keep all your images in the respective folder and include the image copy step in the above bash script:

/*
This IIFE appends a '/' to all img[src] so that in-page images point to a single image repo at the root.
This reflects a GitHub-based authorship model where documentation authors are creating md files in individual GitHub directories and then dragging and dropping documentation images into an `images` subdirectory.
 */
(function() {
    var allImages = document.querySelectorAll('img');
    if (allImages.length > 0) {
        for (var i = 0; i < allImages.length; i++) {
            var originalSource = allImages[i].getAttribute('src');
            var newSrc = '/' + originalSource;
            allImages[i].setAttribute('src', newSrc);
        }
    }
})();

I will eventually be tweaking the above bash script as part of a Wercker auto-deployment. I really should just write a blog post for all of this…stay tuned.


How to add lunr.js to your site?
#2

Hi @rdwatters
Thanks for taking time to write this up. I really appreciate it.
I believe there is a tiny error in the code, namely in “$index 1”:

{{ if and $index (gt $index 1) }},{{ end }}

I believe this causes the first comma between blocks to be missing in JSON file (try running your JSON file through JSONLint). I found when the above line is updated to the following, no commas are missing (and json-minify npm library accepts it):

{{ if and $index (gt $index 0) }},{{ end }}

I’m trying to use your way of generating Lunr.js index file via Hugo because I’m having trouble with hugo-lunr library.

I’m currently in the middle of integrating npm scripts, Hugo and Lunr.js. Here’s an excerpt from my package.json — the Lunr site index /public/json/index.html is being minified and saved into Hugo root directory:

"scripts": {
...
"index:prepare": "json-minify public/json/index.html > public/site-index.json",
...
}

Again, thanks for sharing!
Roy


#3

Hey @royston thanks for the feedback.

As it turns out, I hacked together the above example by extracting from a vpn-only site I’m putting together for my team/company. Here is the actual template I’m using at layouts/sections/singles.html. singles is a content type I put together for a bunch of one-off pages on the site, most of which I did not want to include in the search index.

It’s been a couple weeks, but I seem to recall the same issue that you’re bringing up w/r/t {{ if and $index (gt $index 1) }},{{ end }}, and I remember using 0 first instead of 1 only to get the invalid JSON you mentioned. That said, I’ve run 200+ wercker builds with the following template without issue:

[{{ range $index, $page := .Site.Pages }}
	{{ if ne $page.Type "singles" }}
		{{ if and $index (gt $index 1) }},{{ end }}
		{"url": "{{ $page.Permalink }}",
		"title": "{{ $page.Title }}",
		{{ if $page.Params.subtitle }}
			"subtitle": "{{ $page.Params.subtitle }}",
			{{ else }}
			"subtitle":"",
		{{ end }}
		"section": "{{ .Section }}",
		"tags": [{{ range $tindex, $tag := $page.Params.tags }}{{ if $tindex }}, {{ end }}"{{ $tag }}"{{ end }}],
		"description": {{if .Description }}"{{.Description}}",
		{{else}} 
		"{{$page.Params.bio_short }}",
		{{end}}
		"content": "{{$page.PlainWords}}"}
		{{ end }}
{{ end }}]

$page.Params.bio_short relates to a team-member content type I put together with my colleagues bio in case you’re wondering. I just tested it locally with my build script by adding a few random words (eg, “cheeseburger,” “toilet paper,” “European,” “introvert”) and lunr.js is working like a charm. Are you thinking about keeping your builds local to your machine or if you’re thinking about using an automated deployment tool (viz. wercker). If so, here’s something I’ve been using thanks to the patience and expertise of @ArjenSchwarz. Perhaps it will be of some utility to you as well.

wercker.yml

box: golang
build:
  steps:
  - script:
    name: echo "Recursively moving all images to static images directory"
    code: |
      curdir=$(pwd)
      find ${curdir}/content \( -iname '*.jpg' -o -iname '*.png' -o -iname '*.gif' \) -type f -exec mv -v -- {} ${curdir}/static/images/ \;
  - arjen/hugo-build:
    version: "0.15"
    config: config.toml
    disable_pygments: false
  - script:
    name: echo "Moving /singles/index.html to /assets/site-index.json"
    code: |
      currentdir=$(pwd)
      cp ${currentdir}/public/singles/index.html ${currentdir}/static/assets/site-index.json
deploy:
  steps:
  - arjen/s3sync:
    key-id: $AWS_KEY
    key-secret: $AWS_SECRET_KEY
    bucket-url: $AWS_BUCKET
    source_dir: public/
    delete-removed: true
    opts: --add-header=Cache-Control:max-age=100

#4

https://github.com/spf13/hugo/pull/1853

@royston Not sure if you’ve been keeping up with some of the new features of 16, but this may help future efforts w/r/t making such indices a little easier to template:


#5

Thank you for wercker example! It’s very helpful, I haven’t automated my website’s deployment to such level yet, it’s very interesting.


#6

@rdwatters Did you check the functionality on iOS? I adapted the code and got it working but on desktop only. Nothing happens on the latest iOS x iPhone. I’m wondering, is it me or we need jQuery after all…


#7

@royston my site is still okay with lunr.js on iOS. Are you using a CDN or setting cache control maybe? I do not have jQuery running on the site - just a single concatenation/minified/uglified script tag. Are you running any iOS emulators? I believe that the dev tools in Safari for OSX let’s you run the site as if it were Safari on iOS. You might want to check the console there.


#8

The only other issue that springs to mind is where the lunrjs script is being called, if you’re adding any async/defer attributes on the script tags, etc. Browsers are not always as predictable when it comes to the way they handle external scripts. If you’re using a build tool, you might want to make the call to lunr.js synchronously in the head of the document, then make everything else via a single asynchronous/deferred call just above the closing body tag. Let me know if that helps. If this is a public repo, I’d be happy to take a look. Just point me in the right direction…


#9

Does this layouts/section/json.html still work for you in Hugo 0.18.1? I get a 404 while trying to access


#10

OK, a quick fix that worked for my setup: create a folder in content called json and stick a blank _index.md file in there.


#11

Yeah, I haven’t had any issues with 18, but it looks like you figured it out already. Also, now that Hugo recognizes json as a MIME type, I’m going to need to refactor this and make it cleaner/easier. Then there is no need for the additional copying with the bash script…


#12

Thanks for the reply! Could you elaborate on the last sentence? Must have missed this development


#13

See my wercker.yml above; it includes copying images and copying over the html file to a .json extension. I would search the forums for search too. Bep put together a sample that he uses for his site.


#14

Thanks!


#15

Awesome. Just ran across this little gem (but not the Ruby kind) as well and thought I’d drop it in here for those who are smart enough not to pay for services like Agolia when such things can be had for free using open source technology: https://gist.github.com/sebz/efddfc8fdcb6b480f567


#16

Hi All,

I have deployed my site on an ubuntu AWS EC2 machine. When I access the site via the IP link (http://10.20.30.40/) , the search functionality works well. Now I went ahead and added an ELB and Route53 DNS name to the site. When i access the site from the DNS name, i am not able to search documents with tags.I tried to troubleshoot this checking the browser console logs and get the below error. I doubt this is something on the browser side that’s blocking the search js.

search.js:51 Uncaught TypeError: Cannot read property ‘search’ of undefined at search (search.js:51) at suggestions (search.js:64) at HTMLInputElement.loading (horsey.js? [sm]:122) search @ search.js:51 suggestions @ search.js:64 loading @ horsey.js? [sm]:122

Any help is appreciated.