# \[SOLVED\] How to create a JSON index when there's code in content?

**URL:** <https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901>\
**Category:** support\
**Created:** [December 24, 2016, 8:54am UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901 "2016-12-24T08:54:05Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jura](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/jura/32/13581_2.png) [@Jura](https://discourse.gohugo.io/u/Jura)\
**Post date:** [December 24, 2016, 8:54am UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/1 "2016-12-24T08:54:05Z")

</div>

Continuing the discussion from [Possible to get markdown escaped as just plain text if not JSON?](https://discuss.gohugo.io/t/possible-to-get-markdown-escaped-as-just-plain-text-if-not-json/2607/4):

I’ve searched the forum and Github, but I cannot make a JSON valid search index in Hugo. Each time there’s code inside the content (between ` ~~~` and `~~~ `), JSON fails.

I’ve had a lot of help with the information on the forum – my code is based on @rdwatters contributions – but cannot make it work. I really hope someone can help, because with this my site cannot have a search.

## Starting point

My Hugo code is:

```
[
{{ range $index, $page := where .Site.Pages "Section" "!=" "" }}
{{ if and $index (gt $index 0) }},{{ end }}
    {{ if (ne $page.Section "") }}
    {
        "objectID": "{{ $page.UniqueID }}",
        "url": "{{ $page.RelPermalink }}",
        "title": "{{ $page.Title }}",
        "platform": "{{ index $page.Params.categories 0 }}",
        "content": "{{ $page.Plain }}"
    }
    {{ end }}
{{ end }}
]

```

With the [JSON lint validator](http://jsonlint.com/), this gives the unexpected EOF error:

 ![](https://canada1.discourse-cdn.com/flex036/uploads/gohugo/original/2X/6/66f458ff976fcfb9401f2c14c330c883ac796119.png)

As we can see here, the `"content"` JSON element breaks whenever there’s code come across.

## Using `jsonify` to encode the content to JSON

Hugo code:

```
[
{{ range $index, $page := where .Site.Pages "Section" "!=" "" }}
{{ if and $index (gt $index 0) }},{{ end }}
    {{ if (ne $page.Section "") }}
    {
        "objectID": "{{ $page.UniqueID }}",
        "url": "{{ $page.RelPermalink }}",
        "title": "{{ $page.Title }}",
        "platform": "{{ index $page.Params.categories 0 }}",
        "content": "{{ $page.Content | jsonify }}"
    }
    {{ end }}
{{ end }}
]

```

Error:

 ![](https://canada1.discourse-cdn.com/flex036/uploads/gohugo/original/2X/7/78665f2592ee60ee03ed67e0a3e990946a952078.png)

## JSONfying plain words

Hugo code:

```
[
{{ range $index, $page := where .Site.Pages "Section" "!=" "" }}
{{ if and $index (gt $index 0) }},{{ end }}
    {{ if (ne $page.Section "") }}
    {
        "objectID": "{{ $page.UniqueID }}",
        "url": "{{ $page.RelPermalink }}",
        "title": "{{ $page.Title }}",
        "platform": "{{ index $page.Params.categories 0 }}",
        "content": "{{ $page.PlainWords | jsonify }}"
    }
    {{ end }}
{{ end }}
]

```

Error:

 ![](https://canada1.discourse-cdn.com/flex036/uploads/gohugo/original/2X/e/ed85da8894a2c186938bbdc911f74679a156100a.png)

## Just PlainWords

When using PlainWords as [used in this topic](https://discuss.gohugo.io/t/live-hugo-site-search-with-lunr-js/2857):

```
[
{{ range $index, $page := where .Site.Pages "Section" "!=" "" }}
{{ if and $index (gt $index 0) }},{{ end }}
    {{ if (ne $page.Section "") }}
    {
        "objectID": "{{ $page.UniqueID }}",
        "url": "{{ $page.RelPermalink }}",
        "title": "{{ $page.Title }}",
        "platform": "{{ index $page.Params.categories 0 }}",
        "content": "{{ $page.PlainWords }}"
    }
    {{ end }}
{{ end }}
]

```

Error:

 ![](https://canada1.discourse-cdn.com/flex036/uploads/gohugo/original/2X/0/0285045d3601480638e802a191ca26d326a16247.png)

(Again JSON breaks whenever there’s code come across.)

## `RawContent` with `jsonify`

Hugo code:

```
[
{{ range $index, $page := where .Site.Pages "Section" "!=" "" }}
{{ if and $index (gt $index 0) }},{{ end }}
    {{ if (ne $page.Section "") }}
    {
        "objectID": "{{ $page.UniqueID }}",
        "url": "{{ $page.RelPermalink }}",
        "title": "{{ $page.Title }}",
        "platform": "{{ index $page.Params.categories 0 }}",
        "content": "{{ $page.RawContent | jsonify }}"
    }
    {{ end }}
{{ end }}
]

```

Error:

 ![](https://canada1.discourse-cdn.com/flex036/uploads/gohugo/original/2X/8/88e53d0990d543e5d5d0fe150c65b4d59c369140.png)

* * *

I’m using:

```
PS I:\site> hugo env
Hugo Static Site Generator v0.18 BuildDate: 2016-12-19T14:42:56+01:00
GOOS="windows"
GOARCH="amd64"
GOVERSION="go1.7.4"

```

The above JSON troubles also happened with Hugo 0.17 (although I did not test `RawContent` on that Hugo version).

* * *

Thanks in advance for any suggestion or idea!

---

<div class="post-metadata">

**Author:** ![moorereason](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/moorereason/32/592_2.png) [@moorereason](https://discourse.gohugo.io/u/moorereason)\
**Post date:** [December 24, 2016, 5:39pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/2 "2016-12-24T17:39:21Z")

</div>

Go HTML templates are pretty picky about their context. Things to try:

```auto
        "content": {{ $page.RawContent | jsonify }}
        "content": {{ $page.RawContent | jsonify | safeJS }}
        "content": "{{ $page.RawContent | jsonify | safeJS }}"

```

Watch [this thread](https://discuss.gohugo.io/t/custom-output-content-types/4584/4) and [this issue](https://github.com/spf13/hugo/issues/2828) for a real, long-term solution.

---

<div class="post-metadata">

**Author:** ![Jura](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/jura/32/13581_2.png) [@Jura](https://discourse.gohugo.io/u/Jura)\
**Post date:** [December 25, 2016, 1:35pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/3 "2016-12-25T13:35:40Z")

</div>

Thanks for replying. Unfortunately no luck with that. The generated JSON file still trips over the double quote usage.

* * *

I like to use `.PlainWords` instead of `.RawContent` because the latter also includes markdown formatting of links and my shortcodes. `.PlainWords` is a bit cleaner in that sense.

But the issue with `.PlainWords` is that it doesn’t translate `"` to `&quote;` in the resulting output _when_ that `"` sign is included in a code block.

So I thought about using a regex like this:

`"{{ $page.PlainWords | replaceRE "\"" "$1" "&quote;" }}"`

I’ve also tried other regex approaches, but this fails (possible because Hugo already expects a double quote that specifies the regex itself). (Validating the regex pattern against an online validator does show it’s correct and matches with `.PlainWords`.)

Can someone help me with the regex/Hugo approach for this? I’m out of my skill zone here.

> [@moorereason](#):
>
> Watch this thread and this issue for a real, long-term solution.

I thought this pull request would be a real solution: [Add generator for a search index by digitalcraftsman · Pull Request #1853](https://github.com/spf13/hugo/pull/1853).

---

<div class="post-metadata">

**Author:** ![bep](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/bep/32/3332_2.png) [@bep](https://discourse.gohugo.io/u/bep)\
**Post date:** [December 25, 2016, 2:29pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/4 "2016-12-25T14:29:21Z")

</div>

You may get some inspiration/help here:

> **[Simple JSON site index in Hugo](https://discourse.gohugo.io/t/simple-json-site-index-in-hugo/2854)**
>
> DISCLAIMER: The following will only work with the latest build of Hugo (0.16-DEV). It has been a slow Sunday, no weather for skiing, so I have fiddled around with getting the foundation for site search on my blog. The actual search parts are still...

I think you will get less friction if you first create the object you want to serialize, then use `jsonify` on that object instead of your manual string manipulations.

This will be better once we get this fixed:

> <https://github.com/gohugoio/hugo/issues/2828>

#1853 is too specific, so I don’t think that will get into Hugo master.

---

<div class="post-metadata">

**Author:** ![Jura](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/jura/32/13581_2.png) [@Jura](https://discourse.gohugo.io/u/Jura)\
**Post date:** [December 25, 2016, 3:43pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/5 "2016-12-25T15:43:00Z")

</div>

> [@bep](#):
>
> You may get some inspiration/help here

Thanks for the suggestion, but the approach that you used in that topic (with `.Plain`) I’ve tried already. Using `jsonify` on that object then only make it (the double quotes problem) worse.

I figure from your reply that you have no trouble with getting a JSON file when you use double quotes in a code block on Hugo 0.18?

* * *

After more testing and tweaking, I think there’s an issue in `.PlainWords`/`.Plain` which explains why `.PlainWords` with or without `jsonify` doesn’t work.

On Hugo 0.18, this markdown code:

```
~~~csharp
Console.WriteLine("Testing 'firstValue' against 'secondValue':");
~~~

```

Generates with `.PlainWords` as:

`Console.WriteLine(&quot;Testing 'firstValue' against 'secondValue':&quot;);`

This is correct, and how I’d expect `.PlainWords` to work.

But now create a code block like this:

```
<pre><code class="language-csharp full">
Console.WriteLine("Testing 'firstValue' against 'secondValue':");
</code></pre>

```

That generates an output like:

`Console.WriteLine("Testing 'firstValue' against 'secondValue':");`

In this case, `.PlainWords` fails to strips the double quotes from the content, and that gives me all the issues I’m having to create a valid JSON output.

So how can I make `.PlainWords` _also_ take `<pre><code>` tags into account? Is it supposed to do so? If so, is there an alternative that makes `.Content` plain with taking HTML tags into consideration? Because this way `.PlainWords` doesn’t work 100% for people with my use case.

---

<div class="post-metadata">

**Author:** ![bep](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/bep/32/3332_2.png) [@bep](https://discourse.gohugo.io/u/bep)\
**Post date:** [December 25, 2016, 4:30pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/6 "2016-12-25T16:30:58Z")

</div>

> [@Jura](#):
>
> I figure from your reply that you have no trouble with getting a JSON file when you use double quotes in a code block on Hugo 0.18?

No, that was not my intention. I have no idea what your problem is.

---

<div class="post-metadata">

**Author:** ![Jura](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/jura/32/13581_2.png) [@Jura](https://discourse.gohugo.io/u/Jura)\
**Post date:** [December 25, 2016, 4:46pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/7 "2016-12-25T16:46:20Z")

</div>

> [@bep](#):
>
> I have no idea what your problem is

My problem is that my JSON file breaks due to double quotes (`"`) in the page’s content. Those quotes are there because `.Plain` and `.PlainWords` do not remove them from inside code highlights marked with `<pre><code>`. (They do remove quotes from code highlights beginning with `~~~` though.)

Besides `.Plain`/`.PlainWords` I don’t know of another way to get a page’s content in valid JSON format.

To solve my issue, I need to get a page’s content in plain text suitable for insertion in a JSON file.

---

<div class="post-metadata">

**Author:** ![bep](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/bep/32/3332_2.png) [@bep](https://discourse.gohugo.io/u/bep)\
**Post date:** [December 25, 2016, 4:53pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/8 "2016-12-25T16:53:04Z")

</div>

> [@Jura](#):
>
> My problem is that my JSON file breaks due to double quotes (") in the page’s content. Those quotes are there because .Plain and .PlainWords do not remove them from inside code highlights marked with \<pre\>\<code\>. (They do remove quotes from code highlights beginning with ~~~ though.)

Quotes are perfectly valid as “plain text” in any definition. What you need is likely an `escape` template func, which we probably could add if we found a suitable func in Go’s stdlib.

---

<div class="post-metadata">

**Author:** ![bep](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/bep/32/3332_2.png) [@bep](https://discourse.gohugo.io/u/bep)\
**Post date:** [December 25, 2016, 4:55pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/9 "2016-12-25T16:55:47Z")

</div>

Have you tried the `htmlEscape` template func?

---

<div class="post-metadata">

**Author:** ![Jura](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/jura/32/13581_2.png) [@Jura](https://discourse.gohugo.io/u/Jura)\
**Post date:** [December 25, 2016, 5:06pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/10 "2016-12-25T17:06:23Z")

</div>

(I see we’re are already on the same page, but I’ll keep the additional explanation here in case it might prove useful later.)

* * *

> [@bep](#):
>
> I have no idea what your problem is.

Let’s me explain why `.Plain` and `.PlainWords` prevent me from creating a JSON valid file.

Let’s write this `blogpost.md`:

`From the Hugo source:`

```
~~~go
import (
    "fmt"
    "html/template"

    "github.com/spf13/hugo/helpers"
)
~~~

Or with only importing the `fmt` package:

<pre><code class="go-highlight full double-size">
import (
    "fmt"
)
</code>
</pre>

The end of the article.

```

Now let’s add the content of this article to our search index with `.PlainWords`:

```
[From the Hugo source: import ( &quot;fmt&quot; 
&quot;html/template&quot; 
&quot;github.com/spf13/hugo/helpers&quot; ) Or with only 
importing fmt: import ( "fmt" ) The end of the article. Hello world, 
this is my first blog post on my Hugo static website!]

```

To highlight, those two double quotes from the `<pre><code>` tags make the whole thing JSON invalid:

 ![](https://canada1.discourse-cdn.com/flex036/uploads/gohugo/original/2X/9/963a6af1f47b414c046fbda04673ff27425e3804.png)

* * *

> [@bep](#):
>
> What you need is likely an escape template func, which we probably could add if we found a suitable func in Go’s stdlib.

> [@bep](#):
>
> Have you tried the htmlEscape template func?

The `htmlEscape` function looks promising from the description in the docs, but I don’t know how to use it. All of the ways mentioned below generate an error in `general.go` for me.

```
{{ .PlainWords htmlEscape }}
{{ htmlEscape .PlainWords }}
{{ .PlainWords | htmlEscape }}
{{ htmlEscape (printf .PlainWords "%v") }}
{{ htmlEscape (printf .PlainWords "%s") }}

```

> ERROR: 2016/12/25 18:06:00 general.go:236: Error while rendering page json-search-index.md: template: theme/json/single.  
> html:11:24: executing "theme/jso.

---

<div class="post-metadata">

**Author:** ![digitalcraftsman](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/digitalcraftsman/32/4142_2.png) [@digitalcraftsman](https://discourse.gohugo.io/u/digitalcraftsman)\
**Post date:** [December 26, 2016, 4:43pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/11 "2016-12-26T16:43:44Z")

</div>

> [@bep](#):
>
> 1853 is too specific, so I don’t think that will get into Hugo master.

I agree that custom content types offer much more flexibility. #1853 could be used as a reference for a template that creates a search index. I favor your appraoch @bep, so should I close my pull request?

---

<div class="post-metadata">

**Author:** ![bep](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/bep/32/3332_2.png) [@bep](https://discourse.gohugo.io/u/bep)\
**Post date:** [December 26, 2016, 4:54pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/12 "2016-12-26T16:54:53Z")

</div>

> [@digitalcraftsman](#):
>
> so should I close my pull request?

You might as well.

---

<div class="post-metadata">

**Author:** ![digitalcraftsman](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/digitalcraftsman/32/4142_2.png) [@digitalcraftsman](https://discourse.gohugo.io/u/digitalcraftsman)\
**Post date:** [December 26, 2016, 5:04pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/13 "2016-12-26T17:04:44Z")

</div>

#1853 has been closed.

---

<div class="post-metadata">

**Author:** ![Jura](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/jura/32/13581_2.png) [@Jura](https://discourse.gohugo.io/u/Jura)\
**Post date:** [January 1, 2017, 2:27pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/14 "2017-01-01T14:27:14Z")

</div>

Does someone know how to address the issue below? I’m also happy with a suggestion or speculation. 🙂

> [@Jura](#):
>
> The htmlEscape function looks promising from the description in the docs, but I don’t know how to use it. All of the ways mentioned below generate an error in general.go for me.
> 
> {{ .PlainWords htmlEscape }}  
> {{ htmlEscape .PlainWords }}  
> {{ .PlainWords | htmlEscape }}  
> {{ htmlEscape (printf .PlainWords “%v”) }}  
> {{ htmlEscape (printf .PlainWords “%s”) }}
> 
> ERROR: 2016/12/25 18:06:00 general.go:236: Error while rendering page json-search-index.md: template: theme/json/single.html:11:24: executing "theme/jso.

---

<div class="post-metadata">

**Author:** ![Jura](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/jura/32/13581_2.png) [@Jura](https://discourse.gohugo.io/u/Jura)\
**Post date:** [January 2, 2017, 5:53pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/15 "2017-01-02T17:53:42Z")

</div>

I succeeded in creating a valid JSON index in Hugo by using `{{ htmlEscape .Plain }}`. 🙂

@bep Do you want me to file a Github issue for the error message that `{{ htmlEscape .PlainWords }}` generates? If `.PlainWords` is undocumented because it’s unsupported, then I don’t want to add to the work that the developers already have.

---

<div class="post-metadata">

**Author:** ![bep](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/bep/32/3332_2.png) [@bep](https://discourse.gohugo.io/u/bep)\
**Post date:** [January 2, 2017, 5:59pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/16 "2017-01-02T17:59:40Z")

</div>

Not sure what error message you refer to, but I fixed the “clipped off error messages” in Hugo 0.18.1. PlainWords is supported, not sure why it’s undocumented.

---

<div class="post-metadata">

**Author:** ![rhewitt](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/rhewitt/32/332_2.png) [@rhewitt](https://discourse.gohugo.io/u/rhewitt)\
**Post date:** [February 28, 2017, 3:06pm UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/17 "2017-02-28T15:06:59Z")

</div>

Would you mind posting your final solution until we get native support for custom output formats?

---

<div class="post-metadata">

**Author:** ![Jura](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/jura/32/13581_2.png) [@Jura](https://discourse.gohugo.io/u/Jura)\
**Post date:** [March 1, 2017, 7:36am UTC](https://discourse.gohugo.io/t/solved-how-to-create-a-json-index-when-theres-code-in-content/4901/18 "2017-03-01T07:36:13Z")

</div>

My final solution was the code in the opening post coupled with what I said in the previous post that generates the content in a valid approach.

Or do you mean something else? (Because I did not create a custom output (i.e., a `.json` file) format you’re asking about.)