Do not index certain pages

hendry · June 30, 2020, 6:44am

Hi! Say I have content that looks like

content/
├── 2020
│   └── first-blog.md
└── internal-boring-do-not-index
    └── ref.md

How do I prevent internal-boring-do-not-index from appearing in:

index.xml
sitemap.xml
internal-boring-do-not-index/index.xml

I don’t want to use robots.txt because I don’t want to draw attention to this content.

Many thanks!

martignoni · July 2, 2020, 9:40am

I do this with a parameter in front matter:

---
title: This page will be excluded from sitemap
robotsdisallow: true
---

Then using such a construct in your template, e.g. sitemap.xml:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
  xmlns:xhtml="http://www.w3.org/1999/xhtml">
  {{ range where .Site.RegularPages "Params.robotsdisallow" "!=" true }}
    ...
  {{ end }}
</urlset>

Live example here

github.com

moodlebox/moodlebox.net/blob/master/layouts/_default/sitemap.xml

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
  xmlns:xhtml="http://www.w3.org/1999/xhtml">
  {{ range where .Site.RegularPages "Params.robotsdisallow" "!=" true }}
  <url>
    <loc>{{ .Permalink }}</loc>{{ if not .Lastmod.IsZero }}
    <lastmod>{{ safeHTML ( .Lastmod.Format "2006-01-02T15:04:05-07:00" ) }}</lastmod>{{ end }}{{ with .Sitemap.ChangeFreq }}
    <changefreq>{{ . }}</changefreq>{{ end }}{{ if ge .Sitemap.Priority 0.0 }}
    <priority>{{ .Sitemap.Priority }}</priority>{{ end }}{{ if .IsTranslated }}{{ range .Translations }}
    <xhtml:link
                rel="alternate"
                hreflang="{{ .Lang }}"
                href="{{ .Permalink }}"
                />{{ end }}
    <xhtml:link
                rel="alternate"
                hreflang="{{ .Lang }}"
                href="{{ .Permalink }}"
                />{{ end }}
  </url>
  {{ end }}

This file has been truncated. show original

bep · June 30, 2020, 7:47am

Also see

hendry · June 30, 2020, 8:49am

Really struggling to understand those confusing build options.

[hendry@t480s show]$ hugo version
Hugo Static Site Generator v0.73.0/extended linux/amd64 BuildDate: unknown
[hendry@t480s show]$ cat content/do-not/foo.md
---
title: Do no index me
_build:
  render: false
  list: never
  publishResources: false
---


Boo!
[hendry@t480s show]$ grep do-not public/index.xml
[hendry@t480s show]$ vim public/index.xml
[hendry@t480s show]$ grep not public/index.xml
[hendry@t480s show]$ grep not public/sitemap.xml
    <loc>http://example.org/do-not/</loc>

I’m probably better off creating a Go template.

bep · June 30, 2020, 9:04am

That is not the same URL as the one you “unlisted”.

hendry · June 30, 2020, 9:57am

I’m sorry, Why? Because it’s a section?

bep · June 30, 2020, 10:20am

Because it’s not where you foot the _build configuration. I’m not sure how to explain it differently.

hendry · July 1, 2020, 3:14am

Oh, so I need to use the _index.md with _build stanzas to exclude a section?

Did I understand correctly?

hendry · July 2, 2020, 9:40am

Thanks @martignoni … I ended up with

 ...
  {{ range .Data.Pages -}}
  {{- if or (eq .Section "services") (eq .Section "2020")}}
...

in my layouts/_default/sitemap.xml