Intro The docs have an example of how to block every page using robots.txt but, I wanted to make a robots.txt template that blocks only pages with a certain frontmatter param. As others have pointed out in the comments and elsewhere, robots.txt is not a sure thing. Web crawlers have to be set to ho…

[ThankYou-BW] Lol! That’s the last thing on my mind, currently. You might want those terms and conditions pages to be indexed/searched, though. I know I would, if I were to sign up for something. I forget those policies and don’t always know what it is I’ve signed up for. It’d be nice if they’…

[image] KalikaKay: You might want those terms and conditions pages to be indexed/searched, though. I know I would, if I were to sign up for something. I forget those policies and don’t always know what it is I’ve signed up for. It’d be nice if they’d pop up before I used your results. Actu…

That’s a perfect example of the use of the robot.txt file and I appreciate that now, do I not only know the purpose of it, I know how and when to use it. I can also choose not to use it and if I do enable the setting, I’ll still need to add some configurations to get it working properly. […

Thank you for this great tutorial. When I wrote {{ with .Params.robotsdisallow }}<meta name="robots" content="noindex">{{ end }} in head.html, then I get a line for the meta tag the have the line enableRobotsTXT = "true" in the frontmater. <meta name="robots" content="noindex"> But all other pag…

@Joerg , extending my {{ with ...}}, I would do: {{ with .Params.robotsdisallow }}<meta name="robots" content="noindex, nofollow, noarchive">{{ else }}<meta name="robots" content="index, follow, archive">{{ end }} Just tested and it works. If you just need content="index, follow", then just leave t…

Extended the tutorial a bit with a section on setting sitemap changefreq and priority in frontmatter. See above.

[image] RickCogley: Good point. Looks like you’d need to do something like: [image] Allow taxonomy with limitations support This would still only work for well-behaved search crawlers, just as a warning to the unfamiliar. People can still download anything they want for free from the command…

Yeah, all these things are “polite requests” only. :wink:

Hi, Rick. I’ve got confused with my robots.txt. Trying follow thar artical Custom Robots.txt and sitemap.xml Templates Something going wrong((( I can not find my Robots.txt and I have three Layout (take a look a scan) That’s ok? [layouts_scan] Every time I have 404 page instead

Custom Robots.txt and sitemap.xml Templates

tips & tricks

RickCogley June 1, 2018, 8:54am 17

Extended the tutorial updating the custom sitemap template by adding a default “x-default” lang, recommended by Google and Yoast.

Topic		Replies	Views
Do not index certain pages support	8	931	July 2, 2020
Custom robots.txt template not working support taxonomy	5	1226	March 21, 2022
Where to put enableRobotsTXT support	7	1954	June 22, 2020
Multilingual site always get a noindex tag in root index.html support	4	2152	September 4, 2018
Crawling is blocked	1	741	February 2, 2020

Custom Robots.txt and sitemap.xml Templates

Related topics