How to configure robots(search engines) to follow and to index?

support
1

This appears in the published site:

META NAME=“ROBOTS” CONTENT=“NOINDEX, NOFOLLOW”

I’d like search engines to crawl and index the entire site, every page, to index and to follow.

How do I accomplish that?

2

This will depend on the Theme that you’re using. I can’t tell from your post, so I’m really guessing here, but check your theme’s help documentation - at a guess there should be a setting on your config.toml file for this somewhere.

If you need more info on creating a robots file, check here https://support.google.com/webmasters/answer/6062596?hl=en

1 Like
3

Based on your recent post I assume you still are using Anake. Please ask that question in their issue tracker or post a link to your repo here with details WHERE the error comes up. This is a theme related issue and probably has to be solved with front matter.

1 Like
4

Thanks, that link is helpful. I see that there’s a robot.txt in the root of the site and crawling preferences can be specified in meta tags.

5

Thanks for pointing me in the right direction. I was able to solve the problem.

Yes, the problem is related to the Ananke theme.

The theme uses input from robots.txt is in /themes/ananke/layouts/robots.txt which has the following code:

There’s a an error in the code above: Both cases result in Disallow. For production, changed Disallow: to Allow: /

Then used the following command to set the OS environment variable:
HUGO_ENV=production export HUGO_ENV

Now it works. It creates a robots.txt file in /public and has the correct meta data in the site pages.

1 Like
6

Hi Team,

i want how to add Robot.txt on this theme pack : Mainroad | Hugo Themes

7

Post here