How to configure robots(search engines) to follow and to index?

This appears in the published site:


I’d like search engines to crawl and index the entire site, every page, to index and to follow.

How do I accomplish that?

This will depend on the Theme that you’re using. I can’t tell from your post, so I’m really guessing here, but check your theme’s help documentation - at a guess there should be a setting on your config.toml file for this somewhere.

If you need more info on creating a robots file, check here

1 Like

Based on your recent post I assume you still are using Anake. Please ask that question in their issue tracker or post a link to your repo here with details WHERE the error comes up. This is a theme related issue and probably has to be solved with front matter.

1 Like

Thanks, that link is helpful. I see that there’s a robot.txt in the root of the site and crawling preferences can be specified in meta tags.

Thanks for pointing me in the right direction. I was able to solve the problem.

Yes, the problem is related to the Ananke theme.

The theme uses input from robots.txt is in /themes/ananke/layouts/robots.txt which has the following code:

There’s a an error in the code above: Both cases result in Disallow. For production, changed Disallow: to Allow: /

Then used the following command to set the OS environment variable:
HUGO_ENV=production export HUGO_ENV

Now it works. It creates a robots.txt file in /public and has the correct meta data in the site pages.

1 Like