SOLVED Unable to set robots.txt content

support
1

I’m trying to set the content of the robots.txt file in my Hugo site, but am unable to change it from just “User-agent: *” (I just want to add two lines after that).

I’ve got the content I want in static/robots.txt, but this seems to not be used or perhaps overwritten.

The site source is at https://github.com/hughgrigg/chineseboost-articles

I’ve used grep and find to try and find where this other content might be coming from. I’m guessing it’s Hugo default.

What should I do to get custom content in robots.txt?

2

Figured it out: the robots.txt file should be at ‘layouts/_default/robots.txt’

3

@hughgrigg did you try just putting your robots.txt in the static folder? The rule for the static folder is that everything therein is copied over as is.

4

Yes, but it is (sometimes) useful to build it from a template (iterate over all the pages etc.)