Custom Robots.txt and sitemap.xml Templates

tips & tricks
21

Hi - that was confusing. I rewrote it. It can be any page you want to exclude. Once you add the param, it should show up in the robots.txt.

Regarding the other question, could you make a new post please. From what you pasted it is hard to tell, and I want to ask if you could please have a look at Requesting Help and provide some more details, in the new post? Thanks.

22

@RickCogley
Loving your code, thanks.

I noticed an issue: Multilingual pages, like example.ru.md, do not get added to robots.txt.

I have enableRobotsTXT = "true" in front matter of both files example.md and example.ru.md.

I expect to get two entries in robots.txt:


/example/
/ru/example/

But I only get one /example/

What do you think is the problem?

23

Hmm, I imagine it is something with the range statement. I don’t have time to set up a test but, can you try {{ .Permalink }} instead? Or, maybe it needs absLangURL.

1 Like
24

Thanks, will give it a try later.

25

Tried using {{ .Permalink }} instead of {{ .RelPermalink }}. Did not work. It just gave me an absolute url output.

Then I tried using{{ .RelPermalink | relLangURL }} and absLangURL. Did not work either.

Did some more research but can not figure it out yet. Any ideas?

26

@RickCogley

Hi,

I have a sitemap.xml and robots.txt given by my client’s SEO person.

As I’m sure about how to connect the steps given by you to my files,

kindly help on the same

sitemap.xml

    <?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
    <!-- Generated by Web-Site-Map.com -->
    <url>
        <loc>https://www.prabhatha.com/</loc>
        <changefreq>weekly</changefreq>
        <priority>1.00</priority>
    </url>
    <url>
        <loc>https://www.prabhatha.com/join-prabhat/register/</loc>
        <changefreq>weekly</changefreq>
        <priority>0.85</priority>
    </url>
    <url>
        <loc>https://www.prabhatha.com/about/about_prabhat/</loc>
        <changefreq>weekly</changefreq>
        <priority>0.85</priority>
    </url>
    <url>
        <loc>https://www.prabhatha.com/repertoire/home/</loc>
        <changefreq>weekly</changefreq>
        <priority>0.85</priority>
    </url>
    <url>
        <loc>https://www.prabhatha.com/training/home/</loc>
        <changefreq>weekly</changefreq>
        <priority>0.85</priority>
    </url>
    <url>
        <loc>https://www.prabhatha.com/costumes/home/</loc>
        <changefreq>weekly</changefreq>
        <priority>0.85</priority>
    </url>
    <url>
        <loc>https://www.prabhatha.com/gallery/home/</loc>
        <changefreq>weekly</changefreq>
        <priority>0.85</priority>
    </url>
</urlset>

and robots.txt

User-agent: *

Sitemap: https://www.prabhatha.com/sitemap.xml
27

@TenSketch Best would be to open a new topic / request, and share your repo etc w/ what you tried, so someone can assist.