Good SEO practice of netlify deployment url and custom domain

regis · February 2, 2018, 12:25pm

Hugo can generate robots.txt for you. Allow or disallow according to the environnement (Netlify or other) this is how I did it for this project:

User-agent: *
{{ if ne ( getenv "HUGO_ENV" ) "production" -}}
Disallow: /
{{- else -}}
Allow: /
Disallow: bogus/
{{- end -}}

You must enable robots txt generating in your configuration file. Checkout the doc.

Topic		Replies	Views
Deployment to netlify with custom domain still shows netlify in metadata support	6	884	March 11, 2020
URL in Subpages is pointing to the wrong URL support	5	732	May 28, 2018
Next in Section link points to Netlify subdomain instead of baseURL support netlify	2	693	January 30, 2021
Using Netlify to Deploy support	4	658	August 13, 2019
Should my base URL have www or non-www URL? support configuration	9	1015	June 27, 2023