Support for llms.txt standard for AI crawlers

support
3

I generate an llms.txt file from within the robots.txt layout using a custom definition llms-txt: https://example.com/llms.txt, similar to the Sitemap: https://example.com/sitemap.xml definition. I use a frontmatter value for pages called sitemap-exclude, which can be set to true to exclude a page from the sitemap, and then I exclude this from the llms.txt as well.

The primary reason for including it in the robots.txt file is for loading the .Permalink so the template can be stored in the assets directory instead of static.

The following is an example that produces 3 levels of sections and includes titles, permalinks, and descriptions for each page.

config/_default/params.toml

[robots]
  llmsTXT = true

layouts/robots.txt

User-agent: *
Sitemap: {{ "sitemap.xml" | absURL }}
{{ range where .Pages "Params.sitemap_exclude" "eq" true }}
Disallow: {{ .RelPermalink }}{{ end }}

{{/* LLMS */}}
{{- $llmsGoTXT := resources.Get "llms.go.txt" -}}
{{- if and $llmsGoTXT .Site.Params.robots.llmsTXT -}}
  {{- $llmsTXT := $llmsGoTXT | resources.ExecuteAsTemplate "llms.txt" . -}}
  llms-txt: {{ $llmsTXT.Permalink }}
{{- end -}}

assets/llms.go.txt

{{ with .Site.Title -}}
    # {{ . }}
{{- end }}

{{ with .Site.Params.Description -}}
> {{ . }}
{{- end }}

{{ range (where (sort ((.Site.GetPage "/").Pages) "Weight" "asc" "Date" "desc" "Lastmod" "desc") "Params.sitemap_exclude" "ne" true) -}}
    - [{{ .Title }}]({{ .Permalink }}): {{ .Description }}
{{ end -}}

{{/* Sections */}}
{{ range (where (sort ((.Site.GetPage "/").Sections) "Weight" "asc" "Date" "desc" "Lastmod" "desc") "Params.sitemap_exclude" "ne" true) -}}
{{ with .Title -}}
    ## {{ . }}
{{- end }}

{{ with .Description -}}
    > {{ . }}
{{- end }}

{{ range (where (sort .Pages "Weight" "asc" "Date" "desc" "Lastmod" "desc") "Params.sitemap_exclude" "ne" true) -}}
    {{ if .Title -}}
        - [{{ .Title }}]({{ .Permalink }}){{ with .Description }}: {{ . }}{{ end }}
    {{- end }}
{{ end -}}

{{/* Sub-Sections */}}
{{ range (where (sort .Sections "Weight" "asc" "Date" "desc" "Lastmod" "desc") "Params.sitemap_exclude" "ne" true) -}}
{{ with .Title -}}
    ### {{ . }}
{{- end }}

{{ with .Description -}}
    > {{ . }}
{{- end }}

{{ range (where (sort .Pages "Weight" "asc" "Date" "desc" "Lastmod" "desc") "Params.sitemap_exclude" "ne" true) -}}
    {{ if .Title -}}
        - [{{ .Title }}]({{ .Permalink }}){{ with .Description }}: {{ . }}{{ end }}
    {{- end }}
{{ end }}
{{ end -}}

{{ end -}}

Example outputs from this configuration:

2 Likes