Hugo sites over 100,000+ pages (issues, remedies)?

support
1

I am in process of creating 2 sites with hugo, one is around 120k pages and the other far exceeds this.

Running already into issues on the build process on my M1 mac laptop. 80k pages minified was too much on it.

It would be great to hear from my Hugo developers on ways Hugo allows for scaling for large websites with even over 1 million pages. Also is there any support for incremental builds with Hugo? As if there was a large site that had a content folder such as “articles” that they needed to add more to the site each month and already had 20k but wanted to add 5k more. Instead of rebuilding all pages each time it would be great for a way to incrementally build.

Any thoughts, advice direction would be appreciated.

Just wanted also to express to Hugo developers who see this how much I love the system, and ease of use once you understand the Hugo logical layout for content.

1 Like
2

https://gohugo.io/configuration/segments/

3

Wow not sure how I missed this documentation page! So if I am seeing this correctly I could split up the content data into segments by folder so all files in a specific folder would be written.

   [[segments.segment1.includes]]
      path = '{/docs,/questions/**}'

so for instance this above segment would be calling for all files in the /questions/ folder to be processed if I ran the command on build:

hugo --renderSegments segment1

Do you have to state all the exclusions or can you just call for that one folder? If this folder is processed with MD files would hugo also update taxonomy?

Can you elaborate abit on this. Thank you so much for the link.

4

I also found out that

hugo --minify

is quite server intensive, running just

hugo

takes much less resources with lots of pages, I think minification takes 2-4 mins longer. One solution I am thinking is to run

npx html-minifier-terser --input-dir public --output-dir public --collapse-whitespace --remove-comments --minify-css true --minify-js true

that terser afterward for segements of the files at a time to make it managable resource wise on my M1 mac laptop.

5

To familiarize yourself with this feature, I suggest creating a small test site and running some experiments. You can build a functional test site with:

hugo new theme my-site --themesDir .

The syntax above is a bit strange, but it works.

6

The --renderSegments flag is just about RENDERING, you will still have up to date access to all meta data for all files (even those outside of the path Globs).

In my experience, for big sites, the rendering is what takes the bulk of time/memory. So if you have a way of partition old/static content with paths, that would be a great remedy.

7

What I am doing is just bulking content into build folders and running them ie 1-20k md files per folder:

questions/build1.0
questions/build2.0
questions/build3.0

In the front matter for each md file I specify the the url path structure via the slug:

---
title: "coly, what does it mean?"
summary: "The term 'coly' is an archaic or dialectal word with multiple meanings, including 'dark' or 'black,' and it may also refer to a type of bird or a surname."
date: 2025-07-12T00:00:00
slug: "col/coly-what-does-it-mean"
topics: "Term"
type: "questions"
---

how would I have topics sort into the folders subsystem ie /public/te/term

we do have the 2 letter folders made ie: ab, te, etc here is the file for one:

---
title: "AB Topics"
date: 2025-07-07T12:05:00+05:30
layout: "topiclist"
letters: "ab"
---

Browse through our extensive list of topics starting with AB . Each topic page provides an overview and a comprehensive list of questions and answers to help you learn more.

currently by default hugo just puts them all the topics folder ie.
/public/topics/ab/
/public/topics/term/
/public/topics/angicanism/
/public/topics/te

etc