First your average duration is very high (at least for whyt I used to see).
For me it takes 11ms to build 8 pages. And I am very sure that on my machine (worse CPU and RAM then you) the IOs are bottlenecking and not the CPU or the RAM.
I am not sure tho how good HUGO is when it comes to multicore support etc. But may you want to give use some more infos about:
how much is the computer (CPU/RAM/IO) utilized when building?
what IO storage device (HDD, SSD, NVMe?) do you have?
I personal run everything what is related to web and to servers on linux but running on windows should not impact this to much.
Anyway a “partial build” command would be nice to first check where changes has been made and then render these pages (and the respecting other files like sitemap etc) instead of always the whole website.
This at least looks like it does not utilize all cores effectively.
More shocking for me is that it utilizes 46GB RAM. But its still not bottlenecking as you have more.
The 100 in the left column tells you to replace all calls to partial/assets.html with partialCached. Then the most time is used up in taxonomy/list.html and partials/paginator.html. Trying to increase the “posts per page” value will decrease your pagination time. But that will always be the most time-hungry task. Try working with the internal tools like nextpage/prevpage instead of a nice numbered pagination (if you use one). Not sure about the taxonomy/list.html issues, but if you post the layout here we might find out about it.
The index.html shouldn’t take that long, because it’s just one page (the home page).
And lastly: There might be something wrong with your seo partial, because I would expect it to be a 0% cacheable layout. It should be much different per page or it won’t help with any seo if it is the same over multiple pages.
So that’s a lot of pagination for a large site, with very little functional return. I can’t imagine anyone paging through (75000/10 = 7500) pages to find the post they reallly, really want. That’s what search is for.
It seems like using a “recent posts” approach might be better. For example, limit the home page to the 30 most recent posts, and page through that (3 pages of posts instead of 7500).
For the test, I replaced “partial” with “partialCached” in all theme files and it reduced the build time by about 30%, but expectantly the active pagination page visualization stopped working.
When building large projects, the time increases not linearly. When I enable debugging, I see that the starting pagination pages are generated faster than the following pages, with no shortage of resources. With a large number of pages it may takes up to several minutes to generate one pagination page. I think there is a problem with the processing logic.
When the pagination size is increased the speed of building increases significantly, but it still takes a lot of time.
I understand the general point of speed vs volume, and the general interest for this.
But just for my own sake, @Andrey1 what is the rationale for you keeping the pagination for 440k post ?
I think this is actually a good challenge, because is shows problems in scalablility.
If big Players like NewYork Times etc once want to switch over to Hugo with their 20million subsites how long would it take to generate and how to optimize that render/build time. Even for small sites people should optimize their render/build times as it safes time.
I always try to see thing in extremities, very less pages, many pages to identify bottlenecks and to makes things more effective and efficient. So I personally like the topic and that he could at least speed up build time by about 30%!
I think everyone agree with you, especially Bep who already stated clearly that he has some good ideas about working on those area, and some WIP.
But the point for this main post is : What is the use case for having a full pagination work for 750.000 pages. Where is the human who will search something this way ?
So even if 750.000 pages with pagination will build under 10sec, I suspect this doesn’t makes sense from a design/UI/user point of view
Clearly, the conclusion is that it’s easier to cut out pagination. But what about the tags, as a result, the tags will also contain 10,000 posts, we cut them out as well. As a result, we cut everything out and leave only the search to the user.
Despite the seeming logic: who will flip through a lot of posts on the page, this is not logical, since most sites have full pagination.
The store may need it, and hugo is not particularly good for stores, but it can still be made for a store. On portfolio sites it would be nice to have pagination too. Websites of illustrators, there is also no need for a search.
I’m not arguing against pagination.
Pagination is good, and your examples are fine.
Point is how much objects will you paginate ?
A shop with 100 page for searching items ? Mmmmm, not sure about his conversion rate.
A portfolio with 100 pages ? Mmmmm are you giving up before 90 ? before 70 ?
since most sites have full pagination.
And all those site are efficients for their goals ?
That is as if you would say “it’s better not to transplant hearts anymore, they take to long to transplant.”
Your main aim should be the usability and accessibility of the resulting website (apart from interesting content) for the end-user, not how convenient or inconvenient the act of creating the site is.
If a page is a catalogue page of for instance all marvel characters, there is no need to have a pagination so people can go 10 characters forward or 20 characters forward. Have a proper search so I can skip right to Korg.
You could sort your taxonomies like it’s done on wikia, with a small box for each item with a title and a link, instead of a full-page preview. This way you can put more items on one list page and reduce the time to compile pagination for that (50 per page, 100 per page).
And lastly: if it’s indeed a weblog with thousands of entries, why not trying something along the lines of prev-page, next-page (for the timeline readers) and prev-month, prev-year, next-month, next-year (for the skimmers). this would be pseudo-static for the month/year (partialCached per month/year) and prev/next might be easier on the compilation time.
From my practice, it follows that through pagination, tags, and filters Google is better at indexing sites, including online stores, than other ways. Again, this is an assumption based only on the results of practical experiments and it may not coincide with other assumptions in other cases.
As you may have noticed, @jmooring and @bep are currently working on speeding up builds, so maybe there will be a hugo release in the near future. That’s great. The faster build speed will allow me to move from Django to static sites on many projects, and improve, among other things, the recent relevance of Core Web Vitals.
The posted metrics are quite clear: