'hugo server' is very slow for me

I run hugo server for previewing my site locally with this PowerShell script:

[console]::WindowWidth=120
[console]::WindowHeight=30
[console]::BufferWidth=[console]::WindowWidth

$host.ui.RawUI.WindowTitle = "Test Server"

Set-Location -LiteralPath (Split-Path -Parent (Split-Path -Parent $MyInvocation.MyCommand.Path))
Remove-Item "public" -Recurse -ErrorAction "Ignore"
Remove-Item "resources" -Recurse -ErrorAction "Ignore"
# Fast Render: Home page + Content page last edited (if any) + Content page last visited (up to 10)
hugo server ā€“renderToMemory --printMemoryUsage --disableFastRender -D -p 80

But site rebuilding takes too much time, especially if only few words has been changed in single file.

Watching for changes in [PATH TO SITE SOURCE]\{archetypes,assets,content,data,i18n,layouts,package.hugo.json,package.json,static}
Watching for config changes in [PATH TO SITE SOURCE]\config\_default, [PATH TO SITE SOURCE]\config\development, [PATH TO SITE SOURCE]\go.mod
Start building sites ā€¦
hugo v0.125.7-b1d808bc373f53ad37c8966bb02a6aea095db5f8+extended windows/amd64 BuildDate=2024-05-08T14:46:24Z VendorInfo=gohugoio

Alloc = 402.7 MB
TotalAlloc = 1.2 GB
Sys = 536.5 MB
NumGC = 24

Alloc = 914.6 MB
TotalAlloc = 2.2 GB
Sys = 967.6 MB
NumGC = 26

Alloc = 1.5 GB
TotalAlloc = 3.2 GB
Sys = 1.6 GB
NumGC = 27

Alloc = 1.2 GB
TotalAlloc = 4.4 GB
Sys = 1.9 GB
NumGC = 29

Alloc = 1.2 GB
TotalAlloc = 5.3 GB
Sys = 2.0 GB
NumGC = 30

Alloc = 1.6 GB
TotalAlloc = 6.5 GB
Sys = 2.1 GB
NumGC = 31

                   | KO
-------------------+------
  Pages            | 501
  Paginator pages  | 203
  Non-page files   |   1
  Static files     |   4
  Processed images |   9
  Aliases          |  62
  Cleaned          |   0

Built in 30342 ms

Alloc = 1.8 GB
TotalAlloc = 6.6 GB
Sys = 2.1 GB
NumGC = 31

Environment: "development"
Serving pages from disk
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Press Ctrl+C to stop

Change detected, rebuilding site (#1).
2024-05-12 18:03:02.704 +0900
Source changed /b/game/the-division/016-eagle-sharer-2/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 71126 ms

Change detected, rebuilding site (#2).
2024-05-12 18:07:37.207 +0900
Source changed /b/game/the-division/016-eagle-sharer-2/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 68745 ms

Change detected, rebuilding site (#3).
2024-05-12 18:09:54.698 +0900
Source changed /b/game/the-division/001-happy-picture-mode/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 66554 ms

Change detected, rebuilding site (#4).
2024-05-12 18:11:01.253 +0900
Source changed /b/game/the-division/001-happy-picture-mode/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 70131 ms

Change detected, rebuilding site (#5).
2024-05-12 18:12:16.211 +0900
Source changed /b/game/the-division/002-todays-division-2/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 69935 ms

As you can see, initial site building time is not that bad (around 30 seconds), but site rebuilding takes too much time (more than 60 seconds), even if all Iā€™ve changed is just a word in single file.

I also tested if removing --disableFastRender would speed up rebuild process, but that didnā€™t change anything. Site rebuilding still takes long time, more than 60 seconds.

Since both Hugo executable and site source is on SSD, I donā€™t think it is related to slow I/O.

Here is what hugo --templateMetrics --templateMetricsHints shows. I posted it in separate site because it is too long.

This is what hugo env says.

hugo v0.125.7-b1d808bc373f53ad37c8966bb02a6aea095db5f8+extended windows/amd64 BuildDate=2024-05-08T14:46:24Z VendorInfo=gohugoio
GOOS="windows"
GOARCH="amd64"
GOVERSION="go1.22.2"
github.com/sass/libsass="3.6.5"
github.com/webmproject/libwebp="v1.3.2"

What can be the reason of this slow site rebuild? How can I solve it?

PS. Iā€™ve set my site source repository as private. If you need anything about my siteā€™s source, please tell me.

Would you mind sharing your repo to me for reproducing this issue?

1 Like

I added you as collaborator in my repository.

I was thinking about asking to you if this was not Hugoā€™s problemā€¦ What a coincidence!

1 Like

OK, will check later, seems the repo is too large, need to time to clone.

1 Like

Things making hugo server slow (in my experience):

  • pagination with page numbers in it ( like << < 1 2 3 current 5 6 7 > >>)
  • images and other resources that are not cached
  • the related posts feature of hugo (using where and some magic works better)
  • when your layout files have a loop that ā€œneedsā€ to time out by itself.

To debug it do run the server with these parameters:

  --printI18nWarnings        print missing translations
  --printMemoryUsage         print memory usage to screen at intervals
  --printPathWarnings        print warnings on duplicate target paths etc.
  --printUnusedTemplates     print warnings on unused templates.
  --templateMetrics          display metrics about template executions
  --templateMetricsHints     calculate some improvement hints when combined with --templateMetrics

The metrics hints are great to see what template needs many re-runs and lots of time.

3 Likes

I realized you linked your template metrics.

I think your image generation is faulty. It needs 23k runs and 3+ minutesā€¦ partials/images/image.html is a candidate for a deeper look. Further down is 23k runs of a partials/images/picture.html which looks like you re-doing things again. Lots of other time hogs are up there. Look into all templates that take 3min+ to generate and start using caching for repeated things. The cache potential of 59 means, that from 100 runs 59 do the same thing.

I suspect something more sinister along the lines of too many partials doing the same task (something that happens to me a lot).

partials/images/functions/params can be cached 100%. so just replace the partial call with partialCached.

step 1: look what template runs often with a high cache potential and start caching them
step 2: look what other templates run long - probably non-cached images.

I hope you donā€™t delete resources between your runs. Keep that folder and it will speed up much.

All this is a black box response as I do not know your repo and layout files.

1 Like

Here are the source of mentioned partials/images/image.html, partials/images/picture.html and partials/images/functions/params.html. If I understood the code correctly, partials/images/functions/params.html is cached.

Also, I have my own shortcode which calls images/image like this:

{{/* gallery/image shortcode */}}
{{/* This requires github.com/hugomods/images */}}
{{- $srcset := .Get "src" -}}
{{- $align := default "center" (.Get "align") -}}
{{- $width := default "100%" (.Get "width") -}}
{{- $caption := default "" (.Get "caption") -}}

{{- $id := delimit (shuffle (seq 1 9)) "" -}}
{{- $page := .Page -}}
{{- $style := "" -}}

{{- if strings.Contains $srcset "|" -}}
  {{- $srcset = split $srcset "|" -}}
{{- else -}}
  {{- $srcset = slice $srcset -}}
{{- end -}}

{{- if gt (len $srcset) 3 -}}
  {{- errorf "[gallery/image] Too many images. Up to 3 images are allowed. @ %s" $page.Title -}}
{{- else if lt (len $srcset) 1 -}}
  {{- errorf "[gallery/image] Not enough image. At least 1 image is required. @ %s" $page.Title -}}
{{- end -}}

{{- if eq (len $srcset) 1 -}}
  {{- $style = "width: 100%" -}}
{{- else if eq (len $srcset) 2 -}}
  {{- $style = "width: 49.5%" -}}
{{- else -}}
  {{- $style = "width: 32.5%" -}}
{{- end -}}
<div class="gallery" align="{{ $align }}">
  <div id="gallery-container-{{ $id }}" style="width: {{ $width }}">
    <figure class="gallery-figure">
      <div class="gallery-container">
        {{- range $srcset -}}
          {{- $src := (printf "%s" .) | default "" -}}
          {{/* images/image will do stuff like internal/external URL check stuff. Pass src directly to images/image. */}}
          {{- if eq (path.Ext $src) "" -}}
            {{- $src = printf "%s.webp" $src -}}
          {{- end -}}
        <div class="gallery-inner" style="{{ $style }}">{{ partial "images/image" (dict "Page" $page "Filename" $src) }}</div>
        {{- end -}}
      </div>
      {{- if ne $caption "" -}}
      <figcaption>{{ (printf "%s" $caption) | markdownify }}</figcaption>
      {{- end -}}
    </figure>
  </div>
</div>

This shortcode is called a lot (up to 100 times per article), and if I remember correctly, the images/image is also called for every image in markdown format (![]()). This can be the main culprit, but I donā€™t know how should I fix it.

I delete resources directory before running hugo server because I think those generated files are not properly re-generated after source file change. Iā€™ll try commenting the line that deletes resources directory.

I appreciate you for the insight, despite of limited information. I personally donā€™t want to make my stuff public yet, so thatā€™s why I set my repo as private.
Iā€™ll wait for further information from the author of the module, razon, too.


ADD:
Commenting out these line didnā€™t help. It still takes more than 60 seconds to rebuild the site.

Set-Location -LiteralPath (Split-Path -Parent (Split-Path -Parent $MyInvocation.MyCommand.Path))
#Remove-Item "public" -Recurse -ErrorAction "Ignore"
#Remove-Item "resources" -Recurse -ErrorAction "Ignore"
# Fast Render: Home page + Content page last edited (if any) + Content page last visited (up to 10)
hugo server ā€“renderToMemory --printMemoryUsage --disableFastRender -D -p 80

Iā€™ve tested your repo on my WSL2 (hugo v0.125.7), it takes 5-6s to build site, re-render is also fast (1s), Iā€™m not sure if there is a IO issue on your local environment, you can checkout the video on https://streamable.com/5n5zbb (expires in 2 days).

Will check and see how Windows (without WSL2) perform on your site later.

I think I found the culprit: Windows Defender.

I thought I did everything I could do for speeding up (re)build process, so I re-checked these:

  • Location of site source
    The path isnā€™t that long, and it is on SSD, the fastest storage I have.
  • Windows Defender
    I separated directory I store git repositories into two, one on SSD (for my site source), and another one on HDD (for other projects).
    I didnā€™t update Windows Defender exclusion to exclude both directories.

After excluding both directories, the build speed is massively improved.

Watching for changes in [MY SITE SOURCE DIR]\{archetypes,assets,content,data,i18n,layouts,package.hugo.json,package.json,static}
Watching for config changes in [MY SITE SOURCE DIR]\config\_default, [MY SITE SOURCE DIR]\config\development, [MY SITE SOURCE DIR]\go.mod
Start building sites ā€¦
hugo v0.125.7-b1d808bc373f53ad37c8966bb02a6aea095db5f8+extended windows/amd64 BuildDate=2024-05-08T14:46:24Z VendorInfo=gohugoio

Alloc = 1.5 GB
TotalAlloc = 5.0 GB
Sys = 1.9 GB
NumGC = 29

                   | KO
-------------------+------
  Pages            | 501
  Paginator pages  | 203
  Non-page files   |   1
  Static files     |   4
  Processed images |   9
  Aliases          |  62
  Cleaned          |   0

Built in 7744 ms
Environment: "development"
Serving pages from disk

Alloc = 1.3 GB
TotalAlloc = 6.6 GB
Sys = 2.1 GB
NumGC = 31

Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Press Ctrl+C to stop

Change detected, rebuilding site (#1).
2024-05-13 15:53:44.394 +0900
Source changed /b/game/the-division/058-3000/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 18545 ms

Change detected, rebuilding site (#2).
2024-05-13 15:55:28.891 +0900
Source changed /b/game/the-division/017-eagle-eagle-eagle/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 17551 ms
Watching for changes in [MY SITE SOURCE DIR]\{archetypes,assets,content,data,i18n,layouts,package.hugo.json,package.json,static}
Watching for config changes in [MY SITE SOURCE DIR]\config\_default, [MY SITE SOURCE DIR]\config\development, [MY SITE SOURCE DIR]\go.mod
Start building sites ā€¦
hugo v0.125.7-b1d808bc373f53ad37c8966bb02a6aea095db5f8+extended windows/amd64 BuildDate=2024-05-08T14:46:24Z VendorInfo=gohugoio

Alloc = 1.5 GB
TotalAlloc = 6.2 GB
Sys = 2.1 GB
NumGC = 31

                   | KO
-------------------+------
  Pages            | 501
  Paginator pages  | 203
  Non-page files   |   1
  Static files     |   4
  Processed images |   9
  Aliases          |  62
  Cleaned          |   0

Built in 6956 ms

Alloc = 1.6 GB
TotalAlloc = 6.6 GB
Sys = 2.1 GB
NumGC = 31

Environment: "development"
Serving pages from disk
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Press Ctrl+C to stop

Change detected, rebuilding site (#1).
2024-05-13 17:29:33.755 +0900
Source changed /b/chit-chat/001-shit/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 14429 ms

Change detected, rebuilding site (#2).
2024-05-13 17:31:01.252 +0900
Source changed /b/chit-chat/001-shit/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 17031 ms

Change detected, rebuilding site (#3).
2024-05-13 17:34:54.757 +0900
Source changed /b/chit-chat/002-shit-2/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 20631 ms

Change detected, rebuilding site (#4).
2024-05-13 17:35:41.751 +0900
Source changed /b/chit-chat/002-shit-2/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 22453 ms

Change detected, rebuilding site (#5).
2024-05-13 17:43:25.756 +0900
Source changed /b/chit-chat/003-what-mp3-player-to-choose/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 20150 ms

Change detected, rebuilding site (#6).
2024-05-13 17:44:14.759 +0900
Source changed /b/chit-chat/003-what-mp3-player-to-choose/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 24755 ms

Change detected, rebuilding site (#7).
2024-05-13 17:56:53.762 +0900
Source changed /b/chit-chat/004-long-tough-road-of-customization/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 21914 ms

Change detected, rebuilding site (#8).
2024-05-13 18:00:11.757 +0900
Source changed /b/chit-chat/004-long-tough-road-of-customization/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 20437 ms

Change detected, rebuilding site (#9).
2024-05-13 18:00:58.758 +0900
Source changed /b/chit-chat/004-long-tough-road-of-customization/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 24863 ms

Change detected, rebuilding site (#10).
2024-05-13 18:01:36.254 +0900
Source changed /b/chit-chat/004-long-tough-road-of-customization/index.md
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 27380 ms

Change detected, rebuilding site (#11).
2024-05-13 18:03:59.253 +0900
Asset changed /hb/modules/custom/scss/index.scss
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 23810 ms

Change detected, rebuilding site (#12).
2024-05-13 18:09:30.860 +0900
Asset changed /hb/modules/custom/scss/index.scss
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 26692 ms

Change detected, rebuilding site (#13).
2024-05-13 18:14:48.246 +0900
Asset changed /hb/modules/custom/scss/index.scss
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 28016 ms

Change detected, rebuilding site (#14).
2024-05-13 18:16:45.751 +0900
Asset changed /hb/modules/custom/scss/index.scss
Web Server is available at http://localhost:80/ (bind address 127.0.0.1)
Total in 28418 ms

I need to test few more times to make sure it is really improved or not. Because, I rarely had few times that (re)build time was fast like this, even when Windows Defender still scans these directories.

This isnā€™t fast like running on WSL, but I think this is improvement.

  • Initial Build Time: 30 seconds ā†’ 8 seconds (6 second on WSL)
  • Rebuild Time: >60 seconds ā†’ 18~30 seconds (not sure on WSL)

If I need more speed, Iā€™ll try Dev Drive with Windows Defender filter detached, but Iā€™m afraid that Dev Drive VHD wonā€™t be mounted after reboot.

Thank you for testing!

PS. If you are no longer interested in further testing, please tell me and Iā€™ll wrap up things.

Windows is slower than Linux (same machine), it takes 8/9s to build site, and 1/2s to re-render on changes. Will check if there is a cacheable templates to improve the build performance, track this on Improve build time Ā· Issue #823 Ā· hbstack/blog Ā· GitHub.

You can remove me from your repoā€™s collaborators.

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.