Hugo is crashing upon start of "Hugo server -D" when serving a lot of images

Hi frjo, thank you.

We’re talking about the whole website, right? There’s about 750 MB of data in it because of all the images. I didn’t setup a repository so far but I could. Is there a preferred platform, like GitHub, GitLab or anything else?

Oh, btw, is there any sensitive data I should remove from any of the config files before uploading them to a public Git repository?

I mostly use GitHub myself but it doesn’t really matter.

There should be anything sensitive to worry about unless you have put it there.

Here it is: https://github.com/ulfklose/blog.ulfklose.de

Thanks for your help.

I see that the folder of the theme you are using is empty. we will need to see that. there seems to be an issue with processing something

Can you try hugo server --renderToDisk and see what happens.

That might be because I did something wrong while committing to the Git repository as some themes are submodules while others aren’t as far as I understood so far. The behavior doesn’t change when changing the theme in use so I guess it’s not a theme related problem.

Meh.

❯ hugo server --renderToDisk
Start building sites …

                   |  EN
-------------------+-------
  Pages            | 1084
  Paginator pages  |  141
  Non-page files   | 2675
  Static files     |   23
  Processed images |    0
  Aliases          |  305
  Sitemaps         |    1
  Cleaned          |    0

Built in 5845 ms
Watching for changes in /Users/ulf/Sites/blog.ulfklose.de/{archetypes,content,data,layouts,static,themes}
Watching for config changes in /Users/ulf/Sites/blog.ulfklose.de/config.toml
fatal error: pipe failed

goroutine 1 [running]:
runtime.throw(0x5c92adc, 0xb)
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/panic.go:1116 +0x72 fp=0xc002adf860 sp=0xc002adf830 pc=0x4039332
runtime.sigNoteSetup(0x6d7c740)
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/os_darwin.go:98 +0xc5 fp=0xc002adf888 sp=0xc002adf860 pc=0x4036085
os/signal.signal_enable(0xd181da6a00000002)
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/sigqueue.go:198 +0xa5 fp=0xc002adf8a8 sp=0xc002adf888 pc=0x406dc05
os/signal.enableSignal(...)
	/usr/local/Cellar/go/1.15.2/libexec/src/os/signal/signal_unix.go:49
os/signal.Notify.func1(0x2)
	/usr/local/Cellar/go/1.15.2/libexec/src/os/signal/signal.go:144 +0x88 fp=0xc002adf8c8 sp=0xc002adf8a8 pc=0x55ae748
os/signal.Notify(0xc001353320, 0xc002adfae0, 0x2, 0x2)
	/usr/local/Cellar/go/1.15.2/libexec/src/os/signal/signal.go:164 +0x162 fp=0xc002adf940 sp=0xc002adf8c8 pc=0x55ae162
github.com/gohugoio/hugo/commands.(*commandeer).serve(0xc000a6c270, 0xc0000ae900, 0x2f, 0x40)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/commands/server.go:480 +0x625 fp=0xc002adfb60 sp=0xc002adf940 pc=0x55e6e25
github.com/gohugoio/hugo/commands.(*serverCmd).server(0xc0000ae900, 0xc00038bb80, 0xc00059df70, 0x0, 0x1, 0x0, 0x0)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/commands/server.go:274 +0x2b6 fp=0xc002adfca8 sp=0xc002adfb60 pc=0x55e55f6
github.com/gohugoio/hugo/commands.(*serverCmd).server-fm(0xc00038bb80, 0xc00059df70, 0x0, 0x1, 0x0, 0x0)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/commands/server.go:131 +0x52 fp=0xc002adfcf0 sp=0xc002adfca8 pc=0x55f57d2
github.com/spf13/cobra.(*Command).execute(0xc00038bb80, 0xc00059df50, 0x1, 0x1, 0xc00038bb80, 0xc00059df50)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/spf13/cobra@v0.0.7/command.go:838 +0x47c fp=0xc002adfdc0 sp=0xc002adfcf0 pc=0x41f689c
github.com/spf13/cobra.(*Command).ExecuteC(0xc0004d6840, 0xc00059de20, 0x757274206e727574, 0x4c0a0a3b7d0a3b65)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/spf13/cobra@v0.0.7/command.go:943 +0x336 fp=0xc002adfe98 sp=0xc002adfdc0 pc=0x41f73d6
github.com/gohugoio/hugo/commands.Execute(0xc00019c0a0, 0x2, 0x2, 0x4008c05, 0xc00018e058, 0x0, 0x0)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/commands/hugo.go:90 +0xb9 fp=0xc002adff28 sp=0xc002adfe98 pc=0x55d2499
main.main()
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/main.go:23 +0x76 fp=0xc002adff88 sp=0xc002adff28 pc=0x55f6f16
runtime.main()
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/proc.go:204 +0x209 fp=0xc002adffe0 sp=0xc002adff88 pc=0x403bb09
runtime.goexit()
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc002adffe8 sp=0xc002adffe0 pc=0x40719a1

goroutine 6 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0002c6870)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/go.opencensus.io@v0.22.0/stats/view/worker.go:154 +0x105
created by go.opencensus.io/stats/view.init.0
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/go.opencensus.io@v0.22.0/stats/view/worker.go:32 +0x57

goroutine 465 [syscall]:
syscall.syscall6(0x4236680, 0x8, 0x0, 0x0, 0xc000069e88, 0xa, 0x6d7c410, 0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/sys_darwin.go:85 +0x2e
golang.org/x/sys/unix.kevent(0x8, 0x0, 0x0, 0xc000069e88, 0xa, 0x6d7c410, 0x0, 0x0, 0xc0007411d0)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/golang.org/x/sys@v0.0.0-20200501145240-bc7a7d42d5c3/unix/zsyscall_darwin_amd64.go:292 +0xa6
golang.org/x/sys/unix.Kevent(0x8, 0x0, 0x0, 0x0, 0xc000069e88, 0xa, 0xa, 0x6d7c410, 0x0, 0x0, ...)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/golang.org/x/sys@v0.0.0-20200501145240-bc7a7d42d5c3/unix/syscall_bsd.go:413 +0x71
github.com/fsnotify/fsnotify.read(0x8, 0xc000069e88, 0xa, 0xa, 0x6d7c410, 0x0, 0x0, 0xc000069d60, 0x4010cf8, 0x60)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:511 +0x6e
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc004bf3c20)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:274 +0x831
created by github.com/fsnotify/fsnotify.NewWatcher
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:62 +0x199

goroutine 899 [select]:
github.com/gohugoio/hugo/commands.(*commandeer).newWatcher.func1(0xc002501100, 0xc000a6c270, 0xc001af8da8, 0xc002aa41e0)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/commands/hugo.go:873 +0xe5
created by github.com/gohugoio/hugo/commands.(*commandeer).newWatcher
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/commands/hugo.go:871 +0x2ac

goroutine 901 [select]:
github.com/gohugoio/hugo/livereload.(*hub).run(0x6d3ccc0)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/livereload/hub.go:39 +0x1e9
created by github.com/gohugoio/hugo/livereload.Initialize
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/livereload/livereload.go:98 +0x45

goroutine 898 [select]:
github.com/gohugoio/hugo/watcher.(*Batcher).run(0xc002501100)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/watcher/batcher.go:53 +0x174
created by github.com/gohugoio/hugo/watcher.New
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/watcher/batcher.go:42 +0x125

One more:

hugo server --disableLiveReload

Nope :frowning:

❯ hugo server --disableLiveReload
Start building sites …

                   |  EN
-------------------+-------
  Pages            | 1084
  Paginator pages  |  141
  Non-page files   | 2675
  Static files     |   23
  Processed images |    0
  Aliases          |  305
  Sitemaps         |    1
  Cleaned          |    0

Built in 3373 ms
Watching for changes in /Users/ulf/Sites/blog.ulfklose.de/{archetypes,content,data,layouts,static,themes}
Watching for config changes in /Users/ulf/Sites/blog.ulfklose.de/config.toml
fatal error: pipe failed

goroutine 1 [running]:
runtime.throw(0x5c92adc, 0xb)
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/panic.go:1116 +0x72 fp=0xc03cd57860 sp=0xc03cd57830 pc=0x4039332
runtime.sigNoteSetup(0x6d7c740)
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/os_darwin.go:98 +0xc5 fp=0xc03cd57888 sp=0xc03cd57860 pc=0x4036085
os/signal.signal_enable(0xf455821700000002)
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/sigqueue.go:198 +0xa5 fp=0xc03cd578a8 sp=0xc03cd57888 pc=0x406dc05
os/signal.enableSignal(...)
	/usr/local/Cellar/go/1.15.2/libexec/src/os/signal/signal_unix.go:49
os/signal.Notify.func1(0x2)
	/usr/local/Cellar/go/1.15.2/libexec/src/os/signal/signal.go:144 +0x88 fp=0xc03cd578c8 sp=0xc03cd578a8 pc=0x55ae748
os/signal.Notify(0xc053650ba0, 0xc03cd57ae0, 0x2, 0x2)
	/usr/local/Cellar/go/1.15.2/libexec/src/os/signal/signal.go:164 +0x162 fp=0xc03cd57940 sp=0xc03cd578c8 pc=0x55ae162
github.com/gohugoio/hugo/commands.(*commandeer).serve(0xc000a0a000, 0xc00016fec0, 0x2f, 0x40)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/commands/server.go:480 +0x625 fp=0xc03cd57b60 sp=0xc03cd57940 pc=0x55e6e25
github.com/gohugoio/hugo/commands.(*serverCmd).server(0xc00016fec0, 0xc0000f8dc0, 0xc000697680, 0x0, 0x1, 0x0, 0x0)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/commands/server.go:274 +0x2b6 fp=0xc03cd57ca8 sp=0xc03cd57b60 pc=0x55e55f6
github.com/gohugoio/hugo/commands.(*serverCmd).server-fm(0xc0000f8dc0, 0xc000697680, 0x0, 0x1, 0x0, 0x0)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/commands/server.go:131 +0x52 fp=0xc03cd57cf0 sp=0xc03cd57ca8 pc=0x55f57d2
github.com/spf13/cobra.(*Command).execute(0xc0000f8dc0, 0xc000697660, 0x1, 0x1, 0xc0000f8dc0, 0xc000697660)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/spf13/cobra@v0.0.7/command.go:838 +0x47c fp=0xc03cd57dc0 sp=0xc03cd57cf0 pc=0x41f689c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000315080, 0xc000697530, 0x8, 0xc00010ab80)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/spf13/cobra@v0.0.7/command.go:943 +0x336 fp=0xc03cd57e98 sp=0xc03cd57dc0 pc=0x41f73d6
github.com/gohugoio/hugo/commands.Execute(0xc0000ae0a0, 0x2, 0x2, 0x4008c05, 0xc0000a0058, 0x0, 0x0)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/commands/hugo.go:90 +0xb9 fp=0xc03cd57f28 sp=0xc03cd57e98 pc=0x55d2499
main.main()
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/main.go:23 +0x76 fp=0xc03cd57f88 sp=0xc03cd57f28 pc=0x55f6f16
runtime.main()
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/proc.go:204 +0x209 fp=0xc03cd57fe0 sp=0xc03cd57f88 pc=0x403bb09
runtime.goexit()
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc03cd57fe8 sp=0xc03cd57fe0 pc=0x40719a1

goroutine 21 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00013a550)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/go.opencensus.io@v0.22.0/stats/view/worker.go:154 +0x105
created by go.opencensus.io/stats/view.init.0
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/go.opencensus.io@v0.22.0/stats/view/worker.go:32 +0x57

goroutine 611 [syscall]:
syscall.syscall6(0x4236680, 0x8, 0x0, 0x0, 0xc000b9a688, 0xa, 0x6d7c410, 0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/sys_darwin.go:85 +0x2e
golang.org/x/sys/unix.kevent(0x8, 0x0, 0x0, 0xc000b9a688, 0xa, 0x6d7c410, 0x0, 0x0, 0x0)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/golang.org/x/sys@v0.0.0-20200501145240-bc7a7d42d5c3/unix/zsyscall_darwin_amd64.go:292 +0xa6
golang.org/x/sys/unix.Kevent(0x8, 0x0, 0x0, 0x0, 0xc000b9a688, 0xa, 0xa, 0x6d7c410, 0x0, 0x0, ...)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/golang.org/x/sys@v0.0.0-20200501145240-bc7a7d42d5c3/unix/syscall_bsd.go:413 +0x71
github.com/fsnotify/fsnotify.read(0x8, 0xc000b9a688, 0xa, 0xa, 0x6d7c410, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:511 +0x6e
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc05442a2a0)
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:274 +0x831
created by github.com/fsnotify/fsnotify.NewWatcher
	/Users/brew/Library/Caches/Homebrew/go_cache/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:62 +0x199

goroutine 612 [select]:
github.com/gohugoio/hugo/watcher.(*Batcher).run(0xc054413f40)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/watcher/batcher.go:53 +0x174
created by github.com/gohugoio/hugo/watcher.New
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/watcher/batcher.go:42 +0x125

goroutine 595 [select]:
github.com/gohugoio/hugo/commands.(*commandeer).newWatcher.func1(0xc054413f40, 0xc000a0a000, 0xc05440a0a0, 0xc053645860)
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/commands/hugo.go:873 +0xe5
created by github.com/gohugoio/hugo/commands.(*commandeer).newWatcher
	/private/tmp/hugo-20200915-67363-21w9wr/hugo-0.75.1/src/github.com/gohugoio/hugo/commands/hugo.go:871 +0x2ac

OK, I read this a little fast.

Seems like it’s the file watcher that crashes, presumably … too many files. Not sure.

Can you try one more time with:

hugo server --watch=false
2 Likes

I used this minos theme https://github.com/carsonip/hugo-theme-minos since to theme files was included but could not replicate the issue.

Some unrelated suggestions for your site:

  1. Move /content/wp-content/… to /static/wp-content/…. Content is stuff that needs to be rendered, everything in the static dir will just be copied as is.
  2. Set unsafe too true for the goldmark renderer. This allows the HTML you have in posts.
  3. Remove the stray config file in the content dir.
markup:
  goldmark:
    renderer:
      unsafe: true

This actually helped. The site is working now. Thank you.

❯ hugo server --watch=false
Start building sites …

                   |  EN
-------------------+-------
  Pages            | 1084
  Paginator pages  |  141
  Non-page files   | 2675
  Static files     |   23
  Processed images |    0
  Aliases          |  305
  Sitemaps         |    1
  Cleaned          |    0

Built in 4344 ms
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop

Thank you for the hints. The stray config file was generated by a script I used to export my WordPress instance to Hugo. I deleted it.

I moved all the images and videos to /static and replaced every occurrence in the generated Markdown files with the new path. That should be it, right?

Where do I have to put this?

config.yaml (or similar in config.toml)

I have what appears to be the same problem.

@bep’s suggestion of:

hugo server --watch=false

does allow the website to load.

We have been using Hugo to render a very large website since 2017, so I guess we must have just crossed some threshold with the number of files.

My specific question: we have a lot of images files included alongside the markdown pages, so that they can be used as page resources. I could probably devise a different strategy for these, by placing them in the static folder. Is this likely to make a difference?

I do have the same issue. So the problem is with too many files? Moving files to static would help resolve the issue?

Thankss! This solved one issue I had with my site.