"too many open files" under Hugo v0.127.0

support
1

I’m running Hugo v0.127.0 and macOS Sonoma 14.5 and today was the day that my repo got big enough for hugo server to start failing when it begins watching files with this error:

Watching for config changes in /Users/justin/code/searls/justin.searls.co/config.toml
Start building sites … 
hugo v0.127.0+extended darwin/arm64 BuildDate=2024-06-05T10:27:59Z VendorInfo=brew

INFO  static: removing all files from destination that don't exist in static dirs
Built in 0 ms
Error: error copying static files: open /Users/justin/code/searls/justin.searls.co/static: too many open files

My site is not small but not massive.

Just 1146 files tracked by git:

$ git ls-files | wc -l
1146

All files, including pagefind’s index and node_modules is 33,093

$ find . -type f | wc -l
   33093

Neither of these seem absurd to me. But if you’ve been bitten by this error recently (as @bep was quite a long time ago), most of the old tricks no longer work.

  • Setting ulimit is ignored by the system, AFAICT
  • Setting the kernel maxfiles config via launchd and rebooting is also ignored if SIP is enabled
  • Even disabling SIP, it didn’t seem to work

As a result, I’m left trying to figure out what have (1) other Hugo power users done when they’ve run into this, and (2) whether Hugo’s watcher can be configured to ignore certain paths and maybe buy me some time, and (3) whether the watcher can be swapped out by the user for something like watchman which (who knows how) manages to avoid this IME on much, much larger directories

2

It seems like Hugo is using fsnotify for file system events. Apparently fsnotify uses kqueue on macOS for watching files, and kqueue requires a file descriptor be open per watched file.

FSEvents on macOS doesn’t have this limitation. There is an issue open on fsnotify to integrate FSEvents, but it sounds like integration probably won’t be considered until all unix systems support it (see comments on the issue)

Looks like the fsnotify org also has a wrapper around FSEvents (this is different than the main fsnotify repo), but I’m not sure if hugo could use it.

I’d link to more helpful info, but I’m a new user and am limited to only two links.

2 Likes
3

You can use

hugo server --poll 1s

Or something instead.

4

fsevents uses CGO under the hood, which is why Hugo doesn’t use it (there’s a longer story here, but that’s for another day).

1 Like
5

I’d love if there was a way to swap out the watcher to one that doesn’t have this file descriptor limitation, but in the meantime I’m glad to have a workaround. Thank you for pointing me to --poll!

6

I’m running that same combo with some pretty beefy/big sites, and cannot remember the last timeI saw the “too many open files” error.

7

I keep a pretty close watch on this forum, as well as the GitHub issues queue, and I haven’t seen this error reported in several years. We have macOS users building sites with tens of thousands of pages without error.

So this seems like something specific to your system, environment, or setup rather than an endemic problem. If you have the luxury of doing so, testing your site on virgin macOS setup would be informative.

8

Welp, I tried with a fresh working copy and re-cloning and then suddenly the hugo server was watching fine. Then I went to the original (still broken) working copy, ran npm install, and suddenly the server started working there too. Same number of files in the directory.

That’s an unsatisfying solution.

9

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.