mostly all refer to ways to increase the number of open filehandles you can have with the “watcher”, as the latter depends on OS provided primitives. In my case the issue is that I like to use hugo to document my code, so I have a number of examples where a leaf bundle includes a python venv which has scripts which I can use to generate pngs etc… I have just discovered that 3 venvs in one hugo project is enough to hit this limit, depending of course on what you install. I proved this by simply deleting one of those virtual environments and the problem went away.
Given this use case the solution of raising the number of open file handles doesn’t address the root cause, it just postpones it to when you have 5 venvs.
I will of course start using virtual environment managers which centralise the location of the virtual environment to avoid this problem. This is a bit harder for people using npm, where it is very standard to have node_modules where you are
Of course there is no need for the file watcher to watch what is under a venv or a node_modules directory. Is it possible in hugo to
have a file or some configuration like .gitignore (say .hugoignore) where you can put in a list of filenames/directories and startup the watcher to ignore anything under that filename, much as git does.
Modify the output of hugo server to add some lines to tell you the number of files “.hugoignored” (see below) so that you don’t include these files as “non-page” files.
If for whatever reason the file limit is hit - legitimately because your hugo project is really really really big - then output this text before throwing the error so that the user can see .e.g that there are 50000 static files.
I still get this on my M2 Ultra Mac Studio and M3 MacBook Air (each being essentially maxed out and with every single documented trick in the book around raising the file limit).
It actually just happened yesterday, and the only durable solution I’ve found is to delete my working copy, reclone it, run npm install, and then run the server. Works every time. I’ve even done a diff on the before/after of node_modules (assuming the old copy was full of spiders) and the diff was identical. It’s something else.
Hugo only watches mounted directories (see documentation for a definition). node_modules is by default never mounted, so that’s not a relevant example.
In my case I have my python virtual environments under directories like
content/posts/2024-027/venv
where
content/posts/2024-027/
is the leaf bundle with an index.md file. I generate figures with the scripts which I reference in the index.md with the figure shortcode. I’d rather do this than use Jupyter notebooks.
I’ve solved my problem by going to pipenv, which puts the “venvs” in a central location and otherwise manages it. My non-page source counts have reduced accordingly.
With regards the mounts, I found the documentation
I presume I could add an excludeFiles glob to fix this issue as well. I still have a couple of venvs. I might try this to see if it reduces the non-page count.
What do your counts look like when you successfully run hugo server? Do you really have a large number of files?
My guess is that you might be a little too leading edge with M2 and M3 Macs. This functionality is provided by the OS and as far as I can see Hugo is using a third party go module to provide this functionality.
In particular the comments in the source code
refer to a very old issue
which makes me think that any flakiness is unlikely to be fixed in the near term. So it isn’t a question of increasing the limit, it is a question of a bug in the underlying system or something that works differently with the M2 and M3 Macs. I’m on an M1.
$ hugo server --logLevel info --ignoreCache --buildFuture --buildDrafts --cleanDestinationDir --bind 0.0.0.0 --environment development
Output:
INFO build: step render pages 1580 content 837 duration 1.991400041s
INFO build: step render deferred count 0 duration 1.458µs
INFO build: step postProcess duration 197.75µs
INFO build: duration 2.200241541s
| EN
-------------------+-------
Pages | 1513
Paginator pages | 67
Non-page files | 0
Static files | 7989
Processed images | 0
Aliases | 29
Cleaned | 0
Built in 3123 ms
Environment: "development"
Serving pages from disk
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address 0.0.0.0)
Press Ctrl+C to stop
I know I’m jumping into this late, but what is “this”? Or to put it another way, what is the specific error message that you see? Is it the same message as reported by the OP?
Sorry to be unclear. Exact same as my initial report. So not the same point of failure as the OP (hugo_build.lock) but rather at some other point when starting the server
to over 2 seconds