Hugo Embedded Web Server Limitations: ERROR TOCSS

support
1 
Start building sites … 
hugo v0.128.1+extended darwin/arm64 BuildDate=2024-07-02T06:46:41Z VendorInfo=brew

ERROR TOCSS: failed to transform "/hb/modules/search/index.scss" (text/x-scss): "/Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hbstack/search@v0.2.2/assets/hb/modules/search/index.scss:2:1": File to import not found or unreadable: /Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hugomods/search@v0.14.0/assets/search/scss/index.scss.
ERROR TOCSS: failed to transform "hb/scss/index.scss" (text/x-scss): "/Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hbstack/hb@v0.12.3/assets/hb/scss/index.tmpl.scss:4:1": File to import not found or unreadable: /Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/twbs/bootstrap@v5.3.3+incompatible/scss/_functions.scss.

Troubleshooting of this was very painful, it took many hours trying to isolate malformed file (for example).

Finally I was able to shrank large project to smaller one:

hugo v0.128.1+extended darwin/arm64 BuildDate=2024-07-02T06:46:41Z VendorInfo=brew


                   |  EN    
-------------------+--------
  Pages            | 38815  
  Paginator pages  |  4244  
  Non-page files   |     1  
  Static files     |     3  
  Processed images |     9  
  Aliases          | 12755  
  Cleaned          |     0  

Built in 134768 ms

Next step is to add single page to it, abc/index.md, containing “Hello, World!” sentence, and project fails with the command hugo server --renderToMemory! So that I started to suspect web server limitations and decided to run hugo which will only compile files:

Start building sites … 
hugo v0.128.1+extended darwin/arm64 BuildDate=2024-07-02T06:46:41Z VendorInfo=brew


                   |  EN    
-------------------+--------
  Pages            | 38816  
  Paginator pages  |  4244  
  Non-page files   |     1  
  Static files     |     3  
  Processed images |     9  
  Aliases          | 12755  
  Cleaned          |     0  

Total in 151853 ms

Note, number of pages increased by 1. And it was built sucessfully!

So, it is limitations of embedded web server. But then why do we have such a strange error message, “failed to transform SCSS” (the file is there!); also, do we have such limitations documented anywhere? Error message could provide more clue.

This also explains why I don’t have such issues at Netlify.

I tried to use “segments” but I don’t understand it yet…

2

Unfortunately segments feature doesn’t help, it seems Hugo tries to build everything and perhaps filter output.

segment_a:
  excludes:
    - output: rss
  includes:
    - path: '{/glossary/a/**}'

hugo server --renderToMemory --renderSegments segment_a

It should be about 5000 pages filtered, but I still have the same issue. I think Hugo tries to analyze everything (all files); although it fails in 50 seconds instead of 130.

Note that in this “glossary” website there are no errors if I delete all documents and leave only “a,b,…,h” (which is built Ok in 130 seconds); but filter “a”-only throws same error in 50 seconds.

Unfortunately I cannot share this project now; maybe in future.

3

Here is test site with generated 40,000 pages, it fails with hugo server (HTTP server), but it won’t fail with just hugo (just HTML generator)

Note “magic number” could be environment dependant: MacOS vs Windows vs Linux

4

Why do you use --renderToMemory? If you don’t have enough memory, that will probably cause trouble. Apparently, rendering to disk (what hugo without serverdoes) works ok – so why not use just that? It’s the default now with hugo server, anyway.

5

I use “renderToMemory” to isolate the isue: minimize I/O and to build it faster. Memory is more than enough for approx 500Mb of generated HTML, 50k pages 10k each.

It’s the default now with hugo server , anyway.

  • No… see Basic usage | Hugo for details. hugo generates HTMLs in output fiolder, and stops. It doesn’t run embedded server: you don’t need embedded server when you deploy it to Netlify.

Also, I provided test application project showing the issue: hugo server fails (with or without -M option), but hugo doesn’t.

6

I forgot to mention: I use Mac OS, Apple M1 Max 64 GB, and I believe this issue is environment dependant; Windows users may or may not experience it.

7

Ok, it is bug with ARM64 build for Apple MacOS, it happened slightly different on another machine, MacBook Air M1 16G:

fefendi@Fuads-MacBook-Air large-site-test % hugo server
hugo: downloading modules …
hugo: collected modules in 54891 ms
Watching for changes in /Users/fefendi/java/web/large-site-test/{archetypes,assets,content,layouts,package.json,static}
Watching for config changes in /Users/fefendi/java/web/large-site-test/config/_default, /Users/fefendi/java/web/large-site-test/config/development, /Users/fefendi/java/web/large-site-test/go.mod
Start building sites … 
hugo v0.128.2+extended darwin/amd64 BuildDate=2024-07-04T08:13:25Z VendorInfo=brew

WARN  [hugopress] [hb-custom] hook does't exist: layouts/partials/hugopress/modules/hb-custom/hooks/hb-blog-sidebar.html
ERROR [en] REF_NOT_FOUND: Ref "/_": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:8": page not found
ERROR [en] REF_NOT_FOUND: Ref "/a": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:34": page not found
ERROR [en] REF_NOT_FOUND: Ref "/b": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:60": page not found
ERROR [en] REF_NOT_FOUND: Ref "/c": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:86": page not found
ERROR [en] REF_NOT_FOUND: Ref "/d": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:112": page not found
ERROR [en] REF_NOT_FOUND: Ref "/e": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:138": page not found
ERROR [en] REF_NOT_FOUND: Ref "/f": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:164": page not found
ERROR [en] REF_NOT_FOUND: Ref "/g": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:190": page not found
ERROR [en] REF_NOT_FOUND: Ref "/h": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:216": page not found
ERROR [en] REF_NOT_FOUND: Ref "/i": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:242": page not found
ERROR [en] REF_NOT_FOUND: Ref "/j": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:268": page not found
ERROR [en] REF_NOT_FOUND: Ref "/k": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:294": page not found
ERROR [en] REF_NOT_FOUND: Ref "/l": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:320": page not found
ERROR [en] REF_NOT_FOUND: Ref "/m": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:346": page not found
ERROR [en] REF_NOT_FOUND: Ref "/n": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:372": page not found
ERROR [en] REF_NOT_FOUND: Ref "/o": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:398": page not found
ERROR [en] REF_NOT_FOUND: Ref "/p": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:424": page not found
ERROR [en] REF_NOT_FOUND: Ref "/q": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:450": page not found
ERROR [en] REF_NOT_FOUND: Ref "/r": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:476": page not found
ERROR [en] REF_NOT_FOUND: Ref "/s": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:502": page not found
ERROR [en] REF_NOT_FOUND: Ref "/t": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:528": page not found
ERROR [en] REF_NOT_FOUND: Ref "/u": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:554": page not found
ERROR [en] REF_NOT_FOUND: Ref "/v": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:580": page not found
ERROR [en] REF_NOT_FOUND: Ref "/q": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:606": page not found
ERROR [en] REF_NOT_FOUND: Ref "/x": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:632": page not found
ERROR [en] REF_NOT_FOUND: Ref "/y": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:658": page not found
ERROR [en] REF_NOT_FOUND: Ref "/z": "/Users/fefendi/java/web/large-site-test/content/dictionary/_index.md:17:684": page not found
ERROR TOCSS: failed to transform "/hb/modules/search/index.scss" (text/x-scss): "/Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hbstack/search@v0.2.2/assets/hb/modules/search/index.scss:2:1": File to import not found or unreadable: /Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hugomods/search@v0.14.0/assets/search/scss/index.scss.
ERROR TOCSS: failed to transform "hb/scss/index.scss" (text/x-scss): "/Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hbstack/hb@v0.12.3/assets/hb/scss/index.tmpl.scss:4:1": File to import not found or unreadable: /Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/twbs/bootstrap@v5.3.3+incompatible/scss/_functions.scss.
Built in 6446892 ms
Error: error building site: TOCSS: failed to transform "hb/scss/index.scss" (text/x-scss): "/Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hbstack/hb@v0.12.3/assets/hb/scss/index.tmpl.scss:4:1": File to import not found or unreadable: /Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/twbs/bootstrap@v5.3.3+incompatible/scss/_functions.scss.
fefendi@Fuads-MacBook-Air large-site-test %

I’ll report it as a bug.

8

I have looked at this, and you need to provide a more realistic test case for me to be able to look a it. Having a 40K page site each with 9 distinct taxonomy entries isn’t something I could imagine someone building in real life. Taxonomies are used to build relations, and for that you want common values.

9

I have this issue constantly, I have few real websites in production having this issue on MacOS, and not having with Netlify build. I spent many hours trying to find malformed page, and could not. Finally I suspected that this is Hugo limitation for Apple MacOS, I generated 40,000 files to reproduce. Taxonomy is high, it is not just 9 distinct values.

I noticed this issue for few websites having number of pages closer to 36000 and above. It took time and many tests to realize that this is Mac ARM64 specific, that it is not hardware issue, and that it is reproducible.

To reproduce, try to build on Mac OS: it will fail. Then, remove random 10000 files, it won’t fail: therefore, it is not malformed file causing the issue.

Also, note: absolutely unrelated error message “File to import not found or unreadable”

Perhaps, some internal limitations somewhere for MacOS, at least, should be documented & workaround suggested.

Interestingly, now I am realizing it too, I was able to build this website locally on Mac, it has over 200,000 distinct pages: https://UltimateLexicon.com

However, test auto-generated website fails with only 40,000 entries: perhaps it really blows up because of super high cardinality of taxonomies.

I have two other realistic websites with 36,000 - 50000 pages which have exactly the same issue. Deployed to production, Netlify (Linux) builds it Ok (60-90 minutes), but Mac OS has the same issue.

10

“Distinct” was maybe not the right word … Unique? The main thing is that you end up with 500k pages and a not very realistic setup.

11

Yes, correct, this random site doesn’t look realistic; but you can try to disable taxonomies in hugo.yaml and try to build, build should be 10x faster, and it will still fail

In my realistic website such as UltimateLexicon, 200,000+ pages, I use only category taxonomy and build on Mac M1 Pro takes about 70 minutes, it doesn’t fail; but I have another website with 36000 pages with category and tags which fails.

I am trying to rebuild test website now with disabled taxonomies; with taxonomies enabled it takes about 15 minutes to build on MacBook M1 Pro, and 2 hours on Mac Air M1

12

I just noticed, randomly generated test site has only category taxonomy enabled:

taxonomies:
  # tags: tags
  categories: categories
  # series: series
  # authors: author      # multiple authors.

So, Now I am disabling it too and trying to build; will report.

13

And build fails in approx. two minutes (with completely disabled taxonomies) instead of fifteen (with categories taxonomy enabled):

taxonomies:
  # tags: tags
  # categories: categories
  # series: series
  # authors: author      # multiple authors.

And now, hugo server command doesn’t fail, but hugo command fails:

large-site-test git:(main) ✗ hugo server
port 1313 already in use, attempting to use an available port
Watching for changes in /Users/fefendi/java/web/large-site-test/{archetypes,assets,content,layouts,package.json,static}
Watching for config changes in /Users/fefendi/java/web/large-site-test/config/_default, /Users/fefendi/java/web/large-site-test/config/development, /Users/fefendi/java/web/large-site-test/go.mod
Start building sites … 
hugo v0.128.1+extended darwin/arm64 BuildDate=2024-07-02T06:46:41Z VendorInfo=brew

WARN  [hugopress] [hb-custom] hook does't exist: layouts/partials/hugopress/modules/hb-custom/hooks/hb-blog-sidebar.html

                   |  EN    
-------------------+--------
  Pages            | 30124  
  Paginator pages  |   270  
  Non-page files   |     0  
  Static files     |     3  
  Processed images |     9  
  Aliases          |    59  
  Cleaned          |     0  

Built in 123729 ms
Environment: "development"
Serving pages from disk
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:58597/ (bind address 127.0.0.1) 
Press Ctrl+C to stop


 large-site-test git:(main) ✗ hugo
Start building sites … 
hugo v0.128.1+extended darwin/arm64 BuildDate=2024-07-02T06:46:41Z VendorInfo=brew

WARN  [hugopress] [hb-custom] hook does't exist: layouts/partials/hugopress/modules/hb-custom/hooks/hb-blog-sidebar.html
Total in 132021 ms
Error: error building site: POSTCSS: failed to transform "css/hb.css" (text/css): Error: Cannot find module '/Users/fefendi/.npm-global/lib/node_modules/@fullhuman/postcss-purgecss'
Require stack:
- /Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hbstack/hb@v0.12.3/postcss.config.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1212:15)
    at Module._load (node:internal/modules/cjs/loader:1038:27)
    at wrapModuleLoad (node:internal/modules/cjs/loader:212:19)
    at Module.require (node:internal/modules/cjs/loader:1297:12)
    at require (node:internal/modules/helpers:123:16)
    at global.requireX (/Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hbstack/hb@v0.12.3/postcss.config.js:10:16)
    at Object.<anonymous> (/Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hbstack/hb@v0.12.3/assets/hb/purgecss.config.js:38:18)
    at Module._compile (node:internal/modules/cjs/loader:1460:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1544:10)
    at Module.load (node:internal/modules/cjs/loader:1275:32) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hbstack/hb@v0.12.3/postcss.config.js'
  ]
}
➜  large-site-test git:(main) ✗

I can add more files, let’s say 50,000 (some magic number), with disabled taxonomies, issue is reproducible, there is some “magic number” after which we have this issue. In one specific case I found exact threshold: adding single page to site with 38815 pages breaks the build (see Strange issue "ERROR TOCSS" - resolved? · hbstack · Discussion #68 · GitHub)

Not sure if we should waste time on it; could be just bug in Mac internals, or something like ulimit nofile, but strange and confusing non-related error message… thanks for having a look.

We documented this and it is already helpful if someone encounters the same:

Error: error building site: TOCSS: failed to transform

14

I can also suspect SCSS compiler issue, it is different between Mac and Windows, but I am not specialist. I am blind in this area; all errors were mentioning CSS/SCSS; very possible issue will be implicitly fixed in the future. Most probably it is not Hugo-related; to be more realistic, I should try it with different theme. Error messages saying “file not found”, but file is there. Multithreading etc… When I published it initially, I didn’t relize yet that Windows and Linux don’t fail (which proves that it is not Hugo-related).

15

OPS sorry… I forgot that I deleted 10,000 pages locally that’s why hugo server didn’t fail; but just plain hugo fails. With all taxonomies disabled.

If I enable categories and tags for this test website build may take few days, don’t try it at home :wink:

16

Right word is “cardinality”, cardinality of taxonomies in generated pages is “over the top”, page subsets do not intersect by common taxonomy; I understand what you mean.

To confirm, I made mistake in original post, I forgot to run “npm install” command. Also, I completely disabled taxonomies via ‘hugo.yaml’ settings and build runs 3 minutes. “Hugo” command, which builds HTMLs and stores locally, does NOT fail. But “hugo server” which starts embedded HTTP server fails. We need embedded HTTP server for local tests. I can configure “segments” to only use small subset of pages but unfortunately it fails too! I am Ok with HTTP test server having limitations, but unfortunately “segments” fails too.

17

If you have a sensible (as in: not 500k taxonomy pages) site that fails on MacOS, please create a new GitHub issue and I will have a look at it.

18

Taxonomies disabled via hugo.yaml settings.

I can run script which will physically remove tags and categories from front matter, simple RegEx “replace in files” in Visual Studio Code can do it too; I’ll do it later tonight.

I can also copy-paste smaller real-life production-quality site folders to make is larger, doubling each time.

19

Hi Eric, thanks for checking it,

I cleaned all taxonomies from test site, absolutely all.

Still have the same:

large-site-test git:(main) ✗ hugo server
Watching for changes in /Users/fefendi/java/web/large-site-test/{archetypes,assets,content,layouts,package.json,static}
Watching for config changes in /Users/fefendi/java/web/large-site-test/config/_default, /Users/fefendi/java/web/large-site-test/config/development, /Users/fefendi/java/web/large-site-test/go.mod
Start building sites … 
hugo v0.128.2+extended darwin/arm64 BuildDate=2024-07-04T08:13:25Z VendorInfo=brew

ERROR TOCSS: failed to transform "/hb/modules/search/index.scss" (text/x-scss): "/Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hbstack/search@v0.2.2/assets/hb/modules/search/index.scss:2:1": File to import not found or unreadable: /Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hugomods/search@v0.14.0/assets/search/scss/index.scss.
ERROR TOCSS: failed to transform "hb/scss/index.scss" (text/x-scss): "/Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hbstack/hb@v0.12.3/assets/hb/scss/index.tmpl.scss:4:1": File to import not found or unreadable: /Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/twbs/bootstrap@v5.3.3+incompatible/scss/_functions.scss.
Built in 183665 ms
Error: error building site: TOCSS: failed to transform "hb/scss/index.scss" (text/x-scss): "/Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/hbstack/hb@v0.12.3/assets/hb/scss/index.tmpl.scss:4:1": File to import not found or unreadable: /Users/fefendi/Library/Caches/hugo_cache/modules/filecache/modules/pkg/mod/github.com/twbs/bootstrap@v5.3.3+incompatible/scss/_functions.scss.

Note: if you delete any subset of 10,000 files, the issue disappear. Also, you can build with hugo, no issues; the issue is only with embedded HTTP server, hugo server.

(same link to test repo; latest commit removes all taxonomies)

20

I have reopened Apple MacOS ARM64 broken build for large sites "Error: error building site: TOCSS: failed to transform" · Issue #12649 · gohugoio/hugo · GitHub