Very large website can not be generated - no space left on device

support
1

Hello, I’m trying to migrate one of my websites to hugo. It has around 250k files in it. I’m stuck at the deployment phase. Locally I get this error:

hugo
Start building sites … 
Total in 923162 ms
Error: Error building site: open hugo/public/post/abc/index.html: no space left on device

Netlify fails with

10:17:50 AM: ────────────────────────────────────────────────────────────────
10:17:50 AM:   "build.command" failed                                        
10:17:50 AM: ────────────────────────────────────────────────────────────────
10:17:50 AM: ​
10:17:50 AM:   Error message
10:17:50 AM:   Command was killed with SIGKILL (Forced termination): hugo
10:17:50 AM: ​
10:17:50 AM:   Error location
10:17:50 AM:   In Build command from Netlify app:
10:17:50 AM:   hugo

Gitlab pages fails with:

$ hugo
Start building sites … 
ERROR: Job failed: execution took longer than 1h0m0s seconds

Now i bumped the gitlab pages job timeout to 3h. Let’s see.

Meanwhile any recommendation is welcome. I just have way too many posts (+130k) and each post has a data json with itself. Maybe I can embed json data directly to posts md files directly instead of using GetJSON.

Any idea is welcome.

2

Too many data files cause problems. This has been known for a number of years now and there is an open GitHub issue somewhere about it.

Content files in Hugo can have JSON front-matter.

Therefore try changing the extension of your .json files to .md (via an external shell script for speed) and then fetch the files’ content in the templates by referencing each JSON key as a Page Parameter.

You should see improvement in your build time as iterating over regular content files is faster than data files.

3

I never tried this version, but I just thought of this: how about writing a Github workflow that creates the site on Githubs infinite space and pushes the finished website back into the repository? Try local with a small subset of posts to see if it works in general and then use a contentdir parameter to point the Github workflow to the main content directory.

Just a thought, never tested.

4

What is GitHub Infinite Space?
You probably mean GitHub Codespace, right?

If so, that is just an online editor similar to VS Code, that modifies files directly into a GitHub repo.

5

infinite = never-ending, not limited. There are no limits to workflows in space as far as I know. Runtime yes, but not space.

6

Build time is capped as far as I know. eg. Gitlab free quota gives 400 minutes of free CI Build and my build takes around 6h on my workstation which i’d say powerful.