Memory allocation panic errno-487 if static files > 550 MB

support
1

Liking Hugo a lot and now migrating my old hand-coded static site.

There is a large tree with thousands of static image and data files but after adding the whole tree in bulk, hugo fails with memory allocation errno 487.

After trimming the tree and then re-adding dirs, it seems that a static tree of 550MB is OK, but 565MB fails. So a slow increment does not seem to be a workaround, looks like a limit to the total tree :frowning: Running hugo 0.30.2 on Win7, same result on 32bit 4MB RAM and 64bit 16MB RAM.

Thanks for any advice

hugo -D -v server
INFO 2017/10/29 10:38:09 Using config file: D:\hugo\sites\DoPECC-hugo\config.toml
INFO 2017/10/29 10:38:09 using a UnionFS for static directory comprised of:
INFO 2017/10/29 10:38:09 Base: D:\hugo\sites\DoPECC-hugo\themes\bootswatch-BS3\static
INFO 2017/10/29 10:38:09 Overlay: D:\hugo\sites\DoPECC-hugo\static
INFO 2017/10/29 10:38:09 syncing static files to D:\hugo\sites\DoPECC-hugo\

runtime: VirtualAlloc of 1048576 bytes failed with errno=487
fatal error: runtime: cannot map pages in arena address space

runtime stack:
runtime.throw(0xc71a7c, 0x30)
/usr/local/go/src/runtime/panic.go:605 +0x7c
runtime.sysMap(0x514b0000, 0x100000, 0x38fe01, 0x1b26238)
/usr/local/go/src/runtime/mem_windows.go:124 +0x113
runtime.(*mheap).sysAlloc(0x1b14000, 0x100000, 0x0)
/usr/local/go/src/runtime/malloc.go:470 +0x231
runtime.(*mheap).grow(0x1b14000, 0x37, 0x0)
/usr/local/go/src/runtime/mheap.go:887 +0x4e
runtime.(*mheap).allocSpanLocked(0x1b14000, 0x37, 0x1b26248, 0xb0)
/usr/local/go/src/runtime/mheap.go:800 +0x2e4
runtime.(*mheap).alloc_m(0x1b14000, 0x37, 0x49a00101, 0x40f7b556)
/usr/local/go/src/runtime/mheap.go:666 +0x115
runtime.(*mheap).alloc.func1()
/usr/local/go/src/runtime/mheap.go:733 +0x40
runtime.systemstack(0x38ff1c)
/usr/local/go/src/runtime/asm_386.s:409 +0x8

------------ snip --------------

2

I searched for those error messages and I found a lot of discussion around Windows and memory issues with Go. Iā€™d check those. I donā€™t think this is something Hugo can fix. Maybe there is a newer version of Go you can install on the 64-bit machine?

3

With 4MB or RAM I would not be surpised here, but I assume you mean GB?

Note that if you use the distributed Hugo 0.30.2, that is compiled with the newest Go 1.9.1, so no use in testing with something newer.

4

I have drilled in a bit more: I have a \static folder with just over 1000 files, these are images and pdfs, mostly 1MB approx in size, a few pdfs up to 20MB, nothing bigger.

Hugo processes and server starts fine if Win7 reports 1040 files in the \static tree with size on disk of 412 028 928 bytes (I know Win7 numbers are ) This is on Win7 with 16GB (yes, GB, my oops before)

If I add 23 MB in 1 file, hugo fails with a go memory allocation errno 487. In this case, size on disk reports as 435 523 584. Remove the 23MB file and hugo is happy again. Add 20MB in many small files, hugo fails again.

So, it seems to me that (in this case) asking hugo to process a \static tree that is more than 412 MB hits the virtual memory allocation issue that has been reported with Go on Windows.

So I am wondering:

  • is this reproducible elsewhere - do all hugo \static trees fail if > the magic size (approx 412 MB)?
  • is there a workaround? eg I could
    1. remove the content from the static tree
    2. run hugo in non-server mode to generate an uploadable site
    3. patch the uploadable site by adding back the content that I removed in 1)
    4. test this on a separate server.
      much less convenient that the hugo live server :frowning: but seems like it should work.

and/or
If this is reproducible, can hugoā€™s \static processing be adjusted?

Thanks for any more help/advice/thoughts.

5

More information:

Hugo builds the site with a static tree >500MB in non-server mode ie hugo -D -v succeeds and builds an uploadable /public tree that seems to be OK

hugo -D -v server fails with errno 487

So it looks to me that hugo server mode cannot cope with the large /static tree.

Hope this is useful/helpful

6

That is not true as a general case. I have built Hugo sites with many GB inside /static.

7

Have you done that in Windows? I donā€™t have that OS available, so I canā€™t test. If we canā€™t fix this due to an OS-specific bug, we can establish that as the likely outcome for others searching this error.

I read that and literally palmed my forehead. Shows my level of involvement in Hugo/Go, my ā€œtraditionalā€ troubleshooting queries are less useful. :slight_smile:

8

Oh, OK.

So Iā€™m puzzled.
If i run hugo in build mode, all is OK
If i run in server mode it crashes with errno 487 after the "syncing static filesā€¦ " message.

I have a few big pdfs in the static tree so I removed these, leaving all files in static < 10MB - no difference. Static tree is now 490 127 360 bytes in 1231 files - so not so big at all.

At this stage looks like I must build manually and test with another server that looks at the /public output. Thatā€™s a shame because the hugo live server was really good!

I canā€™t think of anything else to try. I guess itā€™s somehow related to the know issues with Go mem allocation on Win, so maybe out of hugoā€™s control

Suggestions appreciated - thanks for the interest and feedback bep. Can supply the offending source if wanted.

9

I find myself in threads about the Hugo server for some reason, and BjĆørn has explained to me the ways the server is robust for serving live sites, but it is ultimately included for development. Is that still your use case, for development?

If you are using it for development, you might be able to pass parameters that change the baseURL to something that reflects your local file system. I donā€™t really know how that works in Windows, but it would allow you to navigate your site locally, sans a separate server. Try it out!

I recognize that as a shoddy replacement, but I am also thinking it is a Windows and Go memory thing. But I also asked if you had the most recent version of Go installed, so what do I know? :stuck_out_tongue:

10

OK, that makes a little more sense.

Try:

hugo server --renderToDisk
11

Bingo!! That looks like it is working now, thanks so much B!

I will exercise it for a day or so and confirm back.

For completeness and for others, can you give a few guides as to when --renderToDisk should be used?

Thanks again!

12

The default for hugo server is to render to memory. So that flag will typically make it slightly slower (still very fast), but it will help in ā€œlow resource situationsā€ (i.e. either very big sites or a PC with little memory).

1 Like
13

OK, verified that hugo server --renderToDisk solves the issue, all working well

Thanks again!!