# Suggestions for speeding up generation on server with 1GB RAM?

**URL:** <https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354>\
**Category:** support\
**Created:** [April 3, 2018, 10:48am UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354 "2018-04-03T10:48:02Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![skippy](https://avatars.discourse-cdn.com/v4/letter/s/bcef8e/32.png) [@skippy](https://discourse.gohugo.io/u/skippy)\
**Post date:** [April 3, 2018, 10:48am UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/1 "2018-04-03T10:48:02Z")

</div>

I have about 10K posts in my site. I am using pagination, at 50 items per page; and I am not using taxonomies. My theme is [available here](https://github.com/skpy/hugo-themes/tree/master/skippy.net/layouts) for review. The 10K pages of content has not been published yet, but it’s a mix of long-form blog posts and a complete Twitter export of all my tweets converted to Markdown.

When I generate my site on my MacBook, with 16GB of RAM, hugo takes about 6 seconds.

When I generate my site on my server, with 1GB of RAM, hugo takes between 15 and 20 seconds.

I fully understand that the low-memory server will require more time to generate the site. I’m mostly just curious if there are any suggestions for ways to shave off some time? Am I doing anything obviously sub-optimal within my theme files that could be improved?

I’d eventually like to start publishing via a [micropub](https://indieweb.org/Micropub) endpoint directly to my server, and have that endpoint invoke Hugo to generate the new content. Waiting ~20 seconds isn’t going to ruin my life, but I am concerned about the scripts timing out eventually. I can raise the max execution time of the scripts; but I’d like to know if there are any other things I can do to improve the hugo experience.

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![skippy](https://avatars.discourse-cdn.com/v4/letter/s/bcef8e/32.png) [@skippy](https://discourse.gohugo.io/u/skippy)\
**Post date:** [April 3, 2018, 10:55am UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/2 "2018-04-03T10:55:37Z")

</div>

Here’s the relevant bits from `hugo --verbose --stepAnalysis=true`:

```auto
INFO 2018/04/03 06:25:53 Using config file: /home/skippy/hugo/config.yaml
Building sites … WARN 2018/04/03 06:25:53 No translation bundle found for default language "en"
WARN 2018/04/03 06:25:53 Translation func for language en not found, use default.
WARN 2018/04/03 06:25:53 i18n not initialized, check that you have language file (in i18n) that matches the site language or the default language.
WARN 2018/04/03 06:25:53 Unable to find Static Directory: /home/skippy/hugo/themes/skippy.net/static/
INFO 2018/04/03 06:25:53 syncing static files to /home/skippy/hugo/public/
initialize:
  32.338486ms (67.345886ms) 5.24 MB 70844 Allocs
load data:
  65.84µs (67.963353ms) 0.00 MB 29 Allocs
load i18n:
  480ns (68.296762ms) 0.00 MB 0 Allocs
read and convert pages from source:
  3.276557237s (3.345290446s) 346.69 MB 2380272 Allocs
INFO 2018/04/03 06:25:56 found taxonomies: map[string]string{"tag":"tags", "category":"categories"}
build Site meta:
  31.669489ms (3.377596319s) 2.23 MB 59462 Allocs
prepare pages:
  1.294691499s (4.716473319s) 73.92 MB 1110156 Allocs
render and write aliases:
  424.344µs (4.71766152s) 0.00 MB 9 Allocs
INFO 2018/04/03 06:26:00 Alias "/note/2007/10/18/archive/1/index.html" translated to "note/2007/10/18/archive/1/index.html"
INFO 2018/04/03 06:26:00 Alias "/note/2007/10/21/archive/1/index.html" translated to "note/2007/10/21/archive/1/index.html"
INFO 2018/04/03 06:26:00 Alias "/note/2007/10/22/archive/1/index.html" translated to "note/2007/10/22/archive/1/index.html"
....
.... a total of 5782 of these INFO lines ....
....
render and write pages:
	10.498777746s (16.212976468s) 236.28 MB 5811240 Allocs
render and write Sitemap:
	570.789543ms (16.784701644s) 34.83 MB 766640 Allocs
render and write robots.txt:
	15.165µs (16.785243846s) 0.00 MB 9 Allocs
WARN 2018/04/03 06:35:10 [en] Unable to locate layout for "404": [404.html theme/404.html]
render and write 404:
	6.782589ms (16.792280159s) 0.00 MB 31 Allocs
render and write pages:
	1.988066285s (18.780614105s) 83.22 MB 1929902 Allocs

                   | EN
+------------------+-------+
  Pages | 15248
  Paginator pages | 190
  Non-page files | 0
  Static files | 21
  Processed images | 0
  Aliases | 2868
  Sitemaps | 1
  Cleaned | 0

Total in 18778 ms

```

It seems like the generation of all of the aliases is where most of the time is spent. In my site, I don’t need or want aliases for content in the `/note/` section. Is there a way I can suppress that entirely for that section?

---

<div class="post-metadata">

**Author:** ![kaushalmodi](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/kaushalmodi/32/2567_2.png) [@kaushalmodi](https://discourse.gohugo.io/u/kaushalmodi)\
**Post date:** [April 3, 2018, 11:44am UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/3 "2018-04-03T11:44:34Z")

</div>

Do `hugo --templateMetrics`. That will give you a good analysis of which parts of your theme form the performance bottleneck.

Btw it’s good to see another Indieweb/microformats2 supporting theme. Thanks for sharing it 🙂

---

<div class="post-metadata">

**Author:** ![skippy](https://avatars.discourse-cdn.com/v4/letter/s/bcef8e/32.png) [@skippy](https://discourse.gohugo.io/u/skippy)\
**Post date:** [April 3, 2018, 12:06pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/4 "2018-04-03T12:06:23Z")

</div>

On my MacBook:

```auto
Building sites …
Template Metrics:

     cumulative average maximum
       duration duration duration count template
     ---------- -------- -------- ----- --------
   1.223657371s 138.563µs 27.011132ms 8831 /tmp/hugo/themes/skippy.net/layouts/note/single.html
   887.034867ms 309.286µs 7.000307ms 2868 _internal/_default/rss.xml
   877.528399ms 306.078µs 20.038929ms 2867 /tmp/hugo/themes/skippy.net/layouts/note/section.html
   351.524765ms 1.840443ms 5.959509ms 191 /tmp/hugo/themes/skippy.net/layouts/index.html
   294.503616ms 433.093µs 115.815023ms 680 /tmp/hugo/themes/skippy.net/layouts/_default/single.html
   235.291371ms 235.291371ms 235.291371ms 1 _internal/_default/sitemap.xml
    72.885477ms 72.885477ms 72.885477ms 1 /tmp/hugo/themes/skippy.net/layouts/allposts/single.html

                   | EN
+------------------+-------+
  Pages | 15248
  Paginator pages | 190
  Non-page files | 0
  Static files | 21
  Processed images | 0
  Aliases | 2868
  Sitemaps | 1
  Cleaned | 0

Total in 3863 ms

```

On my server:

```auto
Building sites …
Template Metrics:

     cumulative average maximum
       duration duration duration count template
     ---------- -------- -------- ----- --------
  10.671292564s 1.20839ms 231.958357ms 8831 /home/skippy/hugo/themes/skippy.net/layouts/note/single.html
   4.844561401s 1.689177ms 151.49244ms 2868 _internal/_default/rss.xml
   4.346622775s 1.516087ms 161.446273ms 2867 /home/skippy/hugo/themes/skippy.net/layouts/note/section.html
   1.058589525s 1.556749ms 191.863092ms 680 /home/skippy/hugo/themes/skippy.net/layouts/_default/single.html
   710.253356ms 3.718603ms 24.372287ms 191 /home/skippy/hugo/themes/skippy.net/layouts/index.html
   490.337219ms 490.337219ms 490.337219ms 1 _internal/_default/sitemap.xml
    138.52623ms 138.52623ms 138.52623ms 1 /home/skippy/hugo/themes/skippy.net/layouts/allposts/single.html

                   | EN
+------------------+-------+
  Pages | 15248
  Paginator pages | 190
  Non-page files | 0
  Static files | 21
  Processed images | 0
  Aliases | 2868
  Sitemaps | 1
  Cleaned | 0

Total in 16850 ms

```

---

<div class="post-metadata">

**Author:** ![kaushalmodi](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/kaushalmodi/32/2567_2.png) [@kaushalmodi](https://discourse.gohugo.io/u/kaushalmodi)\
**Post date:** [April 3, 2018, 12:24pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/5 "2018-04-03T12:24:15Z")

</div>

So looks like 1/2 the time goes in creating the 8k note pages. I believe those are the exported tweets? Suggestion (not an elegant one): May be pack 10 tweets in one page instead of 1 tweet per page? That should bring down the total time by a factor of roughly half (11/20) … It looks like the file IO, time to access that note single layout is the bottleneck. So my suggestion is to just reduce the access to the note single i.e. pack more tweets per page (using your Python script).

Also, you can also review if you need those many RSS pages… You can finely control that, per Page Kind.

---

<div class="post-metadata">

**Author:** ![bep](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/bep/32/3332_2.png) [@bep](https://discourse.gohugo.io/u/bep)\
**Post date:** [April 3, 2018, 12:55pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/6 "2018-04-03T12:55:26Z")

</div>

For the server `--renderToDisk` would be a good flag.

---

<div class="post-metadata">

**Author:** ![skippy](https://avatars.discourse-cdn.com/v4/letter/s/bcef8e/32.png) [@skippy](https://discourse.gohugo.io/u/skippy)\
**Post date:** [April 3, 2018, 1:05pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/7 "2018-04-03T13:05:27Z")

</div>

```auto
$ ./hugo version
Hugo Static Site Generator v0.38 linux/amd64 BuildDate: 2018-04-02T11:03:34Z
:~/hugo$ ./hugo --renderToDisk
Error: unknown flag: --renderToDisk

```

Or were you suggesting that this would be a good flag to implement within the code, because it doesn’t yet exist?

---

<div class="post-metadata">

**Author:** ![kaushalmodi](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/kaushalmodi/32/2567_2.png) [@kaushalmodi](https://discourse.gohugo.io/u/kaushalmodi)\
**Post date:** [April 3, 2018, 1:15pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/8 "2018-04-03T13:15:05Z")

</div>

Looks like there is a server confusion here. OP is talking about running `hugo` on a physical web server. You are talking about the `hugo server` flag.

@skippy `hugo` anyways renders files to disk, `hugo server` does not (by default), and thus that switch exists only for the latter. And `hugo server` is only for development.

---

<div class="post-metadata">

**Author:** ![skippy](https://avatars.discourse-cdn.com/v4/letter/s/bcef8e/32.png) [@skippy](https://discourse.gohugo.io/u/skippy)\
**Post date:** [April 5, 2018, 4:17pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/9 "2018-04-05T16:17:06Z")

</div>

> [@kaushalmodi](#):
>
> Also, you can also review if you need those many RSS pages… You can finely control that, per Page Kind.

Pointer on how to do this? My notes are a section, not a kind. I could convert them all to a kind with front matter, I suppose…

---

<div class="post-metadata">

**Author:** ![kaushalmodi](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/kaushalmodi/32/2567_2.png) [@kaushalmodi](https://discourse.gohugo.io/u/kaushalmodi)\
**Post date:** [April 5, 2018, 4:25pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/10 "2018-04-05T16:25:57Z")

</div>

> [@skippy](#):
>
> My notes are a section, not a kind.

“section” is one of the [Page Kinds](https://gohugo.io/templates/section-templates/#page-kinds). By default RSS is generated for all section pages (like /posts/, /notes/). You might need to use Custom Output Formats and strategically put the templates in your theme to enable/disable RSS for specific sections. i.e. You’d disable the default RSS output format completely, and enable your custom RSS2 (or any name you pick) output format.

… then set the templates only for the `layouts` you want. See:

> **[Custom output formats](https://gohugo.io/templates/output-formats/#output-format-for-pages)**
>
> Hugo can output content in multiple formats, including calendar events, e-book formats, Google AMP, and JSON search indexes, or any custom text format.

> I could convert them all to a kind with front matter, I suppose…

No, Kinds cannot be overridden, but Layouts can be.

---

<div class="post-metadata">

**Author:** ![olikami](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/olikami/32/5273_2.png) [@olikami](https://discourse.gohugo.io/u/olikami)\
**Post date:** [April 9, 2018, 2:08pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/11 "2018-04-09T14:08:15Z")

</div>

I’m not familiar with the MicroPub format, but is is possible to receive the article, save the data and then generate the site asynchronously?

---

<div class="post-metadata">

**Author:** ![skippy](https://avatars.discourse-cdn.com/v4/letter/s/bcef8e/32.png) [@skippy](https://discourse.gohugo.io/u/skippy)\
**Post date:** [April 9, 2018, 6:05pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/12 "2018-04-09T18:05:59Z")

</div>

Yes, this is a possibility, with some caveats.

If I want to use Micropub to upload images (or other media) for inclusion in a post, those images should be made immediately available to the Micropub client. That is to say, the media upload process should put them into the live site, not the source of my site. This is actually relatively easy to accomplish; and indeed I’d likely put the uploaded media into **both** the live site and the source’s `static` directory.

I’ve revised my theme a bit, making use of [this forum post](https://discourse.gohugo.io/t/disabling-rss-for-sections/10479/2) to disable RSS for sections. This is what @kaushalmodi was suggesting above, but this forum post cleared up much of my confusion.

Interestingly, even though Hugo is now generating several thousand fewer items, it takes about the same amount of time: ~4 seconds on my laptop, and ~13 seconds on my server.

I’m happy enough with these changes, even though they don’t really improve the speed. 13 seconds isn’t terrible.

---

<div class="post-metadata">

**Author:** ![moorereason](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/moorereason/32/592_2.png) [@moorereason](https://discourse.gohugo.io/u/moorereason)\
**Post date:** [April 11, 2018, 1:39am UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/13 "2018-04-11T01:39:00Z")

</div>

How do the CPUs compare between the laptop and server? It’s hard to tell if your CPU-bound or I/O-bound. How fast does it build if you use `--renderToMemory` (which removes disk write I/O from the equation)? I don’t think memory is the issue. Hugo doesn’t optimize for low-memory scenarios–it just crashes if you don’t have enough to build the site.

In case it’s not obvious, your laptop builds 18,000+ pages at 200µs per page. The server is building it at just under 1ms per page. That’s not horrible.

Also, when using the `--templateMetrics` option, I recommend also adding `--templateMetricsHints`. Feel free to share the output with the hints.

---

<div class="post-metadata">

**Author:** ![skippy](https://avatars.discourse-cdn.com/v4/letter/s/bcef8e/32.png) [@skippy](https://discourse.gohugo.io/u/skippy)\
**Post date:** [April 11, 2018, 1:13pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/14 "2018-04-11T13:13:36Z")

</div>

Good point, @moorereason!

The server is a single CPU droplet at DigitalOcean:

```auto
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 1
On-line CPU(s) list: 0
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Model name: Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
Stepping: 4
CPU MHz: 2399.998
BogoMIPS: 4799.99
Virtualization: VT-x
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 15360K
NUMA node0 CPU(s): 0
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm retpoline kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust smep erms xsaveopt

```

My laptop is a quad core i7:

```auto
Model Name:	MacBook Pro
Model Identifier:	MacBookPro11,3
Processor Name:	Intel Core i7
Processor Speed:	2.6 GHz
Number of Processors:	1
Total Number of Cores:	4
L2 Cache (per Core):	256 KB
L3 Cache:	6 MB
Memory:	16 GB

```

My entire site – raw content + generated results + all static files – is 167MB, so the entirety of the site can easily live in memory.

You’re absolutely right: this is a CPU bound process.

---

<div class="post-metadata">

**Author:** ![skippy](https://avatars.discourse-cdn.com/v4/letter/s/bcef8e/32.png) [@skippy](https://discourse.gohugo.io/u/skippy)\
**Post date:** [April 11, 2018, 1:19pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/15 "2018-04-11T13:19:13Z")

</div>

Metrics output with my latest theme changes:

```auto
$ hugo --templateMetrics --templateMetricsHints
Building sites …
Template Metrics:

      cache cumulative average maximum
  potential duration duration duration count template
      ----- ---------- -------- -------- ----- --------
          0 4.322870907s 1.507802ms 124.246219ms 2867 /home/skippy/skippy.net/themes/skippy.net/layouts/note/section.html
          0 4.233033164s 845.76µs 121.249375ms 5005 /home/skippy/skippy.net/themes/skippy.net/layouts/_default/single.html
          0 3.768407799s 836.308µs 123.21377ms 4506 /home/skippy/skippy.net/themes/skippy.net/layouts/note/single.html
          0 536.785966ms 536.785966ms 536.785966ms 1 _internal/_default/sitemap.xml
          0 179.387099ms 179.387099ms 179.387099ms 1 /home/skippy/skippy.net/themes/skippy.net/layouts/index.html
          0 174.048444ms 174.048444ms 174.048444ms 1 /home/skippy/skippy.net/themes/skippy.net/layouts/allposts/single.html
          0 70.098322ms 70.098322ms 70.098322ms 1 theme/_default/home.xml

                   | EN
+------------------+-------+
  Pages | 12381
  Paginator pages | 0
  Non-page files | 0
  Static files | 21
  Processed images | 0
  Aliases | 0
  Sitemaps | 1
  Cleaned | 0

Total in 12605 ms

```

---

<div class="post-metadata">

**Author:** ![cmal](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/cmal/32/5324_2.png) [@cmal](https://discourse.gohugo.io/u/cmal)\
**Post date:** [April 25, 2018, 2:08pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/16 "2018-04-25T14:08:41Z")

</div>

> [@skippy](#):
>
> If I want to use Micropub to upload images (or other media) for inclusion in a post, those images should be made immediately available to the Micropub client.

In the end, have you finally worked out your micropub integration?

How about using a completely separate media endpoint and then import remote images from your build script? That’s sort of [my approach](https://framagit.org/csa-ivry/vaydom/blob/master/bin/hooks/pre-build/ext-img.sh) at the moment to integrate remote content into my hugo website, and it plays out nice.

This way you can handle uploads outside your hugo repo, return [HTTP 202](https://micropub.net/draft/#response) with a link to the future post URL, then start your build script to import the image and actually generate the article with the image imported in your hugo repo.

---

<div class="post-metadata">

**Author:** ![skippy](https://avatars.discourse-cdn.com/v4/letter/s/bcef8e/32.png) [@skippy](https://discourse.gohugo.io/u/skippy)\
**Post date:** [April 25, 2018, 4:36pm UTC](https://discourse.gohugo.io/t/suggestions-for-speeding-up-generation-on-server-with-1gb-ram/11354/17 "2018-04-25T16:36:22Z")

</div>

I ended up writing [my own micropub server](https://github.com/skpy/micropub). It supports a media endpoint as well as direct file submissions. What I’ve learned is that not all Micropub clients support media endpoints (yet), so I need to deal with form-encoded media uploads alongside the content.

I don’t currently send an HTTP 202 response because [micropub.rocks](https://micropub.rocks), the test suite I used, doesn’t support that yet. There is an [open issue](https://github.com/aaronpk/micropub.rocks/issues/74) to add support for HTTP 202 to micropub.rocks, though, so hopefully this will get improved.