Content directories per environment

Before I start to set up a clever™ system that copies my content into the content folder on grunt netlify-setup I wondered if there is a way to define content folders per “environment” (dev/production). I have about 2000 old blog entries that slow down recreation of the site in hugo server to (best case) 30 seconds. Because I work quite often still on the design, I wonder if it’s possible to maybe have just 10 posts in one content folder for hugo server to just speed things up and all other (or all) posts in another content folder for the production environment which then grabs content from two folders?

If that is not possible, what would your approach be? I am thinking about a simple copy job when building my site on netlify.

You shouldn’t need to do that if you do development while running hugo server. By default (unless you pass the --disableFastRender switch or set that variable in site config), the FastRender feature will cause hugo server to render only the last few touched files, making most of the site functional and accessible.

Have you already tried that?

I don’t pass the fastrender parameter. Still… it’s extremely slow. If I change content files it’s between 40 to 70 seconds, changing template files sometimes goes up to 120 seconds. Not sure. I weeded out all unused templates recently.

I now run it through --templateMetrics --templateMetricsHints and it looks like though it runs through all my posts that goes through fast. what takes time are the archive pages which get recreated each time (home.html and pagination.html)?

Change detected, rebuilding site
2018-03-13 21:38:27.261 +0700
Source changed "/home/patrick/Projects/samui-samui.de/content/post/2018/03/wider-den-darts/index.md": WRITE

Template Metrics:

      cache     cumulative       average       maximum         
  potential       duration      duration      duration  count  template
      -----     ----------      --------      --------  -----  --------
          0 23.877553119s   59.843491ms  144.344207ms    399  /home/patrick/Projects/samui-samui.de/layouts/_default/home.html
         39 18.251488023s   45.743077ms  104.839292ms    399  partials/pagination/posts.html
          0  948.749622ms  948.749622ms  948.749622ms      1  _default/list.algolia.json
         22  443.030639ms     222.181µs   41.709512ms   1994  partials/article/head.html
         97   351.13488ms     875.648µs    7.471912ms    401  partials/head.html
         18  249.523647ms     125.137µs   41.407957ms   1994  partials/article/meta-top.html
          0  146.186855ms  146.186855ms  146.186855ms      1  _internal/_default/sitemap.xml
          0   92.719357ms   92.719357ms   92.719357ms      1  _internal/shortcodes/figure.html
         92   88.500844ms       220.7µs     710.011µs    401  partials/head/open_graph.html
         99   80.860739ms     201.647µs     626.906µs    401  partials/header/navigation.html
         99   75.094362ms     187.267µs    6.632163ms    401  partials/head/twitter.html
         99    56.22079ms     140.201µs     445.523µs    401  partials/head/metadata.html
          0   19.487508ms   19.487508ms   19.487508ms      1  /home/patrick/Projects/samui-samui.de/layouts/post/single.html
        100   18.683396ms       9.369µs      50.495µs   1994  partials/article/meta-bottom.html
        100   10.262196ms   10.262196ms   10.262196ms      1  partials/tools/related.html
          0    5.907134ms    5.907134ms    5.907134ms      1  _internal/_default/rss.xml
          0    1.739959ms    1.739959ms    1.739959ms      1  /home/patrick/Projects/samui-samui.de/layouts/404.html
        100       154.3µs       154.3µs       154.3µs      1  partials/pagination/single.html
        100     117.427µs     117.427µs     117.427µs      1  partials/tools/sharebuttons.html
          0       5.935µs       5.935µs       5.935µs      1  _internal/_default/robots.txt

Total in 69370 ms

Yea, there is something sub-optimal with your site (rendering the home page 399 times sounds odd … I assume there is pagination going on here, but there seem to be some expensive stuff going on there), but it is hard to tell without the source.

The repo lives at https://bitbucket.org/pkollitsch/samui-samui.de/
I am slightly paranoid about having it public, but for now it might be helpful :slight_smile:

I also wonder why some paths in that debugging table are relative and some are from root. is that an indicator or just how hugo finds things?

The content was imported from WordPress and only in /content/2018/did I start to use some packages… the rest needs some re-work - not sure if that is relevant.

Little hard to say, but you could start by filtering the posts before you send them into the Paginator and not wait after it is created:

{{ range where $paginator.Pages “Params.hidden” “ne” “true” }}

Also consider to raise the paginator to (say) 10.

I did that and sped up by about 50%. Also: It appears that less partials = more speed. I moved all partials like post-head, meta lines and so on into their respective parent templates (home.html, list.html, post.html etc.)

One thing I am not sure about but it felt as if things sped up: removing Go template comments ({{/* commen */}}) containing Go template code ({{/* {{ some Go tag }} */}}) made things faster. My guess is that the inner code is executed and then ignored… or it’s just my imagination.

I am now at around 15 seconds per rebuild.

PS: and I am aware that this post is way off-topic by now.

(moved discussion on Go Template comments to separate thread.)

For partials that do not change eg. the footer use partialCached and your speed will improve further.

1 Like