10,000 page blog takes 30 minutes to deploy on Netlify

NOTE: My github repo is private, but I’ll someone as collaborator if they’re willing to help.

Hi, my blog has 9,700 pages. It builds fairly quickly, but it takes about 30 minutes to upload and process on Netlify. And I haven’t even migrated most of my images because I’m not sure the deploy would ever end.

How can I speed it up?

It builds super fast locally (20 seconds), and the build time on Netlify isn’t bad. It’s the deploy/processing that takes 20+ minutes.

Here’s my metrics:
hugo --templateMetrics
Building sites …
Template Metrics:

 cumulative       average       maximum         
   duration      duration      duration  count  template
 ----------      --------      --------  -----  --------
   5.426369398s    2.102429ms   45.225654ms   2581  _default/list.html
   1.817594382s    1.570954ms   27.310659ms   1157  _default/single.html
   1.810800586s     469.484µs   23.576344ms   3857  partials/footer.html
   1.644528197s     426.374µs   16.803403ms   3857  partials/header.html
   1.286260322s     161.834µs    6.493141ms   7948  _default/li.html
   1.283920136s   11.068277ms   39.983145ms    116  index.html
   1.186835996s     307.709µs   15.551626ms   3857  partials/head.html
   1.156802423s     299.922µs    23.45745ms   3857  partials/link.html
   853.079161ms     370.742µs    16.22766ms   2301  _internal/_default/rss.xml
   573.752401ms     148.871µs   41.873077ms   3854  partials/siteinfo.html
   282.682481ms       73.29µs    6.575491ms   3857  partials/menu.html
   251.659965ms      217.51µs    7.643296ms   1157  partials/related.html
   167.908962ms      43.533µs    9.368193ms   3857  partials/head_custom.html
   143.643834ms  143.643834ms  143.643834ms      1  _internal/_default/sitemap.xml
   112.865371ms   56.432685ms    97.25894ms      2  _default/terms.html
   105.534114ms       39.13µs    5.241093ms   2697  partials/pagination.html
    69.960544ms      18.138µs    3.328558ms   3857  partials/search.html
    20.915103ms       5.422µs    4.943453ms   3857  partials/svgpack-sprite.html
     10.92832ms       2.833µs    1.993562ms   3857  partials/footer_custom.html
     1.753666ms     292.277µs    1.344981ms      6  _internal/shortcodes/youtube.html
      624.361µs     624.361µs     624.361µs      1  404.html

   |  EN   
  Pages            | 5760  
  Paginator pages  |  398  
  Non-page files   |   47  
  Static files     | 5376  
  Processed images |    0  
  Aliases          | 2299  
  Sitemaps         |    1  
  Cleaned          |    0  

Total in 5909 ms


Hi, explain more about this. And give Netlify logs as well.

It usually takes only about 2-3 minutes to begin “Starting deploy” Then it crawls. Here’s the Netlify log:

5:17:54 PM: Build ready to start
5:17:56 PM: build-image version: 9e0f207a27642d0115b1ca97cd5e8cebbe492f63
5:17:56 PM: build-image tag: v3.3.2
5:17:56 PM: buildbot version: ef8d0929ed0baabafd8bbb7d0b021e1fc24180c0
5:17:56 PM: Fetching cached dependencies
5:17:56 PM: Starting to download cache of 1.3GB
5:18:06 PM: Finished downloading cache in 9.933509435s
5:18:06 PM: Starting to extract cache
5:18:21 PM: Finished extracting cache in 14.971249202s
5:18:22 PM: Finished fetching cache in 25.89568038s
5:18:22 PM: Starting to prepare the repo for build
5:18:23 PM: Preparing Git Reference refs/heads/master
5:18:52 PM: Found netlify.toml. Overriding site configuration
5:18:52 PM: Starting build script
5:18:52 PM: Installing dependencies
5:18:52 PM: Started restoring cached node version
5:18:55 PM: Finished restoring cached node version
5:18:56 PM: v10.16.0 is already installed.
5:18:57 PM: Now using node v10.16.0 (npm v6.9.0)
5:18:57 PM: Attempting ruby version 2.6.2, read from environment
5:18:58 PM: Using ruby version 2.6.2
5:18:59 PM: Using PHP version 5.6
5:18:59 PM: Installing Hugo 0.55.6
5:18:59 PM: Hugo Static Site Generator v0.55.6-A5D4C82D2/extended linux/amd64 BuildDate: 2019-05-18T08:08:34Z
5:18:59 PM: Started restoring cached go cache
5:18:59 PM: Finished restoring cached go cache
5:18:59 PM: unset GOOS;
5:18:59 PM: unset GOARCH;
5:18:59 PM: export GOROOT='/opt/buildhome/.gimme/versions/go1.12.linux.amd64';
5:18:59 PM: export PATH="/opt/buildhome/.gimme/versions/go1.12.linux.amd64/bin:${PATH}";
5:18:59 PM: go version >&2;
5:18:59 PM: export GIMME_ENV='/opt/buildhome/.gimme/env/go1.12.linux.amd64.env';
5:18:59 PM: go version go1.12 linux/amd64
5:18:59 PM: Installing missing commands
5:18:59 PM: Verify run directory
5:18:59 PM: Executing user command: hugo --gc --minify
5:18:59 PM: Building sites …
5:19:28 PM:                    |  EN
5:19:28 PM: +------------------+------+
5:19:28 PM:   Pages            | 9683
5:19:28 PM:   Paginator pages  | 1045
5:19:28 PM:  Non-page files   |   47
5:19:28 PM:   Static files     | 5376
5:19:28 PM:   Processed images |    0
5:19:28 PM:   Aliases          | 3316
5:19:28 PM:   Sitemaps         |    1
5:19:28 PM:   Cleaned
5:19:28 PM:  |    0
5:19:28 PM: Total in 28485 ms
5:19:28 PM: Skipping functions preparation step: no functions directory set
5:19:28 PM: Caching artifacts
5:19:28 PM: Started saving pip cache
5:19:28 PM: Finished saving pip cache
5:19:28 PM: Started saving emacs cask dependencies
5:19:28 PM: Finished saving emacs cask dependencies
5:19:28 PM: Started saving maven dependencies
5:19:28 PM: Finished saving maven dependencies
5:19:28 PM: Started saving boot dependencies
5:19:28 PM: Finished saving boot dependencies
5:19:28 PM: Started saving go dependencies
5:19:28 PM: Finished saving go dependencies
5:19:28 PM: Build script success
5:19:28 PM: Starting to deploy site from 'public'
5:19:31 PM: Creating deploy tree asynchronously
5:20:31 PM: 7411 new files to upload
5:20:31 PM: 0 new functions to upload
5:26:41 PM: Starting post processing
5:26:54 PM: Starting post processing
5:28:44 PM: Finished processing build request in 10m48.041340993s
5:41:44 PM: Post processing done
5:41:44 PM: Post processing done
5:41:44 PM: Site is live
5:41:44 PM: Site is live

What netlify post processing options do you have configured?


But, when deploying from netlify deploy using their cli, it looks like a lot of time is spent on CDN diffing and then loading 14,000 pages of tags.

Whoops. That was the first deploy to dummy project. It timed out waiting for site to go live. I made two small changes, rebuilt, and started netlify deploy again. This time, it got through CDN diff and upload in a few seconds, but it’s been churning on Waiting to deploy site for a while again.

I would recommend contacting Netlify Support. Closing this topic.