Moving all static files (/assets) to external CDN... input?

First of all, I’m loving Hugo. Moving a 16yr old blog with over 2500 posts => Hugo & doing a ton of content cleanup/fixes in the process. One downside is my deployments take about 10-15 minutes with the CD auto build+deploy process I’ve set up as some posts have 2-6 aliases in place, creating a deployment of nearly 5,000-6,000 files on each build.

So I’m looking to parallelize the builds & deployments a bit more.

One thing I’m looking to do is take all the content in /static/assets (over 1,500 files) and deploy it to a separate location that’s exposed via CDN. To do that, I’d need to replace all URLs on the build files from “/assets/*” to something like “assets.mydomain.com”, then before the deployment step, separate those files into a different build.

Just throwing an idea out at this point… anyone done something like this or have other suggestions? TIA!

TIA?

Anyways… I did this with one of my sites and I do it by deploying the whole site to TWO hosts (two domains names) and then using the full URL for the assets. In your case that would be https://assets.mydomain.com/assets/ for all “CDN” links. CDN for poor people :wink:

I have a lot of panoramas on my web site, every pano an image pyramid of > 1k files.
I left this files out of hugo and defined an alias for the web server.

apache:

  DocumentRoot /media/html
  Alias /panorama /media/panorama
  <Directory /media/panorama> ...

The panorama pages refer to /panorama/name … and all works fine.

CDNs do not see this.

TIA = Thanks in Advance

Thanks for the suggestion, but this doubles my problem (long deployment cycle of 15m+)… I’m trying to go in the other direction.

I wonder if that deployment cycle is on Netlify :wink: In that case it’s the free package that slows the deployment down. Buy the lowest package and it goes up in speed.

If its a simple SSH or SFTP deployment process I think you can optimize the speed by using rsync, compression and so on…

TIA: I entered the Internet back in the time when we had enough space to write all words outs instead of LOL’ing and TIA’ing and missing all pronouns. But I learn every day :smiley:

Note this is me thinking out loud;

What you may want to consider with the latest Hugo is to partition your site into modules. If one of them is my-assets with 1500 files inside static, you can just tag that as v1.0.0 or something and import it in your main project.

With that happening, you could take advantage of the CI cache. On Netlify this is hardcoded into Hugo and should work right out of the box.

More info here (see the modules cache):

1 Like

Nope… not Netlify. Site codebase lives in a git repo hosted in Azure DevOps. A DevOps build pipeline is setup to build (1) on every push to master & (2) every day at 10a (for content that publishes/expires in the future). After the build (running the hugo tool), all resulting content for the site is pushed to a build artifact. That triggers a deployment pipeline that copies the files => Azure Storage blob container that’s setup to host static sites.

While I’m using Azure, most build/release CI/CD tools follow this model, copying to an artifact. It’s the XCOPY of 6000+ files that’s slow, as you can see from this log from a build (doesn’t include the deployment step):

Thanks for the suggestions, but they aren’t going to work for this scenario…

Modules… I’m not familiar with them so I’ll research. Thanks!

Yea, my thoughts above should apply to any CI vendor. The thing is, your site in /public will in the common case end up on a CDN anyway (I don’t know Azure good enough to tell).

Given that you don’t have millions of files, you can get very far by:

  1. Making sure to cache the Hugo source as effectively as possible (see my comment about modules above)
  2. Making sure to use a “copy strategy” that only uploads changed files. If you use hugo deploy (also new, and it supports Azure), you get this.
1 Like

That’s exactly what I was hoping to find (a copy only changed files strategy)… that would eliminate the issue I’m running into with moving lots of files. I suspect the custom Azure DevOps build task isn’t using this… thanks!

The deploy command looks promising for what I’m trying to do… however, I ran into an issue in my testing… created a new thread as it’s not directly related to the discussion here: ISO clarifying how "deploy" command works

Building and deploying are 2 different things.

In order to reduce build time you could remove image processing at this stage and perform the workload with a cron task / webhook triggered post-deployment, using some package like imageMagick

In order to reduce deployment time you could use Rsync to mirror your public dir on your remote server.
https://rsync.samba.org/

Thanks… build times aren’t really an issue anymore as they are sub-60s.

The deploy command is exactly what I wanted.

I wish there was an option to have it export a JSON / text output list of all files added/updated/deleted so I could use that in my automated deployment pipeline to purge the cache selectively. It’s normal output will work though… I’m just going to massage it first.