Content in /static doubles data usage on server


#1

Hello
I’m a big fan of Hugo, but I stumbled on some efficiency issue. If you put data in the /static directory and you generate your website, these files are copied to the /public folder. This is neglectable with small or little files on space issues, but can be an issue if you have many files and/or other big media files like movies and pdf’s. Should Hugo not implement a
–forceLinkStatic to link to the data, so requirements get less and processing gets even more quickly? This can be implemented on all OS’s where file linking is available (I guess all where Hugo is running) and there could be a difference for hard-
or softlinking (–forceSoftLinkStatic) when the OS has the system call.

Thank you for any replies, (counter)ideas or workarounds!
Regards
Philippe


#2

There is the Assets folder for Resources that are published only if .Permalink, .RelPermalink or .Content is used in templates.

Please have a look in the Hugo Documentation. Particularly the Hugo Pipes Section.


#3

Hi Alexandros
Thank you for the reply, but as I see it, assets are more for handling CSS and js documents to be filtered or extracted somehow.
What I meant is when you have a reference to media files like .jpg, .mp4, or .avi.
e.g. in your content you can write:
! [image] ( /image.jpg ) or
{{< video src="/my_video.mp4" controls=“yes” >}}
So you place this file into your /static folder to be referenced, but Hugo makes a copy to /public first. That gives double space requirements. It’s ok for a couple of files, but what if you want to make a gallery? If a link to that reference would be possible instead of a whole copy, then all this double space is not wasted and the result is the same. Also, all this overhead of copying is left out, so Hugo can process everything more quickly.

Regards
Philippe


#4

Hello Philippe,

Ok. Another way to achieve what you want would be to organize your content by using Headless Page Bundles:

This way if you upload a very high resolution photo only the resize you specify after you use Hugo’s Image Processing will be made public.

However I don’t quite understand what you mean with the following:

To have images and video along side content you need to use Page Bundles.

Hugo does not create double references unless you have the same file twice under different directories within your Hugo project.

Basically you either place an image or video either within a Page Bundle or under /static/ not both.


#5

Hi Alexandros
I give an example:
You want to show a picture on a webpage, so you write in some file.md:
! [image] ( /image.jpg )
You place the file “image.jpg” into your ./static directory, then you process your site with hugo.
If you look in ./public, you will find ./public/image.jpg. The file is copied by Hugo from ./static/image.jpg to ./public. That means you need the double space.
If you have a gallery of pictures you need double space for every file.
But if linking would be available, then you keep only 1 file and some bytes from the link in your local or server disk.

I don’t see how you can do that with Page bundles?


#6

No. The contents of /public/ is what is published on the web when a Hugo project is deployed.

If you use Git the contents of /static/ Headless Bundles and Page Bundles will also occupy space since they are in version control.

But if you simply host to a server you only need to push the /public/ folder.

To ignore a file from Git Version control you need to use a .gitignore file. Use it to exclude the /public/ folder so that it does not occupy redundant space in your repo.

By the way the Hugo Docs have an excellent section about using GitHub and other hosting providers.


#7

Ok. You can deploy only the content of /public/ to the server, but it happens to be that I implemtented the git repository and the hugo site on the same server. I cannot gitignore the /static/, because if someone clones the repository to his local disk he would be left out with an empty /static/.
So the user can clone the repository, make changes and push the content back to the git server, where a local repository and hugo site is active (the git push calls a git post-receive to update the /public/ and the documentroot of the webserver is pointing to this /public/ directory). It all works flawless, except the server is stuck with all these copied media files. That’s why links would come in handy.
I could process the post-receive to checksum all media files and link them afterwards, but I was thinking why Hugo is copying all these files in the first place instead of linking?


#8

Because there are workflows -as the ones described in the Docs and I briefly mentioned above- where this is needed.

Typically Git Version control is separate from the production server.

Anyway maybe someone else could come up with a suggestion to improve your setup.


#9

A lot of folks generate their site locally, then upload the files to a web server somewhere. If Hugo did the file linking as you mentioned, this would break things in those scenarios.


#10

Hi Zachary!
Not at all if you use hardlinks! Unix has it and NTFS. It doesn’t make any difference if you upload with hardlinks to your webserver. Plus, linking is much faster than copying and as I said you save on local or server diskspace.
That’s why I suggested to get a --forceLinkStatic switch for the hugo renderer.


#11

I suggest that you open a GitHub issue with this request so that the ones who are maintaining the project can offer their perspective.

Feel free to reference this topic if you want.


#12

Yes, indeed. I could use deduplication on the server itself. That would solve this copying problem :slight_smile: Anyway, I’m quiet satisfied with the combined git and production setup, it’s all selfcontained. At the meantime I setup a storage outside the documentroot, but inside the production container to store all media. The user can then URL to this location without all the copying hastle. So, I force the users not to use this /static/ anymore.


#13

Well, now that you pointed to it, it has been already suggested it seems:


but it was closed. Not all people use normal filesystems it seems :frowning: He even suggested some hardlink options, same like I suggested.