Replace in go.mod works but HUGO_MODULE_REPLACEMENTS env var doesn't work

Hello,

I have spent some time to figure out how to make HUGO_MODULE_REPLACEMENTS work, but have been unsuccessful. The replace command in go.mod works but I would like to avoid having to edit the go.mod each time I am tweaking my site’s theme. The HUGO_MODULE_REPLACEMENTS env var would fit the bill perfectly, but it doesn’t work for me.

Here is a recipe that reproduces the problem consistently for me. Can anyone else please verify and may be suggest what I could be doing wrong here.

mkdir -p /tmp/hugo_temp
cd /tmp/hugo_temp

# download the theme module
git clone https://gitlab.com/kaushalmodi/hugo-mwe-theme

# download the minimum working example site repo to a site/ directory
git clone --single-branch -b use-hugo-module-theme https://gitlab.com/hugo-mwe/hugo-mwe site

cd site
  • Inside the site directory, you will find a bash script srv.sh that uses HUGO_MODULE_REPLACEMENTS env var to replace the gitlab.com/kaushalmodi/hugo-mwe-theme module reference with a local path.

Testing replace in go.mod (works)

  1. Uncomment the // replace gitlab.com/kaushalmodi/hugo-mwe-theme => ../hugo-mwe-theme line in site/go.mod and save that file.

  2. Run hugo mod graph.

  3. You will see something like:

    gitlab.com/kaushalmodi/hugo_mwe gitlab.com/kaushalmodi/hugo-mwe-theme@v0.1.0 => /tmp/hugo_temp/hugo-mwe-theme
    
  4. This shows that the replace command in go.mod works fine.

Testing the HUGO_MODULE_REPLACEMENTS env var (fails)

Make sure you are in the cloned site/ directory.

  1. Comment the replace line in go.mod once again (or discard the earlier edit in that file by running git checkout HEAD -- go.mod).

  2. Run ./srv.sh (which uses HUGO_MODULE_REPLACEMENTS to do the same replacement as before).

    • That script runs HUGO_MODULE_REPLACEMENTS="gitlab.com/kaushalmodi/hugo-mwe-theme -> ../hugo-mwe-theme" hugo server.
  3. This time, I get this error:

    Error: module “…/hugo-mwe-theme” not found; either add it as a Hugo Module or store it in “/tmp/hugo_temp/site/themes”.: module does not exist

Update This is now fixed by overriding the Hugo themesDir as well:

HUGO_THEMESDIR="$(pwd)/../" \
  HUGO_MODULE_REPLACEMENTS="gitlab.com/kaushalmodi/hugo-mwe-theme -> hugo-mwe-theme" \
  hugo server

  • Hugo version 0.92.0
  • OS: CentOS 7.6

@kaushalmodi When using the env var vs. replace in go.mod you need to remember that the env var operates relatives to the themes dir (even if you are not using a theme), so you need more ../. For a site you would need ../../hugo-mwe-theme and for a module with an exampleSite for which you are generating the exampleSite you would need ../../../hugo-mwe-theme.

HTH

1 Like

I use it every day, so I’m sure it works, but I guess there are some caveats worth knowing:

  • Using HUGO_MODULE_REPLACEMENTS is the same as manually editing the path in your module config in config.toml – which is not the same as a replacement inside go.mod
  • I suspect your problem is a path confusion issue, as relative paths resolves relative to themeDir – I notice in one shell script I use for this purpose is that I use absolute paths on the right side of the fence, which I suspect is easier to reason about.
  • Go 1.18 (soon to be released) comes with a concept of workspaces (go.workspace files), which should make (cross fingers!) this situation much cleaner/better. That would be files that you would (normally) put in .gitignore.
2 Likes

Thanks for your reply.

I missed this bit in the documentation.

HUGO_MODULE_REPLACEMENTS="github.com/bep/myprettytheme -> ../.." . Any relative path is relate to themesDir, and absolute paths are allowed.

It didn’t even occur to me that the path would need to be relative to themesDir because:

  1. I had rm -rfd the themes/ dir, and
  2. the replace in go.mod was working well.

As there is no themes/ dir, I am now doing this :slight_smile:

HUGO_THEMESDIR="$(pwd)/../" \
  HUGO_MODULE_REPLACEMENTS="gitlab.com/kaushalmodi/hugo-mwe-theme -> hugo-mwe-theme" \
  hugo server

ok, I got the path replacement working in the example repo above. But looks like it is still causing problems (even when using absolute path for replacement), if the replaced module depends on other modules.

For example, I have this theme module for my main site that depends on few other modules:

module gitlab.com/kaushalmodi/hugo-theme-refined

go 1.16

require (
	github.com/kaushalmodi/hugo-atom-feed v0.1.0 // indirect
	github.com/kaushalmodi/hugo-debugprint v0.1.0 // indirect
	github.com/kaushalmodi/hugo-jf2 v0.1.2 // indirect
	github.com/kaushalmodi/hugo-search-fuse-js v0.2.0 // indirect
)

When I run this:

export HUGO_THEMESDIR="$(pwd)/../"
# The replacement paths need to be relative to Hugo's themesDir.
export HUGO_MODULE_REPLACEMENTS="gitlab.com/kaushalmodi/hugo-theme-refined -> hugo-theme-refined"

hugo mod graph

it looks like it found the main theme module in “themesDir”, but it failed to get one of the nested modules:

hugo: collected modules in 933 ms
Error: module “GitHub - kaushalmodi/hugo-search-fuse-js: Hugo theme component for implementing static site search using Fuse.js” not found; either add it as a Hugo Module or store it in “/home/kmodi/hugo”.: module does not exist

But /home/kmodi/hugo directory does contain hugo-search-fuse-js/ directory.


Short recipe to reproduce this issue

mkdir -p /tmp/hugo_temp_36746
cd /tmp/hugo_temp_36746

# download the theme module
git clone --single-branch -b add-nested-module https://gitlab.com/kaushalmodi/hugo-mwe-theme

# download the minimum working example site repo to a site/ directory
git clone --single-branch -b use-hugo-module-theme https://gitlab.com/hugo-mwe/hugo-mwe site

cd site
./srv.sh

Error:

go get: added GitHub - kaushalmodi/hugo-search-fuse-js: Hugo theme component for implementing static site search using Fuse.js v0.2.0
hugo: collected modules in 904 ms
Error: module “GitHub - kaushalmodi/hugo-search-fuse-js: Hugo theme component for implementing static site search using Fuse.js” not found; either add it as a Hugo Module or store it in “/tmp/hugo_temp_36746”.: module does not exist

@kaushalmodi I have observed this as well. I am not sure it is a bug per-se (or rather a lack of documentation).

There are a couple of ways to solve the issue. One is add the nested module(s) to the HUGO_MODULE_REPLACEMENTS. The other (IIRC) is to use hugo mod vendor and commit the _vendor directory in the top-level module.

I’m pretty sure what is happening is that when you use ‘replacements’ there isn’t the usual downloading of the module into the module cache directory, nor download of nested module (because the usual code path is not followed). This may be a Go module behavior rather than Hugo-specific.

I probably should have commented that this was happening in the forum, but I was busy and forgot.

Maybe @bep has some thoughts on whether this is something Hugo can handle better, or if it’s more a case of updating the docs.

Yeah… it’s quite inelegant but this works:

set -euo pipefail # http://redsymbol.net/articles/unofficial-bash-strict-mode
IFS=$'\n\t'

port="${1:-3333}" # default

# shellcheck disable=SC2155
export HUGO_THEMESDIR="$(pwd)/../" # The replacement paths need to be relative to Hugo's themesDir.

replacements=("gitlab.com/kaushalmodi/hugo-theme-refined"
              ", github.com/kaushalmodi/hugo-search-fuse-js"
              ", github.com/kaushalmodi/hugo-jf2"
              ", github.com/kaushalmodi/hugo-atom-feed"
              ", github.com/kaushalmodi/hugo-debugprint")

HUGO_MODULE_REPLACEMENTS=""
for mod in "${replacements[@]}"
do
    HUGO_MODULE_REPLACEMENTS="${HUGO_MODULE_REPLACEMENTS}${mod} -> $(basename "${mod}")"
done
export HUGO_MODULE_REPLACEMENTS

echo "Hugo Module Graph:"
hugo mod graph

hugo server \
     --buildDrafts \
     --buildFuture \
     --navigateToChanged \
     --port "${port}"

Yes, I do not see this problem when using the replace command in go.mod.

This may be a Go module behavior rather than Hugo-specific.

But the go.mod based replacement using replace works… may be I will update my hugo serve wrapper script to modify go.mod on the fly.

That does sound like a Hugo issue then. Hopefully bep or @jmooring has some ideas.

To devs: I can also contribute information and samples and such if that proves helpful and you need more than has already been provided here.

Alright… I am sticking with this for now; works robustly (for me at least) for theme modules with nested modules or not:

set -euo pipefail # http://redsymbol.net/articles/unofficial-bash-strict-mode
IFS=$'\n\t'

port="${1:-3333}" # default

mod_base_dir=".."
mods_to_be_replaced=("gitlab.com/kaushalmodi/hugo-theme-refined")

sed -i '/replace .* =>/d' go.mod
for mod in "${mods_to_be_replaced[@]}"
do
    echo "replace ${mod} => ${mod_base_dir}/$(basename "${mod}")" >> go.mod
done

echo "Hugo Module Graph:"
hugo mod graph

hugo server \
     --buildDrafts \
     --buildFuture \
     --navigateToChanged \
     --port "${port}"

I personally make a /config/development/modules.yaml file then add

replacements: "github.com/user/repo -> ../replacement path"

to make it easier (development only replacement)