If you’re wondering what’s up with the whitespace, it’s because I hard break lines at 80 characters in my Markdown file.
In any case, sometimes if I make a change to this file and save it:
Hugo processes that and the link is created successfully
Hugo processes that and it sometimes literally outputs [non-root user]/blog/running-docker-containers-as-a-non-root-user-with-a-custom-uid-and-gid}
Note worthy things:
There’s a trailing } in the output
I’ve seen this happen at least 10 or 15 times now after using Hugo for only a few weeks. It’s not isolated to this specific post. I’ve seen it in others as well.
I’m using: hugo v0.128.2-de36c1a95d28595d8243fd8b891665b069ed0850 linux/amd64 BuildDate=2024-07-04T08:13:25Z VendorInfo=gohugoio
It makes me nervous because it doesn’t throw a warning or error. I’ve also had it happen with --logLevel debug and there’s nothing extra in the output vs a situation where it works.
I fear if I build my site let’s say 20 times, maybe 1 or 2 of those times result in having many broken links. I don’t know if this problem is isolated to the server or if it’ll be an issue when building production versions of my site too.
Do you have a markdown render hook for links in your theme? If not, even if, what template is resulting in errors? I am sure you can trace the “sometimes” wrong output to specific layout files where the content is handled not via page.Content or something like that but via .Render or .markdownify or other methods and I remember faintly that there is one specific render-method that does not render (Hugo) shortcodes.
You might get more help from the experts if you have a (minimal, if your repo is confidential) sample in a git-repo that produces this error.
Faced with this I would try to find out if my render hook has issues and if not I would try to trace the “sometimes” down to specific cases.
Technical issues that come to mind: Maybe Hugo has a problem with not yet existing references when creating content with these shortcodes. There are ways to set how many processes to use in parallel by setting HUGO_NUMWORKERMULTIPLIER before the server/hugo command runs, but I am not sure if that is the right way.
Yes. Here is the exact one with zero edits. I use it to make external links open in a new window and to also add certain query string params to certain domains:
{{- $u := urls.Parse .Destination -}}
{{ $destination := .Destination }}
{{- if in .Page.Site.Params.CourseURLs .Destination }}
{{ $destination = print .Destination "?utm_source=nj&utm_medium=website&utm_campaign=" (.Page.RelPermalink | strings.TrimSuffix "/") }}
{{ end -}}
<a href="{{ $destination | safeURL }}"
{{- with .Title }} title="{{ . }}"{{ end -}}
{{- if and $u.IsAbs (not (in .Destination .Page.Site.BaseURL)) }} target="_blank"{{ end -}}
>
{{- with .Text | safeHTML }}{{ . }}{{ end -}}
</a>
{{- /* chomp trailing newline */ -}}
It’s pretty much taken straight from the docs. I only modified the course URL part.
Beyond this and the link in my original post that is as far as I know everything needed to reproduce this? If I manually make an edit to a post with that link, save it and perform this workflow I can reproduce it to some degree. I can’t say for sure it’ll always happen but I repeated the workflow a few times and it happened within 1-60 tries 3 times. My CPU has 4 cores.
There is no error being reported or anything in the logs to trace. Even with debug logging.
It’s possible, that would indicate this is a problem for both serving and building. Unfortunately if I reduce this to 1 core then builds will likely be N times slower though.
I just confirmed it does happen with production builds too, it’s not related to the watcher. I found broken links on my site in the same way the original post is described here.
I’m going to temporarily revert back to Jekyll until I can figure out how to reliably prevent this. It’s too risky to have any number of ~550 pages happen to have broken links.
Builds go from ~2s with 4 cores to ~6s when setting HUGO_NUMWORKERMULTIPLIER=1 by the way.
As per David’s suggestion I’ll try running with 1 worker
I also added this command to my wrapper script to perform builds
if grep -R "\[.*\]\/" published/blog; then
echo "Aborting, this build produced broken Markdown links!"
exit 1
fi
The basic idea here is if this command finds a match then it will halt my deploy process so it won’t get sent to my server.
I’m not 100% sure if all failures will match that regex but it’s better than nothing and it does match a few broken link outputs I’ve at least seen with my own 2 eyes.
guess you already checked with --printPathWarnings
I haven’t, when I do this I get hundreds of path warnings about duplicates found with (2) for most of them. I don’t know why it does this though. It basically says every asset and blog post is a duplicate.
your baseURL contains a trailing ‘/’
It doesn’t. I have it set to baseURL: "http://localhost:1313" in development. Should it be using a trailing slash? This hook is the only spot where I directly reference baseURL.
if you overwrite these shortcodes
I do overwrite them with {{ (relref . .Params) | strings.TrimSuffix "/" -}} because I don’t want trailing slashes in my URLs. I can’t recall if this issue happened before I overwrote it. I would say maybe 70% yes it did because I recall in development having to sometimes modify and save a page to get links to process correctly and this TrimSuffix strategy is something I did near the end before going live.
untested pseudo code just to visualize
When the issue happens will your errorf output produce new information that the debug logs didn’t provide?
That may produce a number of false positives because I have posts that purposely output {% endfor %} and other Jekyll / Liquid syntax since I’ve blogged about Jekyll in the past. I also have a new Hugo post coming out where I purposely output {{% and other Go template syntax.
there are some cases where it could happen as a false negative - but hundresd of them seem a little strange. Assets should not be touched when the blog is rendered by content md files. If you have sassets that render to same path it will. Impossible to tell without all involved stuff.
Ignoring warnings should be done after thorrowly checking that these are false positives. If so - maybe then it’s a bug in the detection code.
Yes
would have been a valuable info when posting a ref shortcode issue. or?
could be a valid testcenario to not change the ref if you have use [text]({{% ref XX %}}] it will be later converted by the render hook.
also the ref is called from a layout/partial chain I guess to just the hook is not enough - also a good reason to share a reproduceable example always .
The printf would output the page logical path of the page , the params (path) given as input and the calculated ref. so I assueme: yes it would.
The code will just output the info stated above, so other stuff on the page won’t be issued.
if most of the links are correct it depends if you filter the output in Hugo using findRE - narrow the pattern to include only real negatives or ignore the positives.
another way would be to use grep on the result pages with proper patterns and then analyse their layout and partial chain… would not be the first time template code breaks something and not hugo core.
all together:
the core produces some output → if that is all valid (according to hugo style) and jsut not your wanted output it must be somewhere in your code/config
Link hooks are a Hugo construct and I’m using an example from the docs with nothing too crazy
Markdown content is processed by Hugo
I wish I could identify what part of my code is causing this but I’m not able to get any information from Hugo when the problem happens. It doesn’t output even 1 hint that something went wrong.
Because “it” thinks that “it” did the right thing. Sometimes, the output is not what you expect but still “valid” in the sense of “I did what you asked me to do with your input”. Don’t hang up yourself on the missing error messages. The issue is somewhere within the markup or the layout files (in my opinion, not fact) and what you get might be what your input is suggesting it to be the wanted output.
Which brings us back to the “reproduce-able sample code”… You are describing a black-box that receives input and produces output, without issuing any parsing or linting errors.
I output a few divs, there’s a couple of Go ranges and partial calls. The markdown is included in this post minus a bunch of standard Markdown.
I don’t know what I could be doing in my layout file that would cause this. Without something provided by Hugo then I have no idea where to start.
This post has enough details to create a minimal example. It’s a link hook and a content file that uses relref to render a cross link. If you threw that in an infinite loop to keep building, that may be able to reproduce it.
I just updated to v0.131.0 and it’s even worse now. It’s happening more frequently and I’ve seen other issues too like using the official YouTube shortcode. It will sometimes just render the string literally using the exact example from the docs {{< youtube 0RKpf3rK57I >}}.
I’ve also seen it start to happen with images in the same way it happens with relref.
Edit: This is with regularly using Hugo and not defining 1 worker process btw.
Given I can reproduce it occasionally using relref what else could I do to narrow it down?
I mean Hugo is providing all of these constructs that I’m arranging together in a post. I’m not introducing any custom logic.
I am relaxed. I’m just not sure how to proceed because I cannot get any information out of Hugo to begin debugging the issue.
I could try creating a repo using the quickstart tutorial and a link hook from the docs then build things in an infinite loop over night and see if that does anything.
But let’s say it doesn’t produce a single rendering failure. I’m now on one side of the spectrum with a minimal example and then I have my real site where it is a problem. I don’t feel like I have the tools necessary to determine where the issue is based on the outputs I receive from Hugo.
I will say this, my site isn’t doing anything crazy. It’s using all constructs provided by Hugo and Go. The only reason it’s not open source is because I have ~1.5 GB of images and I didn’t want to put them in a repo.
I guess where I’m going with this one is, I’m not sure how having code like this:
Will be related to the links not being processed correctly.
As for image outputs being a new problem with v0.131.0, I’ve posted my full image partial in the other thread where we figured out a solution to the fingerprint delimiter. The YouTube shortcode is straight from the docs / Hugo.
In this post I’ve posted my exact unedited link render hook and how I’m overriding the relref shortcode to strip the trailing slash. That’s the only things related to link processing right?
I haven’t read this entire thread, but I would try to understand and resolve the page collisions first.
As others have mentioned, this might be easier to troubleshoot if we could see the entire picture including site configuration, etc. From your perspective it may seem like you’re providing enough information to reproduce the problem, but from my experience the devil can be in the details.
Include a link to the source code repository of your project, because we really need the context of seeing your templates and partials to be able to help you. It is trivial to do a quick git clone on your repo, then run hugo server in your project, to help you out. On the other hand, recreating your code from screenshots, or sort of guessing at it, is not.
If you can’t share your repository for whatever reason, consider creating a dummy repo that you can share, which reproduces the problem you’re experiencing.
Though not mentioned, you can DM a link to your repo to these folks here trying to help you (if it is a repo of reasonable size). Because you won’t receive any help if you don’t do so and your request will just be ignored.
