How do you test generated endresult


#1

While creating some new posts (migrating picked posts manually from a very old page) and also changing the structure of some existing posts (moved them to another folder) I haven’t noticed that I actually broke it.

What happened is that I only found it in prod when looking through the articles a bit (call it explorative testing) I noticed that a picture is gone on a post and the highlighted code is just concatenated to one big string without any formatting.

As I’m just starting out with hugo experiments, I have automated dev and prod deploys with CI and so on but I’m missing any real tests besides that hugo generates output and doesn’t throw any error.

With a handful posts, explorative testing is no issue but after it grows a bit, if it’s not automated you probably just need luck to not miss it.
As my day job is developing tools and frameworks that developers and testers can easily write automated tests in their projects, I need to stick to my principles and find a good solution at least for my own stuff and block broken output to dig it’s way into prod.

What ways do you use for sort of unit testing the generated code in your project(s)?
Are there already some ideas or solutions implemented?

I’m not looking at usual functional tests (no way this makes sense going trough this maintenance hell when you have lots of posts) but actually using the md files as spec and check it against the generated html.
Side effects could then also be to check for broken links, having specs for how certain shortcodes and template layout components need to look like from the structure, get a impact diff (when doing commits with small changes, see if it did fundamental changes to modified files or even had a side effect.


#2

this won’t cover everything you’re looking for, I don’t think, but is pretty handy: https://github.com/gjtorikian/html-proofer


#3

@rac2030 Would you consider “link checking” within this multi-faceted scope of checking the rendered output?

Some context: there was some discussion surrounding the topic of link checking about a year ago, and I’m considering putting this in as a feature request.


#4

@budparr Yes while something like this surely helps, it would be only part of a full testing solution.
@rdwatters Having link verification (at least internal one) built in together with asset checking (everything referenced as internal assets like relative paths to images or other static assets) would be cool.

Tough I see it also as somehow problematic as some relative urls could still point outside but this could be catched with a sort of whitelist to ignore those.

Applied to the issue that occured for me, it would have given a warning or error as one image was missing, thus may preventing the issue that occurred later in the output.

Will think a bit more about that topic and how it could be taken up in a way that would be useful to all sites regardless of the theme used (which I see as the main pain point in getting a general usage built in).
I will have to try to reproduce my exact issue as the html output produced looked syntactical correct but was missing the rendering from markdown in the html after the said missing image as well as just removing all the highlight wrappers and putting the content as plain md inside the html. Not sure this is failing from hugo itself stopping the blackfriday rendering or if this is actually inside the renderer. I can say more when I reproduced it and can dig into the codebase of hugo. Hopefully I get then some idea on how to best tackle the testing issue :wink:


#5

Would definitely second a link checking feature. +1


#6

Here’s something neat for you: Check my Links is a Google Extension. Very nice visual representation - functioning links appear green and broken links red with error code (e.g. 404). Something else probably exists for Firefox.