# Difference testing

**URL:** <https://discourse.gohugo.io/t/difference-testing/41498>\
**Category:** support\
**Created:** [November 17, 2022, 12:05pm UTC](https://discourse.gohugo.io/t/difference-testing/41498 "2022-11-17T12:05:25Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![drElektron](https://avatars.discourse-cdn.com/v4/letter/d/7ea924/32.png) [@drElektron](https://discourse.gohugo.io/u/drElektron)\
**Post date:** [November 17, 2022, 12:05pm UTC](https://discourse.gohugo.io/t/difference-testing/41498/1 "2022-11-17T12:05:25Z")

</div>

I wanted to plunder the hive mind for views on simple/basic strategies for testing differences between Hugo outputs.

The scenario I’m imagining is one in which a small/medium site is updated (content, resources, Hugo version…) and you want to check how the output compares. Something that is too big to do manually, but doesn’t merit time, cost and effort for a large testing solution.

I thought it might be broadly helpful to get a sense of what people currently do when they want to satisfy themselves that changes are non-breaking, without disappearing down a testing rabbit-hole. Hopefully that makes sense.

---

<div class="post-metadata">

**Author:** ![ju52](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/ju52/32/6749_2.png) [@ju52](https://discourse.gohugo.io/u/ju52)\
**Post date:** [November 17, 2022, 1:34pm UTC](https://discourse.gohugo.io/t/difference-testing/41498/2 "2022-11-17T13:34:29Z")

</div>

If your site is not too big, I would go with a local GIT repo.

---

<div class="post-metadata">

**Author:** ![drElektron](https://avatars.discourse-cdn.com/v4/letter/d/7ea924/32.png) [@drElektron](https://discourse.gohugo.io/u/drElektron)\
**Post date:** [November 21, 2022, 12:10pm UTC](https://discourse.gohugo.io/t/difference-testing/41498/3 "2022-11-21T12:10:13Z")

</div>

Yeah, I think there are tools like [grouse](https://github.com/capnfabs/grouse) that make that kind of approach easier. But as you suggest, it may not be the most efficient thing for anything above a certain size or if a user is running Hugo through some external platform.

It was perhaps a silly question as this kind of thing is very context dependent. I just thought it might be useful for people to read about some of the approaches that others are using to keep ahead of potential breaking changes when performing updates.

---

<div class="post-metadata">

**Author:** ![lkhrs](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/lkhrs/32/22287_2.png) [@lkhrs](https://discourse.gohugo.io/u/lkhrs)\
**Post date:** [November 21, 2022, 2:50pm UTC](https://discourse.gohugo.io/t/difference-testing/41498/4 "2022-11-21T14:50:11Z")

</div>

I’ve been thinking about doing this too, with testing frameworks like [Jest](https://jestjs.io) or [Playwright](https://playwright.dev).

I’d be looking at writing targeted tests, not just comparing entire pages (although you can do that with these frameworks). Checking titles, meta descriptions, Open Graph tags, links, etc.

---

<div class="post-metadata">

**Author:** ![jmooring](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/jmooring/32/4214_2.png) [@jmooring](https://discourse.gohugo.io/u/jmooring)\
**Post date:** [November 21, 2022, 5:23pm UTC](https://discourse.gohugo.io/t/difference-testing/41498/5 "2022-11-21T17:23:15Z")

</div>

> [@drElektron](#):
>
> It was perhaps a silly question

I think it’s a _good_ question.

@ju52 mentioned Git, where you can quickly see a set of **diffs**. Good idea.

@lkhrs mentioned **targeted tests** to reduce noise. Also a good idea.

A very simple approach, using targeted tests with diffs to compare, might look like:

1. Create a section of your site named “tests”. Each page in this section is designed to “stay the same” from one build to the next, regardless of content changes. That means these pages should not include last modified dates, or reference to any pages outside of the “tests” section.
2. Use configuration directories instead of a single configuration file.
  - In your default configuration, use build options to _exclude_ the “tests” directory
  - Create a “testing” configuration directory, using build options to _include_ the “tests” directory. You could also add configuration options to remove everything that you won’t test.

3. Clear your public directory, build your site with “hugo -e testing” and save the public directory somewhere. This will be your “gold” reference for future comparisons.
4. To run a test, build your site using the testing environment (`hugo -e testing`), then use the `diff` command to compare before and after, limited to files generated in `public/tests`.

Here’s an example:

```bash
git clone --single-branch -b hugo-forum-topic-41498 https://github.com/jmooring/hugo-testing hugo-forum-topic-41498
cd hugo-forum-topic-41498

```

See the bash script (test.sh) in the root of the project directory. Make sure you declare the project\_dir in the `main()` function before testing.

---

<div class="post-metadata">

**Author:** ![nternetinspired](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.gohugo.io/nternetinspired/32/19706_2.png) [@nternetinspired](https://discourse.gohugo.io/u/nternetinspired)\
**Post date:** [November 22, 2022, 11:02pm UTC](https://discourse.gohugo.io/t/difference-testing/41498/6 "2022-11-22T23:02:10Z")

</div>

Personally, I would use Netlify branch deploys and enable their split testing feature. Pretty much zero config, splits traffic at the network edge. [Split Testing | Netlify Docs](https://docs.netlify.com/site-deploys/split-testing/)