Hugo Testing - How to Contribute and Help

dev
1

Hello Everyone,

For those of you have been been asking (in particular @voxadam, @bjornerik, @mohae, @tatsushid, @tommasop) how to contribute to Hugo in the “Contributing golang code” thread:

I have said I would start a new dedicated testing thread. Well,err, this is it :slight_smile:

To give everyone some background. I started hacking at Hugo for two reasons. Firstly I use hugo and I like it a lot. I much prefer it to the previous static website builder solution I was using (nanoc on ruby). Converting one of my sites to Hugo got me out of Ruby dependency hell, which I really appreciated. Go’s single binary approach is a big win. Secondly, I liked go. I’d used it previously to move an internal company web service from perl to go. As much as a proof of concept project as much as anything, So, I was looking for a go project to pitch in on :slight_smile: And finally working on the the testing side of a project isn’t what I do for a day job. So that’s a big plus :slight_smile: [For the curious, I’m actually a ScrumMaster in my day-to-day role],

When I asked Steve (@spf13) and Nate (@natefinch) what they were looking for in terms of help both suggested taking a look at testing. After that I just grabbed the code, ran the tests, and looked at the code coverage the tests were currently giving us. Then I just picked the one with the lowest figure an started adding tests. So it was dead easy to start helping out.

So, far I’ve focused on the hugolib/helpers module taking that from 25% coverage to 75%+. See my original testing thread for the details. Note; this is still WIP.
http://discuss.gohugo.io/t/hugolib-helpers-first-stab-at-extra-tests/58

As much as there is a plan for all of this, what I think we should do is:

  • Increase the test coverage generally across the whole project
  • Then look to refactor the code and the tests

If we can achieve higher test coverage then that will enable as much refactoring as may be required. It makes adding new features easier, not only though better (refactored) code, but also because we’ll have a suite of regression tests. It’ll make bug hunting easier to and with a little luck the there will be fewer bugs going forward because of the tests.

So if you want to help out, post a message to the thread and we’ll see what we can all achieve.

Owen

2

Count me in please!!!

3

@tommasop

ok then you’re in :).

Owen

4

@tommasop @teknico @oscar_b @bjornerik @mohae @spf13

Folks, if you’ve not yet seen it @spf13 has merged in some of my tests and fixes into master.
@spf13 - ta, very much :slight_smile:

I’ve just done a quick catchup and I’m seeing test coverage of:

$ go test -cover ./…
? GitHub - owenwaller/hugo: A Fast and Flexible Static Site Generator built with love by spf13 in GoLang [no test files]
ok github.com/owenwaller/hugo/commands 0.023s coverage: 11.5% of statements
? github.com/owenwaller/hugo/create [no test files]
ok github.com/owenwaller/hugo/helpers 0.013s coverage: 46.4% of statements
? github.com/owenwaller/hugo/hugofs [no test files]
ok github.com/owenwaller/hugo/hugolib 0.114s coverage: 65.0% of statements
? github.com/owenwaller/hugo/livereload [no test files]
ok github.com/owenwaller/hugo/parser 0.006s coverage: 42.6% of statements
ok github.com/owenwaller/hugo/source 0.007s coverage: 45.7% of statements
ok github.com/owenwaller/hugo/target 0.005s coverage: 49.2% of statements
ok github.com/owenwaller/hugo/transform 0.009s coverage: 52.2% of statements
ok github.com/owenwaller/hugo/utils 0.002s coverage: 11.1% of statements
? github.com/owenwaller/hugo/watcher [no test files]

So what I’m going to suggest is that we take a package each (or in pairs - however folks want to wok) and see if we can up the coverage.

So I’ll take the utils package as that’s at 11% and I’ll see if I can up helpers back up again. I seems to have gone down post refactor.

Are there any other takers for the other modules?

Owen

5

First, I’ll start improving test coverage of hugolib/pageGroup.go, hugolib/pageSort.go and hugolib/template.go, not a sub package. I’m familiar with them and have written some of them.

6

@tatsushid That’s fine with me. If you’re familiar with them so much the better.

Thanks

7

One slightly related questions that I have been pondering.

Arrest me if I’m wrong here, but in Go there doesn’t seem to be a separation of tests vs actual code other than the test functions (by naming convention).

  • So if you need some temporary struct you can create it inside the function
  • But if you need some temporary function, it is visible in that package. You can prefix it with test, dummy etc., but it’s still there on autocomplete etc. when doing the actual implementation.
  • This becomes kind of ugly when you add some temporary test methods to structs in the domain (Page …)

Am I missing something here? Any good workarounds?

9

@bjornerik, the third case is, for example, doing in my latest PR, isn’t it? Sorry for it.

Go doesn’t have the functionality to insert method to an instance, like Ruby, Perl etc. So, if you just need a small code to reuse in a function, you can define and use a function literal (Closure) but if you need something struct method, there is no way to do it except defining it in package wide scope.

But, Go separates a test code from an actual implementation not only a function name but also a file name. *_test.go is only built into a binary when go test is executed. You can easily see what files are compiled by using -x option with go build and go test. Therefore, only to cut the functions, methods defined in test code from auto completion list, it may be able to do by ignoring *_test.go files.

10

@tatsushid - yes the last case was “you”, and that (great commit - great tests btw) got me thinking, but I have done similar myself.

But what you’re saying about them not getting their way into the final binary is the most important thing. It’s just me that is used to the “Java world” (a lot of bad things to say about that world, too, of course) - where the test and “run” classpath is very separated, but at runtime and in the IDE (test can see the runtime classpath, but not the other way around).

I have tested it now, and, although confusing … it works:

  • I can reference test types from my “real” code and use them in “go test”
  • but “go build” will fail

Go figure.

11

@bjornerik, Go is similar to C or C++. go test builds all codes in a package, links them and makes one binary for test. So the “real” code can refer the codes in test files because those are merged in a same binary, same scope (If you use the same package name in test files). That’s why a test code can call an unexported function and struct in a “real” code.

On the other hand, go build does the same but only use “real” code. It ignores *_test.go files. So the binary built by the command doesn’t have any test codes to be called by “real” code.

12

@tatsushid thanks for that input, very useful.

13

Doing some work on pagination, I was looking for some end-to-end tests I could expand upon, but didn’t find them.

There are lots of unit-tests and lots of semi-integration tests – but I see none that tests full site generation with all the whistles. Am I missing something? I’m a litle bit hesitant to start that task myself …

Maybe @OwenWaller has some input on this?

14

@bjornerik, and Happy New Year.

As far as I am aware(*) your understanding is correct. The sort of end-to-end tests don’t exist (yet) but we absolutely need them. So no you are not missing something and I agree this is likely to be a large task.

Do you know if hugo comes with some sort of “Get started here” or tutorial site, yet(**)? If we had something we can use as a standard site - or rather some content that we can generate this site from - then all we really need to do is diff the output of that standard site against some “master” copy. If there’s no diff then everything works. If the standard content doesn’t exist then I suspect setting this all up may be a fair bit of work, but very much worth while.

What would be your thoughts on how to proceed at this point?

Owen

(*) Sorry I’ve been out of the loop for the past month with work and family commitments. My understanding is based on the code as it was about 6 weeks ago. I am hoping that I can get a little time on Hugo from this week onwards.
(**) Of course if anyone out there knows of a site we can use as a stating point that would be very helpful.

15

This could be a good starting point, maybe:

@ahsanulhaque might chime in with a status.

16

@bjornerik

This seems as good a starting point as any other.
@ahsanulhaque, what do you think? Is there potential in using this as a starting point for a set of test data for testing? How far along are you with this?

Owen

17

I played a little with the great go-fuzz tool today:

https://github.com/bep/gobep-fuzz (uses https://github.com/dvyukov/go-fuzz)

There are some fuzz tests for some areas in Hugo. It found some panics:

18

Hello guys!

I would like to contribute as well. Been learning Go for the past 4~6 months now, and I’ve been using Hugo to convert my Blog and my Website into. So count me in as a huge fan.

Also starting on increasing test coverage is an awesome way to learn how the code is structured.

Cheers,
Gergely.