Hugo Refactor

dev
1

I’ve just pushed a pretty significant change to the way Hugo works internally.

As Hugo grew with contributions and features it’s become more complex. The simple sequential steps we started with have grown complex. I’ve been working on a refactor of how Hugo works with the goal of more delegation and simplifying the code.

Tonight I’ve completed the first major step towards this end. Included in todays push is:

A new handler system.
It’s still a prototype, but already does all that the previous
approach did and more. The concept here is that handlers will be defined for each file type. They are separated into page and file handlers. Page handlers are responsible for rendering into HTML and writing to the destination. File Handlers are only responsible for writing to the destination. I’ve created a couple working examples here. Markdown, Html & Css. The Markdown and Html work the same as they’ve worked for a long time. Css will minify any CSS file as it writes it to the destination. Handlers only work on files in the Content Directory. If you want a CSS file to remain unminified they place it in your Static Directory.

A new hugofs package
I’ve developed a new filesystem abstraction system called Afero. Afero provides a generic filesystem and file interfaces as well as a few implementations of this interface. OsFs simply wraps the corresponding calls from the OS package. MemMapFs is a fully safe and fast memory backed filesystem.

The beginning of the end of Targets
They were a good idea, but we can do more in a far simpler way with Afero Filesystems. Afero’s MemMapFs is far more complete and accurate then the memory target system. It provides a fully accurate mocking and in switching to using it for tests we’ve been able to identify and fix some bugs.

The targets provide translation, but this can be done seamlessly with an Afero Filesystem.

There’s still plenty more to do

Currently hugolib is way too large. The handlers shouldn’t be in hugolib. Neither should the templates and a bunch of other stuff. We need to separate more and delegate more.

I also need feedback on the handlers as well as help developing more and adding more tests around them.

Currently a lot of functionality resides in the Site & Page structs that belongs in the handlers and in filesystems. We need to rewrite this to provide more isolation and separation.

I also want to have a feature where hugo server doesn’t write the destination files to disk, but instead saves it in memory and serves from there. It actually works right now if you just comment out the last line in the hugofs/fs.go file. Using the memory backed filesystem improves build times around 10% according to my rough benchmarks. Yup. Hugo can get even faster.

There’s a lot of things we can do with a memory backed cross platform file system.

Now that the foundation is laid. I’d love some help. A lot of the tickets in the v0.13 milestone depend on this functionality.

Who’s in?

2

I can at least do a rebase of my PR(s) (and as part of that some testing) …

3

I would love to help, but I am a beginner to Go, and would need some help/guidance. My first non-doc bug fix was merged earlier today. :slight_smile:

4

@spf13 Congrats! This is great news; I had been hoping that we would get to see your changes soon.

Your hugo server vision is similar to what I was going to suggest after the refactor had been done. While working on my synchronization package, I had been thinking that Hugo content should mainly be in RAM the disk being used as the persistent store.

Seeing you push out the afero repository raised my hopes that it would be part of the Hugo refactor. I’m happy to see that that is the case and excited at the possibilities it opens up.

I had been trying to limit my changes and pull requests to things that probably wouldn’t be affected by the refactor. Now that it is out, I will focus more on the milestones and issues that affect more of Hugo.

1 Like
5

OK,

I tried to do a rebase from master into my feature branch. The merge itself was smooth sailing, but there are some breaking changes related to sections / section selection.

I would explain it if I understood it, but I cannot find any real pattern here.

One of the symptoms is that this:

    {{ range first 1 (where .Data.Pages "Section" "post") }}

Now returns empty. There are similar confusing issues for the archetype selection.

6

Happy to give as much guidance as needed.

1 Like
7

Go at it. I think this is step 1 of the refactor, but I think it was the largest one and I think that it may be most of what we do prior to the next stable release. I’m a bit hesitant to changing too much at once.

8

The changes that would affect more of the code would be bug fixes. I’ll need to make posts discussing those so nothing will be affected without discussion.

The file stuff is also another discussion, after I get a chance to familiarize myself with afero and the way Hugo is using it now.

9

So hugo was my first real project in go that went beyond a simple script. It was also one of only a few full applications written in Go at the time. In writing hugo I learned a lot about how to write go better. I think the entire community as a whole is still trying to figure out the best ways to write go.

A few learnings are that:

  • Hugo’s design overused struct fields in a few instances where a method would have been superior.
  • Hugo doesn’t use interfaces nearly as often as it should
  • Hugo app structure is too monolithic… In spite of trying very hard to avoid this, it happened. In hindsight we should have separated hugo into a bunch of smaller packages instead of hugolib. This addresses it to some degree, but there’s a lot more to do here.

I also think that we did a lot of things right. Even some of the things I’ve been addressing here I feel strongly that we built the right amount of functionality first and the refactor was essential as Hugo grew, but wouldn’t have made any sense at the beginning.

There’s a bunch of other things I’ve learned through the process. I should probably write a blog post on it, I think it would be pretty interesting.

So Section is a good example of the first point above. It was previously a field on the Page struct. It shouldn’t have been, it’s always a derived value. It should have always been a method.

In head it’s now a method. That did cause some issues as some of the template methods looked only at the fields. PageGroup was written this way and had tests that checked section. I rewrote the pageGroup method to now look at both fields and methods. See https://github.com/spf13/hugo/blob/master/hugolib/pageGroup.go#L98 .

We should add some tests around the where functionality and add the same capability there.

I would also like it if these methods all examined Params as well. I think it would really help usability.

One thing that I think is a bit weird about hugo is that certain front matter values have an elevated status. I understand why I wrote it that way, but in some cases it feels pretty arbitrary. I’d love to erase much of the line between them…

Get should check fields, methods and params, Where and Group should also check fields, methods and params.

10

I look forward to having time to look at this soon—I’ll be happy to join in the discussion and refactoring when I have a bit more time.

1 Like