Pygments is so slow. What about incremental builds?

support
1

Note: I put this for now in «support», because I hope you can say that this is because I’m doing something bad. Anyway, maybe this should be in «feature», because incremental builds seems just logic for me.

Yeah. It seems to be so slow, sorry. At least for me.
As example, my hugo build time right now:

Change detected, rebuilding site
2015-04-30 11:39 +0200
0 draft content
0 future content
24 pages created
4 paginator pages created
99 tags created
6 categories created
3 series created
in 14058 ms

(I’m using stable Hugo, v0.13)

Fiuuuu. 14 loooooong seconds. Live reload is pretty funny right now.
And yeah: I use Pygments highlight for code. Why? Because I like it.

Wait, wait! You use Pygments? That’s so damn SLOW! That’s the reason! It’s not Hugo,
its because you use Pygments, man. Hugo is blazing fast.

Aha. Pelican, generating the same content, but more complete (still doesn’t showing TOCs in Hugo, like example, because I’m still migrating, and I have more tags, more categories…):

11:44 $ pelican
Done: Processed 24 article(s), 0 draft(s) and 0 page(s) in 5.65 seconds.

So, I don’t thing this is about the highlight thing. Another reason?

And I’m still adding content. Every article I add, Hugo turns more and more slow.
Something seems odd to me.
Why Hugo, so modern and fast is said it is, is slower than any other static generator I tried before? (Pelican and Acrylamid, basically).
And why it hasn’t incremental builds? Why regenerate everything everytime you change anything in one file, even a single comma? And this (don’t having incremental builds) is just part of the problem: Pelican also hasn’t incremental builds (acrylamid has), and uses python (supposedly should be slower) and… well, it’s three times faster right now (and Hugo gets slower). And just before that, I edit something in an archive and rerun and only 3 seconds in pelican (five times faster). If I do the same in Hugo, it’s as slower as always.

Please, tell me that I’m doing something terribly wrong and that this has an easy miraculous fix T_T.
And don’t tell me «If Pelican or Acrilamyd are so fast, use them instead», because that’s not what I’m asking ;). Any ideas anyone? If needed any more info, like config bla bla, I can post it.
What do you think about the incremental builds thing?

I’m so sorry if I appear to be impolite here. I’m aware that sometimes the way I explain things can look that way. If so, it’s totally not intended. I tend for exaggerate things, and to illustrate things with dramatic examples and/or overreaction, but there isn’t bad intention in any way :þ.
Thanks to all for your time :).

2

The issue is Pygments which is very slow. The documentation even warns about this. The latest HEAD of Hugo adds caching to the Pygments library which significantly improves this issue. I would suggest using HEAD or stop using Pygments.

3

With those page counts, there is something wrong. Done right, Hugo is blisering fast.

  • Pygments is fairly slow, but not THAT slow. This is fixed in Hugo 0.14-DEV (added cache)
  • Large amounts of files in /data (hundreds of megs) also tends to slow it down

Incremental builds are complex and not the solution to YOUR problem.

Could you share your project?

4

@spf13 Well… I just used sed for replace the highlight shortcodes with the default Markdown code markers, and yes: using highlight.js, it’s really, really faster (424ms the same content). Pretty much faster than Pelican (even without using Pygments, 5.xx seconds). But that’s an obvious point. I guess that with the lastest HEAD with caching, the improvement will be noticeable, and at least, match Pelican speed :). For now, I will switch to client side highlighting then. Thanks. In a time, I will retry with Pygments, to see the changes :).

@bjornerik I can’t share the project (that sounds like a serious thing), just because I still didn’t uploaded it. But it’s a tiny, tiny, tiny blog, with empty /data folder. The most heavy folder is /images (in static), and it’s only 5 MiB xD. The entire site, for now, it’s about 12 MiB xD. So a tiny tiny tiny thing ;).
In any case, I will switch to an js highlighting script for now. Thanks.

5

If it’s a tiny thing then something is wrong.

But, if you cannot share it’s not easy to help.

6

Just to add my twopence, my 20+ pages (with Pygments) get crunched in 70ms on average. So, no, Hugo is not slow :slight_smile:
I’m on 0.14-DEV and MBP with an SSD btw.

7

Adding my two yen, I’m still testing as well, and using highlight.js though, not pygments. My site with about 20 pages renders in ~80ms.

8

I’m happy to complyread that, @royston. Considering that time, I guess that the cache feature implemented in 0.14 really pays off :). I’m running stable hugo, so I will see that in next release ;).
@bjornerik Yeah, I know it, sorry :þ. I just push it to a new branch in my blog repository. You can see it here. Surely, if speed gets so better like @royston pointed (when 0.14 comes out and/or if there’s some funny mistake in my config or something like that) I would prefer to stick with Pygments rather than rely in some js script.

9

@soulchainer I use lots of pygments on my sites, and none of my sites have ever (even pre 0.14) exeeded 1 sec.

1 Like
10

I just look at the hugo help and executed hugo with --verbose=true and --stepAnalysis=true (Why I didn’t think about this BEFORE?), for getting where the time it’s spent and I got this:

$ hugo server -w --verbose=true --stepAnalysis=true
INFO: 2015/05/02 Using config file: /home/soulchainer/.github/soulbits/config.yaml
INFO: 2015/05/02 syncing from /home/soulchainer/.github/soulbits/themes/greyshade/static to /home/soulchainer/.github/soulbits/public/
INFO: 2015/05/02 syncing from /home/soulchainer/.github/soulbits/static/ to /home/soulchainer/.github/soulbits/public/
initialize & template prep:
	40.098832ms (40.74811ms)	    1.41 MB 	17168 Allocs
load data:
	91.632µs (40.923173ms)	    0.01 MB 	100 Allocs
import pages:
	14.845659925s (14.886659469s)	   61.07 MB 	142958 Allocs
INFO: 2015/05/02 found taxonomies: map[string]string{"tag":"tags", "category":"categories", "series":"series"}
build taxonomies:
	488.722µs (14.887197894s)	    0.07 MB 	1544 Allocs
render and write aliases:
	2.721137ms (14.889984411s)	    0.18 MB 	2765 Allocs
render and write taxonomies:
	125.094475ms (15.015255471s)	   87.43 MB 	325183 Allocs
render & write taxonomy lists:
	387.179µs (15.015721685s)	    0.02 MB 	574 Allocs
render and write lists:
	20.826932ms (15.036630236s)	    7.00 MB 	16019 Allocs
render and write pages:
	10.771505ms (15.048798017s)	    5.61 MB 	53779 Allocs
WARN: 2015/05/02 Unable to locate layout for 404 page: [404.html theme/404.html]
render and write homepage:
	22.311796ms (15.071223483s)	    7.26 MB 	24448 Allocs
render and write Sitemap:
	2.0334ms (15.07331836s)	    0.20 MB 	4140 Allocs
0 draft content 
0 future content 
24 pages created 
4 paginator pages created 
123 tags created
6 categories created
3 series created
in 15052 ms

It appears that the great part of the time is wasted in

import pages:
	14.845659925s (14.886659469s)	   61.07 MB 	142958 Allocs

Makes this any sense? According to this, it’s not even related with Pygments, right?
What does exactly this import pages process do?
But this is strange, because, if I use js for highlight, this is bleeding fast, so…
Using js, import pages is about

import pages:
	71.691671ms (133.485083ms)	   64.63 MB 	93096 Allocs
11

Again, without a complete project to look at and test, it is impossible to guess.

12

You mean more complete that the one I already upload specifically for this and comment to you a pair of comments ago?
What’s specifically missing? Git didn’t upload the rest of the folders (like archetypes or data), just because they’re empty. I guess the public/ folder isn’t necessary, so… I don’t know what is really missing >_<.

13

Sorry, I slept little last night … :slight_smile: I will test it on my PC.

14

No problem :). It happens to all of us.

15

Yes, it’s the Pygments… You must have a whole lot of source code.

Hugo 0.14 is as slow as 0.13 on first build, but repeated builds clocks in at 300 ms.

➜  soulchainer.github.io git:(hugo-pygments) hugo-013 
0 draft content 
0 future content 
24 pages created 
4 paginator pages created 
123 tags created
6 categories created
3 series created
in 11229 ms
➜  soulchainer.github.io git:(hugo-pygments) hugo    
ERROR: 2015/05/02 Site's .BaseUrl is deprecated and will be removed in Hugo 0.15. Use .BaseURL instead.
ERROR: 2015/05/02 Paginator's .Url is deprecated and will be removed in Hugo 0.15. Use .URL instead.
0 draft content
0 future content 
24 pages created
4 paginator pages created
123 tags created
6 categories created
3 series created
in 305 ms
16 
find . -name "*.md" | xargs grep -i "/highligh" | wc -l
249

That’s a lot of highlight blocks for just 24 pages.

17

Well, at least it’s clear now :). Thanks for all your help, @bjornerik :).

18

Could someone tone down the title of this thread? It isn’t true and it looks like something a Jekyll fan boy would write.

19

I shifted it to “Pygments is so slow” which more accurately represents the content in the thread.