As git is not good at handling large files, over the time any git-based repo grown ugly if plenty of binary content gets added to it.
It slows down git clone. Recently gitlab (a popular open-source alternative to github) added support for git-annex in their enterprise edition. Details - https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/
My idea is to have hugo config take care of git-annex remote. And everytime I run hugo, remote be synced to local repo. This way git-annex will work in background for a user.