I am afraid that moving assets outside git is the only way to go.
A while back I opened a topic, motivated by similar concerns like yours, about moving a project from git to a Rsync workflow and some suggestions regarding hosting providers, were made over here
However I admit that I still haven’t taken the time to test these recommendations.