My experience with Hugo modules in this area have been very good. There is a module cache which makes it in general very fast (I just tested now with some of my module backed sites, and all of them built in < 100 ms). Not sure what happens in your situation, but that is hard to debug remotely.
Even if there is some startup penalty, you should not see that in server reloads.
What you can look after is if you see log lines that looks like hugo: Downloading modules ... which is an indicator of remote work being done, which should not be needed for every build (because it’s cached).