If you’re building your site within an IDE (e.g, Microsoft VS Code, etc.), the built-in file watcher can have am impact on build time.
If that’s the case, terminate your IDE and build the site from a terminal.
And if you haven’t done so arleady, look at the template metrics:
hugo --templateMetrics --templateMetricsHints