But I’d like to know why this file now exists?
The only info I can find online so far (without digging into Hugo’s source code) is from the release notes
Hugo now writes an empty file named .hugo_build.lock to the root of the project when building (also when doing hugo new mypost.md and other commands that requires a build). We recommend you just leave this file alone. Put it in .gitignore or similar if you don’t want the file in your source repository.
// Used to control concurrency between multiple Hugo instances, e.g.
// a running server and building new content with 'hugo new'.
// It's placed in the project root.
lockFileBuild = ".hugo_build.lock"
When you run hugo new ... with archetype directories with lots of images etc., that would earlier trigger lots of changes if you had a running server and also some issues with building from partially written files.
Now when you do hugo new ... we acquire a write lock (OS level) on .hugo_build.lock before we write to /content and then release the lock when done.
The same lock is acquired and released on every hugo build and also all rebuilds when running a server.
A small addendum to the above: The file locks are standard OS file locks (which, isn’t very standard when you look at it), so it should be possible for external tools that want to dump large amount of files into /content to do the same lock dance.