Hugo’s built in server will not automatically load your custom 404.html file, but you can test the appearance of your custom not-found page by navigating your browser to /404.html.
I have been running into this problem a lot, as I’m trying to customise some themes, including their 404 pages. Whenever I go to a nonexistent page, I get a blank and blinding plaintext 404, despite the existence of custom 404s.
If you want to test your 404 page, just visit localhost:1313/404.html and you’ll see how it would appear.
As for why the 404 is not shown, I can’t remember the explanation but if I understood it correctly, Hugo server (usually localhost:1313) is not a complete server for it to understand a 404 error and load the 404.html.
You will have to install a complete server like Apache or Nginx and generate the Hugo files there, and use it as your localhost server.