I fail to understand what you want, because having a sitemap with Japanese items but not having these pages created will result in errors (or penalties?) in most search engines.
Having said that, I recently learned that Hugo automatically creates sitemap indexes for translated websites. One sitemap file per language and one sitemap index file.
Maybe create a sitemapindex layout that ranges through your structures however you want them to be organized?
Also have a look at @ju52’s repo in this topic.