Very abstract answer:
You have content. That content has a structure.
content/_index.md
content/chapter1/index.md
content/chapter2/index.md
content/appendix/_index.md
content/appendix/glossary.md
...
This is what a menu gives you in Hugo. Without much configuration it will take title/linktitle for the titles and weight frontmatter to sort the entries. You define them to be part of a menu and then range through those menus.
Parts of your content are in a single file.
content/chapter1/index.md
You have headlines inside this file (hopefully marked via #, ##, etc). These headlines can be extracted with a “Table of Contents” (ToC, that’s how Hugo calls them) on a per page base.
Be careful not to confuse your "Table of Contents " with Hugo’s. I would maybe call your structure Book Index or something like that.
These fragments are basically objects containing a single page ToC that you can use in your navigation.
To get to your Book Index I would range through .RegularPages (sorting them by weight before) and then load each single page and display or load the ToC of each page. You can configure the ToC to use only hx to hx headings (depending on your layouts, that might be h2 or h1 to h3 or however deep you wish to show your structure.
Because that touches every single page, I would think about caching this, which might get you into trouble if you want to show some form of “you are here” indicator in your Index (which is solvable via JS or some :has CSS wizardry. But let’s get an Index out of your project first and don’t think about “you are here” until you are done. 