The only way we can assist is if you share your site project, so we can see your templates and config. Please review Requesting Help and post your code! 
Based on your file tree, I’d expect /layouts/books/rss.xml to be picked up first in the section RSS template lookup order, so nothing pops out to me.