Views to show nodes filtered by referenced user
What you want to do is often called displaying sibling items. In your case the connection is the ER user connection (not Taxonomy) but you can find examples with details about how to do it, search answers for it.
In short:

Create Views of type Book, you'll probably want Block Display, it doesn't have to be Entity reference Display type (that is for influencing choices in nodes in ER fields).
Add the Relationship for the User ER field. It should be the forwards Relationship. You can require it if you want to exclude all Books from your list that don't have the User ER filled in.
Add the second Relationship for that field, but the reverse one this time (it will be called different). Make it use the first Relationship. You can require it.
Add Contextual filter for Node ID, make it use the second Relationship (because this way you get the ID of sibling nodes) and configure it the usual way (default value ID from URL, validate Book node type) etc.
The current Book node will also be listed. To exclude it add another Contextual filter for node ID, don't make it use any Relationships, set to default value ID from URL and under &quot;More&quot; choose &quot;Exclude&quot;.
Under Structure &gt; Blocks set the Block to display in a theme region, and configure it to use Contextual options. Don't forget the Visibility settings too.

You should now see the block on Book nodes showing their siblings per shared user ER.
You can test Contextual filter when editing Views by entering values into the Preview field. If you have two Contextual filters you have to enter two values separated by &quot;/&quot; In your case enter a book node twice e.g. &quot;12/12&quot;. &quot;all/all&quot; usually bypasses the Contextual filter results.
You still have to check that it works by placing the Block in a region at the end, to check your Contextual configuration is correct.