Show relationship summary using Entity Reference
First off, you only need an entity reference field on the content type Books that points to Author in your example. It would be a pain in the long run to manually maintain the two relationships and all data needed to describe your relationship is given in the Book->Author field.

Now let's create a View that lists books.



(I've removed the second column in the screenshot from Views as no settings were done there)

Add the entity reference relationship under Relationships.
Add Content: title and use your relationship.
Under Format and Settings, set group by to the Content: Title field you created in the step before.

Now all books are listed and grouped by their author.

In my example I also sort by the authors name. If you rather show the Nid of the content, just switch from Content: Title to Content: Nid.