Address book not visible to registered users
Ok, after checking a bit the commerce module, it looks like this is tied to profile view permissions and also addresses are linked to customer profiles.
I made it work by enabling the following Profile permissions under /admin/people/permissions:

Customer: Create profiles (enables to create new addresses)
Customer: Delete own profiles (enables to delete addresses)
Customer: Update own profiles
Customer: View own profiles (enables the address-book tab for registered customers)
View own profiles (just in case)