Allow access for anonymous to custom user view mode
As you already guessed the permission is checked before building the content, so you will never be able to display any user data without that permission. (At lest not without some programming.)

You could try to grant the View user profiles permission to all user and then use the Field Permissions module and grant access per field. So you can show anonymous users only the fields they should have access to.

UPDATE:

As you have pointed out the above solution introduces some problems with privacy and some additional fields not being hidden on the user profile.

So another suggestion; Remove the Views access check. Views by default bypasses access checks on content (so Views without access checks are considered unsafe). Which might be of benefit to you here. As an alternative you can set a more generic permissions such as View published content to prevent errors from modules like Security Review.



Then under display choose Display Suite and your custom view mode.



You will end up with a View that shows the custom View mode with all selected fields, even for anonymous users. Yet the user profile at example.com/user/1 shows access denied for anonymous users.