Users chosen from a field in a content type see only their result
Views module doesn't control content access
Views module respects the entities permissions set elsewhere in Drupal.
There are Permissions option available in Views but that only controls that particular Views Display global visibility, it doesn't control the nodes/users visibility. You can set Views to be visible by a certain user role or some other permission, but it will only apply to that Views page/block, it will not apply to nodes that are listed in that Views results! Even if a user is restricted from viewing that Views, they will still be able to see the nodes listed in the Views if they visit the nodes through the individual nodes links.
To properly restrict access to content on your Drupal site you need to restrict it on the node level, not Views level. Once you have it sorted out on node level, Views will respect it, check access for each node and show only the result rows that the user has access to view. There is a way to turn of node access check in Views (under Advanced options) but that's not recommended and it's rarely used.
Use Content access modules, Views will respect them
In your case you want to restrict access in a very granular way, to show content to only the user that is referenced from individual content item. You can do that with the help of additional module(s), you should research Access modules on drupal.org that relate to entity references.
Some of the ones I can suggest to test:

Access by Reference
Permissions by field