Control access to node by entity reference on User record
The Node Access User Reference module is probably what you want. From the module page:

Gives content access permissions to users for content that references the users with User reference or Entity reference.
This is great if you want your content authors to be able to choose additional authors, or choose who can view their content, as well as being able to display the list of authors or viewers within your content, or in views displays.