Content access based on on the field reference
There are plenty of node access modules available for Drupal but I honestly am not sure if there is one that would do exactly what you are looking for.

Either way, I would probably avoid a specific node access module for this and use Panels - read the Panels module - to do this since its makes this scenario easy to accomplish.

You can easily override Drupal's standard Node template and create an invariant that matches both the content/node type you want to apply this rule on and add a condition that the logged in user's ID matches the user id in the reference field defined by the node.