How to restrict node access to nodes based on field value?
I think the Node Access User Reference module will do exactly what you want.  It allows you to restrict access to a node based upon a user reference field in that node. It even comes in Drupal 6 and 7 flavors.

From what I can tell though, you want to make sure you use only one node access module for your system, so you may want to look at the Node Access Module Overview to make sure this is the best bet.