Node Access - Change ownership of node before loading
One possible solution might be to add a User Reference field to the node and add the Users you want to grant access to. Then in hook_node_access, check for that field, and return the proper access for view/edit/delete. Don't forget the value of the author field too, to preserve the original functionality. Basically, if the current UID is in the list of referenced users, return TRUE.

I feel like I have done this in the past in 7, but can't locate any code examples at the moment.