How to eliminate access to page based on the user ID?
You will probably need to add a custom hook_node_access code.  In this you can use the view operation to check:


Is the current user the owner (yes - done - allow access)
Is the user an admin (yes - done - allow access)
Does the node have your user access list field (No -done - allow access)
Is the current user's uid in the field values? (Yes - allow access / No -deny access)


This should handle the basic access issues.  You may need to look at how to hide / show links based on access as well.