How to give access to the files of node for individual users?
To do the access checking yourself you'll need to write a hook_file_download(). 

In broad strokes you'll need to get the actual file object (see this question for help with that), find the references with file_get_file_references(), and finally load them to check each reference (probably just one but it would be unwise to assume that's true) to see if the current user is given access to the file.