How to handle private file uploads and downloads?
If the whole node should only be visible to the author, then you should either implement hook_node_access() yourself and correctly deny view access or use a content access module (e.g. Content access) to do that for you. Note that hook_node_access() won't prevent that now to show up in listings, for that you need a content access module (or implement one on your own).

Alternatively, if you just want to hide the field, you could use the Field permission module and configure it accordingly.

Both ways should be able to to this because it is something that Drupal can check, it can't know that you prevent access in hook_node_view().