404 error from node access hook
You can use drupal_goto("custom 404 page") to redirect a user to your custom page. For your second question you can check if user is anonymous using this condition. if($user-&gt;uid == 0)

Thus your hook can be rewritten like this. 

function MODULE_NAME_node_access($node, $op, $account) {
  global $user;
  if ($op == 'view' &amp;&amp; $node-&gt;status == 0 &amp;&amp; $user-&gt;uid == 0){
    drupal_goto("custom_404_page");
   }
}


See here for drupal_goto documentation 

Note: This function has been removed in Drupal 8. In Drupal 8 you can do this by this way. 

return new RedirectResponse(\Drupal::url('user.page'));


or this way 
`use Drupal\Core\Controller\ControllerBase;

class MyControllerClass extends ControllerBase {
   public function foo() {
   //...
   return $this-&gt;redirect('user.page');
  }
}`


See here for Drupal 8 redirect info