How stop HOOK node form submit
Stopping a submit is best done in a validation function. You need to have this code in a custom validation handler that you add to the form, and then you set a validation error on it. 

Learn how to add custom validation to an existing Drupal form

Redirecting on form validation fail seems to be a problem. Look here for workaround:

Redirect form in validation fail

As to hook_node_submit, the docs have this to say:


  Act on a node after validated form values have been copied to it.
  
  This hook is invoked when a node form is submitted with either the
  "Save" or "Preview" button, after form values have been copied to the
  form state's node object, but before the node is saved or previewed.
  It is a chance for modules to adjust the node's properties from what
  they are simply after a copy from $form_state['values']. This hook is
  intended for adjusting non-field-related properties. See
  hook_field_attach_submit() for customizing field-related properties.


This hook is not a good place to try and stop submission, as form validation has already passed. It is therefore assumed that the node should be saved to the db.