form_set_error not working in node form validation
If you check the source of form_set_error(), you will see that if you don't supply the name of the form element for which you are setting the error, error messages will not be set.

As long as the first argument is NULL, form will be submitted. Your code was being executed, but it didn't stop the submission.

What you need to do is to pass the name of the field for which you want the error to show. This goes as the first argument for form_set_error().

Example:

form_set_error('YOUR_FIELD_NAME', t('Your message'));