Why isn't the custom validation handler fired in hook_form_alter()?
Validate and submit handlers can be placed on the form level or on specific buttons. If they are defined on a button, only those will be executed; it only falls back to the top-level if nothing is defined on the pressed button.

The node form, like all entity forms has the callbacks defined on the button, which is complicated by the fact that there are multiple buttons (save as published, save as unpublished and so on). You must define your validate callback on all of them.

Also, for validation of an entity, you should instead use validation constrains on the fields or the entity.