How to validate fields with the Rules module?
It sounds like you don't have the supporting module Rules Forms: https://www.drupal.org/project/rules_forms

This module adds form validation for existing (core) forms and custom forms from contrib modules.

The documentation states that Rules Forms can:

Validate the values of form elements during form validation.
Set form errors when rule-based validation fails.