How do I add the "error" class when a form validation fails?
First try removing your if test and see if you can get the form_set_error to trigger. That way you can make sure your test is correct. You should be able to call form_set_error in a validation function and it will work. 

If that isn't the problem then make sure the name of your field is correct.

To answer the question about simply adding a class in form validation you can use the following:
$form_state['complete form']['aux']['#attributes']['class'][] = 'error';

It won't add to the input itself but to the field wrapper and you can style it from there. But I think figuring out why form_set_error isn't behaving how you want is your best bet. That should really be adding the error class.