Tracking number of attempts via Form API
Just use a variable insider $form_state.

In validate function

if ($validation_failed) {
  $form_state['error_count']++
  $form_state['cache'] = TRUE;
}


Then, in your form function just check $form_state['error_count'].

$form_state will be persistent between page petitions while validation fails thanks to $form_state['cache'].

You can also check for errors using form_get_errors().