Prevent anonymous to input a registered email during commerce checkout through Rules?
With @RyanSzrama's suggestion, I pursue a form validation solution. What I finally use is to implement a hook_commerce_checkout_pane_info_alter. The following is a simplified code snippet for that.

/**
 * Implements hook_commerce_checkout_pane_info_alter
 */
function MYMODULE_commerce_checkout_pane_info_alter(&amp;$checkout_panes) {
  if(isset($checkout_panes['account'])){
    $checkout_panes['account']['callbacks']['checkout_form_validate'] = 'MYMODULE_form_validate';
  }
}

function MYMODULE_form_validate($form, &amp;$form_state, $checkout_pane, $order) {
  if (!user_is_logged_in()) {
    $mail = $form['account']['login']['mail']['#value'];
    $query = sprintf("SELECT COUNT(*) FROM {users} WHERE mail = '%s';", $mail);
    if (db_query($query)-&gt;fetchField()) {
      $msg = t('The email address is registered to an account!');
      form_set_error('[account][login][mail]', $msg);
      return FALSE;
    }
  }
  return TRUE;
}


Hope that also provides some hint or help to others. For form validation, please refer to Adding validation to commerce pane form.