title: scikit-learn train model on two classes for binary classification and test on more than 2 classes with cross validation
body: <p>I have a dataset with samples of 4 different classes. This is for a binary classification problem so the output need to be binary so at the moment I only keep 2 classes in my dataset. However I also would like to test the model on the other classes I didn't use for training all while using cross validation.</p>
  <pre><code>scoring = {
      &quot;balanced_accuracy_score&quot;: make_scorer(balanced_accuracy_score),
      &quot;precision_score0&quot;: make_scorer(precision_score, average=None, labels=[class_healthy]),
      &quot;precision_score1&quot;: make_scorer(precision_score, average=None, labels=[class_unhealthy])
  }
  
  cross_validation_method = RepeatedKFold(
      n_splits=n_splits, n_repeats=n_repeats, random_state=0
  )
  
  scores = cross_validate(
      SVC(kernel=&quot;linear&quot;, probability=True, class_weight=&quot;balanced&quot;),
      X.copy(),
      y.copy(),
      cv=cross_validation_method,
      scoring=scoring,
      n_jobs=-1,
      return_estimator=True,
  )
  </code></pre>
  <p>This works if I make sure that I only keep 2 labels in my training data(X, y) and returns the cross validated metrics I need for both classes. Now if I keep the 4 original labels in my input data, is it possible to specify 2 labels to use for training and return the cross validated metrics for all the labels including the one used for training?</p>
  <p>EDIT: Some clarification on what I am trying to do, I have my model trained on 2 classes (healthy and unhealthy) and it predicts those two outcomes. I want to see how my model predicts the other samples even though I know it is not trained on them this is for 'exploratory' purposes, my output will always be healthy or unhealthy by design whish is fine, so once my binary model is fitted I serialize and export the model in a file and just predict the other samples separately and I look at the prediction outcomes. This works fine but what I wish to do, is to perhaps tag the 'extra' samples during the cross validation an only train on healthy/unhealthy target.</p>