Test Binary Classifier on a Test-set that includes only one class I'm working on a disease binary classification problem.
0 = healthy , 1 = not healthy
The disease is a movement disorder that appears on the patient while moving a specific movement.
I applied leave-one-out cross-validation to train on all patients except one to test on and so on.
The problem is, some patients don't show class not healthy at all in their datasets. Thus, the evaluation metrics f1-score, precision &amp; recall went down sharply because of those patients.
I tried to use SMOTE oversampling, but it didn't work because it will generate new samples from the data of other patients. Additionally, I tried class-weights, but it didn't work as well, because there is not a minor class in the test set to give it a higher priority.
How can I solve this problem?
Give cross-validation or bootstrapping a shot!  Also, for metrics, look at per class performance and macro averaged.