Hyperparameter Optimization for a Machine Learning Algorithm I have a question regarding Hyperparameter Optimization for a Machine Learning Algorithm.

I try to fit a Support Vector Classifier and use Hyperparameter-Tuning (but it could be also another classifier).

My classes are highly imbalanced (20% of one class, lets call it “red” and 80% of the other, lets call it “black”).

Now, the objective of my Hyperparameter-Optimization is the cross-validation loss.

If, say, 20% of the observations are “reds” and 80% are “black”, then a really bad classifier would just label every case as “black” and achieve on average a cross-validation loss of 20%.

Now my question: If I see that the objective of my Hyperparamer-Optimization does not fall significantly below 20%, I could infer that it is useless without further analyzing. Correct? Or is there something I do not understand correctly?

This happens in all optimizations I am running now. I could conclude that my features are not informative.

I guess I have to change the objective for the hyperparameter optimization routine? (If my package allows for that)
You are right, what has to be changed is the objective. You are currently using accuracy as a measure of how good your classifier is. Accuracy is not a good measure when you have class imbalance. For this reason, other objectives have to be used. AUC is a better measure, and so is the log-loss, although not that interpretable. However, both AUC and log-loss need you method to output probabilities, not just predictions of classes. For this reason, it is required that your SVM implementation has a predict probability method.