Using keras with sklearn: apply class_weight with cross_val_score I have a highly imbalanced dataset (± 5% positive instances), for which I am training binary classifiers. I am using nested 5-fold cross-validation with grid search for hyperparameter tuning. 

I want to avoid undersampling, so I have been looking into the class_weight hyperparameter. For sklearn's decisiontree classifier, this works really well and is easily given as a hyperparameter. However, this is not an option for sklearn's neural network (multi-layer perceptron) as far as I can tell. I have been using Keras instead and I can apply class_weight with gridsearchCV, but not with cross_val_score.

Is there a way to use class_weights in keras with cross-validation?
The sklearn MLPClassifier does not implement any option for class weights at the moment. There are at least two paths for you to follow. You could go with writing a custom loss function, which allows you to stay in the sklearn framework without reaching out to keras. Another option is to implement the cross-validation yourself, which is not difficult to do, and run your keras model for each fold. An example can be found here.