How to handle highly Imabalanced classification? I have been dealing with a classification problem. Real issue is the imbalance here
I have ~500,000 -ve samples and ~300 +ve samples.End result is predicted probabilities NOT hard 0-1 classification  Personally I am not a big fan of oversampling/under-sampling since they mess the distribution up. I also tried stratified sampling so that same proportion is kept while training but its not working. For my present approach I take a sample of -ve points (say 100,000) and then train with 300 +ve samples and getting an ROC-AUC of ~85 but almost no case predicted reaches probability &gt;51% (after using model.predict_proba). Any tips on handling such an extreme imbalance? additionally I would only like to stay with tree based or boosting models since they provide interpretability (hence avoiding neural network, anomaly detection , autoencoder for now).
Any hints, resources appreciated!
Below is the code I am using.
df = read_data(path)
Pkl_Filename = path + &quot;/catboost.pkl&quot;
# if not os.path.exists(Pkl_Filename):
X = df.drop(ep_config['target_field'], axis=1)
y = df[ep_config['target_field']]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, stratify=y)
clf = CatBoostClassifier(eval_metric='Recall', use_best_model=True, scale_pos_weight=...)
sm = SMOTE(random_state=27, ratio=1.0)
X_train, y_train = sm.fit_sample(X_train, y_train)
smote = clf.fit(X_train, y_train, eval_set=(X_test, y_test))

# smote_pred = smote.predict(X_test)

# Checking accuracy
# accuracy_score(y_test, smote_pred)
# clf.fit(X_train, y_train, eval_set=(X_test, y_test))
with open(Pkl_Filename, 'wb') as file:
    pickle.dump(smote, file)
# else:
#     clf = pickle.loads(Pkl_Filename)
logging.info('the test roc is :{:.6f}'.format(roc_auc_score(y_test, smote.predict_proba(X_test)[:,1])))
have you already tried using only very little of the -ve cases? So for example to train your model on 900 points total, 600/300? Then stratified sampling should still work fine. Then I'd evaluate your model based on it's ability to predict -ve cases and just monitor the performance it get's on the (in your case) gigantic test dataset that the model hasn't seen before.
Also, you could generate &quot;synthetic&quot; +ve cases by bootstrapping from the real data to see how that influences the model. Similar to what is described here: https://sci2s.ugr.es/keel/pdf/specific/articulo/dupret_science.pdf