High probabilities of success for wrong predictions
It is difficult to diagnose without having more info. But based upon what you mention I point:

Since you have a class imbalance, you MUST use regularizarided models that penalize the cost function  made by the minority class, more strictly than those errors for the majority class (Boosting models, Linear models)

Try using another metric like ROC AUC or Balanced Accuracy to measure the performance of your model so that you can see an adequate point of view of how your model is performing.
An additional advantage of using ROC Score is that you can find the &quot;optimal&quot; threshold being the one that maximizes Area under the roc curve when you make predictions with your model

This is the most important part. Use a technique to debug your model like SHAP values,Partial dependence plot in order to understand how your model is learning and the relationships between the features and the target

If you are using ensemble methods, you can set something named Monotonicity constraints That will allow you to add business knowledge to the model in order to correct potential non-sense relationships that the model may find