title: Binary classifier peaks around 0.5 for signal, still gives decent separation
body: <p>I'm using XGBoost for binary classification. I plot the classifier output for being class A. It peaks on zero for class B, but it always peaks around 0.5 for class A.</p>
  <p>Still, it gives decent -but not stellar- separation, as evidenced by the ROC curve.</p>
  <p>Is this behaviour normal or expected in some way? Or does not peaking on 1 mean that something is definitely wrong with the model?</p>
  <p><strong>Edit:</strong></p>
  <p>Following Jon's tip in the comments, I found out that when I implement balancing of the two unbalanced classes (with scale_pos_weight), the classifier output looks more proper indeed, with the peak moving from 0.5 to 1. <em>However</em> this happens only when I run without folds.</p>
  <p>When I run with folds, I <a href="https://machinelearningmastery.com/cross-validation-for-imbalanced-classification/" rel="nofollow noreferrer">read</a> that instead of balancing one should use stratification. I tried this out by using <code>StratifiedKFold</code> instead of <code>KFold</code>. Unfortunately this doesn't change the situation.</p>
  <p>Please note that my original question still holds, both in the folds and no folds case: Even when the peak moves to 1, was having it at 0.5 wrong?</p>