How to interpret the AUC score in this case? I just run a random forest model on a imbalance dataset. I got the set of AUC and the confusion matrix. The AUC seemed not bad but actually the model predict every instance as positive. So how it happened and how to use AUC properly?



The ROC Curve as below:



I plot out the predicted probability of positive class in test set. The probability was within a tight range (0-0.4).
If you want to figure it out how this ROC happens , you would better LIST the tuples including your &quot;predicted&quot; values and the &quot;truth&quot; values, and SORT with the &quot;predicted&quot; value , then PLOT the ROC .
In your case , the tuples and points should be like this :
  predicted truth (x,y)
    0.53 0  (6/6,14/14)
    0.55 0  (5/6,14/14)
    0.57 1  (4/6,14/14)
    0.59 0  (4/6,13/14)
    0.60 1  (3/6,13/14)
    0.62 1  (3/6,12/14)
    0.63 0  (3/6,11/14)
    0.64 0  (2/6,11/14)
    0.66 1  (1/6,11/14)
    0.68 1  (1/6,10/14)
    0.71 1  (1/6,9/14)
    0.73 1  (1/6,8/14)
    0.77 1  (1/6,7/14)
    0.78 0  (1/6,6/14)
    0.82 1  (0/6,6/14)
    0.86 1  (0/6,5/14)
    0.89 1  (0/6,4/14)
    0.92 1  (0/6,3/14)
    0.94 1  (0/6,2/14)
    0.96 1  (0/6,1/14)
            (0/6,0/14)

Put these points (x,y) on ROC picture , it should be like this :

Just much like that of yours !
By the way , if you want to know how these points are figured out , you can check the code written in scala in the below :
    def computeAuc(predict: BDV[Double],groundTruth: BDV[Double]): Double
    = {
    
        // retrieve number of positive and negative samples in ground truth
        val nPos = groundTruth.toArray.filter(_&gt;0).length
        val nNeg = groundTruth.toArray.filter(_&lt;=0).length
    
        // tuple predict with ground truth , and sort with predict
        val pair = predict.toArray.zip(groundTruth.toArray)
        val sortedPair = pair.sortBy(_._1)
        var auc = 0.0.toDouble
        val x = BDV.zeros[Double](predict.length + 1)
        val y = BDV.zeros[Double](predict.length + 1)
        x(0) = 1.0
        y(0) = 1.0
    
        // calculate auc incrementally
        var i = 1.toInt
        while(i &lt; sortedPair.length) {
          y(i) = (1.0 * sortedPair.slice(i,pair.length).filter(_._2 &gt; 0).length) / nPos
          x(i) = (1.0 * sortedPair.slice(i,pair.length).filter(_._2 &lt;= 0).length) / nNeg
          auc = auc + (((y(i) + y(i - 1))*(x(i - 1) - x(i)))/2.0)
          i += 1
        }
        auc = auc + ((y(i - 1) * x(i - 1))/2.0)
    
        auc
   }

Finally , your imbalance problem is severe , and you would better do down-sampling or up-sampling before training .
Hopes this contributes you , good luck -)