Area Under Curve with probability
In your code the Area Under the Curve (AUC) is used to calculate the area under the Cumulative Distribution Function (CDF). Let's go through the code to see how this is done. However, in the last section, I do not agree with the two lines used to calculate the area. If all the labels have a ground truth label of 1, and we have a perfect classifier where the probability of being Class 1 is always 100%, we should have an AUC of 1. However, we will get 50%. I will show this later. You should note that the way this code is written will only work for a binary classifier, with labels 0 and 1. 

We will first calculate the probability of each instance being in class 0 and class 1 by

probs = forest.predict_proba(X_train)


Then we will extract the second column, this contains the probability of each instance being in Class 1.

probas = probs[:,list(forest.classes_).index(1)]


Then we will join the probabilities of each instance with its ground truth label. Then we will sort the groupings based on their probability. The instances with a probability of 1 will be at the start of the list and those with a probability of 0 will be at the end of the list. This is done using

joint = zip(probas, truth)
joint = sorted(list(joint), key=lambda x:x[0], reverse=True)


Then we will separate the probabilities and the ground truth labels into separate lists using

probas = [x[0] for x in joint]
truth = [x[1] for x in joint]


The blue line is the probabilities and the ground truth labels is the orange line.





This is where the problems start. The next two lines do not include the probabilities but only the ground truth labels.

truth_cumulative = np.cumsum(truth)
truth_cumulative = truth_cumulative / np.sum(truth)


This means that if we had all class 1, with probability 1. We would get an AUC of 0.5. This does not make any sense. 

For example, if I force it by doing this

probas[100:100000] = 1
truth[100:100000] = 1

There will only be a few 0 values left. Then after the ordering and cumulative sum we get the plot as follow, which is obviously wrong.





We need to do exactly what is written in the comment but not done through code. We will get the false positive rates for each false negative rate. Much like what is described below in the Receiver Operator Curve (ROC).

This can be done with 

# Training Set
probas_train = forest.predict_proba(X_train)
fpr_t, tpr_t, thresholds_t = roc_curve(y_train, probas_train[:, 1])

# Testing Set
probas_ = forest.predict_proba(X_test)
fpr, tpr, thresholds = roc_curve(y_test, probas_[:, 1])
roc_auc = auc(fpr, tpr)

train, = plt.plot(fpr_t, tpr_t, label = 'Train')
test, = plt.plot(fpr, tpr, label = 'Test')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend()
plt.show()

print('Area Under the Curve (AUC): ', roc_auc)



  Training Set  Area Under the Curve (AUC):  0.998140359403  Testing Set 
  Area Under the Curve (AUC):  0.937729144675






Receiver Operator Curve (ROC)

This was invented during WWII as a means to detect aircraft using radar. For example, if we have a sensor which requires some kind of threshold to detect planes. We can determine the true positive rate (TPR) and false positive rate (FPR) which results in our experiment. We will then plot these as the x-axis as the false positive rate, and the y-axis as the true positive rate.

$TPR = \frac{\text{True Positives}}{\text{Positives}}$

$FPR = \frac{\text{False Positives}}{\text{Negatives}}$

If the sensor is perfect we will always have $0$ false positives and always a $100\%$ true positives. This results in a curve which looks like the blue curve. If the sensor was completely random and garbage, then you would have a random guess which is the diagonal line.



A good classifier will be as close to the blue line as possible. Note, that a line below the diagonal is also good, you should just invert your decision. You have a consistent classifier but you are inverting the decision. 

Intuitively, consider a sensor which reads a value and it is above a threshold $\theta$ we call it a detection. If we lower the threshold of detection, we should expect more detection, however if we end up with an equal amount of true positives and false negatives, then we are going along the diagonal line. This is a bad classifier.



Here we have 2 sensors, red and yellow, we want to decide which sensor is better. Just looking at the curves is not very effective because it's hard to see which one is definitely higher. So we can devise a metric named the Area Under the Curve (AUC). This is exactly what it means in the name. Then we an say the curve with the higher AUC is the better one.