Could I make a ROC curve using TP vs. FP instead of TPR vs. FPR?
No, plotting TP against FP does not make sense, in your case it would make more sense to use the precision/recall curve. On one axis, you plot the precision (aka positive predictive value) and on the other axis you plot the recall (aka sensitivity).
The precision is given by TP/(TP+FP) and recall is given by TP/(TP+FN), where TP is true positives, FP is false positives and FN is false negatives.
For context, I'm currently working on a machine learning problem where there exists fairly large imbalances in our dataset, and we've been trying to find ways to work around our data imbalance to extract useful metrics/measures of performance. One metric we've really wanted to use is a ROC curve plot for each of our classes, but we've found that since a large portion of our data is negative in nature (meaning that most of the time, our images don't contain anything in them), the False Positive Rate is basically getting "squished" close to 0 since our number of True Negatives is so large, so our ROC plots look basically perfect when in reality they are being heavily skewed by our high TN count.
For example, a given class can have as many as 30,000 Negative cases, but only 2,000 or so positive cases, so 90+% accuracy can be achieved if the model were to simply always predict negatively. Our problem is also a multi-class problem, so it's not feasible to simply remove negative cases from the data since removing a negative case could be inadvertently removing a positive case for another class in a given image. We've done what data thinning/reduction we can by removing all images where there are no occurrences of any class in a given image, but there still exists a large data imbalance in our data after this thinning process.
In the meantime we've switched over to using precision-recall plots, and those have yielded more reasonable results since precision and recall don't depend on the True Negative count, but we've recently wanted to switch back to using ROC curves, and I'm trying to find a solution to address the skewed plots we were seeing prior.
One idea I had was simply plotting the #True Positives vs. #False Positives, and since doing this I've seen more reasonable looking curves. By doing this, I am able to avoid the issue of the False Positive Rate getting squished down due to the high TN count, and the plot essentially conveys the same information (the trade-off between True Positives and False Positives). What's wrong with this idea? Is there any other approach I can take to getting better results from my ROC plots?
One idea my colleague proposed was as follows:
" Do you think you can redo the ROC curves by removing most of these non-events from the statistics? Maybe keep about as much non-events as for the class with the highest number of events? "
I wasn't sure exactly what he meant, so I asked him for some clarification and he said this:
" You can still re-balance the data for your ROC analysis by randomly excluding most of the "nothing" events so that they do not dominate the statistics in your confusion matrix. Assuming your have enough random samples of the "nothing events" in your analysis, that smaller subset can be an accurate representation of the true "nothing" population "
I'm still not sure what exactly he means by this or how I am supposed to implement it, and I've asked him again for clarification, but in the mean time I was hoping I could get some more insight elsewhere.