Technical Note: Towards ROC Curves in Cost Space
  ROC curves and cost curves are two popular ways of visualising classifier
performance, finding appropriate thresholds according to the operating
condition, and deriving useful aggregated measures such as the area under the
ROC curve (AUC) or the area under the optimal cost curve. In this note we
present some new findings and connections between ROC space and cost space, by
using the expected loss over a range of operating conditions. In particular, we
show that ROC curves can be transferred to cost space by means of a very
natural way of understanding how thresholds should be chosen, by selecting the
threshold such that the proportion of positive predictions equals the operating
condition (either in the form of cost proportion or skew). We call these new
curves {ROC Cost Curves}, and we demonstrate that the expected loss as measured
by the area under these curves is linearly related to AUC. This opens up a
series of new possibilities and clarifies the notion of cost curve and its
relation to ROC analysis. In addition, we show that for a classifier that
assigns the scores in an evenly-spaced way, these curves are equal to the Brier
Curves. As a result, this establishes the first clear connection between AUC
and the Brier score.
