Probability vs recall for time series classification task
[edited]

The Area Under the Curve (AUC) is a good fit for this kind of problem, but it's not exactly a standard case. The ROC curve is built by ranking your instances by their predicted probability of fault, for example:

p(faulty)   gold
0.03        0
0.14        1
0.19        0
0.23        0
0.31        1
0.32        0
0.65        1
0.78        1
0.83        0
0.90        1


Each level of probability corresponds to a possible threshold for the binary classification problem. The number of True/False Positive/Negative can be calculated for every possible level of threshold:

p(faulty)   gold TP TN FP FN
                 5  0  5  0
0.03        0    5  1  4  0
0.14        1    4  1  4  1   
0.19        0    4  2  3  1 
0.23        0    4  3  2  1
0.31        1    3  3  2  2
0.32        0    3  4  1  2
0.65        1    2  4  1  3
0.78        1    1  4  1  4
0.83        0    1  5  0  4
0.90        1    0  5  0  5


At this stage one obtains exactly the data required for a standard ROC curve: from these values precision and recall scores can be computed for every level and plotted on the ROC curve. The AUC is used in the usual way. 

This method takes into account the continuity of the probabilities while matching predictions against binary classes. An example of problem where this evaluation method is used: https://pan.webis.de/clef15/pan15-web/author-identification.html