How to measure a detector's performance under extremely complex real-world environment? I am trying to formalize my understanding of how best to measure the performance of a binary detector. (See previous question here). I had looked into ROC curves for doing this however based on previous research and feedback here I think ROC curves are too simplistic. Here is the problem I face:

Context:

I have a microphone that is setup in a park. The mic takes in inputs and an algorithm runs in the background that is supposed to detect the rustle of leaves. (Animal approaching, something in the bushes, etc). It does this by insert time-frequency-detection-scheme here, but that is not the issue. The end result of course is a 'Yes' or 'No', (So a binary detector), indicating whether or not leaves are indeed rustling. 

Main Question(s): 


How can/should I characterize this detector's performance, vis-a-vis true positive and false positive rates? (If that is the wrong way to go about it, then what is the right way?)
I understand that to generate a false-positive analysis, requires a false-positive model. In this case, the false-positive model is not something simple like white noise, but spans almost any signal possibility. Therefore, how can we model all those possibilities?


Thoughts:

My current solution/thought is to measure a true-positive rate and false-positive rate to characterize detector performance. (For given settings/thresholds).

Thus, for given and fixed detector settings, I test its true-positive rate by rustling leaves (within a specified range), and see if it detects the incidents. This will give me a true-positive rate. (A consistent 'Yes').

However I am stumped as to how to measure the false-positive rate. 

I do not understand how to do this, because the mic is outside and subject to anything - cars, people, animals, thunder storms, just life. In other words, just about any combination or permutation of signals imaginable is possible. How then does one test its false positive rate?
on an ROC curve your plotting FP(x-axis) vs TP(y-axis) They are calculated by:  

True-positives = # of correctly detected positives/# of actual positives 

False-Positives = # of positives that are not true-positives/ number of positives 

Now on your ROC curve, because you did not mention incuding this into your question above, you most likely did not include the "coin-flip" line. That is the line y = x (from x = 0-1)

This line represents how well your algorithm performs by just guessing randomly (Yes, No) 

If your algorithm is a decent detector, the ROC curve will it will be above the coin-flip line, if it is below this line it is better at being an anti-detector, i.e., when there is no event and would be a poor detector for your application. 

I've worked extensively on this for my master's thesis on onset detection. Below is an example of ROC curves plotted against the coin-flip line.