Real world performance evaluation on problem similar to malware detection I have a question on how to evaluate the performance of models similar to malware detection in the real world. For example, I have created a malware detection model to predict whether the engine will be down. And the model seems to do a decent job so I push it to production. However, in real-world, when the model predicts the engine is going to be down, I will send someone to fix it. The tricky part is I am doing the prevention by sending someone to fix the engine before it has been down. Are there any good methods to evaluate the performance of the model? Thanks!
Can you measure a proxy measurement that will be measurable even if the system never goes down? For example, does the engineer find “malware” or whatever the analogue is in this situation?

If not, and if it is safe to do so, why not simply take no action in X% of cases and use the data gathered in those cases as evaluation data?