What does test score provide in terms of real application?
Evaluation is a crucial part of any serious ML project, but when it comes to evaluation choices there is no perfect answer. Generally speaking one evaluates a system in order to know how well it performs (i.e. how reliable are its predictions), usually to know which level of quality to expect when used in production (but not only). However the evaluation results are only useful to some extent:


the evaluation method/measures should be chosen so that it actually represents "quality" for the target task. In general this is imperfect because no evaluation score can fully represent the diversity of a specific task (and that's assuming good evaluation choices)   
the test data should be representative of the production data, and in reality it rarely is from the same distribution. 
There's almost always a chance factor that one tries to eliminate using cross-validation or other methods, but again that's imperfect.


Besides these unavoidable simplifications, an evaluation score can be interpreted for exactly what it is. For example a precision score of 80% for class X means that one should expect an instance predicted X as truly of class X in 80% of the cases in average. Predicting a specific probability for a particular prediction requires either using a probabilistic model or devising a specific model which predicts a confidence score/probability... But in both cases the model could still can be wrong about the probability it predicts!