How reliable are model performance reportings?
The question of the measured test error of a classification model is reliable, hence if the test error on unknown set $T_1$ is the same as on $T_2$ is hard two answer. It depends on the following factors:


How many digits of the error are reported?
How many samples have $T_1$ and $T_2$? The more digits are reported, the more samples you need. As a rule of thumb, make sure that any change in the reported error means at least 3 samples have changed. So if you use accuracy and report two decimal places (e.g. 12.34%), then 0.01% must be bigger than 3 => $3 &lt; \frac{0.01}{100} \cdot |T_1| \Leftrightarrow 30000 &lt; |T_1|$
The distribution must be similar. The simplest part is the distribution of classes. The more difficult part is how the features look like.


For other forms of error analysis, you might want to look into my Master's thesis Analysis and Optimization of Convolutional Neural Network Architectures