(3.3.11) Pernot : To answer the question of Trickey about the intercomparison of DFAs, a small difference in MAEs between two DFAs certainly cannot be relied upon without additional information. This difference might be an artefact of the limited size of a nonexhaustive reference data set. As a result, there may be a high probability of rank inversion when the data set is perturbed by adding or suppressing a few points. A set of tools has recently been proposed to address this problem -for example, by estimating a rank inversion probability, P inv , or by using statistics based on system-wise comparisons, such as the systematic improvement probability. (3.3.12) Pernot : Validation requires the comparison of calculated values with a set of high-quality reference data. The resulting errors are used to estimate validation statistics. An important fact to have in mind is that the distribution of errors has no reason to be normal, 343 essentially because the errors are dominated by systematic contributions from all the approximations involved in a calculation -level of theory, basis set quality, values of parameters, and so on. This is why statistics such as the MAE should not be used to estimate the confidence level of a DFA. There are more pertinent metrics such as Q 95 (the 95th quantile of the absolute error distribution), on which a level of confidence can be based. We then know that there is a 5% risk to get an absolute error above Q 95 . By comparison, the probability for absolute errors to be above the MAE has been observed to vary between 0.2 and 0.45. From a MAE value of let us say 0.5 kcal/mol, one cannot estimate the probability/risk for the errors to exceed 1 kcal/mol.