Model selection: large mean and variance vs small mean and variance This question was always in my mind. Imagine you are doing 5-10 fold cross validation and one model gives you mean accuracy of 0.8, but with 0.2 standard deviation and the other one gives 0.7 with 0.05 standard deviation. Which one is better?
This is exactly a question I asked Sebastian Raschka (author of the great Python Machine Learning book), here you can find his answer saying &quot;I also recommend the 1-standard error method, which basically means that you select the best model from k-fold based on pure performance, and then you select the simplest model that is within 1 standard error of that model&quot;.
The more extended explanation of his answer can be found on his github link.