Metrics to determine K in K-cross fold validation Consider a scenario where the dataset in hand is quite large, let's assume 50000 samples (quite well balanced between two classes).  What metrics can be used to decide the K value in a K-fold cross-validation? In other words, can a 5-fold CV be enough or should I go for a 10-fold CV? 

The rule of thumb is the higher K, the better. But, putting aside the computational costs, what can be used to decide the value of K? Should we look at the overall performance, e.g. average accuracy? That is, if accuracy (5CV) ~ accuracy(10CV), we can opt for 5-fold CV?. Is the standard deviation between the performance of different folds important? That is, the lower the better?
The rule of thumb is the higher K, the better.


I think a better rule of thumb is: The larger your dataset, the less important is $k$.

However, it is useful to have a general understanding of the impact of $k$ on the performance estimator (leaving aside computational costs):


Increasing $k$ decreases the bias because the training set better represents the data
Increasing $k$ increases the variance of the estimator because the training data sets are becoming more similar


Also note that there is no unbiased estimator for the variance of the $k$-fold CV. Together this means that there is no metric that can tell you the best $k$ if you leave computational costs aside. Some empirical studies suggest that 10 is a reasonable default. 

And to be clear, $k$ is not a hyper-parameter you want to tune to find the best accuracy. If you start performing $k_2$-fold CV to find the best $k_1$ something hopefully feels wrong.