Why is 10 considered the default value for k-fold cross-validation?
The statement:


  We most often use k=10 because evidence shows it's the best value for k. Smaller values don't give as good estimates, and larger values don't provide much better results either.


Is just categorically false. The reason people default to K=10 is because they don't know how changing K effects their estimates of the generalization error and they (like you) heard somewhere along the line that K=10 was good.

To understand what makes a good value of K (and whether or not K=10 is in fact better than say K=9 or K=11) you need to understand what changing this value has on your estimate. 

As K decrease the bias in your estimate increases. This is because with lower values of K you are training on less data. For example K = 2 trains on only half the data, thus you will have a pessimistic bias in your estimate since you've decreased the amount of data available for your model to learn from. K = 3 trains on two thirds of your data, more data available to train on, better performance.

It used to be thought that there was a bias/variance trade-off in that a decrease in K would cause a decrease in variance (to go along with your increased bias) and while this is partially true it does not always hold. Lower values of K will have lower variance due to the fact that your training sets are less correlated. Think of the extreme example where K = N (leave one out). All of the training sets will look extremely similar, meaning that the estimate you receive is highly dependent on the sample you have to train on. If you were to draw many samples from the population and estimate error using leave one out you would have large variance in your estimates because of the variance between samples. This was the original reasoning for believing there existed a bias/variance trade-off with the choice of K.

However, this post outlines that this is not the case and that there is no universal truth as to what happens to the variance as K increases or decreases. Some studies show the variance increases with K, some show it decreases with K.

https://stats.stackexchange.com/questions/61783/bias-and-variance-in-leave-one-out-vs-k-fold-cross-validation/357749#357749

The other thing to consider is computational complexity. If you are dealing with datasets with millions of records it may be infeasible to use a very large value of K, especially if you are doing nested and/or repeated cross validation. So many people make their choice of K based simply on time to compute.