Cross Validation - Why does more folds increase variation?
This has been a discussion for quite a while. For a more theoretical point of view you can find a good summary here. 

From a practical point of view I'd look at it as follows. With increasing $k$ two things happen:


Your $k-1$ training folds increase in size
Your validation folds decreases in size


From the first point you can draw the conclusion that your $k$ models become more similar since your training data becomes more similar since you splitt off less data for the validation sets in the $k$-th fold. Which might lead to less between model-variance. 

Moreover, since you are training on more data, the relative model complexity (i.e. compared to your data) decreases. In the bias-variance-trade-off graph (taken from [1])



this means we are moving towards the left, i.e. we trade less model variance for more model bias (please note that model variance here has a more general and conceptual meaning than the calculated variance or standard deviation between folds). The reason is that we are fitting a model with constant complexity to more data as $k$ increases, i.e. it becomes harder to learn the training data. 

However, we are not only increasing the size of our training set with increasing $k$. We are also decreasing the validation set in size (see point two from above). Therefore, there might be higher between-fold variance with regards to our validation sets. This might be more relevant if the overall dataset carries higher variance or more outliers.  



[1] "The Elements of Statistical Learning" by Hastie et al