How to decide to use K-fold validation or not?
The main criterion is that you need enough data in your training set to get a good model fit - which is both a function of the data quantity and the model complexity.

When you don't have much training data, you want as much data in your training set as possible - in this case, you can do "leave one out" cross-validation.  This is like k-fold, except it's N-fold, where N is the number of observations you have. This means you hold one observation out, build a model using everything else, validate using that one sample, and repeat N times.

When you have more data with a simple model, you can use a much more loose validation strategy. For example, if you have a million records and are doing linear regression with one predictor, you can get away with using a few thousand samples to build your model, since you'll get really good parameter estimates from thousands of samples (or fewer).

Those are the extremes. The suggestions you've seen live somewhere in-between.  When you pick high K, you have to build lots of models (this takes time), but each model is built with more data (good for complex models).  When you pick low K, you have to build fewer models, but each model is built with less data.

Personally, I pick a low value for K when I'm doing model exploration, but a higher value when I'm ready for validation of a good candidate model.  Actually, in this case you should have three sets - a training set, a validation set (that you use to find a good set of parameters), and a second validation set that you've held out the ENTIRE time (since, in a way, the other validation sets are helping you 'train' the model parameters).