k-fold cross-validation: model selection or variation in models when using k-fold cross validation
There are two main (set of) things your model needs to learn from training: Parameters and Hyperparameters.

Parameters are those that can be inferred (learnt) from data. These are internal to the model and will be calculated in the training phase during. The hyperparameters however, cannot be learnt from data and need to be decided (make assumptions) before training. As you come from statistics, an example could be modelling a distribution from a set of samples. Parameters could be mean and std, and hyperparameter could be selecting a Normal distribution or a Beta distribution.

If you don't have hyperparameters (e.g. simple regression model), you only need to train the model and let the algorithm find the optimal weights (parameters) using the data. However, if you want to use let's say Support Vector Machines, then you need to provide some hyperparameters such as the cost, kernel, parameters of the kernel, and so on... before starting the training.

Cross Validation can be used to find the optimal hyperparameters. It is done by creating a grid of potential hyperparameter values and training each combination of them against the whole data, performing k trainings of k-1 portions (or folds) of data. For each specific combination of hyperparameters the model is then trained and evaluated k times, getting a number of predictions equal to the number of folds (k) for each hyperparameter combination. The combination that yields better results based on the specified metric, is considered the best option (final model).

Have a look at this nice blog for further clarification.

Cross validation procedure: