Should I repeat the testing/training split while repeating K-fold cross validation? I'm fitting random forest regressions on my data, and using 10 K-fold cross-validation to evaluate model performance. While re-runing the cross-validation, I noticed that the results differed between each run, sometimes by a lot. So, I decided to repeat the cross-validation calculation 20 times, creating a for-loop, and then summarising the results afterwards. Just to illustrate, I'm doing something like this:
for (i in 1:20) {

trainIndex &lt;- createDataPartition(data$response, p = .8, list = FALSE, times = 1)
data_train &lt;- data[ trainIndex,]
data_test  &lt;- data[-trainIndex,]

train.control &lt;- trainControl(method = &quot;cv&quot;, 
                              number = 10)

model &lt;- train(formula, data = data_train, method = &quot;rf&quot;,
               trControl = train.control)


pred &lt;- predict(model, data_test)
t&lt;- postResample(pred = pred, obs = data_test$estimate)
t &lt;- t[[2]]

result[[i]] &lt;- t }

So, essentially, each re-run I am splitting my dataset into training and testing sets again. This results in a lot of variance, when ploting the resuls of each run together they vary from ~0.2 to 0.6 R.squared. If I don't do this within the loop (i.e. if I split into training/testing before the loop), then the results of the 20 runs are very similar.
Which way is the right way to go about this?
The purpose of k-fold cross validation is to fit the model multiple times and average out to results to estimate the model's predictive performance.
If having done 10-fold CV once and got one result and then repeating it again and getting a significantly different result, probably means the number of folds is too low, and the k-fold process isn't capturing the inherent variability in the data sufficiently well.
A better test is to change the number of folds and see what happens to the average and variance of the combined output.