Training, cross validation and testing accuracy (RMSE and R2) differs when using different shuffles and splits I have a very small data set of 60 observations. My training, cross-validation and testing accuracy (RMSE and R-squared) differ in a considerable amount when using different random states while performing shuffling and then splitting. The training, testing and cross validation accuracy changes each time a different random state is used. How can I solve such an issue and how to really assess the performance of the model?
In terms of evaluation, the best you can do with a very small amount of data is repeating $k$-fold cross-validation many times (i.e. very large $k$), and consider the whole distribution of scores as the performance (in particular take into account the variance across folds).

It's going to be difficult anyway to obtain a reliable measure of performance with such a small dataset. Two options come to mind:


obtain more instances, possibly by using some interpolation method to generate artificial data (but it's not as good as real data).
make the model less complex by reducing the number of features, as this is likely to reduce the variance in the performance.