Cross-validation for model comparison: use the same folds?
There seem to be a few elements of confusion here. I'll try to answer clearly, but would definitely appreciate input from others.

1. What is cross-validation for?

Cross-validation is generally used as a mechanism for figuring out which of several models might be a better fit for data based on a training set before choosing a model to apply to the test set. This is commonly done when tuning hyperparameters (such as penalty factors in regularized regressions), and in other scenarios as well.

In such a case you would want to use the same folds for cross-validating each model, because you want to identify differences in performance attributable to the models themselves, and not due to effects of randomly partitioning observations into different folds. Importantly, cross-validation is part of the model training process, not final model evaluation.

Cross-validation is not for finding the "best" performing model overall. Model performance is based on application to a held-out test set of observations. The idea is that the model performance will often be artificially high in the training data due to idiosyncrasies among the training data set which are not present in observations overall, and application to the test data suggests how the model might generalize to totally novel observations. Even if cross-validation results are discussed in a study, they are not the appropriate value to compare a model prepared independently and trained on different data.

2. During cross-validation, how would one assess which putative model is "better"?

This question can be a bit controversial, namely in that accuracy (such as of classification) is not a very strong method of identifying the best model during cross-validation. It will work with varying degrees of quality depending on what you're modelling. But better measures are varieties of loss functions, which could be mean squared error, binomial deviance, or many others (it's very situation dependent).

How you would compare the output of those loss functions is usually fairly straightforward-- you want minimum error, minimum deviance, etc. Performing a paired t-test isn't really appropriate. Such a test would tell you something about the distributions underlying your variables for data points that are connected in time (like measures of some variable before and after a treatment is applied) in the context of a specific null hypothesis and alternative hypothesis set. That sort of paired relationship doesn't apply to observations subjected to different models one-at-a-time, and in a mutually exclusive way.

3. What should be done instead?

As outlined above, model performance is assessed on novel data rather than training data. In model development that will be the test set, but it can just as well be application of the model to totally new data. If the models to be compared are similar in complexity, a proper loss function is still ideal. But in many situations classification accuracy is necessary (if that's what your boss is asking for, that may be what you have to provide).

In that case simpler counts of correct classification (9 out of 15 were correctly classified) are appropriate. Better still are measures of model sensitivity and specificity (the model's ability to identify true positives and true negatives, respectively, as in binomial classification) and positive/negative predictive value (how reliable any given classification suggested by the model is) can be very effective.



tl;dr: Cross-validation is the wrong stage to be comparing an already-trained model to a different model that is being trained, and t-tests are not informative in comparing cross-validations. Loss functions or classification accuracy are more appropriate (to varying degrees in varying situations) and should be based on data not used in training.