My understanding is that with cross validation and model selection we try to address two things:

**P1**. Estimate the expected loss on the population when training with our sample

**P2**. Measure and report our uncertainty of this estimation (variance, confidence intervals, bias, etc.)

Standard practice seems to be to do repeated cross validation, since this reduces the variance of our estimator.

However, when it comes to reporting and analysis, my understanding is that **internal validation is better than external validation** because:

It is better to report:

* The statistics of our estimator, e.g. its confidence interval, variance, mean, etc. on the full sample (in this case the CV sample).

than reporting: 

* The loss of our estimator on a hold-out subset of the original sample, since:

(i) This would be a **single measurement** (***even if we pick our estimator with CV***)

(ii) Our estimator for this single measurement would have been trained on a set (e.g. the CV set) that is smaller than our initial sample since we have to make room for the hold-out set. This results in a **more biased** (pessimistic) estimation in **P1** .

Is this correct? If not why? 

### Background:

It is easy to find textbooks that recommend dividing your sample into two sets: 

* The **CV** set, which is subsequently and repeatedly divided into **train** and **validation** sets.
* The **hold-out** (test) set, only used at the end to report the estimator performance

My question is an attempt to understand the merits and advantages of this textbook approach, considering that our goal is to really address the problems **P1** and **P2** at the beginning of this post. It looks to me that reporting on the hold-out test set is **bad practice** since the analysis of the CV sample is more informative.

### Nested K-fold vs repeated K-fold:

One can in principle combine **hold-out** with regular **K-fold** to obtain **nested K-fold**. This would allow us to measure the variability of our estimator, but it looks to me that for the same number of total models trained (total # of folds) repeated K-fold would yield estimators that are **less biased** and **more accurate** than nested K-fold. To see this:

* Repeated K-fold uses a larger fraction of our total sample than nested K-fold for the same K (i.e. it leads to lower bias)
* 100 iterations would only give 10 measurements of our estimator in nested K-fold (K=10), but 100 measurements in K-fold (more measurements leads to lower variance in **P2**)

What's wrong with this reasoning?