I have $k$ Machine Learning models, trained on the same training set, and I want to test the hypothesis that their performance on a fresh test set is the same. See EDIT 1 below for what I mean with performance. I'm looking for frequentist answers. Also, I'd prefer an answer that refers to a paper or preprint on this topic (testing the null that $k$ ML models have the same performance on a finite-sample test set). The models are Computer Vision models.

1. Is there a **distribution-free, exact** test that allows me to test the above hypothesis?
2. If there's no such test, is there at least a **distribution-free, approximate** test?
3. As a worst-case scenario, I could be contented with a parametric, asymptotic test. But it seems to me a terrible option: this is ML, so we know nothing about the models, and we know nothing about the test set data distribution (apart from the fact that it's an iid sample). Yeah, I know "if the size of the test set $n\_{test}\to\infty$ we can assume that...". Point is, $n\_{test}$ isn't going anywhere. Acquiring and labelling the test set was very expensive, and I'm not getting new labeled data anytime soon. Thus, I'd rather get an answer that doesn't rely on the size of the test set being very large.

**EDIT 1**: the models are Computer Vision models, meaning that the test set $S=\{(X\_i,Y\_i)\}\_{i=1}^{n\_{test}}$ is made of couples (image, structured label). The problem is that, for each image $X\_i$, the corresponding label $Y\_i$ is not just a categorical variable (we're not doing image classification here), but it's a list of $n\_i$ categorical variables (the classes of each of the $n\_i$ objects presents in image $X\_i$) and $4n\_i$ real numbers (the coordinates of the bounding boxes of each object: see [here](https://www.v7labs.com/blog/object-detection-guide) for details). Thus, it's not so simple to say if, for image $X\_i$, the prediction $f(X\_i)$ of the model is right (1) or wrong (0). I naively thought of solving this using some aggregate metric, since for each of the $k$ models, we can compute a loss over the whole test set, which is a nonnegative number. However, as correctly noted by Jacques Wainer, this approach doesn't work for hypothesis testing. Let's consider then a different setting, where each of the $k$ models (each of the $k$ treatments, in statistical parlance) returns a real number in $[0,1]$ for each of the $n\_{test}$ elements of the test set. Thus, *for a single metric*, our dataset is a matrix $M$ of $k\times n\_{test}$ numbers in $[0,1]$. See EDIT 3 for the multiple metrics case.

**EDIT 2**: there's something that I think was overlooked in comments & answers. I think the $k\times n\_{test}$ entries of $M$ are not iid. All models are trained on the same training set, and are tested on the same images, thus I would expect the model results for the a given image $f\_1(X\_i),\dots,f\_k(X\_i)$ to be highly correlated.

**EDIT 3** (hopefully the final one): let's consider now more than one metric for each image (the multiple hypotheses case I mentioned in earlier versions of this question). For $d$ metrics, I'd get $d$ matrices $M\_1,\dots, M\_d$ of size $k\times {n\_{test}}$. As duly noted by Christian Hennig, different metrics for the same image are likely to be correlated (see also EDIT 2 above), and thing soon get very complicated. So let's forget about multiple hypothesis testing. I'll accept Christian Hennig's suggestion:

> 
> decide in advance to use only one test as "major test of interest", interpret that one, and give only a rough "exploratory" interpretation of all the other tests in relation to the major test
> 
> 
>