Comparing models using P-value I have 2 different ML architectures for a translation task, I evaluate them using BLEU score (higher is better)

I've run them 9 times each, yielding the following scores

Architecture 1 | 36.52 |
36.27 |
35.9 |
35.22 |
37.13 |
35.53 |
35.3 |
34.14 |
35 

Architecture 2 | 36.85 |
35.64 |
36.37 |
36.82 |
36.74 |
36.46 |
35.77 |
37.31 |
36.68

Means are 35.67 for the first and 36.52 for the second

I want to calculate a P-value that confirms (or not) the superiority of the second model, with alpha = 0.05

Thanks in advance
I think you can just apply a t-test or Wilcoxon test if you're don't believe in the normality of your observations.

In R:

A &lt;- c(36.52 , 36.27 , 35.9 , 35.22 ,37.13 , 35.53 , 35.3 , 34.14 , 35)
B &lt;- c(36.85 , 35.64 , 36.37 , 36.82 , 36.74,36.46 , 35.77 , 37.31 , 36.68)
wilcox.test(A,B,alternative="less",paired=T)
t.test(A,B,alternative="less",paired=T)


I've done paired tests under the assumption that you did the same folds for each architecture. If that is false, then do unpaired tests.

Edit: You may be interested in a question I posted a few months ago. Cross-validation for model comparison: use the same folds?