Vectorising orthogonal-triangular decomposition for 3D matrix
Your code looks fairly good, but there are a few things I would do differently:


It's very confusing that you call B_rep for B, and call B the mean of B_rep. The comment and code here looks very strange. You should call B_mean = mean(B_rep, 1) to stick with the general naming convention in your code.

B_meanB=bsxfun(@minus,B_rep,B); % B minus mean values of B

bsxfun performs better than repmat, so instead of B_rep=repmat(matrice2,[1 1 n2]); you can do:

B_rep = bsxfun(@times, matrice2, ones(1,1,n2)); 

Instead of ', you should use .' when transposing an array. The first one is the complex conjugated transpose.  
i and j are bad variable names in Matlab.
You should try to use more spaces. It makes the code much easier to read.




I don't have the statistical toolbox, so I can't test your code myself. I'm not sure how it can be vectorized, so I can't help with much when it comes performance gain I'm afraid.