MIMO with STBC example - How to derive SNR?
Maybe, you should check if all squares are correct in your post (in present form it makes no sense to me). The following will be consistently without squares, but you can add them everywhere. I will still be correct.

\$u\$ and \$v\$ are vectors. The expression \$\text{max}_{u}\frac{| u^H H u|}{|u|^2}\$ refers to the maximum value of the expression \$\frac{| u^H H u|}{|u|^2}\$ when arbitrary vectors \$u\$ are inserted. This is also known as the Rayleight quotient (see wikipedia).

Note, that \$\text{max}_{u,v}\frac{| u^H H v|}{|u||v|}\$ (with different vectors) is the same as the above, because the maximum of the inner term will be attained when \$u\$ and \$v\$ are equal (this argument needs, I think, that the channel matrix is physical, i.e. is positive definite).

Luckily, there is a simplification for the Rayleight-Quotient: It equals the largest eigenvalus of the matrix $H$. This is most likely the value \$\lambda\$ what occurs in the last term.

Sidenote: in MIMO often eigenvalues or singular values of the channel matrix occur.