Calculating incoherency in compressive sensing
As a starting-point, you might try something like this:

Step 1: create random (complex-valued) matrices

rows = 32;   %number of output samples from compression matrix 
cols = 2048;  %number of input samples supplied to compression matrix
matrixA = randn(rows, cols) + 1i*randn(rows, cols); 
matrixB = randn(rows, cols)+ 1i*randn(rows, cols);


Step 2: Compute Maximum Dot-Product Magnitude

%compute all possible cross-correlations
xc = abs(matrixA * matrixB');
%find the maximum
result = max(max(abs(matrixA * matrixB')));
%todo:  scale by sqrt(n)
%...


Step 3: Beware of Amplitude Scaling Impacts

Since the coherence equation that you provided depends upon dot-products, the amplitude scaling of your matrix will effect the result.  Most likely the equation you provided assumes the matrices are scaled in amplitude.  For example, each column in matrixA &amp; matrixB might need to be scaled by it's "Euclidean Norm" or "L2-Norm".  You might find it helpful to read about correlation coefficients which are scaled accordingly, or revisit your equation reference just to be safe.