Coherence Calculation in Sparse Sensing
Update

I would like to rephrase the original problem with slightly different notation and claim that there is no need to measure the coherence of the product of the circulant matrix A and the DCT matrix B as asked by the OP. 



In the compressive sensing setting, the coherence of a sensing matrix $A$ is useful when we are using it for constructing measurements $y = A \alpha$ where $\alpha$ is a sparse vector. Then, the coherence properties of $A$ can be used to give an estimate of number of measurements required in $y$ so that $\alpha$ can be recovered stably from $y$. $\alpha$ is actually a sparse representation of some signal $x$ in some orthonormal basis $\Psi$ which is represented by the equation $x = \Psi \alpha$. In this particular case, $x$ is the image 32x32, $\Psi$ is the IDCT2 transform, $\alpha$ is the sparse representation of $x$ obtained by DCT2 transformation which is the equation $\alpha = \Psi^T x$. Thus, the matrix $B$ from OP is $\Psi^T$. So the relationship between the measurements $y$ and the original signal $x$ is
$y = A \Psi^T x$. The sparse recovery of $x$ from $y$ would happen in two steps.


Construct $\alpha$ from $y$ by using some algorithm $\Delta$. We represent this algorithmic recovery step as:  $\alpha = \Delta(A, y)$. $\Delta$ could be basis pursuit or OMP or anything else.
Construct $x$ from $\alpha$ by using the equation $x = \Psi \alpha$. 


In this overall recovery process, the matrix $A B = A \Psi^T$ plays no role. Hence, there is no need to measure the coherence of $AB$.   



The old answer was incorrect. The coherence of a matrix $A$ doesn't change if it is pre-multiplied by an orthogonal matrix $B$ as $BA$. Coherence of a matrix can change in arbitrary ways if it is post-multiplied by an orthogonal matrix as $AB$. I am not aware of any particular results of relating the coherence of $A$ with $AB$. 



Below is old answer.

You need to compute the coherence of the matrix A. B is an orthonormal matrix. So it doesn't impact coherence of the combined matrix AB .  For computing the coherence of A, normalize its columns and then take pairwise inner products of the columns. The coherence is the highest inner product in magnitude of the pairs of normalized columns of A.

For more information about coherence, see my tutorial notes here .

You may also want to learn about Babel function which is a generalization of coherence.