DCT of 4x4 checkerboard matrix - what's the correct result? I'm trying to work out what the correct solution is for a DCT of a 4x4 checkerboard matrix (let's call it A) and for a matrix of ones (let's call it B). So A is:

   0   1   0   1
   1   0   1   0
   0   1   0   1
   1   0   1   0


And B is:

   1   1   1   1
   1   1   1   1
   1   1   1   1
   1   1   1   1


With Octave for dct(A) I get:

   1.00000   1.00000   1.00000   1.00000
  -0.38268   0.38268  -0.38268   0.38268
   0.00000   0.00000   0.00000   0.00000
  -0.92388   0.92388  -0.92388   0.92388


And for dctmtx(4) * A * transpose(dctmtx(4)) (which should be equivalent) I get:

   2.00000   0.00000  -0.00000  -0.00000
   0.00000  -0.29289  -0.00000  -0.70711
  -0.00000  -0.00000   0.00000  -0.00000
  -0.00000  -0.70711  -0.00000  -1.70711


And then dct(B):

   2   2   2   2
   0   0   0   0
   0   0   0   0
   0   0   0   0


And dctmtx(4) * B * transpose(dctmtx(4)): 

   4   0   0   0
   0   0   0   0
   0   0   0   0
   0   0   0   0


What's going on? Which is correct?
As far as I can see, there are two little mistakes. First of all, you should use dct2() instead of dct(). And second, you shouldn't use the 'transpose' function because you want the DCT matrix not only transposed but also conjugated. So you should do the following:
$$\tt{T=dctmtx(4); DCTB = T*B*T'}$$
And this should equal $\tt dct2(B)$.

EDIT: As correctly pointed out by Peter K., the DCT matrix is real-valued, so indeed the 'transpose' operation and the ' operation give the same result. So I only noticed ONE little mistake ...