DCT of 4x4 checkerboard matrix - what's the correct result?
As far as I can see, there are two little mistakes. First of all, you should use dct2() instead of dct(). And second, you shouldn't use the 'transpose' function because you want the DCT matrix not only transposed but also conjugated. So you should do the following:
$$\tt{T=dctmtx(4); DCTB = T*B*T'}$$
And this should equal $\tt dct2(B)$.

EDIT: As correctly pointed out by Peter K., the DCT matrix is real-valued, so indeed the 'transpose' operation and the ' operation give the same result. So I only noticed ONE little mistake ...