discrete cosine transformation (dct-II) own implementation with loops
You are missing a $\sqrt{\frac{2}{N}}$ scaling factor. Change the following line:

tmp(k+1,n+1) = sqrt(2/N)*cos((n+0.5)*k*pi/N);


You can also simply use Matlab's built-in function dctmtx.

C = dctmtx(n)