How can extract the cosine transform formula used for 2D by scipy.fft.dct
I'd take bets that the 2D-DCT is just separable into identical row-wise and column-wise DCTs. Anything else would be surprising.
Anyways, you know the 2D-DCT is a reversible linear operation: It maintains dimensionality, and has only a trivial null space. Therefore,  you can analyze what it does directly from applying it to an arbitrarily chosen basis of your matrix space.