What's the difference between using DFT, IDFT or DCT to calculate cepstrum of a power spectrum?
They are all somewhat different, in a family of potential candidates,  depending on the system you use and the features you are interested in.

Cepstra play on anagrams related to standard Fourier terms:  quefrency alanysis, liftering, cepstrum, saphe (for the phase)  etc. Not surprisingly, one can find some mixing in cepstrum definitions. You can think of it as a spectrum of a spectrum, scrambled.

The original cepstrum was a squared magnitude of the inverse Fourier transform of the logarithm of the squared magnitude of the Fourier transform of a signal (Bogert et al. 1963). But people have developed generalized versions (related to homomorphic systems). This is where the apparent inconsistency arises.

From a very mundane point of view, take a transformation $F$ (often orthogonal) that gives you some kind of spectrum. Separate (or not) a phase component ($\angle\left( F\left( x\right)\right)$, which could be the sign, some official phase or a mere constant) and an amplitude component $|F\left( x\right)|$. Put an exponent $p$ to the latter, add a little constant $k$ to it to avoid it from vanishing. Pour everything in a companding function $L$: the  $\log$, or even a Box-Cox transform: $(x^\lambda-1)/\lambda$. Now, transform the result  with $F^*$, which can either be $F$ or its inverse $F^{-1}$:

$$F^*\left(\angle\left( F\left( x\right)\right).L\left(k+ |F\left( x\right)|^p\right)\right)\,.$$

Now you can look at the absolute value of this, or a power again, or some inverse of the function $L$.

The key features are:


what is a meaningful frequency information for you or your system? If your system natively uses a DCT, then it is natural to compute a spectrum in a DCT domain. 
computations: indeed, a few orthogonal transforms have been used for cepstra: sine, cosine, or Hadamard, which can be quite fast.
features looked at: from basic to invertible.