What's the difference between using DFT, IDFT or DCT to calculate cepstrum of a power spectrum? I've seen different equations that calculate cepstrum from power spectrum, but the equations are not consistent. Some people use Fourier transform, some use the inverse Fourier transform, and some use the Cosine transform. I.e.:

$$ C(k) = IDFT(\log_{10}P(n)) \\
   C(k) = DFT(\log_{10}P(n)) \\
   C(k) = DCT(\log_{10}P(n)) \\
$$

For example to calculate LPCC (Linear Prediction Cepstral Coefficients), I see either IDFT and DFT being used equally commonly, while MFCC (Mel Frequency Cepstral Coefficients) uses DCT exclusively.

Are they different? If so what's the different? If not why is there an inconsistency?
They are all somewhat different, in a family of potential candidates,  depending on the system you use and the features you are interested in.

Cepstra play on anagrams related to standard Fourier terms:  quefrency alanysis, liftering, cepstrum, saphe (for the phase)  etc. Not surprisingly, one can find some mixing in cepstrum definitions. You can think of it as a spectrum of a spectrum, scrambled.

The original cepstrum was a squared magnitude of the inverse Fourier transform of the logarithm of the squared magnitude of the Fourier transform of a signal (Bogert et al. 1963). But people have developed generalized versions (related to homomorphic systems). This is where the apparent inconsistency arises.

From a very mundane point of view, take a transformation $F$ (often orthogonal) that gives you some kind of spectrum. Separate (or not) a phase component ($\angle\left( F\left( x\right)\right)$, which could be the sign, some official phase or a mere constant) and an amplitude component $|F\left( x\right)|$. Put an exponent $p$ to the latter, add a little constant $k$ to it to avoid it from vanishing. Pour everything in a companding function $L$: the  $\log$, or even a Box-Cox transform: $(x^\lambda-1)/\lambda$. Now, transform the result  with $F^*$, which can either be $F$ or its inverse $F^{-1}$:

$$F^*\left(\angle\left( F\left( x\right)\right).L\left(k+ |F\left( x\right)|^p\right)\right)\,.$$

Now you can look at the absolute value of this, or a power again, or some inverse of the function $L$.

The key features are:


what is a meaningful frequency information for you or your system? If your system natively uses a DCT, then it is natural to compute a spectrum in a DCT domain. 
computations: indeed, a few orthogonal transforms have been used for cepstra: sine, cosine, or Hadamard, which can be quite fast.
features looked at: from basic to invertible.