Mathematical expression of Multicarrier frequency equalizer
Hint: the convolution is not circular in that case as we used DCT instead of DFT.

Luckily, that's not true! At the receiver, you're doing a DFT, and you've prepared the transmit signal such that a convolution with the channel looks like a circular one by prepending the CP.
The fact that the data inside has something to do with the DCT is irrelevant there.

Then, the signal is recovered by multiplying by the DCT matrix

Yep, because that inverts $D'$.

I  think it's called frequency domain equalizer,

No, the frequency-domain equalizer is $\text{IFFT}(\text{FFT}/H)$.