Mathematical expression of Multicarrier frequency equalizer I have a signal $X$ with length of $N$, multiplied with any unitary matrix, i.e  the transpose of DCT matrix as:
$x = D' X$
where $D'$ is the transpose of the of DCT matrix. Then let's add $CP$ guard interval into the signal $x$. The resultant signal after adding the cp into the head of the signal is $x_2$.
We transmit $x_2$ through a channel $h$ leading to have the signal $y$ as follows:
$y = h*x_2$   ......  where $*$ is the convolution operation.
Hint: the convolution is not circular in that case as we used DCT instead of DFT.
At the receiver, after removing the delay of the channel and the CP guard interval, I used the following steps:
$x_3 = ifft(fft(y)./H)$ .....  I used ZF equalizer, $H$ is the frequency-domain channel found by $H = fft(h,N)$
Then, the signal is recovered  by multiplying by the DCT matrix
$x_4 = D * x_3$
My question,  how does it work the step when I get $x_3$?? is there mathematical expression for that,  , I think it's called frequency domain equalizer, but I didn't get its mathematical expression. Most importantly, why does it work however we don't have circular convolution when getting $y$? !!
thank you
Hint: the convolution is not circular in that case as we used DCT instead of DFT.

Luckily, that's not true! At the receiver, you're doing a DFT, and you've prepared the transmit signal such that a convolution with the channel looks like a circular one by prepending the CP.
The fact that the data inside has something to do with the DCT is irrelevant there.

Then, the signal is recovered by multiplying by the DCT matrix

Yep, because that inverts $D'$.

I  think it's called frequency domain equalizer,

No, the frequency-domain equalizer is $\text{IFFT}(\text{FFT}/H)$.