Verifying Channel Response Identity If $x[n]$ is input signal, $y[m]$ is output signal, $h[m]$ is impulse response of the channel, then the following relation holds

$\lvert Y(\omega)\rvert = \lvert H(\omega)\rvert^2 . \lvert X(\omega)\rvert$

where $F(\omega)$ is the DFT of $f[n]$.

To verify this, I wrote following MATLAB code

x = randn(1,1000);
h = [1 2 3 4 5];
y = conv(x,h);
plot((abs(fft(h,1024)).^2).*(abs(fft(x,1024))));
hold on
plot(abs(fft(y,1024)),'r')


Result shows that second plot is at a much lower level than first plot. When I multiply second plot by $1024$ i.e. NFFT then both the plots overlap.

What is wrong with the original equation? I remember learning it that way in class.
I think you made a mistake between the power spectrum and the Fourier Transform.
This is the right form of the Convolution Theory (Nothing is squared):

$$ \lvert Y(\omega)\rvert = \lvert H(\omega) X(\omega)\rvert $$

Try this and it will work for you on MATLAB.

MATLAB Code

% Calculating the Magnitude of the Filtered Signal

vFilterCoeff = [1; 2; 3; 4; 5];
vInputSignal = randn(1000, 1);

vOutputSginal = conv(vInputSignal, vFilterCoeff);

fftNumPoints = 1024;

vOutputSignalDftMagnitude = abs(fft(vOutputSginal, fftNumPoints));
vFilteredSignalDftMagnitude = abs(fft(vFilterCoeff, fftNumPoints) .* fft(vInputSignal, fftNumPoints));

hFigure = figure();
hAxes   = axes();
plot([vOutputSignalDftMagnitude, vFilteredSignalDftMagnitude]);


The result is:



As can be seen there's a full equality.