Verifying Channel Response Identity
I think you made a mistake between the power spectrum and the Fourier Transform.
This is the right form of the Convolution Theory (Nothing is squared):

$$ \lvert Y(\omega)\rvert = \lvert H(\omega) X(\omega)\rvert $$

Try this and it will work for you on MATLAB.

MATLAB Code

% Calculating the Magnitude of the Filtered Signal

vFilterCoeff = [1; 2; 3; 4; 5];
vInputSignal = randn(1000, 1);

vOutputSginal = conv(vInputSignal, vFilterCoeff);

fftNumPoints = 1024;

vOutputSignalDftMagnitude = abs(fft(vOutputSginal, fftNumPoints));
vFilteredSignalDftMagnitude = abs(fft(vFilterCoeff, fftNumPoints) .* fft(vInputSignal, fftNumPoints));

hFigure = figure();
hAxes   = axes();
plot([vOutputSignalDftMagnitude, vFilteredSignalDftMagnitude]);


The result is:



As can be seen there's a full equality.