Calculate Impulse Response of a System From Its Input/Output Waveform via FFT
It would be best to upload the exact data for $x[n]$ and $y[n]$, but the reason I think of, is that either at least one of your signals is not a perfect baseband signal and has considerable amount of energy everywhere, or (this one is more probable) the signals are baseband but your sampling rate is not high enough, hence the Nyquist criterion is not satisfied and aliasing has occurred. In the latter case, try increasing the number of samples per each signal. Note that this is also an approximate, because the DFT package of MATLAB operates on discrete signals, yet with truncation in both time and frequency.