MATLAB appears to be applying my filter to the spectrum twice, once on each side So it looks like MATLAB is applying the filter to the "negative" and "positive" frequencies separately and I end up with two band pass regions with a null around zero. I want the whole spectrum filtered once.

[b,a] = butter(2,[.18 .3739],'bandpass');
hm = filter(b,a,Y);
figure;
plot(20*log10(abs((fftshift(fft(hm))))))


Any ideas?
Your filter has a real-valued impulse response, which means it has a two-sided spectrum.  Your filter is only applied once, but that application changes both positive and negative frequencies: -0.3739 to -0.18 and from 0.18 to 0.3729 (modulo the transition region of the filter).  If the filter did not do this, you would have a low pass filter and not a bandpass filter.