Loop Bandwidth for Carrier Demodulation loop
Your results are not showing that the data is being "filtered out" necessarily but that you are using a non-linear phase filter resulting in group delay variation over your passband, as evidenced by the spread in phase. 

Don't use a Butterworth filter to model this. Instead use filters derived from either firls or firpm (remez in Octave) which are linear phase and have constant group delay.