How does transition width of a low-pass filter affect a message which is well within the pass-band? I'm taking a communication systems course and we use GNURadio in the lab. In an exam, we had a simple question where we were asked to demodulate a Dual Side Band (with carrier) AM signal in GNURadio. Carrier frequency was known to be precisely 500KHz and message was audio (known to be &lt; 15KHz). The incoming signal has sampling rate of 1.92MHz. 

EDIT: i.e incoming signal is given to be of the form $(1 + k_am(t))cos(2{\pi}{f_c}t)$ with $|k_am(t)| &lt; 1$

To do that, I took the absolute value of the incoming signal, and used a Low pass Filter with cutoff frequency 17KHz and transition width 100Hz to obtain the audio clip. I was listening to the audio sampled at 48KHz (with proper decimation of 40 set in the Low Pass filter block). However the received audio clip was very jittery. I could listen and identify the song, but there were periodic higher frequency spiky noises. 

The TA observed this, changed the transition width of the LPF from 100Hz to 1KHz and it was  completely fine now - I could hear it neat and clear! I argued that I only tried to make the filter better by making it more ideal - brick-wall type, and also, this transition is happening where I don't have my message (transition occurs at 17KHz and my message &lt; 15KHz). He replies that I pay a price when I make a filter transit faster, and added I would understand that when I do the digital communications course. 

Could someone kindly explain to me, what price I pay (in digital domain) when I decrease the transition width of the Low Pass filter? Thanks in advance.
For an FIR filter there are 3 main components that determine the filter length for equiripple designs:


Passband ripple
Stopband attenuation level
Transition width (Width from the edge of the passband to the edge of the stopband)


For other filter designs the filter order may be related to flatness of the passband and the rate of fall-off in the stopband (and initial stopband attenuation). The transition width is often expressed as a fraction of the original sampling frequency.

In your case the original fraction is $\frac{100}{1.92\times10^6}\approx 52.1\times10^{-6}$ which is very small and is going to require a very long filter. Depending on the algorithm used to design the filter, the algorithm may run into numerical difficulties. As an example, designing a 2000 tap FIR equiripple filter using Matlab's Remez type routines is often problematic.

Increasing the transition width by a factor of 10 will roughly reduce the filter length by a factor 10 as well. This is just a rough approximation and results will vary based on the design routine and the filter specs being used.

If you really need such a small transition width, it is better to perform you decimation in steps, e.g, break it down to decimate by 10 and then by 4. In the first decimate by 10 you can use a wider transition width. The transition width in the last filter will be the same as the original requirement, but because of the reduced sampling rate the filter length will be much smaller. Crochiere and Rabiner's "Multirate Signal Processing" goes into more details.