Output of IIR filter not matching the frequency response - Proximity to fs/4 After not having dealt with digital filters for a long time, I have now been playing around with filter design in octave and I am observing a behavior which I do not understand.
After designing a low-pass Chebychev filter with cheby1(1,3,0.4) and plot its frequency response, it can be seen that the -3dB point is exactly at the specified cutoff frequency.
However, when I test the filter with an input wave x=cos(2*pi*cutoff*t), using the filter function y=filter(b,a,x), the amplitude of the output signal can get significantly lower than the expected $0.707$, $0.63$ for this example in particular.
It seems that the output of the filter better approximates the plotted frequency response for cutoff frequencies either close to Nyquist frequency, or close to $0$, but as it gets closer to the middle point between these two, the output steps away from the expected attenuation.
What is the reason behind this?
From the octave documentation on cheby1.m you can see that the cut-off frequency in radians is given by pi*Wc, where Wc is the cut-off frequency input argument of cheby1.m. So for a function call [b,a] = cheby1(n,Rp,Wc), you should test the filter's gain at the cut-off frequency with an input sequence cos(Wc*pi*n). Furthermore, choose a sufficiently long input signal so that you can observe the filter's steady-state behavior.
Apart from that it's a matter of sampling the time domain function. Note that the samples will generally not occur at the maxima of the sinusoid. I.e., the maximum magnitude of the output signal
$$y[n]=|H(\omega_c)|\cos(n\omega_c+\phi(\omega_c))$$
will not necessarily attain its theoretical maximum value $|H(\omega_c)|$.