Matlab applying an FIR filter to a signal, generated in Matlab I want to create an FIR in Matlab and apply it to a signal (also generated in Matlab). You will find the code below. I will put my questions in order from here.

1) The filter coefficients vector "b" should contain coefficients for a bandpass with a filter order of 100 and a passband of 1000Hz to 2000Hz. My sampling frequency is 50.000Hz. Matlab uses normalized frequency. As far as I have understood, my formula to transform my frequency to a normalized frequency is f_norm = (2*f)/f_sample. So in my example I would get 0.04 for 1000Hz and 0.08 for 2000Hz. Is that correct?

2)I am creating a test signal sinuses with the frequency 200Hz, 2500Hz and 5500Hz and compute the two sided fft of the amplitude. The graph is attached

This looks good, since the overall amplitude should be about 0.33 for every signal part. Now, when I apply my filter to the signal and plot the two sided fft of the result, it shows the next graphFFT Signal after filter 

I mean I can see the 5500Hz being filtered out, but the amplitude of the 2500Hz signal drops by a factor of 10 and the 200Hz signal is also not filtered out completely. What are some things that I can improve here? 3) Also, when I only raise the filter order, I am getting worse results with more suppression on the 2500Hz signal and less suppression on the signals that I want to have filtered out.



%fs = 50000
b = fir1(100, [0.04 0.08], 'bandpass');    %passband 1000Hz bis 2000Hz
figure(1);
freqz(b,1,2001);

f1 = 100;                  %Hz
f2 = 5500;
f3 = 2500;
A = 1.00;                  %Amplitude
sampleRate = 50000;
sampleTime = (1/sampleRate);
t = 0:sampleTime:1;

s1 = A/3 * sin(2 * pi * f1 * t);
s2 = A/3 * sin(2 * pi * f2 * t);
s3 = A/3 * sin(2 * pi * f3 * t);

signal = s1 + s2 + s3;

Y = abs(fft(signal));
samplePoints = length(Y);
Y = Y / samplePoints;
F = linspace(0,50000,50001);

figure(2);
plot(F,Y);
grid on;

y = filter(b,1,signal);

Y = abs(fft(y));
samplePoints = length(Y);
Y = Y / samplePoints;
F = linspace(0,50000,50001);

figure(3);
plot(F,Y);
grid on;
In general your result match expectations, I don't see anything in there that's unusual or unexpected.

Filter design is a complicated trade off between complexity/cost, stopband attenuation, pass band ripple, transition steepness, phase distortion, time domain ringing, non-causality, latency, etc. The best way to go about is to clearly articulate the requirements for your specific application and then evaluate any design against these requirements line-by-line. 

I'd also be careful with spectral analysis: direct FFT works sometimes, but something like pwelch() is typically much more robust.