Out of memory problem with filter design tool in Matlab
Your sampling frequency is currently set to 5000000000 MHz, which is 5000 terahertz!  Notice that the units on the frequency specification should all be in MHz. Change the Fs value to be 5000 (MHz) and you will be much better off.

However, even once you fix this problem, you still may run into trouble.  You are specifying transition bands that have a width of less than 0.01% of your sampling rate.  This is not going to be practical, especially as an FIR filter.  I recommend that you consider a good low-pass filter followed by lots of down sampling.  Then you will be able to realistically extract the band of interest.