FIRFilter with low cut-off compared to sampling frequency
If the desired cutoff frequency is very low relative to the sample rate, then yes, you would need a very high-order FIR filter to implement it. However, in these sort of cases, it's usually a better idea to use a multistage approach. 

For example, say you wanted to decimate a signal by a factor of 10000. If you implement this with a decimating lowpass filter, it would need to have a cutoff frequency less than $\omega = \frac{\pi}{10000}$. This would require a filter with tens to hundreds of thousands of taps, depending on how much attenuation you require in the stopband. This is a lot of processing, especially if you're trying to do real-time processing. What's more, with really long filters like this, you can run into numerical precision issues; the filter that you get might not even give you the frequency response that you designed it for!

Instead, consider an approach with two stages: two identical filters that each decimate by a factor of 100. In this case, your cutoff frequency must be less than $\omega = \frac{\pi}{100}$. And you'll only need hundreds to thousands of taps in order to implement each filter. The computational cost of two few-thousand-tap filters is much less than the cost of one filter with hundreds of thousands of taps. This only becomes more apparent when you notice that the first filter decimates by a factor of 100, making the cost of the second one negligible compared to the first stage.

In short, when large rate changes like this are desired, multi-stage approaches are almost always what you want. The same logic follows for interpolation as well. Lyons has an easy-to-understand discussion of this topic in his introductory text Understanding Digital Signal Processing.