Filter design - Rules of thumb regarding ratio of sampling and stop/pass frequency
In the most general case, there isn't really a formula for how long the filter should be. There are papers which show the empirical relationships between the filter transition width, passband ripple, and stopband attenuation. You typically have - for a fixed sampling frequency:


A smaller transition width (from the stopband to the passband), as a percentage of the sampling frequency, requires a longer filter. A 100 Hz transition width for fsamp=5kHz would require a smaller filter than a 100 Hz transition width with fsamp=15kHz - assuming other requirements are equal.
Smaller ripples in the passband, requires a longer filter
More attenuation in the stopband requires a longer filter


That said, for remez type designs there are a couple of published formulas that give you an estimated length of the filter. In Matlab you could use the remezord() function. These formulas work fairly well, but they do run into problems when the transition frequencies are near 0 Hz or fs/2.

There are some published papers that give equations for the length of filter using a Kaiser window for a windowed Sinc design - but you don't have as much control in the specification of the filter - if I recall correctly.