What if the Down sampling factor is not an integer?
Resampling (by rational or even irrational ratios) can be done by low-pass filtering in conjunction with high quality interpolation of all the samples needed for the new rate, directly.  No two-step up-sampling following by downsampling is required (although that is one possible implementation for simple ratios).

A Sinc kernel, being a reconstruction formula, is the ideal interpolator for band-limited sampled waveforms.  In practice, a windowed Sinc interpolator is usually as good or better than the FIR or IIR filters used inside other resampling methods.

Also, information may be lost in going to any lower sample rate (even in your original 4:1 resampler) if the original sample data contains higher frequency spectrum that has to be removed by the anti-alias low pass filter (required to prevent aliasing at the lower sample rate).