Is FIR decimation the fastest possible method for integer downsampling?
Another way to design a FIR filter involves matching the impulse response to an analog prototype's impulse response.  The idea being that  an IIR can often be truncated to FIR. The FIR will not be symmetric  and not have linear phase but  the group delay can be much smaller. 
I believe that Friedlander has a paper  on how to do the approximation. There was a Matlab Signal Processing Toolkit function that did this but I haven't had much cause to use it. 
Comet also can do nonsymetric FIR designs. 

I also suggest that you try using Parks Mclelan if it can give you a shorter filter than using the window method