Is FIR decimation the fastest possible method for integer downsampling? For a digital music synthesizer I am developing, I have a stream of incoming samples $x_i$ and would like to downsample by $M$ to obtain a stream of samples $y_{Mi}$. I am currently bandpassing the signal with an FIR which is a brick-wall filter windowed by Blackman-Harris, and it sounds great to my ears. But I have to compute $y_{Mi} = \sum_{j=0}^{N-1} x_{Mi-j} h_j$ for every outgoing sample. $N$ is usually 64 to 256, and $M$ is usually 8 or 16, so an FFT convolution doesn't seem worth it for performance.

If there a better method than naive FIR for integer decimation of audio at these scales? Can I make some reasonable tradeoffs in quality to achieve better performance? For example, polyphase decimation is a nice trick, but since RAM/cache is not an issue, I believe they should run in the same speed since they involve an equal number of multiplications.
Another way to design a FIR filter involves matching the impulse response to an analog prototype's impulse response.  The idea being that  an IIR can often be truncated to FIR. The FIR will not be symmetric  and not have linear phase but  the group delay can be much smaller. 
I believe that Friedlander has a paper  on how to do the approximation. There was a Matlab Signal Processing Toolkit function that did this but I haven't had much cause to use it. 
Comet also can do nonsymetric FIR designs. 

I also suggest that you try using Parks Mclelan if it can give you a shorter filter than using the window method