How to prevent distortion after down-sampling?
Read up on the sampling theorem, e.g. https://en.wikipedia.org/wiki/Nyquist%E2%80%93Shannon_sampling_theorem

If your feature vectors are still in the time domain and linearly derived from the original audio wave forms, you just need to low pass filter before down sampling. 

You need to low-pass filter to avoid aliasing. There many different types of lowpass filters and the best choice depends on the specific requirements of your application (causality, latency, phase distortion, transients, size of transition band, MIPS &amp; Memory, required aliasing rejection, pass band ripple tolerance, etc.) 

As you lowpass filter, you lose information. If you need to down sample to a 500 Hz sampling rate all information above 250 Hz (in practice more like 220 Hz) is lost.

Lowpass filters are can be tricky to design depending on your requirements but the code to run them is very straight forward and can easily be done from scratch. Just make sure you use cascaded second order sections and Direct From I or Transposed Form  II.