Is using electrolytic polarised capacitor this way wrong?
Tantalum electrolytic capacitors can live with some small reverse current, but aluminum electrolytics will not do as well.  I would not use a polarized capacitor in this way for any circuit in production. Your options are high dielectric ceramic capacitors (but they have poor change in capacitance with voltage) , and non-polarized electrolytics, which are basically two electrolytic capacitors back-to-back. Both of these solutions exhibit a high value of dielectric absorption. 
If possible, make all your impedances larger, so you can use smaller value capacitors. Taking this an extreme will force you to use some active components (that is, op amps) in your signal conditioning.