Would a capacitor between base and emitter make relay amplifier less noisy?
Yes, it can help. 

I suggest to use capacitor 1...10uF here, it will make time constant (R*C) about 2...20ms, but the real OFF time will be about 10x shorter, because only starting part of capacitor unloading curve will be effective.

The voltage on capacitor will change between 0V and about 0.7V - so you can use electrolytic capacitor for any voltage.

BTW: if the flyback diode is not enought, it means that you probably have problems with ground wiring (to thick wire, too long wire etc).