Why does my amplifier oscillate when I try to reduce the bandwidth?
Your amp seems to have pins for a deliberate compensation cap.  They added those for a reason, implying that the usual capacitor from the output directly to the negative input might not work so well.  Looks like you confirmed that.  If you want the amp to be more stable, increase C1.

To reduce out of band noise, try adding a little low pass filtering of the input signal.  For example, you could split R1 into two halves and put a capacitor to ground from the middle point.