How to eliminate 50 Hz noise from amplifier's output
It's not the oscilloscope probe behaving like an antenna -- it's YOU. You are capacitively coupled to noise in the environment. The cap prevents DC signals from reaching the amplifier, not AC signals.

If you don't want to see the 50Hz noise, hook the input up to the signal you're trying to amplify, and don'd touch the input pins.