Differential amplifier with a notch on output
You have probably got a significant common mode voltage problem i.e. the common mode voltage of the signal source is probably several volts different to the ground connection on your measurement circuit.
You get rid of this CM voltage by shorting both grounds and the circuit behaves.
Any differential amplifier has to deal with both the signal voltage extremes AND the common mode voltage together. This is usually achieved by making R58 and R4 much bigger in value. I can see you are on the way with this but, I suspect you might not have evaluated the true common mode voltage of the source relative to measurement ground.
I think the problem is amplified by you choosing the MCP634 - the highest voltage allowed on the input pins is Vdd - 1.3 volts and with you operating at 3.3 volts on your supply, the largest voltage allowed (before you get silly things happening) is 2 volts. So, if you say there is 2 volts across R5 then things go wrong when ENC1A is greater than 8 volts. Can this happen? Does it happen?