"Disconnecting" an opamp from a potentiometer circuit
I would use a PhotoMOS relay, which uses an LED and photocell to control back-to-back MOSFETs for purely resistive output with no strings attached (unlike conventional SSRs which steal power from the load). It's the closest you can get to a mechanical relay, but much smaller and more reliable.

In your application there is a large resistance in series and low voltage, so a device like the AQY282SX would be fine. It has max. 2.5 Ω 'on' resistance and only needs 5 mA of LED drive current.