Noise and potentiometer types
The contact resistance variation is 3% of the value (not great) but this could be improved by using the pot as a potential divider and feeding the wiper directly to the inverting input. The wiper feeding the inverting input means the current through the wiper is effectively zero other than op-amp bias currents (nA to pA).

So, in this new configuration, it doesn't matter so much about wiper contact resistance changes. If you had shown all your circuit including what feeds the inverting input I might have been a bit more definite about this.

EDIT this is what I recommend to try: -







simulate this circuit &ndash; Schematic created using CircuitLab

Hope it works OK