Difference amplifier with potentiometer on input
Because some current flows through R1, which "ruins" your voltage divider.

I.e. your voltage divider only works "properly" when negligible current flows through the wiper of the potentiometer. When R1 is disconnected, no current can flow, and the pot acts as a "proper" voltage divider. When you connect R1, a non-negligible current flows through R1, causing your voltage divider to "droop".