Voltage source for Vref or Voffset
Yes, low impedance is important. 

The differential input impedance (looking into either input) is 696 ohms when the diffamp is in balance, so if you're looking for 0.1% accuracy you need the source impedance to be well under an ohm.  

You can use another of the same kind of op-amp (LT1818) to buffer the 1.5V reference, which could be derived from a higher voltage reference with a voltage divider. 

This is not the usual application for a differential amplifier, seems like a lot of parts just to remove 1.5V from the input signal. 

Edit: For a voltage divider you can use something like this: 







simulate this circuit &ndash; Schematic created using CircuitLab

Try to make R2 about 1/5 to 1/10 of the pot value, and choose R1 and R3 to get enough range to account for all variations for sure, but not too much more.