Subtracting two voltages using an op-amp
This should work, but usually there's also a resistor from the non-inverting input to ground:



If \$\dfrac{R4}{R2} = \dfrac{R3}{R1}\$ then:


  \$V_{OUT} = \dfrac{R_{3}}{R_{1}} \times (V_{IN+} - V_{IN-})\$


To minimize offset error both inputs have to see the same impedance, and therefore \$R1 = R2\$ and \$R3 = R4\$.
Omitting R4 will only give a scaling factor for the voltage on the non-inverting input, namely:  


  \$V_{OUT} = 2 \times V_{IN+} - V_{IN-}\$


but changing the setting for the lower potmeter should definitely have a result in the output. Did you measure the voltages on both inputs?

What happens if you set the lower voltage to 2.5V and the upper to 1V? The inverting input should also be 2.5V, and the output 4V. What do you measure?  

Note: especially the lower voltage follower is not necessary in your version; the opamp's input current is low enough to be negligible for most uses, and by the way, you're connecting the potmeter's wiper to an exactly same input!



Further reading
differential amplifier tutorial (interesting site overall!)