Subtract Diff-Signal (Millivolts range) on top of common mode voltage (volts) with opamp - very inaccurate
Consider the effect of a small mismatch between the resistor values (actually the mismatch between two resistor ratios is what is important).

If you used 1% resistors, the non-inverting input can be 1% off worst case, and the feedback likewise, so you could see an error of as much as 2% of the common mode voltage. At 20V, that's 400mV, or 40x your entire full-scale signal. Chances are it would be somewhat less in practice, but probably more than 1000% error.

While you could buy very expensive 0.01% resistor networks you'd still have a 4% error and even a tiny shift with temperature would cause a huge error in the output. That's why that kind of differential amplifier is  generally not used when the signal is small compared to the common-mode voltage- the sensitivity to component tolerances is far too great. 

You could use the classic three amplifier in-amp configuration and avoid buying a commercial part- the key is to have enough amplification in the first stage (R1/Rgain) that the resistor ratios in the output amplifier are not too critical. Also, watch for saturation. 



Also, this is not a huge source of error, but note that the amplifier inputs have some current draw. If you put the inverting on the low side of the shunt the load looks like 20K to ground (1mA) so it will measure that current (but not the 1mA the other input draws since it doesn't go through the shunt). The instrumentation amplifiers have high-impedance inputs so that error current will likely be maybe 5-7 orders of magnitude less.