Measuring mOhm resistance using LT006 precision amplifier and 4-Wire Method
Use an AC current source, and AC preamplifiers on the small developed voltage.
There's no 'offset' if you use an AC signal instead of direct-coupled DC.AN-98 See Fig. 27

The use of FFT oscilloscope, or a phase-locked amplifier, might accomplish
the desired result without having to wire up a dedicated measurement circuit.

A low-voltage measurement with DC sensitivity will always show artifacts, if not from offset voltage, then from thermocouple effects.   AC measurements are the
only good solution.