Increasing the opamp gain is decreasing the DAC resolution
Your resolution is
$$
\huge
V_\text{Resolution}=\frac{V_{DAC_\text{output range}}}{2^{DAC_\text{bit_number}}}\times GAIN
$$
it is the smallest increment you can achieve for the ADJ voltage.

To get a better resolution, you can:


diminishing the OP AMP gain, but won't be able to have a 0-5V range for the ADJ voltage
diminishing the DAC output_range (by diminishing the DAC voltage reference), but then you also won't be able to have a 0-5V range for the ADJ voltage
get a DAC with a higher bit number


Given your constraints (getting a 0-10V range for the ADJ voltage with a 1mV resolution) you need at least a 14-bit DAC.