Difference between using large vs small test resistance to check internal DMM/scope resistance
I suspect you ran into a calculation accuracy error. Let me explain.

If \$R_1\$ and \$R_2\$ are much less than the scope/DMM input resistance, the circuit you showed is practically unloaded (i.e. we assume infinite \$R_{dmm}\$).

Therefore, since \$R_1 = R_2\$, then \$V_{BB}=0.5 V_{AA}\$ (equal branch voltage divider) and \$ R_1 + R_2 = 2 R_1\$. Then look at the denominator of that formula:

$$
V_{in} R_2 - V_{out} (R_1 + R_2) 
= V_{AA} R_2 - V_{BB} (R_1 + R_2) 
= V_{AA} R_1 - 0.5 V_{AA} \times 2 R_1 
= 0
$$

In other words, the denominator would be theoretically zero, giving an infinite \$R_{dmm}\$, as expected.

The problem is, that in practice the denominator won't be exactly zero, and any slight approximation of the values of \$R_1\$ and \$R_2\$ or any slight error in the measurement of \$V_{AA}\$ or \$V_{BB}\$ will greatly affect the result!

In other words, when the denominator approaches zero, it becomes much more numerically sensitive to the errors of the parameters used to calculate it.