Johnson–Nyquist Noise of Open Circuit
The big factor that you didn't consider is that "free space" has capacitance and therefore, an open circuit of (say) 1 G ohms shunted with (say) 1 pF (for example) has a bandwidth of: -

$$\dfrac{1}{2\pi RC}$$

Numerically this is 0.159 Hz and the equivalent noise bandwidth is that multiplied by \$\pi/2\$. This restricts noise voltage very significantly.


  Johnson–Nyquist Noise increases with resistance - so does it become
  infinite for an open circuit?


No it doesn't because of the effect of capacitance.