How surface refractivity affect radio propagation?
The paper you reference is quite thorough in giving ranges of temperature and height over which its approximations are valid, but it is totally silent on frequency. Therefore, we can assume that the refractive index is independent of frequency, at least over the commercial communication range that is the ITU's remit.

However, we know that long wave (100s kHz) and medium wave (~1MHz) signals can operate over the horizon, while VHF (~100MHz) and above operates more or less line of sight, so what's going on?

The clue is in the ratio of the signal's wavelength to the rate at which refractive index changes with height, see equation 11. A radio signal, to the extent that it propagates 'quasi-optically', does so over a 'beam' that is a few wavelengths wide, see Fresnel Zone on wikipedia for a proper definition of 'a few'. A long wavelength signal therefore experiences the medium as having a varying index across the beam width, and bends. A short wavelength signal sees a medium with negligible variation over a few wavelengths, and travels in straight lines.