What is the math behind high FM frequency components being more noise prone?
In a radio communication system the ratio of received power \$P_r\$ to transmit power \$P_t\$ is given by Friis transmission equation

$$\frac{P_r}{P_t} = \frac{A_r A_t \lambda^2}{(4\pi d)^2},$$

where \$A_r\$ is the receive antenna gain, \$A_t\$ is the transmit antenna gain, \$d\$ is the distance between the antennas, and \$\lambda\$ is the wavelength. The wavelength is related to the frequency by \$\lambda = c / f\$. Hence

$$\frac{P_r}{P_t} = \frac{A_r A_t c^2}{(4\pi d)^2 f^2}.$$

Thus the higher frequencies will be attenuated more than the lower frequencies. Usually this is not a problem when the bandwidth is low compared to the carrier frequency. It can become a problem, if the bandwidth is large. The difference at 100 MHz with 200 kHz bandwidth is 0.4%.



Another source of the same effect is the FM demodulator. The FM demodulator consists of a differentiator and an envelope detector.



source

The output of the envelope detector is

$$A[\omega_c + k_f m(t)],$$

where \$m(t)\$ is the message. An FM system is non-linear, but for high SNR it can be shown it is approximately linear. Hence the noise is amplified by the differentiator, since the transfer function is

$$H(f) = 2\pi j f.$$

Thus the noise is increased at higher frequencies.