What is the mathematical relationship between MOSFET switching frequency (motor speed control) and motor speed in rpm?
There isn't one really.

The switching frequency of the MOSFETs is a function of the inductance and resistance of the motor and is used to control current ripple (and therefore torque ripple).

The speed of the motor shaft is a function of how fast you change which of the three phases are being powered.

That being said, there are limits. For instance the switching losses incurred from a high switching frequency will limit the amount of power available to the motor and therefore limit the speed with which you can commutate the motor.