SD Card data rate and frequency
SDR104 operating mode specifies various things, including


maximum power drawn by the card
communication voltage
maximum allowed clock speed


You can operate the interface at a lower clock speed, if you wish to do so, but you still need to meet the other requirements of SDR104 mode.

If you decide that 50MB/s is enough, you can design your system to meet the relaxed requirements of SDR50 mode. If you decide that you require 105MB/s, you need to choose the entirely different UHS-II mode.