How do I calculate maximum frequency supported by an optocoupler say for example PC817 and what factors it depends on?
It's depend on the rise and fall time of the optocoupler, or rather I should say the propagation delay.
In the best case you'll get 143 kHZ (tr+tf)

 (see the datasheet).
If your frequency is 5MHZ, the PC817 optocoupler isn't the best choice. The optocoupler propagation delay is more than the SPI clock period (1/frequency).
In this link you'll understand how to choose a suitable optocoupler to your SPI bus.