Why are state transitions in MDPs probabilistic rather than deterministic?
Your understanding is right!

Using a probabilistic transition function allows the model to explore a bigger search space before making a decision. One of the most important use cases of MDP is in POS tagging in NLP using a Hidden Markov Model.

In case of a deterministic model, search space is limited by the number of transitions and hence at each step, a definite decision is made. This does not take into regard the possibility of relationship between the previous states, rather only deals with the current and next state. These model are good for solving a certain range of tasks like decision trees etc.

When it come to tasks such as weather prediction, there is significance to the historical weather data. In such cases, we cannot make use of a deterministic approach. You always predict the chance of rainfall etc.



This example can also be extended to predict the weather for future days