Is my understanding of the differences between MDP, Semi MDP and POMDP correct?
Yes, the core differences between the different categories of problems are correct as you've described them.
For SMDPs, I'd like to remark that the water boiling example is maybe not the best. That looks more like an example of &quot;delayed rewards&quot;, but not one of &quot;durative actions&quot;: when the agent takes that action to raise the temperature, it takes some time before the reward comes in, but the agent's action itself doesn't take that much time. The agent could maybe do something else in the meantime. Delayed rewards are not restricted to SMDPs, they can also show up in regular MDPs (you just have to make sure to include some data in the state representation to indicate the time since the temperature was set or something like that, such that the Markov property is not violated).
A typical example of an SMDP with durative actions would be a grid with rooms, split by walls but connected by smaller doors inside those walls. A single &quot;primitive&quot; action would just take a single step in the larger grid, but longer-duration &quot;macro-actions&quot; would make the agent navigate from somewhere within a room to one of the doors &quot;on auto-pilot&quot;. This would still take as much time as it would to navigate explicitly in multiple smaller actions, but the larger more complex behaviour being encapsulated in a single &quot;macro-action&quot; can speed up learning.