Approximate value iteration for continuous state space MDPs
There are two primary methods to deal with continuous state MDPs. 
1. State-space discretization.
2. Value function approximation.
As for value function approximation, you can either go for a deterministic/stochastic black-box model or opt for fitted value iteration. You can find the algorithm for the same in pages 10-13 of this link.