Approximate value iteration for continuous state space MDPs I have a continuous state space MDP as a generative model. I input the state and action and it outputs the reward and the next state. Assume that I sampled $n$ state-reward-states. I wonder how I can implement value iteration using a function approximator. I couldn't find any implementation example online. Can you please point me some references?
There are two primary methods to deal with continuous state MDPs. 
1. State-space discretization.
2. Value function approximation.
As for value function approximation, you can either go for a deterministic/stochastic black-box model or opt for fitted value iteration. You can find the algorithm for the same in pages 10-13 of this link.