Temporal Difference Learning Getting Stuck I'm trying to implement a temporal difference algorithm that learns the maximum revenue over a period of time using prices as the actions, inventory as the state, and revenues realized as the reward. The problem I'm having is that I can't seem to get it to converge on an optimal policy. It seems like it gets stuck at a revenue somewhere around 60% of optimization and then won't budge anymore. Are there some common pitfalls that might be causing this? I've tried playing with the rate of exploration a little bit, but that hasn't seemed to help it much.

EDIT:
Ok, so I went back through everything, and it seems like the problem is that, when it stops exploring, it just keeps increasing the Q-values of states that it's already deemed the best in the past. So for example, it visits the price 5 at a certain state and time and gets a reward. Then, in the next several episodes, it continues to visit 5, and continues getting a reward, adding that reward to the Q-value until it's pretty high. At that point, even if it were to explore at the same state and time, the reward it gets isn't enough to overcome the inflated Q-value, so it just goes right back to 5 in the next episode. Here are the steps I'm trying to follow.
Sounds like an interesting application. To debug RL applications, I like to perform rollouts. First identify a state where the current policy appears to be clearly wrong; (maybe sample some states and look at them individually). Then switch off the learning and run a sample from that state for both the current RL policy and the action you think is correct. This should give you the true action values. Hopefully, that will give you a hint as to what's going wrong. If it's an exploration problem, then the current policy won't be trying any actions similar to the action you think is correct. If the form of the value function is wrong, then it won't fit the values coming from your samples.