Can you implement Replay Buffers for Reinforcement Learning when most experiences give zero reward?
What if the reward is only given at a terminal state? Or, most of the experiences rarely give immediate rewards but the reward at the terminal state is sort of "what really matters"?
Cause in that case, in that sample of a random minibatch, if it does not include the terminal state, then you would most likely have many 0 immediate rewards. Would DDPG algorithm with Replay Buffer still work for such situation?
Yes it will work just fine. The point of replay buffer is not to find non-zero rewards, it stores observed state transitions, and these are critical to resolving the credit assignment problem. Every time the algorithm process a transition with zero reward, it also processes the link between state, action and next state. This link between states is critical to picking correct actions when rewards are sparse.
If all the experience only has zero rewards, then of course there is nothing useful to learn yet (the agent may as well take random actions as far as it knows). Also, environments with very sparse rewards are harder to learn.
However, there is no problem in general if a minibatch contains only samples with zero reward. The agent can still learn just fine in that case, as the learning targets are not just based on reward on the single step, but also predictions of future reward.
Biased sampling towards non-zero rewards might help in some cases, but a more robust related approach is to bias towards action values where there was a large change recently to next state's action values - that is the premise of prioritised sweeping.
Not the answer you're looking for? Browse other questions tagged machine-learning neural-networks bootstrap gradient-descent reinforcement-learning or ask your own question.
Reinforcement Q Learning Algorithm Behaviour: What Can Be Inferred from Charts?
Can someone please explain the target update for deep Q learning?
How can Deep Q Learning be applied to scenarios with rewards only received in a final step?
Why does Hindsight experience replay help addressing the problem of sparse rewards?
How to set up a Q function approximator using neural net for DDPG?