title: prioritized experience replay in deep Q-learning
body: <p>i was implementing DQN in <a href="https://gym.openai.com/envs/MountainCar-v0" rel="nofollow noreferrer">mountain car problem</a> of openai gym. this problem is special as the positive reward is very sparse. so i thought of implementing prioritized experience replay as proposed in this <a href="https://arxiv.org/pdf/1511.05952.pdf" rel="nofollow noreferrer">paper by google deep mind</a>.</p>
  
  <p>there are certain things that are confusing me:</p>
  
  <ul>
  <li>how do we store the replay memory. i get that p<sub>i</sub> is the priority of transition and there are two ways but what is this P(i)?</li>
  <li>if we follow the rules given won't P(i) change every time a sample is added.</li>
  <li>what does it mean when it says "we sample according to this probability distribution". what is the distribution.</li>
  <li>finally how do we sample from it. i get that if we store it in a priority queue we can sample directly but we are actually storing it in a sum tree.</li>
  </ul>
  
  <p>thanks in advance </p>