Difference between advantages of Experience Replay in DQN2013 paper
Isn't the third advantage just another case of breaking correlation?


It could be viewed that way, but it is a different kind of correlation.

The second advantage is about breaking correlation due to samples being from adjacent time steps on the same trajectory. This is a more important problem when state vectors evolve slowly/incrementally per time step. 

The third advantage is about breaking correlation due to samples being taken from the same policy. This is a more important problem when some actions have very similar reward and state progression independently of state, or may result in no change to state (e.g. an agent tries to move into a wall - agents getting stuck in corners due to runaway feedback of the direction being "best" is a thing you can sometimes observe whilst DQN is learning).

Both can also be a factor when an early event puts the remainder of a trajectory into a single part of the overall space, which can occur in environments where state history is important - in those cases, almost all of the time steps in a single episode could be correlated. Think of a resource management game, where expending or keeping a key resource early on in the game has a large influence on eventual success at a task. This is impacted by elements of second and third advantages - i.e. the states in the trajectory are correlated due to state of the resource throughout, and across multiple episodes the current policy might prefer expending or keeping that resource at a specific stage.

The correlations avoided by the second and third advantages typically occur over different scales of time, although that does depend strongly on the specific problem. The second advantage might be gained with a relatively small replay memory (depending on how the state evolves and lengths of episodes). The third advantage will usually require a larger memory, in order that it captures episodes with varying policies. In addition the replay memory should include exploratory actions which also help with this third issue.