Guidelines to debug REINFORCE-type algorithms?
The only resource I could find so far : 

https://github.com/williamFalcon/DeepRLHacks



For my specific case, I made a few errors :


Frozen some part of the network that shouldn't be frozen
Wrong learning rate


Even if I could overfit a small dataset, it didn't mean anything : while training on the whole dataset, the average reward was not going up.

You should look for a reward going up.



I'm not accepting this answer as I believe it is not complete : it lacks general and systematic guidelines to debug a Reinforcement Learning algorithm.