Policy Gradient not "learning"
I can't say for sure but I think the issue here is you're not subtracting the mean of the rewards.

The idea is that actions with above average reward are positive after mean normalization, while actions with below average reward are negative after mean normalization. 

Your update step is -log(P(action))*reward, which you then minimize with your optimizer. 

P(action)&lt;1 therefore log(P(action))&lt;0, -log(P(action))&gt;0

If reward&gt;0, -log(P(action))*reward&gt;0. Minimizing this value is the same maximizing log(P(action))*reward&lt;0, which is maximized when P(action)=1.

Conversely, if reward&lt;0, -log(P(action))*reward&lt;0. This has the opposite effect, where P(action) is driven to 0.

The important part is the different sign on above average/below average rewards causes actions associated with good rewards to have their probability increased, while actions associated with bad rewards have their probability decreased.