Understanding REINFORCE loss The loss used in REINFORCE algorithm is confusing me.

From Pytorch documentation :

loss = -m.log_prob(action) * reward


We want to minimize this loss.  



If a take the following example :


Action #1 give a low reward (-1 for the example)
Action #2 give a high reward (+1 for the example)


Let's compare the loss of each action considering both have same probability for simplicity :

p(a1) = p(a2)
=> m.log_prob(a1) = m.log_prob(a2)
And loss(a1) = -m.log_prob(a1) * reward(a1) = m.log_prob(a1)
And loss(a2) = -m.log_prob(a2) * reward(a2) = -m.log_prob(a2) = -m.log_prob(a1)
Then loss(a1) &lt; loss(a2) since m.log_prob(X) &lt; 0

Here I don't understand this conclusion : the loss being minimized, it means a small loss is good compared to a high loss. 

So here it would mean action #1 is good compared to action #2 ? But the reward says the opposite !
I think the intuition here is that you want to push the negative loglikelihood (NLL) * the reward to be as negative as possible. Since our reward is often not differentiable because it is obtained through sampling, we can only change the NLL. For actions with high reward, we have greater &quot;pressure&quot; / gradient to push the NLL down, than if the reward was low, as the reward acts as a multiplier on the gradient on the NLL. So you'd imagine at convergence/equilibrium, the high reward actions would have greater likelihood than the low reward ones. Let me know if that makes sense!