Is A2C loss function taking smaller steps for larger mistakes?
So this question is due to a misunderstanding I had with how the loss function works in A2C. 

Usually loss functions are always positives, but in A2C it can be negative as well. I thought that minimizing the loss function means moving it closer to zero, but in fact reducing the loss is indifferent to the sign of the loss value and is about making the loss value smaller.

In other words, the loss for 0.1 is smaller than the loss for 0.9 prediction for a disadvantageous action.

When the advantage is positive, $-log (p) * advantage$ will be smaller if $p$ is growing, and when the advantage is negative, $-log (p) * advantage$ will be smaller if $p$ is decreasing.