Why is the reward signal normalized in openAI's REINFORCE?
It's just a hack to normalize the rewards to make optimization less problematic in cases where the reward might have very large/small values. It's fine because scaling the rewards by a constant factor won't change what the optimal policy is. You can think of the mean reward as the baseline.