How to scale all positive continuous reward?
I’ll try to find where I found it, but normalizing the rewards has always worked for me. Assuming you have a list of the discounted returns for each action, you subtract the whole list by its average value then divide it by its standard deviation. In Python with NumPy, that would look like:
returns -= np.mean(returns)
returns /= np.std(returns)

This puts the returns in a small and consistent range that keeps learning similar with different rewards.