How to calculate v min and v max for C51 DQN
If you're using a discount factor less than 1, you should be able to compute a maximum return (likewise, a minimum return) based on the max (min) reward you can earn at each timestep. However, this issue you bring up is usually cited as a difficulty with C51. I think people tend to simply use fixed values for the min/max return (or just make rough estimates). If you want to avoid this, I recommend looking into the QR-DQN algorithm which circumvents this issue altogether and is more theoretically sound.