Why is the logarithm of the standard deviation used in this implementation of proximal policy optimization?
As you have mentioned using log is nicer because it makes multiplications to additions etc etc(it helps in numerical stability issues). But I think over here, the reason they are doing it like that is because of enforcing a simple constraint in a much more simpler way. In the __init__ we are noticing that the log_std is being formulated instead of the std itself. Would it be wrong if you formulated the std itself? No. But it would be a bit messy to have some constrained imposed on it. For example, if we modeled std it could have been possible for the std to become 0 or negative. Well that's not a correct values of a std. But over here that is being enforced automatically, this kind of little things makes the learning for the model easier. Even if the log_std becomes negative or zero it does not matter because the exp will take care of it by exponentiating it to a positive number.