how to compute bernoulli entropy?
Good observation, and yes, they are in fact equivalent ways of computing the entropy of a bernoulli random variable. 

To begin, you have to notice that in the openai code, we do not have the value of $ p $ passed to the function, instead we have $ logit(p) $ which is defined to be:

$ logit(p) = log\frac{p}{1-p} = log(p) - log(1-p)$

Also, two formula to keep in mind. The first is the sigmoid function which is:

$ sigmoid(x) = \frac{1}{1+e^{-x}} $

And the second is the sigmoid of the logit which is:

$ sigmoid(logit(p)) = \frac{1}{1+\frac{1-p}{p}} = p$

(i.e. the inverse of the logit function is the sigmoid function)

Now we are ready to go from the first equation (used in the code) to the second equation (the generic equation to compute the bernoulli entropy)

$ entropy = (1 - sigmoid(logit(p)))*logit(p) - log(sigmoid(logit(p))) $

$ = (1 - p)*logit(p) - log(p) = (1 - p)*(log(p)-log(1-p)) - log(p) $

$ = (1 - p)*log(p) - (1-p)*log(1-p) - log(p) $

$ = -p*log(p) - (1-p)*log(1-p) = entropy $

Thus the formula used in the code is equivalent to the second formula you provided.