derivative f'(x). The next sections discuss these two in a
reinforcement learning example. For more details see Gradient
Estimation Using Stochastic Computation Graphs .
Score function
When the probability density function is differentiable with respect
to its parameters, we only need "sample()" and "log_prob()" to
implement REINFORCE:
\Delta\theta  = \alpha r \frac{\partial\log
   p(a|\pi^\theta(s))}{\partial\theta}
where \theta are the parameters, \alpha is the learning rate, r is the
reward and p(a|\pi^\theta(s)) is the probability of taking action a in
state s given policy \pi^\theta.
In practice we would sample an action from the output of a network,
apply this action in an environment, and then use "log_prob" to
construct an equivalent loss function. Note that we use a negative
because optimizers use gradient descent, whilst the rule above assumes
gradient ascent. With a categorical policy, the code for implementing
REINFORCE would be as follows:
probs = policy_network(state)