Because expectations are with respect to p(τ ) and β(z|τ ), neither of which depend on η, we avoid the need to invoke a reparameterization trick in gradient estimation. This supports both on-policy (i.e., for β(z|τ ) = stop grad(π η (z|τ ))) and off-policy learning (e.g., using a replay buffer to store past episodes).