Since Ψ τ does not depend on η, we have ∇ η L α (π η , r τ ) = ∇ η L α (π η , R τ ), and thus optimization of the VRB with respect to the policy parameters η does not require knowledge of the normalizing constant. However, for mass-covering α > 1, optimization can be challenged by pronounced variance in the estimated gradient, which is unfortunate since this is the regime that is generally of interest.