where D(π η ∥r τ ) ≥ 0 is a divergence from the policy π η (z|τ ) to the target distribution r τ (z) = Ψ -1 τ R τ (z) and Ψ τ = z R τ (z) dz is the (unknown) normalizing constant of the tempered reward function. Here, we amortize optimization with respect to τ > 0 by specifying a temperature distribution p(τ ), e.g., an exponential distribution with rate parameter λ > 0, and minimize the expected divergence from the policy to the tempered reward marginalized over p(τ ).