To sample from πη , we first randomly initialize k latent codes by sampling from the policy π η and we then run stochastic gradient Langevin dynamics (SGLD) on the latent codes for T iterations using -E(z 1 , . . . , z k ) as the target unnormalized log density. While the proximity term in the energy function requires O(k 2 ) distance evaluations, we can form an MC approximation using a subset of ℓ ≪ (k -1)k/2 pairs sampled randomly each SGLD step to ensure tractability when k is large.