Question: represent the article for finding a single-sentence tldr

Question: Deep generative neural networks have proven effective at both conditional and unconditional modeling of complex data distributions. Conditional generation enables interactive control, but creating new controls often requires expensive retraining. In this paper, we develop a method to condition generation without retraining the model. By post-hoc learning latent constraints, value functions identify regions in latent space that generate outputs with desired attributes, we can conditionally sample from these regions with gradient-based optimization or amortized actor functions. Combining attribute constraints with a universal “realism” constraint, which enforces similarity to the data distribution, we generate realistic conditional images from an unconditional variational autoencoder. Further, using gradient-based optimization, we demonstrate identity-preserving transformations that make the minimal adjustment in latent space to modify the attributes of an image. Finally, with discrete sequences of musical notes, we demonstrate zero-shot conditional generation, learning latent constraints in the absence of labeled data or a differentiable reward function. Generative modeling of complicated data such as images and audio is a long-standing challenge in machine learning. While unconditional sampling is an interesting technical problem, it is arguably of limited practical interest in its own right: if one needs a non-specific image (or sound, song, document, etc.), one can simply pull something at random from the unfathomably vast media databases on the web. But that naive approach may not work for conditional sampling (i.e., generating data to match a set of user-specified attributes), since as more attributes are specified, it becomes exponentially less likely that a satisfactory example can be pulled from a database. One might also want to modify some attributes of an object while preserving its core identity. These are crucial tasks in creative applications, where the typical user desires fine-grained controls BID0.One can enforce user-specified constraints at training time, either by training on a curated subset of data or with conditioning variables. These approaches can be effective if there is enough labeled data available, but they require expensive model retraining for each new set of constraints and may not leverage commonalities between tasks. Deep latent-variable models, such as Generative Adversarial Networks (GANs; BID8 and Variational Autoencoders (VAEs; BID15 BID26, learn to unconditionally generate realistic and varied outputs by sampling from a semantically structured latent space. One might hope to leverage that structure in creating new conditional controls for sampling and transformations BID2 .Here, we show that new constraints can be enforced post-hoc on pre-trained unsupervised generative models. This approach removes the need to retrain the model for each new set of constraints, allowing users to more easily define custom behavior. We separate the problem into creating an unsupervised model that learns how to reconstruct data from latent embeddings, and leveraging the latent structure exposed in that embedding space as a source of prior knowledge, upon which we can impose behavioral constraints. Our key contributions are as follows:Figure 1: (a) Diagram of latent constraints for a VAE. We use one critic D attr to predict which regions of the latent space will generate outputs with desired attributes, and another critic D realism to predict which regions have high mass under the marginal posterior, q(z), of the training data. (b) We begin by pretraining a standard VAE, with an emphasis on achieving good reconstructions. (c) To train the actor-critic pair we use constraint-satisfaction labels, c, to train D to discriminate between encodings of actual data, z ∼ q(z|x), versus latent vectors z ∼ p(z) sampled from the prior or transformed prior samples G(z ∼ p(z), y). Similar to a Conditional GAN, both G and D operate on a concatenation of z and a binary attribute vector, y, allowing G to learn conditional mappings in latent space. If G is an optimizer, a separate attribute discriminator, D attr is trained and the latent vector is optimized to reduce the cost of both D attr and D realism. (d) To sample from the intersection of these regions, we use either gradient-based optimization or an amortized generator, G, to shift latent samples from either the prior (z ∼ p(z), sampling) or from the data (z ∼ q(z|x), transformation).• We show that it is possible to generate conditionally from an unconditional model, learning a critic function D(z) in latent space and generating high-value samples with either gradient-based optimization or an amortized actor function G(z), even with a nondifferentiable decoder (e.g., discrete sequences).• Focusing on VAEs, we address the tradeoff between reconstruction quality and sample quality (without sacrificing diversity) by enforcing a universal "realism" constraint that requires samples in latent space to be indistinguishable from encoded data (rather than prior samples).• Because we start from a VAE that can reconstruct inputs well, we are able to apply identitypreserving transformations by making the minimal adjustment in latent space needed to satisfy the desired constraints. For example, when we adjust a person's expression or hair, the is still clearly identifiable as the same person (see Figure 5). This contrasts with pure GAN-based transformation approaches, which often fail to preserve identity.• Zero-shot conditional generation. Using samples from the VAE to generate exemplars, we can learn an actor-critic pair that satisfies user-specified rule-based constraints in the absence of any labeled data. Decoder-based deep generative models such as VAEs and GANs generate samples that approximate a population distribution p (x) by passing samples from some simple tractable distribution p(z) (often p(z) N (0, I)) through a deep neural network. GANs are trained to fool an auxiliary classifier that tries to learn to distinguish between real and synthetic samples. VAEs are fit to data using a variational approximation to maximum-likelihood estimation: DISPLAYFORM0 Figure 2: Typical VAEs use a pixel-wise data likelihood, N (µ x (z), σ x I), with σ x = 1 to produce coherent samples at the expense of visual and conceptual blurriness (Row 3). Some reconstructions (Row 2) actually change attributes of the original data. Decreasing σ x to 0.1 maximizes the ELBO (supplemental TAB3) and increases the fidelity of reconstructions (Row 4) at the cost of sample realism (Row 5). Using an actor to shift prior samples to satisfy the realism constraint, we achieve more realistic samples without sacrificing sharpness (Row 6). The samples are mapped to the closest point in latent space that both satisfies the realism constraint and has the same attributes as the original data.where the "encoder" distribution q(z | x) is an approximation to the posterior p(z | x), π(x; g(z)) p(x | z) is a tractable likelihood function that depends on some parameters output by a "decoder" function g(z), and q and g are fit to maximize the evidence lower bound (ELBO) L ELBO. The likelihood π(x; g) is often chosen to be a product of simple distributions such as π(x; g) = N (x; g, σ DISPLAYFORM1 GANs and VAEs have complementary strengths and weaknesses. GANs suffer from the "modecollapse" problem, where the generator assigns mass to a small subset of the support of the population distribution-that is, it may generate realistic samples, but there are many more realistic samples that it cannot generate. This is particularly problematic if we want to use GANs to manipulate data rather than generate new data; even GAN variants that include some kind of inference machinery (e.g., BID5 BID6 BID24 to determine what z best matches some x tend to produce reconstructions that are reminiscent of the input but do not preserve its identity. On the other hand, VAEs (especially those with simple likelihoods π) often exhibit a tradeoff between sharp reconstructions and sensible-looking samples (see Figure 2). That is, depending on what hyperparameters they are trained with (e.g., latent dimensionality and the scale of the likelihood term), VAEs tend to either produce blurry reconstructions and plausible (but blurry) novel samples, or bizarre samples but sharp reconstructions. It has been argued BID20 that this is due to the "holes" problem; the decoder is trained on samples from the marginal posterior q(z) 1 N n q(z | x n), which may have very high KL divergence to the presupposed marginal p(z) BID10. In particular, if the decoder, g(z), can reconstruct arbitrary values of x with high accuracy (as in the case of small σ x) then the typical posterior p(z | x) will be highly concentrated. We show this experimentally in supplemental FIG7. If q(z | x) underestimates the posterior variance (as it usually does), then the marginal posterior q(z) will also be highly concentrated, and samples from p(x) = z p(z)p(x | z)dz may produce that are far from typical reconstructions DISPLAYFORM2 If we tune σ x to maximize the ELBO BID1, we find the optimal σ x ≈ 0.1 (supplemental TAB3). Figure 2 shows that this choice does indeed lead to good reconstructions but strange-looking samples. We look at the two latent dimensions that have the lowest average posterior standard deviation on the training set, taking low variance in z space as a proxy for influence over the generated images. All other latent dimensions are held fixed at their original values (from a sample from p(z) on the left, and from a sample from q(z | x) for a held-out x on the right). Gray x marks correspond to the points in latent space of the generated images to the right. The cross-section on the left, taken from a prior sample, shows contours that point towards more realistic looking digits. In the cross-section on the right, a sample from the validation set (indicated by orange squares) resides within a local maximum of the critic, as one would hope. Conditional GANs (CGAN; BID21 and conditional VAEs (CVAE; BID28 can generate samples conditioned on attribute information when available, but they must be trained with knowledge of the attribute labels for the whole training set, and it is not clear how to adapt them to new attributes without retraining from scratch. Furthermore, CGANs and CVAEs suffer from the same problems of mode-collapse and blurriness as their unconditional cousins. We take a different approach to conditional generation and identity-preserving transformation. We begin by training an unconditional VAE with hyperparameters chosen to ensure good reconstruction (at the expense of sample quality). We then train a "realism" critic to predict whether a given z maps to a high-quality sample. We also train critics to predict whether a given z maps to a sample that manifests various attributes of interest. To generate samples that are both realistic and exhibit desired attributes, one option is to optimize random z vectors until they satisfy both the realism and attribute critics. Alternately, we can amortize this cost by training an "actor" network to map a random set of z vectors to a subregion of latent space that satisfies the constraints encoded by the critics. By encouraging these transformed z vectors to remain as close as possible to where they started, we alleviate the mode-collapse problem common to GANs. Our approach is summarized visually in Figure 1. The details follow in sections 3, 4, 5, and 6. We define the realism constraint implicitly as being satisfied by samples from the marginal posterior q(z) 1 N n q(z | x n) and not those from p(z). By enforcing this constraint, we can close the gap between reconstruction quality and sample quality (without sacrificing sample diversity).As shown in Figure 1, we can train a critic D to differentiate between samples from p(z) and q(z). The critic loss, L D (z), is simply the cross-entropy, with labels c = 1 for z ∼ q(z | x) and c = 0 for z ∼ p(z). We found that the realism critic had little trouble generalizing to unseen data; that is, it was able to recognize samples from q(z | x held−out) as being "realistic" FIG0 ).Sampling from the prior is sufficient to train D for models with lower KL Divergence, but if the KL Divergence between q and p is large, the chances of sampling a point p(z) that has high probability under q(z) becomes vanishingly small. This leads to poor sample quality and makes it difficult for D to learn a tight approximation of q(z) solely by sampling from p(z). Instead, we use an inner-loop of gradient-based optimization, DISPLAYFORM0, to move prior samples to points deemed more like q(z) by D. For clarity, we introduce the shorthand DISPLAYFORM1 ). This gives us our critic loss for the realism constraint: DISPLAYFORM2 Figure 4: Conditional generation with a CGAN actor-critic pair acting in the latent space of a VAE with σ x = 0.1. Each row starts from a different prior sample and maps it to a new point in latent space that satisfies both the attribute constraints and the realism constraint. The attribute constraints are changed one at a time to produce as smooth a transition as possible from left to right. The bottom CGAN is regularized during training to prefer small shifts in latent space (λ dist = 0.1), while the top is not (λ dist = 0.0). Compared to the images generated by the unregularized model, the images generated by the regularized model are much less diverse across columns, suggesting that the regularization does indeed enforce some degree of identity preservation. The regularized model produces images that are somewhat more diverse across rows, suggesting that the regularization fights mode collapse (arguably at the expense of image quality). For each column, the complete list of attributes is given in supplemental TAB2.Since this inner-loop of optimization can slow down training, we amortize the generation by using a neural network as a function approximator. There are many examples of such amortization tricks, including the encoder of a VAE, generator of a GAN, and fast neural style transfer BID31 BID18. As with a traditional GAN, the parameters of the function G are updated to maximize the value D ascribes to the shifted latent points. One of the challenges using a GAN in this situation is that it is prone to mode-collapse. However, an advantage of applying the GAN in latent space is that we can regularize G to try and find the closest point in latent space that satisfies D, thus encouraging diverse solutions. We introduce a regularization term, L dist (z, z) = 1/σ z 2 log(1 + (z − z) 2 ) to encourage nearby solutions, while allowing more exploration than a mean square error term. As a VAE utilizes only a fraction of its latent dimensions, we scale the distance penalty of each dimension by its utilization, as indicated by the squared reciprocal of the scale σ z (x) of the encoder distribution q(z | x), averaged over the training dataset, DISPLAYFORM3 We want to generate samples that are realistic, but we also want to control what attributes they exhibit. Given binary attribute labels y for a dataset, we can accomplish this by using a CGAN in Figure 5: Identity-preserving transformations with optimization. Two separate critics are trained, one for attributes and one for the realism constraint. Starting at the latent points corresponding to the data reconstructions, we then perform gradient ascent in latent space on a weighted combination of critic values (1.0 attribute, 0.1 marginal posterior), stopping when a threshold value is passed for both critics. Images remain semantically close to the original because the pixel-wise likelihood of VAE training encourages identity-preserving reconstructions, and the dynamics of gradient ascent are naturally limited to finding solutions close in latent space. Panels are black for attributes of the original image, as the procedure just returns the original point in latent space.the latent space, which amounts to replacing D(z) and G(z) with conditional versions D(z, y) and G(z, y) and concatenating y to z as input. If both the actor and critic see attribute information, G must find points in latent space that could be samples from q(z) with attributes y. This procedure is computationally inexpensive relative to training a generative model from scratch. In most of our experiments, we use a relatively large CGAN actor-critic pair (4 fully connected ReLU layers of 2048 units each), which during training uses about 96× fewer FLOPs/iteration than the unconditional VAE. We also trained a much smaller CGAN actor-critic pair (3 fully connected ReLU layers of 256 units), which uses about 2884× fewer FLOPs/iteration than the VAE, and achieves only slightly worse than the larger CGAN (supplemental FIG5 and TAB0). Figure 4 demonstrates the quality of conditional samples from a CGAN actor-critic pair and the effect of the distance penalty, which constrains generation to be closer to the prior sample, maintaining similarity between samples with different attributes. The regularized CGAN actor has less freedom to ignore modes by pushing many random z vectors to the same area of the latent space, since it is penalized for moving samples from p(z) too far. The increased diversity across rows of the regularized CGAN is evidence that this regularization does fight mode-collapse (additional qualitative evidence is in supplemental Figures 7 and 8). However, without a distance penalty, samples appear more a bit realistic with more prominent attributes. This is supported by TAB0: Accuracy of a separate model trained to classify attributes from images, evaluated on test data and generated images. We condition and evaluate the generated images on the same labels as the test data. For comparison, the of a similar task using invertible CGANs for generation BID24 are provided. However, since the full list of salient attributes was not given in the paper, we emphasize that they are not directly comparable as the two experiments use a slightly different set of attribute labels. We also measure the distance in latent space that prior samples are shifted, weighted by 1/σ z 2. Actors trained with a latent distance penalty λ dist have slightly worse accuracy, but find latent points much closer to the prior samples and produce a greater diversity of images (see supplemental Figures 7 and 8). Interestingly, an actor trained without a distance penalty achieves higher classification accuracy than the test set itself, possibly by generating images with more exaggerated and distinctive features than real data. A "small model" CGAN with 85x fewer parameters (3 fully connected layers of 256 units) generates images (supplemental FIG5 of comperable quality. Due to the smaller capacity, the model finds more local solutions (smaller z M SE) that have slightly less attribute accuracy, but are more visually similar to the prior sample without an explicit regularization term. Although we used a VAE as the base generative model, our approach could also be used to generate high-quality conditional samples from pretrained classical autoencoders. We show in supplemental FIG6 that we obtain reasonably good conditional samples (albeit with high-frequency spatial artifacts) as σ x → 0 (equivalent to a classical autoencoder). Learning the decoder using VAE training encourages q(z) to fill up as much of the latent space as possible (without sacrificing reconstruction quality), which in turn encourages the decoder to map more of the latent space to reasonable-looking images. The prior p(z) = N (0, I) also imposes a natural scale on the latent variables. If we have a VAE that can produce good reconstructions of held-out data, we can transform the attributes of the output by gradient-based optimization. We simply need to train a critic, D attr (z), to predict the attribute labels p(y | z) of the data embeddings z ∼ q(z | x), and use a cross-entropy loss to train. Then, starting from a data point, z ∼ q(z | x), we can perform gradient descent on the the realism constraint and attribute constraint jointly, L D real (z) + λ attr L Dattr (z). Note that it is helpful to maintain the realism constraint to keep the image from distorting unrealistically. Using the same procedure, we can also conditionally generate new samples (supplemental Figure 9) by starting from z ∼ p(z). Figure 5 demonstrates transformations applied to samples from the held-out evaluation dataset. Note that since the reconstructions are close to the original images, the transformed images also maintain much of their structure. This contrasts with supplemental Figure 10, where a distance-penalty-free CGAN actor produces transformations that share attributes with the original but shift identity. We could preserve identity by introducing a distance penalty, but find that it is much easier to find the correct weighting of realism cost, attribute cost, and distance penalty through optimization, as each combination does not require retraining the network. So far, we have assumed access to labeled data to train attribute classifiers. We can remove the need to provide labeled examples by leveraging the structure learned by our pre-trained model, using it to generate exemplars that are scored by a user-supplied reward function. If we constrain the reward Figure 6: Transformations from a prior sample for the Melody VAE model. In each 16-bar pianoroll, time is in the horizontal direction and pitch in the vertical direction. In the prior sample, notes falling outside of the C Major scale are shown in red. After transformation by G P=CMaj,d=0, all sampled notes fall within the scale, without a significant change to note density. After transformation of the original z by G P=CMaj,d=192, all sampled notes lay within the scale and the density increases beyond 192. Synthesized audio of these samples can be heard at https://goo.gl/ouULt9. function to be bounded, c(x): R N →, the problem becomes very similar to previous GAN settings, but now the actor, G, and critic, D, are working together. D aims to best approximate the true value of each latent state, E x∼p(x|z) c(x), and G aims to shift samples from the prior to highvalue states. The critic loss is the cross-entropy from c(x), and the actor loss is the same as L G in equation 3, where we again have a distance penalty to promote diversity of outputs. Note that the reward function and VAE decoder need not necessarily be differentiable, as the critic learns a value function to approximate the reward, which the actor uses for training. To highlight this, we demonstrate that the output of a recurrent VAE model can be constrained to satisfy hardcoded rule-based constraints. We first train an LSTM VAE (details in the Appendix) on melodic fragments. Each melody, m, is represented as a sequence of categorical variables. In order to examine our ability to constrain the pitch classes and note density of the outputs, we define two reward functions, one that encourages notes from a set of pitches P, and another for that encourages melodies to have at least d notes: Figure 6 gives an example of controlling the pitch class and note density of generated outputs, which is quantitatively supported by the in Table 2. During training, the actor goes through several phases of exploration and exploitation, oscillating between expanding to find new modes with high reward and then contracting to find the nearest locations of those modes, eventually settling into high value states that require only small movements in the latent space (supplemental Figure 11). DISPLAYFORM0 Conditional GANs BID21 and VAEs BID28 introduce conditioning variables at training time. BID28 allow these variables to affect the distribution in latent z space, but still require that p(z | y) be a tractable distribution. BID24 use CGANs to adjust images, but because CGANs cannot usually reconstruct arbitrary inputs accurately, they must resort to image-space processing techniques to transfer effects to the original input. BID32 propose adding "attribute vectors" to samples from p(z) as a simple and effective heuristic to perform transformations, which relies heavily on the linearity of the latent space. Table 2: Average rewards and constraint satisfaction rates (in parentheses) for unconditional (Prior) and conditional generation. Samples from the prior receive low rewards, on average, and near zero satisfaction rates from both the pitch class (C Major) and note density (≥ 192 notes) constraints. After applying an actor optimized only for the C Major scale (G P=CMaj,d=0), the pitch class constraint is fully satisfied 70.8% of the time with only a minor effect on density. The average value close to 1 also indicates that when the constraint is not satisfied, it is typically off by only a few notes. Applying an actor function optimized for the C Major scale and high density (G P=CMaj,d=192) causes both constraints to be satisfied at high rates, with a slightly larger shift in latent space. Some recent work has focused on applying more expressive prior constraints to VAEs BID26 BID29 BID3 BID30. The prior that maximizes the ELBO is p (z) = q(z) BID10; one can interpret our realism constraint as trying to find an implicit distribution that is indistinguishable from q(z). Like the adversarial autoencoder of BID20, our realism constraint relies on a discriminative model, but instead of trying to force q(z) to equal some simple p(z), we only weakly constrain q(z) and then use a classifier to "clean up" our . Like this work, the recently proposed adversarially regularized autoencoder BID13 uses adversarial training to generate latent codes in a latent space discovered by an autoencoder; that work focuses on unconditional generation. Gómez-Bombarelli et al. FORMULA0 train classifiers in the latent space of a VAE to predict what latent variables map to molecules with various properties, and then use iterative gradient-based optimization in the latent space to find molecules that have a desired set of properties. On molecule data, their procedure generates invalid molecules rarely enough that they can simply reject these samples, which are detected using off-the-shelf software. By contrast, the probability of generating realistic images under our pretrained VAE is astronomically small, and no simple criterion for detecting valid images exists. BID11 also use a classifier to constrain generation; they use a Deep Q-network as an auxiliary loss for training an LSTM. Closest to Section 6, BID22 generate very high quality conditional images by optimizing a sample from the latent space of a generative network to create an image that maximizes the class activations of a pretrained ImageNet classifier. Our work differs in that we learn an amortized generator/discriminator directly in the latent space and we achieve diversity through regularizing by the natural scale of the latent space rather than through a modified Langevin sampling algorithm. We have demonstrated a new approach to conditional generation by constraining the latent space of an unconditional generative model. This approach could be extended in a number of ways. One possibility would be to plug in different architectures, including powerful autoregressive decoders or adversarial decoder costs, as we make no assumptions specific to independent likelihoods. While we have considered constraints based on implicit density estimation, we could also estimate the constrained distribution directly with an explicit autoregressive model or another variational autoencoder. The efficacy of autoregressive priors in VAEs is promising for this approach BID16. Conditional samples could then be obtained by ancestral sampling, and transformations by using gradient ascent to increase the likelihood under the model. Active or semisupervised learning approaches could reduce the sample complexity of learning constraints. Real-time constraint learning would also enable new applications; it might be fruitful to extend the reward approximation of Section 6 to incorporate user preferences as in BID4. For images, we use the MNIST digits dataset BID17 and the Large-scale CelebFaces Attributes (CelebA) dataset BID19. MNIST images are 28×28 pixels and greyscale scaled to. For attributes, we use the number class label of each digit. CelebA images are centercropped to 128 × 128 pixels and then downsampled to 64 × 64 RGB pixels and scaled to. We find that many of the attribute labels are not strongly correlated with changes in the images, so we narrow the original 40 attributes to the 10 most visually salient: blond hair, black hair, brown hair, bald, eyeglasses, facial hair, hat, smiling, gender, and age. For melodies, we scraped the web to collect over 1.5 million publicly available MIDI files. We then extracted 16-bar melodies by sliding a window with a single bar stride over each non-percussion instrument with a 4 4 time signature, keeping only the note with the highest pitch when multiple overlap. This produced over 3 million unique melodies. We represent each melody as a sequence of 256 (16 per bar) categorical variables taking one of 130 discrete states at each sixteenth note: 128 note-on pitches, a hold state, and a rest state. All encoders, decoders, and classifiers are trained with the Adam optimizer BID14, with learning rate = 3e-4, β 1 = 0.9, and β 2 = 0.999.To train D real (z), D attr (z) and G(z) we follow the training procedure of BID9, applying a gradient penalty of 10, training D and G in a 10:1 step ratio, and use the Adam optimizer with learning rate = 3e-4, β 1 = 0.0, and β 2 = 0.9. While not necessary to converge, we find it improves the stability of optimization. We do not apply any of the other tricks of GAN training such as batch normalization, minibatch discrimination, or one-sided label smoothing BID25. As samples from p(z) are easier to discriminate than samples from G(p(z)), we train D by sampling from p(z) at a rate 10 times less than G(p(z)). For actors with inner-loop optimization, G opt, 100 iterations of Adam are used with with learning rate = 1e-1, β 1 = 0.9, and β 2 = 0.999. To model the MNIST data, we use a deep feed-forward neural network FIG0 ).The encoder is a series of 3 linear layers with 1024 outputs, each followed by a ReLU, after which an additional linear layer is used to produce 2048 outputs. Half of the outputs are used as the µ and the softplus of the other half are used as the σ to parameterize a 1024-dimension multivariate Gaussian distribution with a diagonal covariance matrix for z. The decoder is a series of 3 linear layers with 1024 outputs, each followed by a ReLU, after which an additional linear layer is used to produce 28x28 outputs. These outputs are then passed through a sigmoid to generate the output image. To model the CelebA data, we use a deep convolutional neural network FIG0 ).The encoder is a series of 4 2D convolutional layers, each followed by a ReLU. The convolution kernels are of size 3 × 3, 3 × 3, 5 × 5, and 5 × 5, with 2048, 1024, 512, and 256 output channels, respectively. All convolutional layers have a stride of 2. After the final ReLU, a linear layer is used to produce 2048 outputs. Half of the outputs are used as the µ and the softplus of the other half are used as the σ to parameterize a 1024-dimension multivariate Gaussian distribution with a diagonal covariance matrix for z. The decoder passes the z through a 4x4x2048 linear layer, and then a series of 4 2D transposed convolutional layers, all but the last of which are followed by a ReLU. The deconvolution kernels are of size 5×5, 5×5, 3×3, and 3×3, with 1024, 512, 256, and 3 output channels, respectively. All deconvolution layers have a stride of 2. The output from the final deconvolution is passed through a sigmoid to generate the output image. The classifier that is trained to predict labels from images are identical to the VAE encoders except that they end with a sigmoid cross-entropy loss. Music is fundamentally sequential, so we use an LSTM-based sequence VAE for modelling monophonic melodies FIG0 ).The encoder is made up of a single-layer bidirectional LSTM, with 2048 units per cell. The final output in each direction is concatenated and passed through a linear layer to produce 1024 outputs. Half of the outputs are used as the µ and the softplus of the other half are used as a σ to parameterize a 512-dimension multivariate Gaussian distribution with a diagonal covariance matrix for z. Since musical sequences often have structure at the bar level, we use a hierarchical decoder to model long melodies. First, the z goes through a linear layer to initialize the state of a 2-layer LSTM with 1024 units per layer, which outputs 16 embeddings of size 512 each, one per bar. Each of these embeddings are passed through a linear layer to produce 16 initial states for another 2-layer LSTM with 1024 units per layer. This bar-level LSTM autoregressively produces individual sixteenth note events, passing its output through a linear layer and softmax to create a distribution over the 130 classes. This categorical distribution is used to compute a cross-entropy loss during training or samples at inference time. In addition to generating the initial state at the start of each bar, the embedding for the current bar is concatenated with the previous output as the input at each time step. For G(z), we use a deep feed-forward neural network (Figure 12a) in all of our experiments. The network is a series of 4 linear layers with 2048 outputs, each followed by a ReLU, after which an additional linear layer is used to produce 2 * dim(z) outputs. Half of the outputs are used as the δz and the sigmoid of the other half are used as gates. The transformed z is the computed as (1 − gates) * z + gates * δz. This aids in training as the network only has to then predict shifts in z. When conditioning on attribute labels, y, to compute G(z, y), the labels are passed through a linear layer producing 2048 outputs which are concatenated with z as the model input. For D(z), we use a deep feed-forward neural network (Figure 12b) in all of our experiments. The network is a series of 4 linear layers with 2048 outputs, each followed by a ReLU, after which an additional linear layer is used to produce a single output. This output is passed through a sigmoid to compute D(z).When conditioning on attribute labels, y, to compute D(z, y), the labels are passed through a linear layer producing 2048 outputs which are concatenated with z as the model input. Figure 7: Additional generated CelebA faces by G CGAN with λ dist = 0. Full attribute labels are given in supplementary TAB2 Figure 8: Additional generated CelebA faces by G CGAN with λ dist = 0.1. Full attribute labels are given in supplementary TAB2 Figure 9: Optimization of samples drawn from the prior to satisfy both the realism constraint and attribute constraints (drawn from the test set). The optimization takes 100 steps, and images are shown at 0, 10, 30, 50 and 100 steps. D is trained with inner-loop optimization, G opt, as described in Section 9.2 Figure 10: Identity-distorting transformations with CGAN actor-critic. Without a penalty to encourage small moves in latent space, the actor maps the latent vectors of the original data points to generated images that have the correct attributes, but a different identity. Panels are black for attributes of the original image, as the procedure just returns the same image as the reconstruction. | x), averaged over the training set per a dimension. The x-axis is sorted from lowest to highest variance. Tighter posteriors correspond to more utilization of the latent dimension, and we scale our distance regularization the square inverse on a per-dimension basis.

A new approach to conditional generation by constraining the latent space of an unconditional generative model.

represent the short summary for finding the corresponding scientific article (long)!
Examples:
Provided: "In this paper, we proposed a novel algorithm, GenDICE, for general stationary distribution correction estimation, which can handle both discounted and average off-policy evaluation on multiple behavior-agnostic samples." Match: "An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain. In many real-world applications, access to the underlying transition operator is limited to a fixed set of data that has already been collected, without additional interaction with the environment being available. We show that consistent estimation remains possible in this scenario, and that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions, derived from fundamental properties of the stationary distribution, and exploiting constraint reformulations based on variational divergence minimization. The ing algorithm, GenDICE, is straightforward and effective. We prove the consistency of the method under general conditions, provide a detailed error analysis, and demonstrate strong empirical performance on benchmark tasks, including off-line PageRank and off-policy policy evaluation. Estimation of quantities defined by the stationary distribution of a Markov chain lies at the heart of many scientific and engineering problems. Famously, the steady-state distribution of a random walk on the World Wide Web provides the foundation of the PageRank algorithm . In many areas of machine learning, Markov chain Monte Carlo (MCMC) methods are used to conduct approximate Bayesian inference by considering Markov chains whose equilibrium distribution is a desired posterior . An example from engineering is queueing theory, where the queue lengths and waiting time under the limiting distribution have been extensively studied . As we will also see below, stationary distribution quantities are of fundamental importance in reinforcement learning (RL) (e.g.,). Classical algorithms for estimating stationary distribution quantities rely on the ability to sample next states from the current state by directly interacting with the environment (as in on-line RL or MCMC), or even require the transition probability distribution to be given explicitly (as in PageRank). Unfortunately, these classical approaches are inapplicable when direct access to the environment is not available, which is often the case in practice. There are many practical scenarios where a collection of sampled trajectories is available, having been collected off-line by an external mechanism that chose states and recorded the subsequent next states. Given such data, we still wish to estimate a stationary quantity. One important example is off-policy policy evaluation in RL, where we wish to estimate the value of a policy different from that used to collect experience. Another example is off-line PageRank (OPR), where we seek to estimate the relative importance of webpages given a sample of the web graph. Motivated by the importance of these off-line scenarios, and by the inapplicability of classical methods, we study the problem of off-line estimation of stationary values via a stationary distribution corrector. Instead of having access to the transition probabilities or a next-state sampler, we assume only access to a fixed sample of state transitions, where states have been sampled from an unknown distribution and next-states are sampled according to the Markov chain's transition operator. This off-line setting is distinct from that considered by most MCMC or on-line RL methods, where it is assumed that new observations can be continually sampled by demand from the environment. The off-line setting is indeed more challenging than its more traditional on-line counterpart, given that one must infer an asymptotic quantity from finite data. Nevertheless, we develop techniques that still allow consistent estimation under general conditions, and provide effective estimates in practice. The main contributions of this work are: • We formalize the problem of off-line estimation of stationary quantities, which captures a wide range of practical applications. • We propose a novel stationary distribution estimator, GenDICE, for this task. The ing algorithm is based on a new dual embedding formulation for divergence minimization, with a carefully designed mechanism that explicitly eliminates degenerate solutions. • We theoretically establish consistency and other statistical properties of GenDICE, and empirically demonstrate that it achieves significant improvements on several behavior-agnostic offpolicy evaluation benchmarks and an off-line version of PageRank. The methods we develop in this paper fundamentally extend recent work in off-policy policy evaluation by introducing a new formulation that leads to a more general, and as we will show, more effective estimation method. We first introduce off-line PageRank (OPR) and off-policy policy evaluation (OPE) as two motivating domains, where the goal is to estimate stationary quantities given only off-line access to a set of sampled transitions from an environment. The celebrated PageRank algorithm defines the ranking of a web page in terms of its asymptotic visitation probability under a random walk on the (augmented) directed graph specified by the hyperlinks. If we denote the World Wide Web by a directed graph G = (V, E) with vertices (web pages) v ∈ V and edges (hyperlinks) (v, u) ∈ E, PageRank considers the random walk defined by the Markov transition operator v → u: P (u|v) = (1−η) where |v| denotes the out-degree of vertex v and η ∈ is a probability of "teleporting" to any page uniformly. Define d t (v):= P (s t = v|s 0 ∼ µ 0, ∀i < t, s i+1 ∼ P(·|s i)), where µ 0 is the initial distribution over vertices, then the original PageRank algorithm explicitly iterates for the limit The classical version of this problem is solved by tabular methods that simulate Equation 1. However, we are interested in a more scalable off-line version of the problem where the transition model is not explicitly given. Instead, consider estimating the rank of a particular web page v from a large web graph, given only a sample D = {(v, u) from a random walk on G as specified above. We would still like to estimate d(v) based on this data. First, note that if one knew the distribution p by which any vertex v appeared in D, the target quantity could be re-expressed by a simple importance ratio d Policy Evaluation (Preliminaries) An important generalization of this stationary value estimation problem arises in RL in the form of policy evaluation. Consider a Markov Decision Process (MDP) M = S, A, P, R, γ, µ 0 , where S is a state space, A is an action space, P (s |s, a) denotes the transition dynamics, R is a reward function, γ ∈ is a discounted factor, and µ 0 is the initial state distribution. Given a policy, which chooses actions in any state s according to the probability distribution π(·|s), a trajectory β = (s 0, a 0, r 0, s 1, a 1, r 1, . . .) is generated by first sampling the initial state s 0 ∼ µ 0, and then for t ≥ 0, a t ∼ π(·|s t), r t ∼ R(s t, a t), and s t+1 ∼ P(·|s t, a t). The value of a policy π is the expected per-step reward defined as: Here the expectation is taken with respect to the randomness in the state-action pair P (s |s, a) π (a |s) and the reward R (s t, a t). Without loss of generality, we assume the limit exists for the average case, and hence R(π) is finite. Behavior-agnostic Off-Policy Evaluation (OPE) A natural version of policy evaluation that often arises in practice is to estimate, where p (s, a) is an unknown distribution induced by multiple unknown behavior policies. This problem is different from the classical form of OPE, where it is assumed that a known behavior policy π b is used to collect transitions; in the behavior-agnostic scenario we are considering here, standard importance sampling (IS) estimators cannot be applied. Let d π t (s, a) = P (s t = s, a t = a|s 0 ∼ µ 0, ∀i < t, a i ∼ π (·|s i), s i+1 ∼ P(·|s i, a i)). The stationary distribution can then be defined as From this definition note that R(π) and R γ (π) can be equivalently re-expressed as Here we see once again that if we had the correction ratio function τ (s, a) = is an empirical estimate of p (s, a). In this way, the behavior-agnostic OPE problem can be reduced to estimating the correction ratio function τ, as above. We note that and also exploit Equation 5 to reduce OPE to stationary distribution correction, but these prior works are distinct from the current proposal in different ways. First, the inverse propensity score (IPS) method of assumes the transitions are sampled from a single behavior policy, which must be known beforehand; hence that approach is not applicable in behavior-agnostic OPE setting. Second, the recent DualDICE algorithm is also a behavior-agnostic OPE estimator, but its derivation relies on a change-of-variable trick that is only valid for γ < 1. This previous formulation becomes unstable when γ → 1, as shown in Section 6 and Appendix E. The behavior-agnostic OPE estimator we derive below in Section 3 is applicable both when γ = 1 and γ ∈. This connection is why we name the new estimator GenDICE, for GENeralized stationary DIstribution Correction Estimation. As noted, there are important estimation problems in the Markov chain and MDP settings that can be recast as estimating a stationary distribution correction ratio. We first outline the conditions that characterize the correction ratio function τ, upon which we construct the objective for the GenDICE estimator, and design efficient algorithm for optimization. We will develop our approach for the more general MDP setting, with the understanding that all the methods and can be easily specialized to the Markov chain setting. The stationary distribution µ π γ defined in Equation 4 can also be characterized via At first glance, this equation shares a superficial similarity to the Bellman equation, but there is a fundamental difference. The Bellman operator recursively integrates out future (s, a) pairs to characterize a current pair (s, a) value, whereas the distribution operator T defined in Equation 6 operates in the reverse temporal direction. When γ ≤ 1, Equation 6 always has a fixed-point solution. For γ = 1, in the discrete case, the fixedpoint exists as long as T is ergodic; in the continuous case, the conditions for fixed-point existence become more complicated and beyond the scope of this paper. The development below is based on a divergence D and the following default assumption. Assumption 1 (Markov chain regularity) For the given target policy π, the ing state-action transition operator T has a unique stationary distribution µ that satisfies D(T • µ µ) = 0. In the behavior-agnostic setting we consider, one does not have direct access to P for element-wise evaluation or sampling, but instead is given a fixed set of samples from P (s |s, a) p (s, a) with respect to some distribution p (s, a) over S × A. Define T p γ,µ0 to be a mixture of µ 0 π and T p; i.e., let Obviously, conditioning on (s, a, s) one could easily sample a ∼ π (a |s) to form (s, a, s, a) ∼ T p ((s, a), (s, a)); similarly, a sample (s, a) ∼ µ 0 (s) π (a |s) could be formed from s. Mixing such samples with probability γ and 1 − γ respectively yields a sample (s, a, s, a) ∼ T p γ,µ0 ((s, a), (s, a)). Based on these observations, the stationary condition for the ratio from Equation 6 can be re-expressed in terms of T p γ,µ0 as where is the correction ratio function we seek to estimate. One natural approach to estimating τ * is to match the LHS and RHS of Equation 8 with respect to some divergence D (· ·) over the empirical samples. That is, we consider estimating τ * by solving the optimization problem min Although this forms the basis of our approach, there are two severe issues with this naive formulation that first need to be rectified: • τ, which in general involves an intractable integral. Thus, evaluation of the exact objective is intractable, and neglects the assumption that we only have access to samples from T p γ,µ0 and are not able to evaluate it at arbitrary points. We address each of these two issues in a principled manner. To avoid degenerate solutions when γ = 1, we ensure that the solution is a proper density ratio; that is, the property τ ∈ Ξ:= {τ (·) ≥ 0, E p [τ] = 1} must be true of any τ that is a ratio of some density to p. This provides an additional constraint that we add to the optimization formulation min With this additional constraint, it is obvious that the trivial solution τ (s, a) = 0 is eliminated as an infeasible point of Eqn, along with other degenerate solutions τ (s, a) = cτ * (s, a) with c = 1. Unfortunately, exactly solving an optimization with expectation constraints is very complicated in general , particularly given a nonlinear parameterization for τ. The penalty method provides a much simpler alternative, where a sequence of regularized problems are solved min with λ increasing. The drawback of the penalty method is that it generally requires λ → ∞ to ensure the strict feasibility, which is still impractical, especially in stochastic gradient descent. The infinite λ may induce unbounded variance in the gradient estimator, and thus, divergence in optimization. However, by exploiting the special structure of the solution sets to Equation 11, we can show that, remarkably, it is unnecessary to increase λ. Theorem 1 For γ ∈ and any λ > 0, the solution to Equation 11 is given by τ * (s, a) = u(s,a) p(s,a). The detailed proof for Theorem 1 is given in Appendix A.1. By Theorem 1, we can estimate the desired correction ratio function τ * by solving only one optimization with an arbitrary λ > 0. The optimization in Equation 11 involves the integrals T p γ,µ0 • τ and E p [τ] inside nonlinear loss functions, hence appears difficult to solve. Moreover, obtaining unbiased gradients with a naive approach requires double sampling . Instead, we bypass both difficulties by applying a dual embedding technique (; . In particular, we assume the divergence D is in the form of an f -divergence ds da where φ (·): R + → R is a convex, lower-semicontinuous function with φ = 0. Plugging this into J (τ) in Equation 11 we can easily check the convexity of the objective Theorem 2 For an f -divergence with valid φ defining D φ, the objective J (τ) is convex w.r.t. τ. The detailed proof is provided in Appendix A.2. Recall that a suitable convex function can be represented as φ (x) = max f x·f −φ * (f), where φ * is the Fenchel conjugate of φ (·). In particular, we have the representation 1 2 x 2 = max u ux − 1 2 u 2, which allows us to re-express the objective as Applying the interchangeability principle , one can replace the inner max in the first term over scalar f to maximize over a function f (·, ·): This yields the main optimization formulation, which avoids the aforementioned difficulties and is well-suited for practical optimization as discussed in Section 3.4. In addition to f -divergence, the proposed estimator Equation 11 is compatible with other divergences, such as the integral probability metrics (IPM) (Müller, 1997;), while retaining consistency. Based on the definition of the IPM, these divergences directly lead to min-max optimizations similar to Equation 13 with the identity function as φ * (·) and different feasible sets for the dual functions. Specifically, maximum mean discrepancy (MMD) requires f H k ≤ 1 where H k denotes the RKHS with kernel k; the Dudley metric requires f BL ≤ 1 where f BL:= f ∞ + ∇f 2; and Wasserstein distance requires ∇f 2 ≤ 1. These additional requirements on the dual function might incur some extra difficulty in practice. For example, with Wasserstein distance and the Dudley metric, we might need to include an extra gradient penalty , which requires additional computation to take the gradient through a gradient. Meanwhile, the consistency of the surrogate loss under regularization is not clear. For MMD, we can obtain a closed-form solution for the dual function, which saves the cost of the inner optimization , but with the tradeoff of requiring two independent samples in each outer optimization update. Moreover, MMD relies on the condition that the dual function lies in some RKHS, which introduces additional kernel parameters to be tuned and in practice may not be sufficiently flexible compared to neural networks. We have derived a consistent stationary distribution correction estimator in the form of a min-max saddle point optimization Equation 13. Here, we present a practical instantiation of GenDICE with a concrete objective and parametrization. We choose the χ 2 -divergence, which is an f -divergence with φ (x) = (x − 1) 2 and φ There two major reasons for adopting χ 2 -divergence: i) In the behavior-agnostic OPE problem, we mainly use the ratio correction function for estimating, which is an expectation. Recall that the error between the estimate and ground-truth can then be bounded by total variation, which is a lower bound of χ 2 -divergence. ii) For the alternative divergences, the conjugate of the KL-divergence involves exp (·), which may lead to instability in optimization; while the IPM variants introduce extra constraints on dual function, which may be difficult to be optimized. The conjugate function of χ 2 -divergence en-joys suitable numerical properties and provides squared regularization. We have provided an additional empirical ablation study that investigates the alternative divergences in Appendix E.3. To parameterize the correction ratio τ and dual function f we use neural networks, τ (s, a) = nn wτ (s, a) and f (s, a) = nn w f (s, a), where w τ and w f denotes the parameters of τ and f respectively. Since the optimization requires τ to be non-negative, we add an extra positive neuron, such as exp (·), log (1 + exp (·)) or (·) 2 at the final layer of nn wτ (s, a). We empirically compare the different positive neurons in Section 6.3. For these representations, and unbiased gradient estimator ∇ (wτ,u,w f) J (τ, u, f) can be obtained straightforwardly, as shown in Appendix B. This allows us to apply stochastic gradient descent to solve the saddle-point problem Equation 14 in a scalable manner, as illustrated in Algorithm 1. We provide a theoretical analysis for the proposed GenDICE algorithm, following a similar learning setting and assumptions to . The main is summarized in the following theorem. A formal statement, together with the proof, is given in Appendix C. Theorem 3 (Informal) Under mild conditions, with learnable F and H, the error in the objective between the GenDICE estimate,τ, to the solution τ where E [·] is w.r.t. the randomness in D and in the optimization algorithms, opt is the optimization error, and approx (F, H) is the approximation induced by (F, H) for parametrization of (τ, f). The theorem shows that the suboptimality of GenDICE's solution, measured in terms of the objective function value, can be decomposed into three terms: the approximation error approx, which is controlled by the representation flexibility of function classes; the estimation error due to sample randomness, which decays at the order of 1/ √ N; and the optimization error, which arises from the suboptimality of the solution found by the optimization algorithm. As discussed in Appendix C, in special cases, this suboptimality can be bounded below by a divergence betweenτ and τ *, and therefore directly bounds the error in the estimated policy value. There is also a tradeoff between these three error terms. With more flexible function classes (e.g., neural networks) for F and H, the approximation error approx becomes smaller. However, it may increase the estimation error (through the constant in front of 1/ √ N) and the optimization error (by solving a harder optimization problem). On the other hand, if F and H are linearly parameterized, estimation and optimization errors tend to be smaller and can often be upper-bounded explicitly in Appendix C.3. However, the corresponding approximation error will be larger. Off-policy evaluation with importance sampling (IS) has has been explored in the contextual bandits (; Dudík et al., 2011;), and episodic RL settings , achieving many empirical successes (e.g., ; Dudík et al., 2011;). Unfortunately, IS-based methods suffer from exponential variance in long-horizon problems, known as the "curse of horizon" . A few variancereduction techniques have been introduced, but still cannot eliminate this fundamental issue (; ;). By rewriting the accumulated reward as an expectation w.r.t. a stationary distribution,; recast OPE as estimating a correction ratio function, which significantly alleviates variance. However, these methods still require the off-policy data to be collected by a single and known behavior policy, which restricts their practical applicability. The only published algorithm in the literature, to the best of our knowledge, that solves agnostic-behavior off-policy evaluation is DualDICE . However, DualDICE was developed for discounted problems and its become unstable when the discount factor approaches 1 (see below). By contrast, GenDICE can cope with the more challenging problem of undiscounted reward estimation in the general behavior-agnostic setting. Note that standard model-based methods , which estimate the transition and reward models directly then calculate the expected reward based on the learned model, are also applicable to the behavior-agnostic setting considered here. Unfortunately, model-based methods typically rely heavily on modeling assumptions about rewards and transition dynamics. In practice, these assumptions do not always hold, and the evaluation can become unreliable. For more related work on MCMC, density ratio estimation and PageRank, please refer to Appendix F. In this section, we evaluate GenDICE on OPE and OPR problems. For OPE, we use one or multiple behavior policies to collect a fixed number of trajectories at some fixed trajectory length. This data is used to recover a correction ratio function for a target policy π that is then used to estimate the average reward in two different settings: i) average reward; and ii) discounted reward. In both settings, we compare with a model-based approach and step-wise weighted IS . We also compare to (referred to as "IPS" here) in the Taxi domain with a learned behavior policy 1. We specifically compare to DualDICE in the discounted reward setting, which is a direct and current state-of-the-art baseline. For OPR, the main comparison is with the model-based method, where the transition operator is empirically estimated and stationary distribution recovered via an exact solver. We validate GenDICE in both tabular and continuous cases, and perform an ablation study to further demonstrate its effectiveness. All are based on 20 random seeds, with mean and standard deviation plotted. Our code will be publicly available for reproduction. Offline PageRank on Graphs One direct application of GenDICE is off-line PageRank (OPR). We test GenDICE on a Barabasi-Albert (BA) graph (synthetic), and two realworld graphs, Cora and Citeseer. Details of the graphs are given in Appendix D. We use the log KL-divergence between estimated stationary distribution and the ground truth as the evaluation metric, with the ground truth computed by an exact solver based on the exact transition operator of the graphs. We compared GenDICE with modelbased methods in terms of the sample efficiency. From the in Figure 1, GenDICE outperforms the modelbased method when limited data is given. Even with 20k samples for a BA graph with 100 nodes, where a transition matrix has 10k entries, GenDICE still shows better performance in the offline setting. This is reasonable since Gen-DICE directly estimates the stationary distribution vector or ratio, while the model-based method needs to learn an entire transition matrix that has many more parameters. We use a similar taxi domain as in , where a grid size of 5 × 5 yields 2000 states in total (25 × 16 × 5, corresponding to 25 taxi locations, 16 passenger appearance status and 5 taxi status). We set the target policy to a final policy π after running tabular Q-learning for 1000 iterations, and set another policy π + after 950 iterations as the base policy. The behavior policy is a mixture controlled by α as π b = (1 − α)π + απ +. For the model-based method, we use a tabular representation for the reward and transition functions, whose entries are estimated from behavior data. For IS and IPS, we fit a policy via behavior cloning to estimate the policy ratio. In this specific setting, our methods achieve better compared to IS, IPS and the model-based method. Interestingly, with longer horizons, IS cannot improve as much as other methods even with more data, while GenDICE consistently improve and achieves much better than the baselines. DualDICE only works with γ < 1. GenDICE is more stable than DualDICE when γ becomes larger (close to 1), while still showing competitive performance for smaller discount factors γ. We further test our method for OPE on three control tasks: a discrete-control task Cartpole and two continuous-control tasks Reacher and HalfCheetah. In these tasks, observations (or states) are continuous, thus we use neural network function approximators and stochastic optimization. Since DualDICE has shown the state-of-the-art performance on discounted OPE, we mainly compare with it in the discounted reward case. We also compare to IS with a learned policy via behavior cloning and a neural model-based method, similar to the tabular case, but with neural network as the function approximator. All neural networks are feed-forward with two hidden layers of dimension 64 and tanh activations. More details can be found in Appendix D. Due to limited space, we put the discrete control in Appendix E and focus on the more challenging continuous control tasks. Here, the good performance of IS and model-based methods in Section 6.1 quickly deteriorates as the environment becomes complex, i.e., with a continuous action space. Note that GenDICE is able to maintain good performance in this scenario, even when using function approximation and stochastic optimization. This is reasonable because of the difficulty of fitting to the coupled policy-environment dynamics with a continuous action space. Here we also empirically validate GenDICE with off-policy data collected by multiple policies. As illustrated in Figure 3, all methods perform better with longer trajectory length or more trajectories. When α becomes larger, i.e., the behavior policies are closer to the target policy, all methods performs better, as expected. Here, GenDICE demonstrates good performance both on averagereward and discounted reward cases in different settings. The right two figures in each row show the log MSE curve versus optimization steps, where GenDICE achieves the smallest loss. In the discounted reward case, GenDICE shows significantly better and more stable performance than the strong baseline, DualDICE. Figure 4 also shows better performance of GenDICE than all baselines in the more challenging HalfCheetah domain. Each plot in the second row shows the average reward case. Finally, we conduct an ablation study on GenDICE to study its robustness and implementation sensitivities. We investigate the effects of learning rate, activation function, discount factor, and the specifically designed ratio constraint. We further demonstrate the effect of the choice of divergences and the penalty weight in Appendix E.3. Model-Based Importance Sampling DualDICE GenDICE (ours) Figure 4: Results on HalfCheetah. Plots from left to the right show the log MSE of estimated average per-step reward over different truncated lengths, numbers of trajectories, and behavior policies in discounted and average reward cases. Effects of the Learning Rate Since we are using neural network as the function approximator, and stochastic optimization, it is necessary to show sensitivity to the learning rate with {0.0001, 0.0003, 0.001, 0.003}, with in Figure 5. When α = 0.33, i.e., the OPE tasks are relatively easier and GenDICE obtains better at all learning rate settings. However, when α = 0.0, i.e., the estimation becomes more difficult and only GenDICE only obtains reasonable with the larger learning rate. Generally, this ablation study shows that the proposed method is not sensitive to the learning rate, and is easy to train. We further investigate the effects of the activation function on the last layer, which ensure the non-negative outputs required for the ratio. To better understand which activation function will lead to stable trainig for the neural correction estimator, we empirically compare using i) (·) 2; ii) log(1 + exp(·)); and iii) exp(·). In practice, we use the (·) 2 since it achieves low variance and better performance in most cases, as shown in Figure 5. We vary γ ∈ {0.95, 0.99, 0.995, 0.999, 1.0} to probe the sensitivity of GenDICE. Specifically, we compare to DualDICE, and find that GenDICE is stable, while DualDICE becomes unstable when the γ becomes large, as shown in Figure 6. GenDICE is also more general than DualDICE, as it can be applied to both the average and discounted reward cases. In Section 3, we highlighted the importance of the ratio constraint. Here we investigate the trivial solution issue without the constraint. The in Figure 6 demonstrate the necessity of adding the constraint penalty, since a trivial solution prevents an accurate corrector from being recovered (green line in left two figures). In this paper, we proposed a novel algorithm GenDICE for general stationary distribution correction estimation, which can handle both the discounted and average stationary distribution given multiple behavior-agnostic samples. Empirical on off-policy evaluation and offline PageRank show the superiority of proposed method over the existing state-of-the-art methods. the existence of the stationary distribution. Our discussion is all based on this assumption. Assumption 1 Under the target policy, the ed state-action transition operator T has a unique stationary distribution in terms of the divergence D (·||·). If the total variation divergence is selected, the Assumption 1 requires the transition operator should be ergodic, as discussed in. Theorem 1 For arbitrary λ > 0, the solution to the optimization Eqn is Proof For γ ∈, there is not degenerate solutions to D T p γ,µ0 • τ ||p · τ. The optimal solution is a density ratio. Therefore, the extra penalty E p(x) [τ (x)] − 1 2 does not affect the optimality for ∀λ > 0. negative, and the the density ratio p(x) leads to zero for both terms. Then, the density ratio is a solution to J (τ). For any other non-negative function τ (x) ≥ 0, if it is the optimal solution to J (τ), then, we have We denote µ (x) = p (x) τ (x), which is clearly a density function. Then, the optimal conditions in Equation 15 imply or equivalently, µ is the stationary distribution of T. We have thus shown the optimal τ (x) = is the target density ratio. Proof Since the φ is convex, we consider the Fenchel dual representation of the f -divergence 2 is also convex, which concludes the proof. Inputs: Convex function φ and its Fenchel conjugate φ, initial state s 0 ∼ µ 0, target policy π, distribution corrector nn wτ (·, ·), nn w f (·, ·), constraint scalar u, learning rates η τ, η f, η u, number of iterations K, batch size B. Sample actions a end for Return nn wτ. We provide the unbiased gradient estimator for ∇ wτ,u,w f J (τ, u, f) in Eqn below: Then, we have the psuedo code which applies SGD for solving Eqn. For convenience, we repeat here the notation defined in the main text. The saddle-point reformulation of the objective function of GenDICE is: To avoid the numerical infinity in D φ (·||·), we induced the bounded version as is still a valid divergence, and therefore the optimal solution τ * is still the stationary density ratio p(x). We denote the J (τ, µ, f) as the empirical surrogate of with optimal (f *, u *), and We apply some optimization algorithm for J (τ, u, f) over space (H, F, R), leading to the output τ, u, f. Under Assumption 2, we need only consider τ ∞ ≤ C, then, the corresponding dual u = E p (τ) − 1 ⇒ u ∈ U:= {|u| ≤ (C + 1)}. We choose the φ * (·) is a κ-Lipschitz continuous, then, the We consider the error betweenτ and τ * using standard arguments Remark: In some special cases, the suboptimality also implies the distance betweenτ and τ *. Specifically,for γ = 1, if the transition operator P π can be represented as P π = QΛQ −1 where Q denotes the (countable) eigenfunctions and Λ denotes the diagonal matrix with eigenvalues, the largest of which is 1. We consider φ (·) as identity and f ∈ F:= span (Q), f p,2 ≤ 1, then the d (τ, τ *) will bounded from below by a metric between τ and τ *. Particularly, we have Rewrite τ = ατ * + ζ, where ζ ∈ span Q \τ *, then Recall the optimality of τ We start with the following error decomposition: • For 1, we have. We consider the terms one-by-one. By definition, we have which is induced by introducing F for dual approximation. For the third term where we define est: Therefore, we can now bound 1 as We consider the terms from right to left. For the term J (τ * H) − J (τ *), we have, which is induced by restricting the function space to H. The second term is nonpositive, due to the optimality of (u *, f *). The final inequality comes from the fact that where the second term is nonpositive, thanks to the optimality ofτ * H. Finally, for the term J (τ * H) − J (τ * H), using the same argument in Equation 21, we have Therefore, we can bound 2 by 2 ≤ C φ,C,λ approx (H) + C P π,κ,λ (F) + 2 est. In sum, we have d (τ, τ *) ≤ 4 est +ˆ opt + 2C P π,κ,λ approx (F) + C φ,C,λ approx (H). In the following sections, we will bound the est andˆ opt. In this section, we analyze the statistical error We mainly focus on the batch RL setting with, which has been studied by previous authors (e.g., ;). However, as discussed in the literature (; ; ;), using the blocking technique of , the statistical error provided here can be generalized to β-mixing samples in a single sample path. We omit this generalization for the sake of expositional simplicity. To bound the est, we follow similar arguments by; via the covering number. For completeness, the definition is given below. The Pollard's tail inequality bounds the maximum deviation via the covering number of a function class: are i.i.d. samples from some distribution. Then, for any given > 0, The covering number can then be bounded in terms of the function class's pseudo-dimension: Lemma 5 (, Corollary 3) For any set X, any points x 1:N ∈ X N, any class F of functions on X taking values in [0, M] with pseudo-dimension D F < ∞, and any > 0, The statistical error est can be bounded using these lemmas. Lemma 6 (Stochastic error) Under the Assumption 2, if φ * is κ-Lipschitz continuous and the psuedo-dimension of H and F are finite, with probability at least 1 − δ, we have Proof The proof works by verifying the conditions in Lemma 4 and computing the covering number. 2 u 2, we will apply Lemma 4 with Z = Ω × Ω, Z i = (x i, x i), and G = h H×F ×U. We check the boundedness of h ζ,u,f (x, x). Based on Assumption 2, we only consider the τ ∈ H and u ∈ U bounded by C and C + 1. We also rectify the f ∞ ≤ C. Then, we can bound the h ∞: where C φ = max t∈[−C,C] −φ * (t). Thus, by Lemma 4, we have Next, we check the covering number of G. Firstly, we bound the distance in G, Denote the pseudo-dimension of H and F as D H and D F, respectively, we have, we obtain the bound for the statistical error: In this section, we investigate the optimization error. Notice our estimator min τ ∈H max f ∈F,u∈U J (τ, u, f) is compatible with different parametrizations for (H, F) and different optimization algorithms, the optimization error will be different. For the general neural network for (τ, f), although there are several progress recently (; ;) about the convergence to a stationary point or local minimum, it remains a largely open problem to quantify the optimization error, which is out of the scope of this paper. Here, we mainly discuss the convergence rate with tabular, linear and kernel parametrization for (τ, f). Particularly, we consider the linear parametrization particularly, i.e., τ (x) = σ w τ ψ (x), f (x) = w f ψ (x), and σ (·): R → R + is convex. There are many choices of the σ (·), e.g., exp (·), log (1 + exp (·)) and (·) 2. Obviously, even with such nonlinear mapping, the J (τ, u, f) is still convex-concave w.r.t (w τ, w f, u) by the convex composition rule. We can bound theˆ opt by the primal-dual gap gap: With vanilla SGD, we have, where T is the optimization steps ., where the E [·] is taken w.r.t. randomness in SGD. We are now ready to state the main theorm in a precise way: Under Assumptions 2 and 1, the stationary distribution µ exists, i.e., *, and the psuedo-dimension of H and F are finite, the error between the GenDICE estimate to τ where E [·] is w.r.t. the randomness in sample D and in the optimization algorithms. opt is the optimization error, and approx (F, H) is the approximation induced by (F, H) for parametrization of (τ, f). Proof We have the total error as where approx:= 2C T,κ,λ approx (F) + C φ,C,λ approx (H). For opt, we can apply the for SGD in Appendix C.3. We can bound the E [est] by Lemma 6. Specifically, we have Plug all these bounds into Equation 25, we achieve the . For the Taxi domain, we follow the same protocol as used in. The behavior and target policies are also taken from (referred in their work as the behavior policy for α = 0). We use a similar taxi domain, where a grid size of 5×5 yields 2000 states in total (25×16×5, cor-responding to 25 taxi locations, 16 passenger appearance status and 5 taxi status). We set our target policy as the final policy π * after running Q-learning for 1000 iterations, and set another policy π + after 950 iterations as our base policy. The behavior policy is a mixture policy controlled by α as π = (1 − α)π * + απ +, i.e., the larger α is, the behavior policy is more close to the target policy. In this setting, we solve for the optimal stationary ratio τ exactly using matrix operations. perform a similar exact solve for |S| variables µ(s), for better comparison we also perform our exact solve with respect to |S| variables τ (s). Specifically, the final objective of importance sampling will require knowledge of the importance weights µ(a|s)/p(a|s). For offline PageRank, the graph statistics are illustrated in Table 1, and the degree statistics and graph visualization are shown in Figure 7. For the BarabasiAlbert (BA) Graph, it begins with an initial connected network of m 0 nodes in the network. Each new node is connected to m ≤ m 0 existing nodes with a probability that is proportional to the number of links that the existing nodes already have. Intuitively, heavily linked nodes ('hubs') tend to quickly accumulate even more links, while nodes with only a few links are unlikely to be chosen as the destination for a new link. The new nodes have a'preference' to attach themselves to the already heavily linked nodes. For two real-world graphs, it is built upon the real-world citation networks. In our experiments, the weights of the BA graph is randomly drawn from a standard Gaussian distribution with normalization to ensure the property of the transition matrix. The offline data is collected by a random walker on the graph, which consists the initial state and next state in a single trajectory. In experiments, we vary the number of off-policy samples to validate the effectiveness of GenDICE with limited offline samples provided. We use the Cartpole, Reacher and HalfCheetah tasks as given by OpenAI Gym. In importance sampling, we learn a neural network policy via behavior cloning, and use its probabilities for computing importance weights π * (a|s)/π(a|s). All neural networks are feed-forward with two hidden layers of dimension 64 and tanh activations. Discrete Control Tasks We modify the Cartpole task to be infinite horizon: We use the same dynamics as in the original task but change the reward to be −1 if the original task returns a termination (when the pole falls below some threshold) and 1 otherwise. We train a policy on this task with standard Deep Q-Learning until convergence. We then define the target policy π * as a weighted combination of this pre-trained policy (weight 0.7) and a uniformly random policy (weight 0.3). The behavior policy π for a specific 0 ≤ α ≤ 1 is taken to be a weighted combination of the pre-trained policy (weight 0.55 + 0.15α) and a uniformly random policy (weight 0.45 − 0.15α). We train each stationary distribution correction estimation method using the Adam optimizer with batches of size 2048 and learning rates chosen using a hyperparameter search from {0.0001, 0.0003, 0.001, 0.003} and choose the best one as 0.0003. For the Reacher task, we train a deterministic policy until convergence via DDPG . We define the target policy π as a Gaussian with mean given by the pre-trained policy and standard deviation given by 0.1. The behavior policy π b for a specific 0 ≤ α ≤ 1 is taken to be a Gaussian with mean given by the pre-trained policy and standard deviation given by 0.4 − 0.3α. We train each stationary distribution correction estimation method using the Adam optimizer with batches of size 2048 and learning rates chosen using a hyperparameter search from {0.0001, 0.0003, 0.001, 0.003} and the optimal learning rate found was 0.003). For the HalfCheetah task, we also train a deterministic policy until convergence via DDPG . We define the target policy π as a Gaussian with mean given by the pre-trained policy and standard deviation given by 0.1. The behavior policy π b for a specific 0 ≤ α ≤ 1 is taken to be a Gaussian with mean given by the pre-trained policy and standard deviation given by 0.2 − 0.1α. We train each stationary distribution correction estimation method using the Adam optimizer with batches of size 2048 and learning rates chosen using a hyperparameter search from {0.0001, 0.0003, 0.001, 0.003} and the optimal learning rate found was 0.003. E.1 OPE FOR DISCRETE CONTROL On the discrete control task, we modify the Cartpole task to be infinite horizon: the original dynamics is used but with a modified reward function: the agent will receive −1 if the environment returns a termination (i.e., the pole falls below some threshold) and 1 otherwise. As shown in Figure 3, our method shows competitive with IS and Model-Based in average reward case, but our proposed method finally outperforms these two methods in terms of log MSE loss. Specifically, it is relatively difficult to fit a policy with data collected by multiple policies, which renders the poor performance of IS. In this section, we show more on the continuous control tasks, i.e., HalfCheetah and Reacher. Figure 9 shows the log MSE towards training steps, and GenDICE outperforms other baselines with different behavior policies. Figure 10 better illustrates how our method beat other baselines, and can accurately estimate the reward of the target policy. Besides, Figure 11 shows GenDICE gives better reward estimation of the target policy. In these figures, the left three figures show the performance with off-policy dataset collected by single behavior policy from more difficult to easier tasks. The right two figures show the , where off-policy dataset collected by multiple behavior policies. Figure 12 shows the ablation study in terms of estimated rewards. The left two figures shows the effects of different learning rate. When α = 0.33, i.e., the OPE tasks are relatively easier, GenDICE gets relatively good in all learning rate settings. However, when α = 0.0, i.e., the estimation becomes more difficult, only GenDICE in larger learning rate gets reasonable estimation. Interestingly, we can see with larger learning rates, the performance becomes better, and when learning rate is 0.001 with α = 0.0, the variance is very high, showing some cases the estimation becomes more accurate. The right three figures show different activation functions with different behavior policy. The square and softplus function works well; while the exponential function shows poor performance under some settings. In practice, we use the square function since its low variance and better performance in most cases. Model-Based Importance Sampling DualDICE GenDICE (ours) Oracle Figure 11: Results on HalfCheetah. Each plot in the first row shows the estimated average step reward over training and different behavior policies (higher α corresponds to a behavior policy closer to the target policy. Although any valid divergence between p · τ and T p γ,µ0 • τ in our estimator is consistent, which will always lead to the stationary distribution correction ratio asymptotically, and any λ > 0 will guarantee the normalization constraint, i.e., E p [τ] = 1, as we discussed in main text, different Figure 12: Results of ablation study with different learning rates and activation functions. The plots show the estimated average step reward over training and different behavior policies. choices of the divergences and λ may incur difficulty in the numerical optimization procedure. In this section, we investigate the empirical effects of the choice of f -divergence and IPM, and the weight of constrant regularization λ. To avoid the effects of other factors in the estimator, e.g., function parametrization, we focus on the offline PageRank task on BA graph with 100 nodes and 10k offline samples. All the performances are evaluated with 20 random trials. We test the GenDICE with several other alternative divergences, e.g., Wasserstein-1 distance, Jensen-Shannon divergence, KL-divergence, Hellinger divergence, and MMD. To ensure the dual function to be 1-Lipchitz, we add the gradient penalty. We use a learned Gaussian kernel in MMD, similar to. As we can see in Figure 13 (a), with these different divergences, the proposed GenDICE estimator can always achieve significantly better performance compared with the model-based estimator, showing that the GenDICE estimator is compatible with many different divergences. Most of the divergences, with appropriate extra techniques to handle the difficulties in optimization and carefully tuning for extra parameters, can achieve similar performances, consistent with phenomena in the variants of GANs . However, KL-divergence is an outlier, performing noticeably worse, which might be caused by the ill-behaved exp (·) in its conjugate function. The χ 2 -divergence and JS-divergence are better, which achieve good performances with fewer parameters to be tuned. The effect of the penalty weight λ is illustrated the in Figure 13(b). We vary the λ ∈ [0.1, 5] with χ 2 -divergence in the GenDICE estimator. Within a large range of λ, the performances of the proposed GenDICE are quite consistent, which justifies Theorem 1. The penalty multiplies with λ. Therefore, with λ increases, the variance of the stochastic gradient estimator also increases, which explains the variance increasing in large λ in Figure 13(b). In practice, λ = 1 is a reasonable choice for general cases. Markov Chain Monte Carlo Classical MCMC aims at sampling from µ π by iteratively simulting from the transition operator. It requires continuous interaction with the transition operator and heavy computational cost to update many particles. Amor-tized SVGD and Adversarial MCMC alleviate this issue via combining with neural network, but they still interact with the transition operator directly, i.e., in an on-policy setting. The major difference of our GenDICE is the learning setting: we only access the off-policy dataset, and cannot sample from the transition operator. The proposed GenDICE leverages stationary density ratio estimation for approximating the stationary quantities, which distinct it from classical methods. Density ratio estimation is a fundamental tool in machine learning and much related work exists. Classical density ratio estimation includes moment matching (Gretton et al.), probabilistic classification , and ratio matching (; ;). These classical methods focus on estimating the ratio between two distributions with samples from both of them, while GenDICE estimates the density ratio to a stationary distribution of a transition operator, from which even one sample is difficult to obtain. developed a reverse-time RL framework for PageRank via solving a reverse Bellman equation, which is less sensitive to graph topology and shows faster adaptation with graph change. still considers the online manner, which is different with our OPR setting."

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.