Question: Represent text

Question: Optimal Transport (OT) naturally arises in many machine learning applications, where we need to handle cross-modality data from multiple sources. Yet the heavy computational burden limits its wide-spread uses. To address the scalability issue, we propose an implicit generative learning-based framework called SPOT (Scalable Push-forward of Optimal Transport). Specifically, we approximate the optimal transport plan by a pushforward of a reference distribution, and cast the optimal transport problem into a minimax problem. We then can solve OT problems efficiently using primal dual stochastic gradient-type algorithms. We also show that we can recover the density of the optimal transport plan using neural ordinary differential equations. Numerical experiments on both synthetic and real datasets illustrate that SPOT is robust and has favorable convergence behavior. SPOT also allows us to efficiently sample from the optimal transport plan, which benefits downstream applications such as domain adaptation. The Optimal Transport (OT) problem naturally arises in a variety of machine learning applications, where we need to handle cross-modality data from multiple sources. One example is domain adaptation: We collect multiple datasets from different domains, and we need to learn a model from a source dataset, which can be further adapted to target datasets BID18 BID8. Another example is resource allocation: We want to assign a set of assets (one data source) to a set of receivers (another data source) so that an optimal economic benefit is achieved BID46 BID17. Recent literature has shown that both aforementioned applications can be formulated as optimal transport problems. The optimal transport problem has a long history, and its earliest literature dates back to Monge. Since then, it has attracted increasing attention and been widely studied in multiple communities such as applied mathematics, probability, economy and geography BID51; BID23. Specifically, we consider two sets of data, which are generated from two different distributions denoted by X ∼ µ and Y ∼ ν.1 We aim to find an optimal joint distribution γ of X and Y, which minimizes the expectation on some ground cost function c, i.e., γ * = arg min γ∈Π(µ,ν) DISPLAYFORM0 The constraint γ ∈ Π(µ, ν) requires the marginal distribution of X and Y in γ to be identical to µ and ν, respectively. The cost function c measures the discrepancy between input X and Y. For crossmodality structured data, the form of c incorporates prior knowledge into optimal transport problem. Existing literature often refers to the optimal expected cost W * (µ, ν) = E (X,Y)∼γ * [c(X, Y)] as Wasserstein distance when c is a distance, and γ * as the optimal transport plan. For domain adaptation, the function c measures the discrepancy between X and Y, and the optimal transport plan γ * essentially reveals the transfer of the knowledge from source X to target Y. For resource allocation, the function c is the cost of assigning resource X to receiver Y, and the optimal transport plan γ To address the scalability and efficiency issues, we propose a new implicit generative learning-based framework for solving optimal transport problems. Specifically, we approximate γ * by a generative model, which maps from some latent variable Z to (X, Y). For simplicity, we denote DISPLAYFORM1 where ρ is some simple latent distribution and G is some operator, e.g., deep neural network or neural ordinary differential equation (ODE). Accordingly, instead of directly estimating the probability density of γ *, we estimate the mapping G between Z and (X, Y) by solving DISPLAYFORM2 We then cast equation 3 into a minimax optimization problem using the Lagrangian multiplier method. As the constraints in equation 3 are over the space of continuous distributions, the Lagrangian multiplier is actually infinite dimensional. Thus, we propose to approximate the Lagrangian multiplier by deep neural networks, which eventually delivers a finite dimensional generative learning problem. Our proposed framework has three major benefits: Our formulated minimax optimization problem can be efficiently solved by primal dual stochastic gradient-type algorithms. Many empirical studies have corroborated that these algorithms can easily scale to very large minimax problems in machine learning BID2; Our framework can take advantage of recent advances in deep learning. Many empirical evidences have suggested that deep neural networks can effectively adapt to data with intrinsic low dimensional structures BID33. Although they are often overparameterized, due to the inductive biases of the training algorithms, the intrinsic dimensions of deep neural networks are usually controlled very well, which avoids the curse of dimensionality; Our adopted generative models allow us to efficiently sample from the optimal transport plan. This is very convenient for certain downstream applications such as domain adaptation, where we can generate infinitely many data points paired across domains BID35.Moreover, the proposed framework can also recover the density of entropy regularized optimal transport plan. Specifically, we adopt the neural Ordinary Differential Equation (ODE) approach in to model the dynamics that how Z gradually evolves to G(Z). We then derive the ODE that describes how the density evolves, and solve the density of the transport plan from the ODE. The recovery of density requires no extra parameters, and can be evaluated efficiently. Notations: Given a matrix A ∈ R d×d, det(A) denotes its determinant, tr(A) = i A ii denotes its trace, A F = i,j A 2 ij denotes its Frobenius norm, and |A| denotes a matrix with [|A|] ij = |A ij |. We use dim(v) to denote the dimension of a vector v. We review some knowledge on optimal transport and implicit generative learning. Optimal Transport: The idea of optimal transport (OT) originally comes from Monge, which proposes to solve the following problem, DISPLAYFORM0 where T (·) is a mapping from the space of µ to the space of ν. The optimal mapping T * is referred to as Monge map, and equation 4 is referred to as Monge formulation of optimal transport. Monge formulation, however, is not necessarily feasible. For example, when X is a constant random variable and Y is not, there does not exist such a map T satisfying T (X) ∼ ν. The Kantorovich formulation of our interest in equation 1 is essentially a relaxation of equation 4 by replacing the deterministic mapping with the coupling between µ and ν. Consequently, Kantorovich formulation is guaranteed to be feasible and becomes the classical formulation of optimal transport in existing literature BID1 BID6 BID16 BID50. Implicit Generative Learning: For generative learning problems, direct estimation of a probability density function is not always convenient. For example, we may not have enough prior knowledge to specify an appropriate parametric form of the probability density function (pdf). Even when an appropriate parametric pdf is available, computing the maximum likelihood estimator (MLE) can be sometimes neither efficient nor scalable. To address these issues, we resort to implicit generative learning, which do not directly specify the density. Specifically, we consider that the observed variable X is generated by transforming a latent random variable Z (with some known distribution ρ) through some unknown mapping G(·), i.e., X = G(Z). We then can train a generative model by estimating G(·) with a properly chosen loss function, which can be easier to compute than MLE. Existing literature usually refer to the distribution of G(Z) as the push-forward of reference distribution ρ. Such an implicit generative learning approach also enjoys an additional benefit: We only need to choose ρ that is convenient to sample, e.g., uniform or Gaussian distribution, and we then can generate new samples from our learned distribution directly through the estimated mapping G very efficiently. For many applications, the target distribution can be quite complicated, in contrast to the distribution ρ being simple. This actually requires the mapping G to be flexible. Therefore, we choose to represent G using deep neural networks (DNNs), which are well known for its universal approximation property, i.e., DNNs with sufficiently many neurons and properly chosen activation functions can approximate any continuous functions over compact support up to an arbitrary error. Early empirical evidence, including variational auto-encoder and generative adversarial networks (GAN, BID21) have shown great success of parameterizing G with DNNs. They further motivate a series of variants, which adopt various DNN architectures to learn more complicated generative models BID45 BID5 BID54 BID10 BID28.Although the above methods cannot directly estimate the density of the target distribution, for certain applications, we can actually recover the density of G(Z). For example, generative flow methods such as NICE BID13, Real NVP BID14 Glow (Kingma & BID31) impose sparsity constraints on weight matrices, and exploit the hierarchical nature of DNNs to compute the densities layer by layer. Specifically, NICE proposed in BID13 denotes the transitions of densities within a neural network as DISPLAYFORM1, where h i represents the hidden units of the i-th layer and f i is the transition function. NICE suggest to restrict the Jacobian matrices of f i's to be triangular. Therefore, f i's are reversible and the transition of density in each layer can be easily computed. More recently, propose a neural ordinary differential equation (neural ODE) approach to compute the transition from Z to G(Z). Specifically, they introduce a dynamical formulation and parameterizing the mapping G using DNNs with recursive structures: They use an ODE to describe how the input Z gradually evolves towards the output G(Z) in continuous time, dz/dt = ξ(z(t), t), where z(t) denotes the continuous time interpolation of Z, and ξ(·, ·) denotes a feedforward-type DNN. Without loss of generality, we choose z = Z and z = G(Z). Then under certain regularity conditions, the mapping G(·) is guaranteed to be reversible, and the density of G(Z) can be computed in O(d) time, where d is the dimension of Z BID22. For better efficiency and scalability, we propose a new framework -named SPOT (Scalable Pushforward of Optimal Transport) -for solving the optimal transport problem. Before we proceed with the derivation, we first introduce some notations and assumptions. Recall that we aim to find an optimal joint distribution γ given by equation 1. For simplicity, we assume that the two marginal distributions X ∼ µ and Y ∼ ν have densities p X (x) and p Y (y) for X ∈ X and Y ∈ Y with compact X and Y, respectively. Moreover, we assume that the joint distribution γ has density p γ. Then we rewrite equation 1 as the following integral form, DISPLAYFORM0 We then convert equation 5 into a minmax optimization problem using the Lagrangian multiplier method. Note that equation 5 contains infinitely many constraints, i.e., the equality constraints need to hold for every x ∈ X and y ∈ Y. Therefore, we need infinitely many Lagrangian multipliers. For notational simplicity, we denote the Lagrangian multipliers associated with x and y by two functions λ X (x): X → R and λ Y (y): Y → R, respectively. Eventually we obtain DISPLAYFORM1 As mentioned earlier, solving p γ in the space of all continuous distributions is generally intractable. Thus, we adopt the push-forward method, which introduces a mapping G from some latent variable DISPLAYFORM2 The latent variable Z follows some distribution ρ that is easy to sample. We then rewrite equation 6 as min DISPLAYFORM3 Note that we have replaced the integrals with expectations, since x∈X p γ (x, y)dx, y∈Y p γ (x, y)dy, p X (x), and p Y (y) are probability density functions. Then we further parameterize G, λ X, and λ Y with neural networks 2. We denote G as the class of neural networks for parameterizing G and similarly F X and F Y as the classes of functions for λ X and λ Y, respectively. Although G, F X, and F Y are finite classes, our parameterization of G cannot exactly represent any continuous distributions of (X, Y) (only up to a small approximation error with sufficiently many neurons). Then the marginal distribution constraints, G X (Z) ∼ µ and G Y (Z) ∼ ν, are not necessarily satisfied. Therefore, the equilibrium of equation 7 does not necessarily exist, since the Lagrangian multipliers can be unbounded. Motivated by BID0, we require the neural networks for parameterizing λ X and λ Y to be η-Lipschitz, denoting as F η X and F η Y, respectively. Here η can be treated as a tuning parameter, and provides a refined control of the constraint violation. Since each η-Lipschitz function can be represented by ηf with f being 1-Lipschitz, we rewrite equation 7 as min DISPLAYFORM4 We apply alternating stochastic gradient algorithm to solve equation 8: in each iteration, we perform a few steps of gradient ascent on λ X and λ Y, respectively for a fixed G, followed by one-step gradient descent on G for fixed λ X and λ Y. We use Spectral Normalization (SN, BID38) to control the Lipschitz constant of λ X and λ Y being smaller than 1. Specifically, SN constrains the spectral norm of each weight matrix W by SN(W) = W/σ(W) in every iteration, where σ(W) denotes the spectral norm of W. Note that σ(W) can be efficiently approximated by a simple one-step power method BID20. Therefore, the computationally intensive SVD can be avoided. We summarize the algorithm in Algorithm 1 with SN omitted. Algorithm 1 Mini-batch Primal Dual Stochastic Gradient Algorithm for SPOT Require: DISPLAYFORM5 Initialized networks G, λ X, and λ Y with parameters w, θ, and β, respectively; α, the learning rate; n critic, the number of gradient ascent for λ X and λ Y; n, the batch size while w not converged do DISPLAYFORM6 Connection to Wasserstein Generative Adversarial Networks (WGANs): Our proposed framework equation 8 can be viewed as a multi-task learning version of Wasserstein GANs BID35. Specifically, the mapping G can be viewed as a generator that generates samples in the domains X and Y. The Lagrangian multipliers λ X and λ Y can be viewed as discriminators that evaluate the discrepancies of the generated sample distributions and the target marginal distributions. By restricting DISPLAYFORM7 essentially approximates the Wasserstein distance between the distributions of G X (Z) and X under the Euclidean ground cost BID51, the same holds for Y ). Denote DISPLAYFORM8 which essentially learns two Wasserstein GANs with a joint generator G through the regularizer R. Extension to Multiple Marginal Distributions: Our proposed framework can be straightforwardly extended to more than two marginal distributions. Consider the ground cost function c taking m inputs X 1,..., X m with X i ∼ µ i for i = 1,..., m. Then the optimal transport problem equation 1 becomes the multi-marginal problem BID42: DISPLAYFORM9 where Π(µ 1, µ 2, · · ·, µ m) denotes all the joint distributions with marginal distributions satisfying X i ∼ µ i for all i = 1,..., m. Following the same procedure for two distributions, we cast equation 10 into the following form min DISPLAYFORM10, where G and λ Xi's are all parameterized by neural networks. Existing methods for solving the multi-marginal problem equation 10 suggest to discretize the support of the joint distribution using a refined grid. For complex distributions, the grid size needs to be very large and can be exponential in m BID51. Our parameterization method actually only requires at most 2m neural networks, which further corroborates the scalability and efficiency of our framework. Existing literature has shown that entropy-regularized optimal transportation outperforms the unregularized counterpart in some applications BID15 BID9. This is because the entropy regularizer can tradeoff the estimation bias and variance by controlling the smoothness of the density function. We demonstrate how to efficiently recover the density p γ of the transport plan with entropy regularization. Instead of parameterizing G by a feedforward neural network, we choose the neural ODE approach, which uses neural networks to approximate the transition from input Z towards output G(Z) in the continuous time. Specifically, we take z = Z and z = G(Z). Let z(t) be the continuous interpolation of Z with density p(t) varying according to time t. We split z(t) into z 1 (t) and z 2 (t) such that dim(z 1) = dim(X) and dim(z 2) = dim(Y). We then write the neural ODE as DISPLAYFORM0 where ξ 1 and ξ 2 capture the dynamics of z(t). We parameterize ξ = (ξ 1, ξ 2) by a neural network with parameter w. We describe the dynamics of the joint density p(t) in the following proposition. Proposition 1. Let z, z 1, z 2, ξ 1 and ξ 2 be defined as above. Suppose ξ 1 and ξ 2 are uniformly Lipschitz continuous in z (the Lipschitz constant is independent of t) and continuous in t. The log joint density satisfies the following ODE: DISPLAYFORM1 where ∂ξ1 ∂z1 and ∂ξ2 ∂z2 are Jacobian matrices of ξ 1 and ξ 2 with respect to z 1 and z 2, respectively. Proposition 1 is a direct of Theorem 1 in. We can now recover the joint density by taking p γ = p, which further enables us to efficiently compute the entropy regularizer defined as DISPLAYFORM2 Then we consider the entropy regularized Wasserstein distance DISPLAYFORM3 is the objective function in equation 8. Note that here G is a functional operator of ξ, and hence parameterized with w. The training algorithm follows Algorithm 1, except that updating G becomes more complex due to involving the neural ODE and the entropy regularizer. To update G, we are essentially updating w using the gradient g w = ∂(L c + H)/∂w, where is the regularization coefficient. First we compute ∂L c /∂w. We adopt the integral form from in the following DISPLAYFORM4 where a(t) = ∂L c /∂z(t) is the so-called "adjoint variable". The detailed derivation is slightly involved due to the complicated terms in the chain rule. We refer the readers to for a complete argument. The advantage of introducing a(t) is that we can compute a(t) using the following ODE, DISPLAYFORM5 Then we can use a well developed numerical method to compute equation 13 efficiently BID12. Next, we compute ∂H/∂w in a similar procedure with a(t) replaced by b(t) = ∂H/∂ log p(t). We then write DISPLAYFORM6 Using the same numerical method, we can compute ∂H/∂w, which eventually allows us to compute g w and update w. We evaluate the SPOT framework on various tasks: Wasserstein distance approximation, density recovery, paired sample generation and domain adaptation. All experiments are implemented with PyTorch using one GTX1080Ti GPU and a Linux desktop computer with 32GB memory, and we adopt the Adam optimizer with configuration parameters 0.5 and 0.999 . We first demonstrate that SPOT can accurately and efficiently approximate the Wasserstein distance. We take the Euclidean ground cost, i.e. c(x, y) = x − y. Then DISPLAYFORM0 essentially approximates the Wasserstein distance. We take the marginal distributions µ and ν as two Gaussian distributions in R 2 with the same identity covariance matrix. The means are (−2.5, 0) and (2.5, 0), respectively. We find the Wasserstein distance between µ and ν equal to 5 by evaluating its closed-form solution. We generate n = 10 5 samples from both distributions µ and ν, respectively. Note that naively applying discretization-based algorithms by dividing the support according to samples requires at least 40 GB memory, which is beyond the memory capability. We parameterize G X, G Y, λ X, and λ Y with fully connected neural networks without sharing parameters. All the networks use the Leaky-ReLU activation BID37. G X and G Y have 2 hidden layers. λ X and λ Y have 1 hidden layer. The latent variable Z follows the standard Gaussian distribution in R 2. We take the batch size equal to 100.WD vs. Number of Epochs. We compare the algorithmic behavior of SPOT and Regularized Optimal Transport (ROT, BID47) with different regularization coefficients. For SPOT, we set the number of units in each hidden layer equal to 8 and η = 10 4. For ROT, we adopt the code from the authors 3 with only different input samples, learning rates, and regularization coefficients. FIG0 shows the convergence behavior of SPOT and ROT for approximating the Wasserstein distance between µ and ν with different learning rates. We observe that SPOT converges to the true Wasserstein distance with only 0.6%, 0.3%, and 0.3% relative errors corresponding to Learning Rates (LR) 10 −3, 10 −4, and 10 −5, respectively. In contrast, ROT is very sensitive to its regularization coefficient. Thus, it requires extensive tuning to achieve a good performance. WD vs. Number of Hidden Units. We then explore the adaptivity of SPOT by increasing the network size, while the input data are generated from some low dimensional distribution. Specifically, the number of hidden units per layer varies from 2 to 2 10. Recall that we parameterize G with two 2-hidden-layer neural networks, and λ X, λ Y with two 1-hidden-layer neural networks. Accordingly, the number of parameters in G varies from 36 to about 2 × 10 6, and that in λ X or λ Y varies from 12 to about 2, 000. The tuning parameter η also varies corresponding to the number of hidden units in λ X, λ Y. We use η = 10 5 for 2 1, 2 2 and 2 3 hidden units per layer, η = 2×10 4 for 2 4, 2 5 and 2 6 hidden units per layer, η = 10 4 for 2 7 and 2 8 hidden units per layer, η = 2 × 10 3 for 2 9, and 2 10 hidden units per layer. FIG1 shows the estimated WD with respect to the number of hidden units per layer. For large neural networks that have 2 9 or 2 10 hidden units per layer, i.e., 5.2 × 10 5 or 2.0 × 10 6 parameters, the number of parameters is far larger than the number of samples. Therefore, the model is heavily overparameterized. As we can observe in FIG1, the relative error however, does not increase as the number of parameters grows. This suggests that SPOT is robust with respect to the network size. We demonstrate that SPOT can effectively recover the joint density with entropy regularization. We adopt the neural ODE approach as described in Section 4. Denote φ(a, b) as the density N (a, b). We take the marginal distributions µ and ν as Gaussian distributions φ and φ(2, 0.5); mixtures of Gaussian 1 2 φ(−1, 0.5) + 1 2 φ(1, 0.5) and 1 2 φ(−2, 0.5)+ 1 2 φ(2, 0.5). The ground cost is the Euclidean square function, i.e., c(x, y) = x−y 2. We run the training algorithm for 6 × 10 5 iterations and in each iteration, we generate 500 samples from µ and ν, respectively. We parameterize ξ with a 3-hidden-layer fully-connected neural network with 64 hidden units per layer, and the latent dimension is 2. We take η = 10 6.Figure 4: Visualization of the marginal distributions and the joint density of the optimal transport plan. Figure 4 shows the input marginal densities and heat maps of output joint densities. We can see that a larger regularization coefficient yields a smoother joint density for the optimal transport plan. Note that with continuous marginal distributions and the Euclidean square ground cost, the joint density of the unregularized optimal transport degenerates to a generalized impulse function (i.e., a generalized Dirac δ function that has nonzero value on a manifold instead of one atom, as shown in Rachev FORMULA0 ; Onural FORMULA1). Entropy regularization prevents such degeneracy by enforcing smoothness of the density. We show that SPOT can generate paired samples (G X (Z), G Y (Z)) from unpaired data X and Y that are sampled from marginal distributions µ and ν, respectively. Synthetic Data. We take the squared Euclidean cost, i.e. c(x, y) = x−y 2, and adopt the same implementation and sample size as in Section 5.1 with learning rate 10 −3 and 32 hidden units per layer. FIG3 illustrates the input samples and the generated samples with two sets of different marginal distributions: The upper row corresponds to the same Gaussian distributions as in Section 5.1. The lower row takes X as Gaussian distribution with mean (−2.5, 0) and covariance 0.5I, Y as (sin(Y 1) + Y 2, 2Y 1 − 3), where Y 1 follows a uniform distribution on, and Y 2 follows a Gaussian distribution N (2, 0.1). We observe that the generated samples and the input samples are approximately identically distributed. Additionally, the paired relationship is as expected -the upper mass is transported to the upper region, and the lower mass is transported to the lower region. Real Data. We next show SPOT is able to generate high quality paired samples from two unpaired real datasets: MNIST and MNISTM BID18. The handwritten digits in MNIST and MNISTM datasets have different s and foregrounds (see FIG2 . The digits in paired images however, are expected to have similar contours. We leverage this prior knowledge 4 by adopting a semantic-aware cost function BID34 to extract the edge of handwritten letters, i.e., we use the following cost function DISPLAYFORM0 where C 1 and C 2 denote the Sobel filter BID49, and x j 's and y j 's are the three channels of RGB images. The operator * denotes the matrix convolution. We set We now use separate neural networks to parameterize G X and G Y instead of taking G X and G Y as outputs of a common network. Note that G X and G Y does not share parameters. Specifically, we use two 4-layer convolutional layers in each neural network for G X or G Y, and two 5-layer convolutional neural networks for λ X and λ Y . More detailed network settings are provided in Appendix A.2. The batch size is 32, and we train the framework with 2 × 10 5 iterations until the generated samples become stable. FIG2 shows the generated samples of SPOT. We also reproduce the of CoGAN with the code from the authors 5 . As can be seen, with approximately the same network size, SPOT yields paired images with better quality than CoGAN: The contours of the paired of SPOT are nearly identical, while the of CoGAN have no clear paired relation. Besides, the images corresponding to G Y (Z) in SPOT have colorful foreground and , while in CoGAN there are only few colors. Recall that in SPOT, the paired relation is encouraged by ground cost c, and in CoGAN it is encouraged by sharing parameters. By leveraging prior knowledge in ground cost c, the paired relation is more accurately controlled without compromising the quality of the generated images. DISPLAYFORM1 We further test our framework on more complex real datasets: Photo-Monet dataset and Edges-Shoes dataset. We adopt the Euclidean cost function for Photo-Monet dataset, and the semantic-aware cost function as in MNIST-MNISTM for Edges-Shoes dataset. Other implementations remain the same as the MNIST-MINSTM experiment. FIG5 demonstrates the generated samples of both datasets. We observe that the generated images have a desired paired relation: For each Z, G X (Z) and G Y (Z) gives a pair of corresponding scenery and shoe. The generated images are also of high quality, especially considering that Photo-Monet dataset is a pretty small but complex dataset with 6,288 photos and 1,073 paintings. Optimal transport has been used in domain adaptation, but existing methods are either computationally inefficient BID7, or cannot achieve a state-of-the-art performance BID48. Here, we demonstrate that SPOT can tackle large scale domain adaptation problems with state-of-the-art performance. In particular, we receive labeled source data {x i} ∼ µ, where each data point is associated with a label v i, and target data {y j} ∼ ν with unknown labels. For simplicity, we use X and Y to denote the random vectors following distributions µ and ν, respectively. The two distributions µ and ν can be coupled in a way that each paired samples of (X, Y) from the coupled joint distribution are likely to have the same label. In order to identify such coupling information between source and target data, we propose a new OT-based domain adaptation method -DASPOT (Domain Adaptation with SPOT) as follows. Specifically, we jointly train an optimal transport plan and two classifiers for X and Y (denoted by D X and D Y, respectively). Each classifier is a composition of two neural networks -an embedding network and a decision network. For simplicity, we denote D X = D e,X • D c,X, where D e,X denotes the embedding network, and D c,X denotes the decision network (respectively for D Y = D e,Y • D c,Y). We expect the embedding networks to extract high level features of the source and target data, and then find an optimal transport plan to align X and Y based on these high level features using SPOT. Here we choose a ground cost c(x, y) = D e,X (x) − D e,Y (y) 2. Let G denote the generator of SPOT. The Wasserstein distance of such an OT problem can be written as DISPLAYFORM0 Meanwhile, we train D X by minimizing the empirical risk DISPLAYFORM1, where E denotes the cross entropy loss function, and train D Y by minimizing DISPLAYFORM2 where [Eventually, the joint training optimize DISPLAYFORM3 where DISPLAYFORM4 is the objective function of OT problem in equation 8 with c defined above, and η s, η da are the tuning parameters. We choose η s = 10 3 for all experiments. We set η da = 0 for the first 10 5 iteration to wait the generators to be well trained. Then we set η da = 10 for the next 3 × 10 5 iteration. We take totally 4 × 10 5 iterations, and set the learning rate equal to 10 −4 and batch size equal to 128 for all experiments. We evaluate DASPOT with the MNIST, MNISTM, USPS BID26, and SVHN BID40 datasets. We denote a domain adaptation task as Source Domain → Target Domain. We compare the performance of DASPOT with other optimal transport based domain adaptation methods: ROT BID48, StochJDOT and DeepJDOT . As can be seen in TAB0, DASPOT achieves equal or better performances on all the tasks. Moreover, we show that DeepJDOT is not as efficient as DASPOT. For example, in the MNIST → USPS task, DASPOT requires 169s running time to achieve a 95% accuracy, while DeepJDOT requires 518s running time to achieve the same accuracy. The reason behind is that DeepJDOT needs to solve a series of optimal transport problems using Sinkhorn algorithm. The implementation of DeepJDOT is adapted from the authors' code 6. Existing literature shows that several stochastic algorithms can efficiently compute the Wasserstein distance between two continuous distributions. These algorithms, however, only apply to the dual of the OT problem equation 1, and cannot provide the optimal transport plan. For example, BID19 suggest to expand the dual variables in two reproducing kernel Hilbert spaces. They then apply the Stochastic Averaged Gradient (SAG) algorithm to compute the optimal objective value of OT with continuous marginal distributions or semi-discrete marginal distributions (i.e., one marginal distribution is continuous and the other is discrete). The follow-up work, BID47, parameterize the dual variables with neural networks and apply the Stochastic Gradient Descent (SGD) algorithm to eventually achieve a better convergence. These two methods can only provide the optimal transport plan and recover the joint density when the densities of the marginal distributions are known. This is prohibitive in most applications, since we only have access to the empirical data. Our framework actually allows us to efficiently compute the joint density from the transformation of the latent variable Z as in Section 4. TAB1 shows the architecture of two discriminators λ X, λ Y. The two networks have identical architechture and do not share parameters. The CNN architecture for USPS, MNIST and MNISTM. PReLU activation is applied BID24. TAB2 shows the architecture of two generators G X and G Y. The last column in TAB2 means whether G X and G Y share the same parameter. TAB3 shows the architecture of two discriminators λ X, λ Y, and two classifiers D X, D Y. The last column in TAB2 uses (·, ·) to denote which group of discriminators share the same parameter. TAB4 shows the architecture of two generators G X and G Y. The last column in TAB4 means whether G X and G Y share the same parameter. The Residual block is the same as the one in BID38. [3 × 3, ch, stride = 1, padding =0] Sigmoid False TAB5 shows the architecture of two discriminators λ X, λ Y, and two classifiers D X, D Y. The last column in TAB5 uses (·, ·) to denote which group of discriminators share the same parameter.

Use GAN-based method to scalably solve optimal transport

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper shows that the Wasserstein distance objective enables the training of latent variable models with discrete latents in a case where the Variational Autoencoder objective fails to do so.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this work, we propose a novel formulation of planning which views it as a probabilistic inference problem over future optimal trajectories. This enables us to use sampling methods, and thus, tackle planning in continuous domains using a fixed computational budget. We design a new algorithm, Sequential Monte Carlo Planning, by leveraging classical methods in Sequential Monte Carlo and Bayesian smoothing in the context of control as inference. Furthermore, we show that Sequential Monte Carlo Planning can capture multimodal policies and can quickly learn continuous control tasks. To exhibit intelligent behaviour machine learning agents must be able to learn quickly, predict the consequences of their actions, and explain how they will react in a given situation. These abilities are best achieved when the agent efficiently uses a model of the world to plan future actions. To date, planning algorithms have yielded very impressive . For instance, Alpha Go BID36 relied on Monte Carlo Tree Search (MCTS) BID23 ) to achieve super human performances. Cross entropy methods (CEM) BID34 have enabled robots to perform complex nonprehensile manipulations BID11 and algorithms to play successfully Tetris BID39. In addition, iterative linear quadratic regulator (iLQR) BID21 BID20 BID41 enabled humanoid robots tasks to get up from an arbitrary seated pose.Despite these successes, these algorithms make strong underlying assumptions about the environment. First, MCTS requires a discrete setting, limiting most of its successes to discrete games with known dynamics. Second, CEM assumes the distribution over future trajectories to be Gaussian, i.e. unimodal. Third, iLQR assumes that the dynamics are locally linear-Gaussian, which is a strong assumption on the dynamics and would also assume the distribution over future optimal trajectories to be Gaussian. For these reasons, planning remains an open problem in environments with continuous actions and complex dynamics. In this paper, we address the limitations of the aforementioned planning algorithms by creating a more general view of planning that can leverage advances in deep learning (DL) and probabilistic inference methods. This allows us to approximate arbitrary complicated distributions over trajectories with non-linear dynamics. We frame planning as density estimation problem over optimal future trajectories in the context of control as inference BID10 BID45 BID43; BID47 BID31. This perspective allows us to make use of tools from the inference research community and, as previously mentioned, model any distribution over future trajectories. The planning distribution is complex since trajectories consist of an intertwined sequence of states and actions. Sequential Monte Carlo (SMC) BID38 BID13 BID27 methods are flexible and efficient to model such a T −1 t≥1 p env (s t+1 |s t, a t) T t≥1 π θ (a t |s t) denotes the probability of a trajectory x 1:T under policy π θ. FIG8.1: O t is an observed optimality variable with probability p(O t |s t, a t) = exp(r(s t, a t)).x t = (s t, a t) are the state-action pair variables considered here as latent. Traditionally, in reinforcement learning (RL) problems, the goal is to find the optimal policy that maximizes the expected return E q θ [T t=1 γ t r t]. However, it is useful to frame RL as an inference problem within a probabilistic graphical framework BID33 BID45 BID30. First, we introduce an auxiliary binary random variable O t denoting the "optimality" of a pair (s t, a t) at time t and define its probability 1 as p(O t = 1|s t, a t) = exp(r(s t, a t)). O is a convenience variable only here for the sake of modeling. By considering the variables (s t, a t) as latent and O t as observed, we can construct a Hidden Markov Model (HMM) as depicted in figure 2.1. Notice that the link s → a is not present in figure 2.1 as the dependency of the optimal action on the state depends on the future observations. In this graphical model, the optimal policy is expressed as p(a t |s t, O t:T).The posterior probability of this graphical model can be written as 2: DISPLAYFORM0 r(s t, a t) + log p(a t).(2.1)It appears clearly that finding optimal trajectories is equivalent to finding plausible trajectories yielding a high return.1 as in BID30, if the rewards are bounded above, we can always remove a constant so that the probability is well defined.2 Notice that in the rest of the paper, we will abusively remove the product of the action priors T t=1 p(at) = exp T t=1 log p(at) from the joint as in BID30. We typically consider this term either constant or already included in the reward function. See Appendix A.2 for details. Many control as inference methods can be seen as approximating the density by optimizing its variational lower bound: BID43. Instead of directly differentiating the variational lower bound for the whole trajectory, it is possible to take a message passing approach such as the one used in Soft Actor-Critic (SAC) BID17 and directly estimate the optimal policy p(a t |s t, O t:T) using the backward message, i.e a soft Q function instead of the Monte Carlo return. DISPLAYFORM1 Since distributions over trajectories are complex, it is often difficult or impossible to directly draw samples from them. Fortunately in statistics, there are successful strategies for drawing samples from complex sequential distributions, such as SMC methods. For simplicity, in the remainder of this section we will overload the notation and refer to the target distribution as p(x) and the proposal distribution as q(x). We wish to draw samples from p but we only know its unnormalized density. We will use the proposal q to draw samples and estimate p. In the next section, we will define the distributions p and q in the context of planning. Importance sampling (IS): When x can be efficiently sampled from another simpler distribution q i.e. the proposal distribution, we can estimate the likelihood of any point x under p straightforwardly by computing the unnormalized importance sampling weights w(x) ∝ p(x) q(x) and using the identity DISPLAYFORM0 is defined as the normalized importance sampling weights. In practice, one draws N samples from q: {x (n) } N n=1 ∼ q; these are referred to as particles. The set of particles {x (n) } N n=1 associated with their weights {w (n) } N n=1 are simulations of samples from p. That is, we approximate the density p with a weighted sum of diracs from samples of q: DISPLAYFORM1, with x (n) sampled from q where δ x0 (x) denotes the Dirac delta mass located as x 0.Sequential Importance Sampling (SIS): When our problem is sequential in nature x = x 1:T, sampling x 1:T at once can be a challenging or even intractable task. By exploiting the sequential structure, the unnormalized weights can be updated iteratively in an efficient manner: w t (x 1:t) = w t−1 (x 1:t−1) p(xt|x1:t−1) q(xt|x1:t−1). We call this the update step. This enables us to sample sequentially x t ∼ q(x t |x 1:t−1) to finally obtain the set of particles {x Sequential Importance Resampling (SIR): When the horizon T is long, samples from q usually have a low likelihood under p, and thus the quality of our approximation decreases exponentially with T. More concretely, the unnormalized weights w (n) t converge to 0 with t → ∞. This usually causes the normalized weight distribution to degenerate, with one weight having a mass of 1 and the others a mass of 0. This phenomenon is known as weight impoverishment. One way to address weight impoverishment is to add a resampling step where each particle is stochastically resampled to higher likelihood regions at each time step. This can typically reduce the variance of the estimation from growing exponentially with t to growing linearly. In the context of control as inference, it is natural to see planning as the act of approximating a distribution of optimal future trajectories via simulation. In order to plan, an agent must possess a model of the world that can accurately capture consequences of its actions. In cases where multiple trajectories have the potential of being optimal, the agent must rationally partition its computational resources to explore each possibility. Given finite time, the agent must limit its planning to a finite horizon h. We, therefore, define planning as the act of approximating the optimal distribution over trajectories of length h. In the control-as-inference framework, this distribution is naturally expressed as p(a 1, s 2, . . . s h, a h |O 1:T, s 1), where s 1 represents our current state. As we consider the current state s 1 given, it is equivalent and convenient to focus on the planning distribution with horizon h: p(x 1:h |O 1:T). Bayesian smoothing is an approach to the problem of estimating the distribution of a latent variable conditioned on all past and future observations. One method to perform smoothing is to decompose the posterior with the two-filter formula BID4 BID26: DISPLAYFORM0 This corresponds to a forward-backward messages factorization in a Hidden Markov Model as depicted in figure 3.1. We broadly underline in orange forward variables and in blue backward variables in the rest of this section. DISPLAYFORM1 Figure 3.1: Factorization of the HMM into forward (orange) and backward (blue) messages. Estimating the forward message is filtering, estimating the value of the latent knowing all the observations is smoothing. Filtering is the task of estimating p(x 1:t |O 1:t): the probability of a latent variable conditioned on all past observations. In contrast, smoothing estimates p(x 1:t |O 1:T): the density of a latent variable conditioned on all the past and future measurements. In the belief propagation algorithm for HMMs, these probabilities correspond to the forward message α h (x h) = p(x 1:h |O 1:h) and backward message β h (x h) = p(O h+1:T |x h), both of which are computed recursively. While in discrete spaces these forward and backward messages can be estimated using the sumproduct algorithm, its complexity scales with the square of the space dimension making it unsuitable for continuous tasks. We will now devise efficient strategies for estimating reliably the full posterior using the SMC methods covered in section 2.2. The backward message p(O h+1:T |x h) can be understood as the answer to: "What is the probability of following an optimal trajectory from the next time step on until the end of the episode, given my current state?". Importantly, this term is closely related to the notion of value function in RL. Indeed, in the control-as-inference framework, the state-and action-value functions are defined as DISPLAYFORM0 T |s h, a h ) respectively. They are solutions of a soft-Bellman equation that differs a little from the traditional Bellman equation (; ; BID35 BID0 . A more in depth explanation can be found in BID30 . We can show subsequently that: DISPLAYFORM1 Full details can be found in Appendix A.3. Estimating the backward message is then equivalent to learning a value function. This value function as defined here is the same one used in Maximum Entropy RL . Using the of the previous subsections we can now derive the full update of the sequential importance sampling weights. To be consistent with the terminology of section 2.2, we call p(x 1:h |O 1:T) the target distribution and q θ (x 1:h) the proposal distribution. The sequential weight update formula is in our case: DISPLAYFORM0 3) is akin to a maximum entropy advantage function. The change in weight can be interpreted as sequentially correcting our expectation of the return of a trajectory. The full derivation is available in Appendix A.4. Our algorithm is similar to the Auxilliary Particle Filter which uses a one look ahead simulation step to update the weights. Note that we have assumed that our model of the environment was perfect to obtain this slightly simplified form. This assumption is made by most planning algorithms (LQR, CEM . . .): it entails that our plan is only as good as our model is. A typical way to mitigate this issue and be more robust to model errors is to re-plan at each time step; this technique is called Model Predictive Control (MPC) and is commonplace in control theory. We can now use the computations of previous subsections to derive the full algorithm. We consider the root state of the planning to be the current state s t. We aim at building a set of particles {x DISPLAYFORM0 and their weights {w DISPLAYFORM1 representative of the planning density p(x t:t+h |O 1:T) over optimal trajectories. We use SAC BID17 for the policy and value function, but any other Maximum Entropy policy can be used for the proposal distribution. Note that we used the value function estimated by SAC as a proxy the optimal one as it is usually done by actor critic methods. Algorithm 1 SMC Planning using SIR 1: for t in {1, . . ., T} do 2: DISPLAYFORM2 3: DISPLAYFORM3 for i in {t, . . ., t + h} do // Update 6: DISPLAYFORM0 7: DISPLAYFORM1 8: DISPLAYFORM2 // Resampling 10: DISPLAYFORM3 12: end for Sample n ∼ Uniform(1, N).14: We summarize the proposed algorithm in Algorithm 1. At each step, we sample from the proposal distribution or model-free agent (line 6) and use our learned model to sample the next state and reward (line 7). We then update the weights (line 8). In practice we only use one sample to estimate the expectations, thus we may incur a small bias. The resampling step is then performed (line 10-11) by resampling the trajectories according to their weight. After the planning horizon is reached, we sample one of our trajectories (line 13) and execute its first action into the environment (line 15-16). The observations (s t, a t, r t, s t+1) are then collected and added to a buffer (line 17) used to train the model as well as the policy and value function of the model-free agent. An alternative algorithm that does not use the resampling step (SIS) is highlighted in Algorithm 2 in Appendix A.6. DISPLAYFORM0 A schematic view of the algorithm can also be found on figure 3.2. We now discuss shortcomings our approach to planning as inference may suffer from, namely encouraging risk seeking policies. DISPLAYFORM0 we have the root white node st−1, the actions a (n) t−1 are black nodes and the leaf nodes are the s (n) t. We have one particle on the leftmost branch, two on the central branch and one on the rightmost branch.• In each tree, the white nodes represent states and black nodes represent actions. Each bullet point near a state represents a particle, meaning that this particle contains the total trajectory of the branch. The root of the tree represents the root planning state, we expand the tree downward when planning. Bias in the objective: Trajectories having a high likelihood under the posterior defined in Equation 2.1 are not necessarily trajectories yielding a high mean return. Indeed, as log E p exp R(x) ≥ E p R(x) we can see that the objective function we maximize is an upper bound on the quantity of interest: the mean return. This can lead to seeking risky trajectories as one very good outcome in log E exp could dominate all the other potentially very low outcomes, even if they might happen more frequently. This fact is alleviated when the dynamics of the environment are close to deterministic BID30. Thus, this bias does not appear to be very detrimental to us in our experiments 4 as our environments are fairly close to deterministic. The bias in the objective also appears in many control as inference works such as Particle Value Functions (a) and the probabilistic version of LQR proposed in BID43.Bias in the model: A distinct but closely related problem arises when one trains jointly the policy π θ and the model p model, i.e if q(x 1:T) is directly trained to approximate p(x 1:T |O 1:T). In that case, p model (s t+1 |s t, a t) will not approximate p env (s t+1 |s t, a t) but p env (s t+1 |s t, a t, O t:T) BID30. This means the model we learn has an optimism bias and learns transitions that are overly optimistic and do no match the environment's behavior. This issue is simply solved by training the model separately from the policy, on transition data contained in a buffer as seen on line 18 of Algorithm 1. In this section, we show how SMCP can deal with multimodal policies when planning. We believe multimodality is useful for exploring since it allows us to keep a distribution over many promising trajectories and also allows us to adapt to changes in the environment e.g. if a path is suddenly blocked. We applied two version of SMCP: i) with a resampling step (SIR) ii) without a resampling step (SIS) and compare it to CEM on a simple 2D point mass environment 4.1. Here, the agent can control the displacement on (x, y) within the square 2, a = (∆x, ∆y) with maximum magnitude ||a|| = 0.05. The starting position (•) of the agent is (x = 0, y = 0.5), while the goal is at g = (x = 1, y = 0.5). The reward is the agent's relative closeness increment to the goal: (a) Sequential Importance Resampling (SIR): when resampling the trajectories at each time step, the agent is able to focus on the promising trajectories and does not collapse on a single mode.(b) Sequential Importance Sampling (SIS): if we do not perform the resampling step the agent spends most of its computation on uninteresting trajectories and was not able to explore as well.(c) CEM: here the agent samples all the actions at once from a Gaussian with learned mean and covariance. We needed to update the parameters 50 times for the agent to find one solution, but it forgot the other one. The proposal distribution is taken to be an isotropic gaussian. Here we plot the planning distribution imagined at t = 0 for three different agents. A darker shade of blue indicates a higher likelihood of the trajectory. Only the agent using Sequential Importance Resampling was able to find good trajectories while not collapsing on a single mode. DISPLAYFORM0 ||st−g|| 2. However, there is a partial wall at the centre of the square leading to two optimal trajectories, one choosing the path below the wall and one choosing the path above. The proposal is an isotropic normal distribution for each planning algorithm, and since the environment's dynamics are known, there is no need for learning: the only difference between the three methods is how they handle planning. We also set the value function to 0 for SIR and SIS as we do not wish to perform any learning. We used 1500 particles for each method, and updated the parameters of CEM until convergence. Our experiment 4.1 shows how having particles can deal with multimodality and how the resampling step can help to focus on the most promising trajectories. The experiments were conducted on the Open AI Gym Mujoco benchmark suite BID5. To understand how planning can increase the learning speed of RL agents we focus on the 250000 first time steps. The Mujoco environments provide a complex benchmark with continuous states and actions that requires exploration in order to achieve state-of-the-art performances. The environment model used for our planning algorithm is the same as the probabilistic neural network used by BID8, it minimizes a gaussian negative log-likelihood model: DISPLAYFORM0 where Σ θ is diagonal and the transitions (s n, a n, s n+1) are obtained from the environment. We added more details about the architecture and the hyperparameters in the appendix A.5.We included two popular planning algorithms on Mujoco as baselines: CEM BID8 and Random Shooting (RS) . Furthermore, we included SAC BID17, a model free RL algorithm, since i) it has currently one of the highest performances on Mujoco tasks, which make it a very strong baseline, and ii) it is a component of our algorithm, as we use it as a proposal distribution in the planning phase. Our suggest that SMCP does not learn as fast as CEM and RS initially as it heavily relies on estimating a good value function. However, SMCP quickly achieves higher performances than CEM and RS. SMCP also learns faster than SAC because it was able to leverage information from the model early in training. Note that our differ slightly from the usually found in the model-based RL literature. This is because we are tackling a more difficult problem: estimating the transitions and the reward function. We are using unmodified versions of the environments which introduces many hurdles. For instance, the reward function is challenging to learn from the state and very noisy. Usually, the environments are modified such that their reward can be computed directly from the state e.g. BID8 3.As in BID18, we assess the significance of our by running each algorithm with multiple seeds (20 random seeds in our case, from seed 0 to seed 19) and we perform a statistical significance test following BID9. We test the hypothesis that our mean return on the last 100k steps is higher than the one obtained by SAC. Our are significant to the 5% for HalfCheetah and Walker2d. See Appendix A.7 for additional details. We also report some additional experimental such as effective sample size and model loss in Appendix A.8. Planning as inference: Seeing planning as an inference problem has been explored in cognitive neuroscience by BID3 and BID37. While shedding light on how Bayesian inference could be used in animal and human reasoning, it does not lead to a practical algorithm usable in complex environments. In the reinforcement learning literature, we are only aware of and BID44 that initially framed planning as an inference problem. However, both works make simplifying assumptions on the dynamics and do not attempt to capture the full posterior distribution. In the control theory literature, particle filters are usually used for inferring the true state of the system which is then used for control BID1. BID22 also combined SMC and MPC methods. While their algorithm is similar to ours, the distribution they approximate is not the Bayesian posterior, but a distribution which converges to a Dirac on the best trajectory. More recently, BID28 achieved promising on a rope manipulation task using generative adversarial network BID12 to generate future trajectories. Model based RL: Recent work has been done in order to improve environment modeling and account for different type of uncertainties. BID8 compared the performance of models that account for both aleatoric and epistemic uncertainties by using an ensemble of probabilistic models. BID16 combined the variational autoencoder BID25 ) and a LSTM BID19 to model the world. BID6 used a model to improve the target for temporal difference (TD) learning. Note that this line of work is complementary to ours as SMCP could make use of such models. Other works have been conducted in order to directly learn how to use a model BID15 BID46 BID7.Particle methods and variational inference: BID14 learn a good proposal distribution for SMC methods by minimizing the KL divergence with the optimal proposal. It is conceptually similar to the way we use SAC BID17 but it instead minimizes the reverse KL to the optimal proposal. Further works have combined SMC methods and variational inference (; b; BID29 to obtain lower variance estimates of the distribution of interest. In this work, we have introduced a connection between planning and inference and showed how we can exploit advances in deep learning and probabilistic inference to design a new efficient and theoretically grounded planning algorithm. We additionally proposed a natural way to combine model-free and model-based reinforcement learning for planning based on the SMC perspective. We empirically demonstrated that our method achieves state of the art on Mujoco. Our suggest that planning can lead to faster learning in control tasks. However, our particle-based inference method suffers some several shortcomings. First, we need many particles to build a good approximation of the posterior, and this can be computationally expensive since it requires to perform a forward pass of the policy, the value function and the model for every particle. Second, resampling can also have adverse effects, for instance all the particles could be resampled on the most likely particle, leading to a particle degeneracy. More advanced SMC methods dealing with this issue such as backward simulation BID32 or Particle Gibbs with Ancestor Sampling (PGAS) have been proposed and using them would certainly improve our . Another issue we did not tackle in our work is the use of models of the environment learned from data. Imperfect model are known to in compounding errors for prediction over long sequences. We chose to re-plan at each time step (Model Predictive Control) as it is often done in control to be more robust to model errors. More powerful models or uncertainty modeling techniques can also be used to improve the accuracy of our planning algorithm. While the inference and modeling techniques used here could be improved in multiple ways, SMCP achieved impressive learning speed on complex control tasks. The planning as inference framework proposed in this work is general and could serve as a stepping stone for further work combining probabilistic inference and deep reinforcement learning. A.1 ABBREVIATION AND NOTATION p(x) Density of interest. Approximation of the density of interest.t ∈ {1, . . . T} time steps.n ∈ {1, . . . N} particle number.h horizon length. The true joint distribution 2.1 in section 2.1 should actually be written: DISPLAYFORM0 In Mujoco environments, the reward is typically written as DISPLAYFORM1 where f is a function of the state (velocity for HalfCheetah on Mujoco for example). The part α||a t || 2 2can be seen as the contribution from the action prior (here a gaussian prior). One can also consider the prior to be constant (and potentially improper) so that is does not change the posterior p(x 1:T |O 1:T). DISPLAYFORM2 By definition of the optimal value function in BID30. DISPLAYFORM3 We use there the forward-backward equation 3.1 for the numerator and the denominator DISPLAYFORM4 A.5 EXPERIMENT DETAILS Random samples: 1000 transitions are initially collected by a random policy to pretrain the model and the proposal distribution. After which the agents start following their respective policy. Data preprocessing: We normalize the observations to have zero mean and standard deviation 1. The model is used to predict the planning distribution for the horizon h of N particles. We then sample a trajectory according to its weight and return the first action of this trajectory. In our experiments, we fix the maximum number of particles for every method to 2500. For SMCP, the temperature and horizon length are described in TAB2.3. We used a custom implementation with a Gaussian policy for both the SAC baseline and the proposal distribution used for both versions of SMCP. We used Adam with a learning rate of 0.001. The reward scaling suggested by BID17 for all experiments and used an implementation inspired by. We used a two hidden layers with 256 hidden units for the three networks: the value function, the policy and the soft Q functions. Model: We train the model p model to minimize the negative log likelihood of p(s t+1 |s t + ∆ t (s t, a t), σ t (s t, a t)). The exact architectures are detailed in TAB2.3. We train the model to predict the distribution of the change in states and learn a deterministic reward function from the current state and predict the change in state. Additionally, we manually add a penalty on the action magnitude in the reward function to simplify the learning. At the end of each episode we train the model for 10 epochs. Since the training is fairly short, we stored every transitions into the buffer. The model is defined as: DISPLAYFORM0 DISPLAYFORM1 7: DISPLAYFORM2 8: DISPLAYFORM3 9:end for 10:Sample n ∼ Categorical(w The significance of our is done following guidelines from BID9 . We test the hypothesis that the mean return of our method is superior to the one of SAC. We use 20 random seeds (from 0 to 19pro) for each method on each environment. For this we look at the average return from steps 150k to 250k for SIR-SAC and SAC, and conduct a Welch's t-test with unknown variance. We report the p-value for each environment tested on Mujoco. A p val < 0.05 usually indicates that there is strong evidence to suggest that our method outperforms SAC.• HalfCheetah-v2: p val = 0.003. There is very compelling evidence suggesting we outperform SAC.• Hopper-v2: p val = 0.09. There is no significant evidence suggesting we outperform SAC.• Walker2d-v2: p val = 0.03. There is compelling evidence suggesting we outperform SAC. A.8.1 EFFECTIVE SAMPLE SIZE More precisely the values are DISPLAYFORM0 where i is the depth of the planning, N is the number of particles and DISPLAYFORM1 We can see that as the proposal distribution improves the ESS also increases. The ESS on HalfCheetah is representative of the one obtained on the other environments. While these values are not high, we are still around 15% thus we do not suffer heavily from weight degeneracy. We also report the negative log likelihood loss of the environment's model during the training on Figure A.2.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Leveraging control as inference and Sequential Monte Carlo methods, we proposed a probabilistic planning algorithm.

Represent the TLDR for finding the corresponding scientific article

We propose a new class of inference models that iteratively encode gradients to estimate approximate posterior distributions.


Question: Represent the scientific article to find a short TLDR!

Question: Training Generative Adversarial Networks (GANs) is notoriously challenging. We propose and study an architectural modification, self-modulation, which improves GAN performance across different data sets, architectures, losses, regularizers, and hyperparameter settings. Intuitively, self-modulation allows the intermediate feature maps of a generator to change as a function of the input noise vector. While reminiscent of other conditioning techniques, it requires no labeled data. In a large-scale empirical study we observe a relative decrease of 5%-35% in FID. Furthermore, all else being equal, adding this modification to the generator leads to improved performance in 124/144 (86%) of the studied settings. Self-modulation is a simple architectural change that requires no additional parameter tuning, which suggests that it can be applied readily to any GAN. Generative Adversarial Networks (GANs) are a powerful class of generative models successfully applied to a variety of tasks such as image generation BID20; ), learned compression BID15, super-resolution , inpainting , and domain transfer BID13 BID23.Training GANs is a notoriously challenging task BID6 BID15 as one is searching in a high-dimensional parameter space for a Nash equilibrium of a non-convex game. As a practical remedy one applies (usually a variant of) stochastic gradient descent, which can be unstable and lack guarantees. As a , one of the main research challenges is to stabilize GAN training. Several approaches have been proposed, including varying the underlying divergence between the model and data distributions ), regularization and normalization schemes BID7 ), optimization schedules , and specific neural architectures (; BID21 . A particularly successful approach is based on conditional generation; where the generator (and possibly discriminator) are given side information, for example class labels;;. In fact, state-of-the-art conditional GANs inject side information via conditional batch normalization (CBN) layers BID3; BID21. While this approach does help, a major drawback is that it requires external information, such as labels or embeddings, which is not always available. In this work we show that GANs benefit from self-modulation layers in the generator. Our approach is motivated by Feature-wise Linear Modulation in supervised learning (; BID3, with one key difference: instead of conditioning on external information, we condition on the generator's own input. As self-modulation requires a simple change which is easily applicable to all popular generator architectures, we believe that is a useful addition to the GAN toolbox. We provide a simple yet effective technique that can added universally to yield better GANs. We demonstrate empirically that for a wide variety of settings (loss functions, regularizers and normalizers, neural architectures, and optimization settings) that the proposed approach yields between a 5% and 35% improvement in sample quality. When using fixed hyperparameters settings our approach outperforms the baseline in 86%(124/144) of cases. Further, we show that self-modulation still helps even if label information is available. Finally, we discuss the effects of this method in light of recently proposed diagnostic tools, generator conditioning and precision/recall for generative models . Several recent works observe that conditioning the generative process on side information (such as labels or class embeddings) leads to improved models (; ;). Two major approaches to conditioning on side information s have emerged: Directly concatenate the side information s with the noise vector z , i.e. z = [s, z]. Condition the hidden layers directly on s, which is usually instantiated via conditional batch normalization BID3 ).Despite the success of conditional approaches, two concerns arise. The first is practical; side information is often unavailable. The second is conceptual; unsupervised models, such as GANs, seek to model data without labels. Including them side-steps the challenge and value of unsupervised learning. We propose self-modulating layers for the generator network. In these layers the hidden activations are modulated as a function of latent vector z. In particular, we apply modulation in a feature-wise fashion which allows the model to re-weight the feature maps as a function of the input. This is also motivated by the FiLM layer for supervised models (; BID3 in which a similar mechanism is used to condition a supervised network on side information. Batch normalization BID12 can improve the training of deep neural nets, and it is widely used in both discriminative and generative modeling ;). It is thus present in most modern networks, and provides a convenient entry point for self-modulation. Therefore, we present our method in the context of its application via batch normalization. In batch normalization the activations of a layer, h, are transformed as DISPLAYFORM0 where µ and σ 2 are the estimated mean and variances of the features across the data, and γ and β are learnable scale and shift parameters. Self-modulation for unconditional (without side information) generation. In this case the proposed method replaces the non-adaptive parameters β and γ with input-dependent β(z) and γ(z), respectively. These are parametrized by a neural network applied to the generator's input FIG0 ). In particular, for layer, we compute In general, it suffices that γ (·) and β (·) are differentiable. In this work, we use a small onehidden layer feed-forward network (MLP) with ReLU activation applied to the generator input z. Specifically, given parameter matrices U and V , and a bias vector b , we compute DISPLAYFORM1 DISPLAYFORM2 We do the same for β(z) with independent parameters. Self-modulation for conditional (with side information) generation. Having access to side information proved to be useful for conditional generation. The use of labels in the generator (and possibly discriminator) was introduced by and later adapted by;. In case that side information is available (e.g. class labels y), it can be readily incorporated into the proposed method. This can be achieved by simply composing the information y with the input z ∈ R d via some learnable function g, i.e. z = g(y, z). In this work we opt for the simplest option and instantiate g as a bi-linear interaction between z and two trainable embedding functions E, E: Y → R d of the class label y, as DISPLAYFORM3 This conditionally composed z can be directly used in Equation 1. Despite its simplicity, we demonstrate that it outperforms the standard conditional models. Discussion. TAB0 summarizes recent techniques for generator conditioning. While we choose to implement this approach via batch normalization, it can also operate independently by removing the normalization part in the Equation 1. We made this pragmatic choice due to the fact that such conditioning is common (; ;).The second question is whether one benefits from more complex modulation architectures, such as using an attention network BID18 whereby β and γ could be made dependent on all upstream activations, or constraining the elements in γ to which would yield a similar gating mechanism to an LSTM cell BID10. Based on initial experiments we concluded that this additional complexity does not yield a substantial increase in performance. We perform a large-scale study of self-modulation to demonstrate that this method yields robust improvements in a variety of settings. We consider loss functions, architectures, discriminator regularization/normalization strategies, and a variety of hyperparameter settings collected from recent studies (; BID7 ; BID15). We study both unconditional (without labels) and conditional (with labels) generation. Finally, we analyze the through the lens of the condition number of the generator's Jacobian as suggested by , and precision and recall as defined in. Loss functions. We consider two loss functions. The first one is the non-saturating loss proposed in BID6: DISPLAYFORM0 The second one is the hinge loss used in: DISPLAYFORM1 Controlling the Lipschitz constant of the discriminator. The discriminator's Lipschitz constant is a central quantity analyzed in the GAN literature (; BID22 . We consider two state-of-the-art techniques: gradient penalty BID7, and spectral normalization . Without normalization and regularization the models can perform poorly on some datasets. For the gradient penalty regularizer we consider regularization strength λ ∈ {1, 10}.Network architecture. We use two popular architecture types: one based on DCGAN , and another from which incorporates residual connections BID8. The details can be found in the appendix. Optimization hyper-parameters. We train all models for 100k generator steps with the Adam optimizer (We also perform a subset of the studies with 500K steps and discuss it in. We test two popular settings of the Adam hyperparameters (β 1, β 2): (0.5, 0.999) and (0, 0.9). Previous studies find that multiple discriminator steps per generator step can help the training BID6 ), thus we also consider both 1 and 2 discriminator steps per generator step 2. In total, this amounts to three different sets of hyperparameters for (β 1, β 2, disc iter): (0, 0.9, 1), (0, 0.9, 2), (0.5, 0.999, 1). We fix the learning rate to 0.0002 as in. All models are trained with batch size of 64 on a single nVidia P100 GPU. We report the best performing model attained during the training period; although the follow the same pattern if the final model is report. Datasets. We consider four datasets: CIFAR10, CELEBA-HQ, LSUN-BEDROOM, and IMAGENET. The LSUN-BEDROOM dataset BID19 contains around 3M images. We partition the images randomly into a test set containing 30588 images and a train set containing the rest. CELEBA-HQ contains 30k images . We use the 128 × 128 × 3 version obtained by running the code provided by the authors 3. We use 3000 examples as the test set and the remaining examples as the training set. CIFAR10 contains 70K images (32 × 32 × 3), partitioned into 60000 training instances and 10000 testing instances. Finally, we evaluate our method on IMAGENET, which contains 1.3M training images and 50K test images. We re-size the images to 128 × 128 × 3 as done in and BID21.Metrics. Quantitative evaluation of generative models remains one of the most challenging tasks. This is particularly true in the context of implicit generative models where likelihood cannot be effectively evaluated. Nevertheless, two quantitative measures have recently emerged: The Inception Score and the Frechet Inception Distance. While both of these scores have some drawbacks, they correlate well with scores assigned by human annotators and are somewhat robust. Inception Score (IS) posits that that the conditional label distribution p(y|x) of samples containing meaningful objects should have low entropy, while the marginal label distribution p(y) should have high entropy. Formally, DISPLAYFORM2 The score is computed using an Inception classifier. Drawbacks of applying IS to model comparison are discussed in BID2.An alternative score, the Frechet Inception Distance (FID), requires no labeled data BID9. The real and generated samples are first embedded into a feature space (using a specific layer of InceptionNet). Then, a multivariate Gaussian is fit each dataset and the distance is computed as DISPLAYFORM3 ), where µ and Σ denote the empirical mean and covariance and subscripts x and g denote the true and generated data, respectively. FID was shown to be robust to various manipulations and sensitive to mode dropping BID9. 2 We also experimented with 5 steps which didn't outperform the 2 step setting. 3 Available at https://github.com/tkarras/progressive_growing_of_gans. In the unpaired setting (as defined in Section 3.2), we compute the median score (across random seeds) and report the best attainable score across considered optimization hyperparameters. SELF-MOD is the method introduced in Section 2 and BASELINE refers to batch normalization. We observe that the proposed approach outperforms the baseline in 30 out of 32 settings. The relative improvement is detailed in TAB3. The standard error of the median is within 3% in the majority of the settings and is presented in TAB7 To test robustness, we run a Cartesian product of the parameters in Section 3.1 which in 36 settings for each dataset (2 losses, 2 architectures, 3 hyperparameter settings for spectral normalization, and 6 for gradient penalty). For each setting we run five random seeds for self-modulation and the baseline (no self-modulation, just batch normalization). We compute the median score across random seeds which in 1440 trained models. We distinguish between two sets of experiments. In the unpaired setting we define the model as the tuple of loss, regularizer/normalization, neural architecture, and conditioning (self-modulated or classic batch normalization). For each model compute the minimum FID across optimization hyperparameters (β 1, β 2, disc iters). We therefore compare the performance of self-modulation and baseline for each model after hyperparameter optimization. The of this study are reported in TAB1, and the relative improvements are in TAB3 and FIG1.We observe the following: When using the RESNET style architecture, the proposed method outperforms the baseline in all considered settings. When using the SNDCGAN architecture, it outperforms the baseline in 87.5% of the cases. The breakdown by datasets is shown in FIG1. FORMULA3 The improvement can be as high as a 33% reduction in FID. We observe similar improvement to the inception score, reported in the appendix. In the second setting, the paired setting, we assess how effective is the technique when simply added to an existing model with the same set of hyperparameters. In particular, we fix everything except the type of conditioning -the model tuple now includes the optimization hyperparameters. This in 36 settings for each data set for a total of 144 comparisons. We observe that selfmodulation outperforms the baseline in 124/144 settings. These suggest that self-modulation can be applied to most GANs even without additional hyperparameter tuning. Conditional Generation. We demonstrate that self-modulation also works for label-conditional generation. Here, one is given access the class label which may be used by the generator and the. We observe that the majority "good" models utilize self-modulation. Figure (c) shows that applying self-conditioning is more beneficial on the later layers, but should be applied to each layer for optimal performance. This effect persists across all considered datasets, see the appendix.discriminator. We compare two settings: Generator conditioning is applied via label-conditional Batch Norm BID3 ) with no use of labels in the discriminator (G-COND). Generator conditioning applied as above, but with projection based conditioning in the discriminator (intuitively it encourages the discriminator to use label discriminative features to distinguish true/fake samples), as in (P-CGAN). The former can be considered as a special case of the latter where discriminator conditioning is disabled. For P-CGAN, we use the architectures and hyper-parameter settings of. See the appendix, Section B.3 for details. In both cases, we compare standard label-conditional batch normalization to self-modulation with additional labels, as discussed in Section 2, Equation 3.The are shown in TAB4. Again, we observe that the simple incorporation of self-modulation leads to a significant improvement in performance in the considered settings. Training for longer on IMAGENET. To demonstrate that self-modulation continues to yield improvement after training for longer, we train IMAGENET for 500k generator steps. Due to the increased computational demand we use a single setting for the unconditional and conditional settings models following and , but using only two discriminator steps per generator. We expect that the would continue to improve if training longer. However, currently from 500k steps require training for ∼10 days on a P100 GPU.We compute the median FID across 3 random seeds. After 500k steps the baseline unconditional model attains FID 60.4, self-modulation attains 53.7 (11% improvement). In the conditional setting Where to apply self-modulation? Given the robust improvements of the proposed method, an immediate question is where to apply the modulation. We tested two settings: applying modulation to every batch normalization layer, and applying it to a single layer. The of this ablation are in FIG1. These suggest that the benefit of self-modulation is greatest in the last layer, as may be intuitive, but applying it to each layer is most effective. Self-modulation is a simple yet effective complementary addition to this line of work which makes a significant difference when no side information is available. In addition, when side information is available it can be readily applied as discussed in Section 2 and leads to further improvements. Conditional Modulation. Conditional modulation, using side information to modulate the computation flow in neural networks, is a rich idea which has been applied in various contexts (beyond GANs). Multiplicative and Additive Modulation. Existing conditional modulations mentioned above are usually instantiated via Batch Normalization, which include both multiplicative and additive modulation. These two types of modulation also link to other techniques widely used in neural network literature. The multiplicative modulation is closely related to Gating, which is adopted in LSTM BID10, gated PixelCNN (van den), Convolutional Sequence-to-sequence networks BID5 and Squeeze-and-excitation Networks BID11. The additive modulation is closely related to Residual Networks BID8. The proposed method adopts both types of modulation. We present a generator modification that improves the performance of most GANs. This technique is simple to implement and can be applied to all popular GANs, therefore we believe that selfmodulation is a useful addition to the GAN toolbox. Our suggest that self-modulation clearly yields performance gains, however, they do not say how this technique in better models. Interpretation of deep networks is a complex topic, especially for GANs, where the training process is less well understood. Rather than purely speculate, we compute two diagnostic statistics that were proposed recently ignite the discussion of the method's effects. First, we compute the condition number of the generators Jacobian. provide evidence that better generators have a Jacobian with lower condition number and hence regularize using this quantity. We estimate the generator condition number in the same was as. We compute the Jacobian (J z) i,j = δG(z)i δzj at each z in a minibatch, then average the logarithm of the condition numbers computed from each Jacobian. Second, we compute a notion of precision and recall for generative models. define the quantities, F 8 and F 1/8, for generators. These quantities relate intuitively to the traditional precision and recall metrics for classification. Generating points which have low probability under the true data distribution is interpreted as a loss in precision, and is penalized by the F 8 score. Failing to generate points that have high probability under the true data distributions is interpreted as a loss in recall, and is penalized by the F 1/8 score. FIG4 shows both statistics. The left hand plot shows the condition number plotted against FID score for each model. We observe that poor models tend to have large condition numbers; the correlation, although noisy, is always positive. This corroborates the observations in . However, we notice an inverse trend in the vicinity of the best models. The cluster of the best models with self-modulation has lower FID, but higher condition number, than the best models without self-modulation. Overall the correlation between FID and condition number is smaller for self-modulated models. This is surprising, it appears that rather than unilaterally reducing the condition number, self-modulation provides some training stability, yielding models with a small range of generator condition numbers. The right-hand plot in FIG4 shows the F 8 and F 1/8 scores. Models in the upper-left quadrant cover true data modes better (higher precision), and models in the lower-right quadrant produce more modes (higher recall). Self-modulated models tend to favor higher recall. This effect is most pronounced on IMAGENET. Overall these diagnostics indicate that self-modulation stabilizes the generator towards favorable conditioning values. It also appears to improve mode coverage. However, these metrics are very new; further development of analysis tools and theoretical study is needed to better disentangle the symptoms and causes of the self-modulation technique, and indeed of others. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. A ADDITIONAL We describe the model structures that are used in our experiments in this section. The SNDCGAN architecture we follows the ones used in. Since the resolution of images in CIFAR10is 32 × 32 × 3, while resolutions of images in other datasets are 128 × 128 × 3.There are slightly differences in terms of spatial dimensions for both architectures. The proposed self-modulation is applied to replace existing BN layer, we term it sBN (self-modulated BN) for short in TAB8, 8, 9, 10. The ResNet architecture we also follows the ones used in. Again, due to the resolution differences, two ResNet architectures are used in this work. The proposed self-modulation is applied to replace existing BN layer, we term it sBN (self-modulated BN) for short in TAB0, 12, 13, 14. For the conditional setting with label information available, we adopt the Projection Based Conditional GAN (P-cGAN) . There are both conditioning in generators as well ad discriminators. For generator, conditional batch norm is applied via conditioning on label information, more specifically, this can be expressed as follows, DISPLAYFORM0 Where each label y is associated with a scaling and shifting parameters independently. For discriminator label conditioning, the dot product between final layer feature φ(x) and label embedding E(y) is added back to the discriminator output logits, i.e. D(x, y) = ψ(φ(x)) + φ(x) T E(y) where φ(x) represents the final feature representation layer of input x, and ψ(·) is the linear transformation maps the feature vector into a real number. Intuitively, this type of conditional discriminator encourages discriminator to use label discriminative features to distinguish true/fake samples. Both the above conditioning strategies do not dependent on the specific architectures, and can be applied to above architectures with small modifications. We use the same architectures and hyper-parameter settings 4 as in. More specifically, the architecture is the same as ResNet above, and we compare in two settings: only generator label conditioning is applied, and there is no projection based conditioning in the discriminator, and both generator and discriminator conditioning are applied, which is the standard full P-cGAN.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

A simple GAN modification that improves performance across many losses, architectures, regularization schemes, and datasets.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.