Question: represent the text to find a very short summary (1-2 sentences max)

Question: Autoencoders provide a powerful framework for learning compressed representations by encoding all of the information needed to reconstruct a data point in a latent code. In some cases, autoencoders can "interpolate": By decoding the convex combination of the latent codes for two datapoints, the autoencoder can produce an output which semantically mixes characteristics from the datapoints. In this paper, we propose a regularization procedure which encourages interpolated outputs to appear more realistic by fooling a critic network which has been trained to recover the mixing coefficient from interpolated data. We then develop a simple benchmark task where we can quantitatively measure the extent to which various autoencoders can interpolate and show that our regularizer dramatically improves interpolation in this setting. We also demonstrate empirically that our regularizer produces latent codes which are more effective on downstream tasks, suggesting a possible link between interpolation abilities and learning useful representations. One goal of unsupervised learning is to uncover the underlying structure of a dataset without using explicit labels. A common architecture used for this purpose is the autoencoder, which learns to map datapoints to a latent code from which the data can be recovered with minimal information loss. Typically, the latent code is lower dimensional than the data, which indicates that autoencoders can perform some form of dimensionality reduction. For certain architectures, the latent codes have been shown to disentangle important factors of variation in the dataset which makes such models useful for representation learning BID7 BID15. In the past, they were also used for pre-training other networks by being trained on unlabeled data and then being stacked to initialize a deep network BID1 BID44. More recently, it was shown that imposing a prior on the latent space allows autoencoders to be used for probabilistic or generative modeling BID18 BID34 BID27.In some cases, autoencoders have shown the ability to interpolate. Specifically, by mixing codes in latent space and decoding the , the autoencoder can produce a semantically meaningful combination of the corresponding datapoints. Interpolation has been frequently reported as a qualitative experimental in studies about autoencoders BID5 BID35 BID30 BID29 BID14 and latent-variable generative models in general BID10 BID33 BID41. The ability to interpolate can be useful in its own right e.g. for creative applications . However, it also indicates that the autoencoder can "extrapolate" beyond the training data and has learned a latent space with a particular structure. Specifically, if interpolating between two points in latent space produces a smooth semantic warping in data space, this suggests that nearby points in latent space are semantically similar. A visualization of this idea is shown in FIG0, where a smooth A critic network is fed interpolants and reconstructions and tries to predict the interpolation coefficient α corresponding to its input (with α = 0 for reconstructions). The autoencoder is trained to fool the critic into outputting α = 0 for interpolants. interpolation between a "2" and a "9" suggests that the 2 is surrounded by semantically similar points, i.e. other 2s. This property may suggest that an autoencoder which interpolates well could also provide a good learned representation for downstream tasks because similar points are clustered. If the interpolation is not smooth, there may be "discontinuities" in latent space which could in the representation being less useful as a learned feature. This connection between interpolation and a "flat" data manifold has been explored in the context of unsupervised representation learning BID3 and regularization BID43.Given the widespread use of interpolation as a qualitative measure of autoencoder performance, we believe additional investigation into the connection between interpolation and representation learning is warranted. Our goal in this paper is threefold: First, we introduce a regularization strategy with the specific goal of encouraging improved interpolations in autoencoders (section 2); second, we develop a synthetic benchmark where the slippery concept of a "semantically meaningful interpolation" is quantitatively measurable (section 3.1) and evaluate common autoencoders on this task (section 3.2); and third, we confirm the intuition that good interpolation can in a useful representation by showing that the improved interpolation ability produced by our regularizer elicits improved representation learning performance on downstream tasks (section 4). We also make our codebase available 1 which provides a unified implementation of many common autoencoders including our proposed regularizer. Autoencoders, also called auto-associators BID4, consist of the following structure: First, an input x ∈ R dx is passed through an "encoder" z = f θ (x) parametrized by θ to obtain a latent code z ∈ R dz. The latent code is then passed through a "decoder"x = g φ (z) parametrized by φ to produce an approximate reconstructionx ∈ R dx of the input x. We consider the case where f θ and g φ are implemented as multi-layer neural networks. The encoder and decoder are trained simultaneously (i.e. with respect to θ and φ) to minimize some notion of distance between the input x and the outputx, for example the squared L 2 distance x −x 2.Interpolating using an autoencoder describes the process of using the decoder g φ to decode a mixture of two latent codes. Typically, the latent codes are combined via a convex combination, so that interpolation amounts to computingx α = g φ (αz 1 +(1−α)z 2 ) for some α ∈ where z 1 = f θ (x 1) and z 2 = f θ (x 2) are the latent codes corresponding to data points x 1 and x 2. We also experimented with spherical interpolation which has been used in settings where the latent codes are expected to have spherical structure BID17 BID45 BID35, but found it made no discernible difference in practice for any autoencoder we studied. Ideally, adjusting α from 0 to 1 will produce a sequence of realistic datapoints where each subsequentx α is progressively less semantically similar to x 1 and more semantically similar to x 2. The notion of "semantic similarity" is problem-dependent and ill-defined; we discuss this further in section 3. As mentioned above, a high-quality interpolation should have two characteristics: First, that intermediate points along the interpolation are indistinguishable from real data; and second, that the intermediate points provide a semantically smooth morphing between the endpoints. The latter characteristic is hard to enforce because it requires defining a notion of semantic similarity for a given dataset, which is often hard to explicitly codify. So instead, we propose a regularizer which encourages interpolated datapoints to appear realistic, or more specifically, to appear indistinguishable from reconstructions of real datapoints. We find empirically that this constraint in realistic and smooth interpolations in practice (section 3.1) in addition to providing improved performance on downstream tasks (section 4).To enforce this constraint we introduce a critic network BID12 which is fed interpolations of existing datapoints (i.e.x α as defined above). Its goal is to predict α fromx α, i.e. to predict the mixing coefficient used to generate its input. When training the model, for each pair of training data points we randomly sample a value of α to producex α. In order to resolve the ambiguity between predicting α and 1 − α, we constrain α to the range [0, 0.5] when feedingx α to the critic. In contrast, the autoencoder is trained to fool the critic to think that α is always zero. This is achieved by adding an additional term to the autoencoder's loss to optimize its parameters to fool the critic. In a loose sense, the critic can be seen as approximating an "adversarial divergence" BID23 BID0 between reconstructions and interpolants which the autoencoder tries to minimize. Formally, let d ω (x) be the critic network, which for a given input produces a scalar value. The critic is trained to minimize DISPLAYFORM0 where, as above, DISPLAYFORM1 for some x (not necessarily x 1 or x 2), and γ is a scalar hyperparameter. The first term trains the critic to recover α from x α. The second term serves as a regularizer with two functions: First, it enforces that the critic consistently outputs 0 for non-interpolated inputs; and second, by interpolating between x andx (the autoencoder's reconstruction of x) in data space it ensures the critic is exposed to realistic data even when the autoencoder's reconstructions are poor. We found the second term was not crucial for our approach, but helped stabilize the convergence of the autoencoder and allowed us to use consistent hyperparameters and architectures across all datasets and experiments. The autoencoder's loss function is modified by adding a regularization term: DISPLAYFORM2 where λ is a scalar hyperparameter which controls the weight of the regularization term. Note that the regularization term is effectively trying to make the critic output 0 regardless of the value of α, thereby "fooling" the critic into thinking that an interpolated input is non-interpolated (i.e., having α = 0). The parameters θ and φ are optimized with respect to L f,g (which gives the autoencoder access to the critic's gradients) and ω is optimized with respect to L d. We refer to the use of this regularizer as Adversarially Constrained Autoencoder Interpolation (ACAI). A diagram of the ACAI is shown in FIG1. Assuming an effective critic, the autoencoder successfully "wins" this adversarial game by producing interpolated points which are indistinguishable from reconstructed data. We find in practice that encouraging this behavior also produces semantically smooth interpolations and improved representation learning performance, which we demonstrate in the following sections. Our loss function is similar to the one used in the Least Squares Generative Adversarial Network BID28 in the sense that they both measure the distance between a critic's output and a scalar using a squared L2 loss. However, they are substantially different in that ours is used as a regularizer for autoencoders rather than for generative modeling and our critic attempts to regress the interpolation coefficient α instead of a fixed scalar hyperparameter. Note that the only thing ACAI encourages is that interpolated points appear realistic. The critic only ever sees a single reconstruction or interpolant at a time; it is never fed real datapoints or latent vectors. It therefore will only be able to successfully recover α if the quality of the autoencoder's output degrades consistently across an interpolation as a function of α (as seen, for example, in fig. 3a where interpolated points become successively blurrier and darker). ACAI's primary purpose is to discourage this behavior. In doing so, it may implicitly modify the structure of the latent space learned by the autoencoder, but ACAI itself does not directly impose a specific structure. Our goal in introducing ACAI is to test whether simply encouraging better interpolation behavior produces a better representation for downstream tasks. Further, in contrast with the standard Generative Adversarial Network (GAN) setup BID12, ACAI does not distinguish between "real" and "fake" data; rather, it simply attempts to regress the interpolation coefficient α. Furthermore, GANs are a generative modeling technique, not a representation learning technique; in this paper, we focus on autoencoders and their ability to learn useful representations. How can we measure whether an autoencoder interpolates effectively and whether our proposed regularization strategy achieves its stated goal? As mentioned in section 2, defining interpolation relies on the notion of "semantic similarity" which is a vague and problem-dependent concept. For example, a definition of interpolation along the lines of "αz 1 + (1 − α)z 2 should map to αx 1 + (1 − α)x 2 " is overly simplistic because interpolating in "data space" often does not in realistic datapointsin images, this corresponds to simply fading between the pixel values of the two images. Instead, we might hope that our autoencoder smoothly morphs between salient characteristics of x 1 and x 2, even when they are dissimilar. Put another way, we might hope that decoded points along the interpolation smoothly traverse the underlying manifold of the data instead of simply interpolating in data space. However, we rarely have access to the underlying data manifold. To make this problem more concrete, we introduce a simple benchmark task where the data manifold is simple and known a priori which makes it possible to quantify interpolation quality. We then evaluate the ability of various common autoencoders to interpolate on our benchmark. Finally, we test ACAI on our benchmark and show that it exhibits dramatically improved performance and qualitatively superior interpolations. Given that the concept of interpolation is difficult to pin down, our goal is to define a task where a "correct" interpolation between two datapoints is unambiguous and well-defined. This will allow us to quantitatively evaluate the extent to which different autoencoders can successfully interpolate. Towards this goal, we propose the task of autoencoding 32 × 32 grayscale images of lines. We consider 16-pixel-long lines beginning from the center of the image and extending outward at an angle Λ ∈ [0, 2π] (or put another way, lines are radii of the circle circumscribed within the image borders). An example of 16 such images is shown in FIG2 (appendix A.1). In this task, the data manifold can be defined entirely by a single variable: Λ. We can therefore define a valid interpolation from x 1 to x 2 as one which smoothly and linearly adjusts Λ from the angle of the line in x 1 to the angle in x 2. We further require that the interpolation traverses the shortest path possible along the data manifold. We provide some concrete examples of good and bad interpolations, shown and described in appendix A.1.On any dataset, our desiderata for a successful interpolation are that intermediate points look realistic and provide a semantically meaningful morphing between its endpoints. On this synthetic lines dataset, we can formalize these notions as specific evaluation metrics, which we describe in detail in appendix A.2. To summarize, we propose two metrics: Mean Distance and Smoothness. Mean Distance measures the average distance between interpolated points and "real" datapoints. Smoothness measures whether the angles of the interpolated lines follow a linear trajectory between the angle of the start and endpoint. Both of these metrics are simple to define due to our construction of a dataset where we exactly know the data distribution and manifold; we provide a full definition and justification in appendix A.2. A perfect alignment would achieve 0 for both scores; larger values indicate a failure to generate realistic interpolated points or produce a smooth interpolation respectively. By choosing a synthetic benchmark where we can explicitly measure the quality of an interpolation, we can confidently evaluate different autoencoders on their interpolation abilities. To evaluate an autoencoder on the synthetic lines task, we randomly sample line images during training and compute our evaluation metrics on a separate randomly-sampled test set of images. Note that we never train any autoencoder explicitly to produce an optimal interpolation; "good" interpolation is an emergent property which occurs only when the architecture, loss function, training procedure, etc. produce a suitable latent space. In this section, we describe various common autoencoder structures and objectives and try them on the lines task. Our goal is to quantitatively evaluate the extent to which standard autoencoders exhibit useful interpolation behavior. Our , which we describe below, are summarized in table 1.Base Model Perhaps the most basic autoencoder structure is one which simply maps input datapoints through a "bottleneck" layer whose dimensionality is smaller than the input. In this setup, f θ and g φ are both neural networks which respectively map the input to a deterministic latent code z and then back to a reconstructed input. Typically, f θ and g φ are trained simultaneously with respect to x −x 2. We will use this framework as a baseline for experimentation for all of the autoencoder variants discussed below. In particular, for our base model and all of the other autoencoders we will use the model architecture and training procedure described in appendix B. As a short summary, our encoder consists of a stack of convolutional and average pooling layers, whereas the decoder consists of convolutional and nearest-neighbor upsampling layers. For experiments on the synthetic "lines" task, we use a latent dimensionality of 64. Note that, because the data manifold is effectively onedimensional, we might expect autoencoders to be able to model this dataset using a one-dimensional latent code; however, using a larger latent code reflects the realistic scenario where the latent space is larger than necessary. After training our baseline autoencoder, we achieved a Mean Distance score which was the worst (highest) of all of the autoencoders we studied, though the Smoothness was on par with various other approaches. In general, we observed some reasonable interpolations when using the baseline model, but found that the intermediate points on the interpolation were typically not realistic as seen in the example interpolation in fig. 3a.Denoising Autoencoder An early modification to the standard autoencoder setup was proposed by BID44, where instead of feeding x into the autoencoder, a corrupted versionx ∼ q(x|x) is sampled from the conditional probability distribution q(x|x) and is fed into the autoencoder instead. The autoencoder's goal remains to producex which minimizes x −x 2. One justification of this approach is that the corrupted inputs should fall outside of the true data manifold, so the autoencoder must learn to map points from outside of the data manifold back onto it. This provides an implicit way of defining and learning the data manifold via the coordinate system induced by the latent space. While various corruption procedures q(x|x) have been used such as masking and salt-and-pepper noise, in this paper we consider the simple case of additive isotropic Gaussian noise wherex ∼ N (x, σ 2 I) and σ is a hyperparameter. After tuning σ, we found simply setting σ = 1.0 to work best. Interestingly, we found the denoising autoencoder often produced "data-space" interpolation (as seen in fig. 3b) when interpolating in latent space. This ed in comparatively poor Mean Distance and Smoothness scores. Variational Autoencoder The Variational Autoencoder (VAE) BID18 BID34 introduces the constraint that the latent code z is a random variable distributed according to a prior distribution p(z). The encoder f θ can then be considered an approximation to the posterior p(z|x). Then, the decoder g φ is taken to parametrize the likelihood p(x|z); in all of our experiments, we consider x to be Bernoulli distributed. The latent distribution constraint is enforced by an additional loss term which measures the KL divergence between approximate posterior and prior. VAEs then use log-likelihood for the reconstruction loss (cross-entropy in the case of Bernoulli-distributed data), which in the following combined loss function: −E[log g φ (z)] + KL(f θ (x)||p(z)) where the expectation is taken with respect to z ∼ f θ (x) and KL(·||·) is the KL divergence. Minimizing this loss function can be considered maximizing a lower bound (the "ELBO") on the likelihood of the training set, producing a likelihood-based generative model which allows novel data points to be sampled by first sampling z ∼ p(z) and then computing g φ (z). A common choice is to let q(z|x) be a diagonalcovariance Gaussian, in which case backpropagation through sampling from q(z|x) is feasible via the "reparametrization trick" which replaces z ∼ N (µ, σI) with ∼ N (0, I), z = µ + σ where µ, σ ∈ R dz are the predicted mean and standard deviation produced by f θ. Various modified objectives BID15 BID47, improved prior distributions BID19 BID39 BID17 and improved model architectures BID37 BID8 BID13 have been proposed to better the VAE's performance on downstream tasks, but in this paper we solely consider the "vanilla" VAE objective and prior described above applied to our baseline autoencoder structure. When trained on the lines benchmark, we found the VAE was able to effectively model the data distribution (see samples, fig. 5 in appendix C) and accurately reconstruct inputs. In interpolations produced by the VAE, intermediate points tend to look realistic, but the angle of the lines do not follow a smooth or short path (fig. 3c). This ed in a very good Mean Distance score but a very poor Smoothness score. Contrary to expectations, this suggests that desirable interpolation behavior may not follow from an effective generative model of the data distribution. Adversarial Autoencoder The Adversarial Autoencoder (AAE) BID27 proposes an alternative way of enforcing structure on the latent code. Instead of minimizing a KL divergence between the distribution of latent codes and a prior distribution, a critic network is trained in tandem with the autoencoder to predict whether a latent code comes from f θ or from the prior p(z). The autoencoder is simultaneously trained to reconstruct inputs (via a standard reconstruction loss) and to "fool" the critic. The autoencoder is allowed to backpropagate gradients through the critic's loss function, but the autoencoder and critic parameters are optimized separately. This effectively computes an "adversarial divergence" between the latent code distribution and the chosen prior. This framework was later generalized and referred to as the "Wasserstein Autoencoder" BID38 One advantage of this approach is that it allows for an arbitrary prior (as opposed to those which have a tractable KL divergence). The disadvantages are that the AAE no longer has a probabilistic interpretation and involves optimizing a minimax game, which can cause instabilities. Using the AAE requires choosing a prior, a critic structure, and a training scheme for the critic. For simplicity, we also used a spherical Gaussian prior for the AAE. We experimented with various architectures for the critic, and found the best performance with a critic which consisted of two dense layers, each with 100 units and a leaky ReLU nonlinearity. We found it satisfactory to simply use the same optimizer and learning rate for the critic as was used for the autoencoder. On our lines benchmark, the AAE typically produced smooth interpolations, but exhibited degraded quality in the middle of interpolations (fig. 3d). This behavior produced the best Smoothness score among existing autoencoders, but a relatively poor Mean Distance score. The Vector Quantized Variational Autoencoder (VQ-VAE) was introduced by (van den) as a way to train discrete-latent autoencoders using a learned codebook. In the VQ-VAE, the encoder f θ (x) produces a continuous hidden representation z ∈ R d z which is then mapped to z q, its nearest neighbor in a "codebook" {e j ∈ R dz, j ∈ 1, . . ., K}. z q is then passed to the decoder for reconstruction. The encoder is trained to minimize the reconstruction loss using the straight-through gradient estimator BID2, together with a commitment loss term β z − sg(z q) (where β is a scalar hyperparameter) which encourages encoder outputs to move closer to their nearest codebook entry. Here sg denotes the stop gradient operator, i.e. sg(x) = x in the forward pass, and sg(x) = 0 in the backward pass. The codebook entries e j are updated as an exponential moving average (EMA) of the continuous latents z that map to them at each training iteration. The VQ-VAE training procedure using this EMA update rule can be seen as performing the K-means or the hard Expectation Maximization (EM) algorithm on the latent codes BID36.We perform interpolation in the VQ-VAE by interpolating continuous latents, mapping them to their nearest codebook entries, and decoding the . Assuming sufficiently large codebook, a semantically "smooth" interpolation may be possible. On the lines task, we found that this procedure produced poor interpolations. Ultimately, many entries of the codebook were mapped to unrealistic datapoints, and the interpolations resembled those of the baseline autoencoder. Adversarially Constrained Autoencoder Interpolation Finally, we turn to evaluating our proposed adversarial regularizer for improving interpolations. For simplicity, on the lines benchmark we found it sufficient to use a critic architecture which was equivalent to the encoder (as described in appendix B). To produce a single scalar value from its output, we computed the mean of its final layer activations. For the hyperparameters λ and γ we found values of 0.5 and 0.2 to achieve good , though the performance was not very sensitive to their values. We use these values for the coefficients for all of our experiments. Finally, we trained the critic using the same optimizer and hyperparameters as the autoencoder. We found dramatically improved performance on the lines benchmark when using ACAI -it achieved the best Mean Distance and Smoothness score among the autoencoders we considered. When inspecting the ing interpolations, we found it occasionally chose a longer path than necessary but typically produced "perfect" interpolation behavior as seen in fig. 3f. This provides quantitative evidence ACAI is successful at encouraging realistic and smooth interpolations. We have so far only discussed on our synthetic lines benchmark. We also provide example reconstructions and interpolations produced by each autoencoder for MNIST BID22, SVHN BID31, and CelebA BID24 in appendix D. For each dataset, we trained autoencoders with latent dimensionalities of 32 and 256. Since we do not know the underlying data manifold for these datasets, no metrics are available to evaluate performance and we can only make qualitative judgments as to the reconstruction and interpolation quality. We find that most autoencoders produce "blurrier" images with d z = 32 but generally give smooth interpolations regardless of the latent dimensionality. The exception to this observation was the VQ-VAE which seems generally to work better with d z = 32 and occasionally even diverged for d z = 256 (see e.g. fig. 9e). This may be due to the nearest-neighbor discretization (and gradient estimator) failing in high dimensions. Across datasets, we found the VAE and denoising autoencoder typically produced more blurry interpolations. AAE and ACAI generally produced realistic interpolations, even between dissimilar datapoints (for example, in fig. 7 bottom). The baseline model often effectively interpolated in data space. We have so far solely focused on measuring the interpolation abilities of different autoencoders. Now, we turn to the question of whether improved interpolation is associated with improved performance on downstream tasks. Specifically, we will evaluate whether using our proposed regularizer in latent space representations which provide better performance in supervised learning and clustering. Put another way, we seek to test whether improving interpolation in a latent representation which has disentangled important factors of variation (such as class identity) in the dataset. To answer this question, we ran classification and clustering experiments using the learned latent spaces of Table 3: Clustering accuracy for using K-Means on the latent space of different autoencoders (left) and previously reported methods (right). On the right, "Data" refers to performing K-Means directly on the data and DEC, RIM, and IMSAT are the methods proposed in BID46 BID20 BID16 respectively. Results marked * are excerpted from BID16 and ** are from BID46. Single-Layer Classifier A common method for evaluating the quality of a learned representation (such as the latent space of an autoencoder) is to use it as a feature representation for a simple, one-layer classifier (i.e. logistic regression) trained on a supervised learning task. The justification for this evaluation procedure is that a learned representation which has effectively disentangled class identity will allow the classifier to obtain reasonable performance despite its simplicity. To test different autoencoders in this setting, we trained a separate single-layer classifier in tandem with the autoencoder using the latent representation as input. We did not optimize autoencoder parameters with respect to the classifier's loss, which ensures that we are measuring unsupervised representation learning performance. We repeated this procedure for latent dimensionalities of 32 and 256 (MNIST and SVHN) and 256 and 1024 (CIFAR-10).Our are shown in table 2. In all settings, using ACAI instead of the baseline autoencoder upon which it is based produced significant gains -most notably, on SVHN with a latent dimensionality of 256, the baseline achieved an accuracy of only 22.74% whereas ACAI achieved 85.14%. In general, we found the denoising autoencoder, VAE, and ACAI obtained significantly higher performance compared to the remaining models. On MNIST and SVHN, ACAI achieved the best accuracy by a significant margin; on CIFAR-10, the performance of ACAI and the denoising autoencoder was similar. By way of comparison, we found a single-layer classifier applied directly to (flattened) image pixels achieved an accuracy of 92.31%, 23.48%, and 39.70% on MNIST, SVHN, and CIFAR-10 respectively, so classifying using the representation learned by ACAI provides a huge benefit. Clustering If an autoencoder groups points with common salient characteristics close together in latent space without observing any labels, it arguably has uncovered some important structure in the data in an unsupervised fashion. A more difficult test of an autoencoder is therefore clustering its latent space, i.e. separating the latent codes for a dataset into distinct groups without using any labels. To test the clusterability of the latent spaces learned by different autoencoders, we simply apply K-Means clustering BID26 to the latent codes for a given dataset. Since K-Means uses Euclidean distance, it is sensitive to each dimension's relative variance. We therefore used PCA whitening on the latent space learned by each autoencoder to normalize the variance of its dimensions prior to clustering. K-Means can exhibit highly variable depending on how it is initialized, so for each autoencoder we ran K-Means 1,000 times from different random initializations and chose the clustering with the best objective value on the training set. For evaluation, we adopt the methodology of BID46 BID16: Given that the dataset in question has labels (which are not used for training the model, the clustering algorithm, or choice of random initialization), we can cluster the data into C distinct groups where C is the number of classes in the dataset. We then compute the "clustering accuracy", which is simply the accuracy corresponding to the optimal one-to-one mapping of cluster IDs and classes BID46.Our are shown in table 3. On both MNIST and SVHN, ACAI achieved the best or second-best performance for both d z = 32 and d z = 256. We do not report on CIFAR-10 because all of the autoencoders we studied achieved a near-random clustering accuracy. Previous efforts to evaluate clustering performance on CIFAR-10 use learned feature representations from a convolutional network trained on ImageNet BID16 which we believe only indirectly measures unsupervised learning capabilities. In this paper, we have provided an in-depth perspective on interpolation in autoencoders. We proposed Adversarially Constrained Autoencoder Interpolation (ACAI), which uses a critic to encourage interpolated datapoints to be more realistic. To make interpolation a quantifiable concept, we proposed a synthetic benchmark and showed that ACAI substantially outperformed common autoencoder models. This task also yielded unexpected insights, such as that a VAE which has effectively learned the data distribution might not interpolate. We also studied the effect of improved interpolation on downstream tasks, and showed that ACAI led to improved performance for feature learning and unsupervised clustering. These findings confirm our intuition that improving the interpolation abilities of a baseline autoencoder can also produce a better learned representation for downstream tasks. However, we emphasize that we do not claim that good interpolation always implies a good representation -for example, the AAE produced smooth and realistic interpolations but fared poorly in our representations learning experiments and the denoising autoencoder had low-quality interpolations but provided a useful representation. In future work, we are interested in investigating whether our regularizer improves the performance of autoencoders other than the standard "vanilla" autoencoder we applied it to. In this paper, we primarily focused on image datasets due to the ease of visualizing interpolations, but we are also interested in applying these ideas to non-image datasets. A LINE BENCHMARK Some example data and interpolations for our synthetic lines benchmark are shown in FIG2. Full discussion of this benchmark is available in section 3.1. We define our Mean Distance and Smoothness metrics as follows: Let x 1 and x 2 be two input images we are interpolating between and DISPLAYFORM0 be the decoded point corresponding to mixing x 1 and x 2's latent codes using coefficient α = n−1 /N−1. The imagesx n, n ∈ {1, . . ., N} then comprise a length-N interpolation between x 1 and x 2. To produce our evaluation metrics, we first find the closest true datapoint (according to cosine distance) for each of the N intermediate images along the interpolation. Finding the closest image among all possible line images is infeasible; instead we first generate a size-D collection of line images D with corresponding angles Λ q, q ∈ {1, . . ., D} spaced evenly between 0 and 2π. Then, we match each image in the interpolation to a real datapoint by finding DISPLAYFORM1 for n ∈ {1, . . ., N}, where C n,q is the cosine distance betweenx n and the qth entry of D. To capture the notion of "intermediate points look realistic", we compute DISPLAYFORM2 We now define a perfectly smooth interpolation to be one which consists of lines with angles which linearly move from the angle of D q 1 to that of D q N. Note that if, for example, the interpolated lines go from Λ q 1 = π /10 to Λ q N = 19π /10 then the angles corresponding to the shortest path will have a discontinuity from 0 to 2π. To avoid this, we first "unwrap" the angles {Λ q 1, . . ., Λ q N} by removing discontinuities larger than π by adding multiples of ±2π when the absolute difference between Λ q n−1 and Λ q n is greater than π to produce the angle sequence {Λ q 1, . . .,Λ q N}. 2 Then, we define a measure of smoothness as DISPLAYFORM3 In other words, we measure the how much larger the largest change in (normalized) angle is compared to the minimum possible value (1 /(N−1)). B BASE MODEL ARCHITECTURE AND TRAINING PROCEDUREAll of the autoencoder models we studied in this paper used the following architecture and training procedure: The encoder consists of blocks of two consecutive 3 × 3 convolutional layers followed by 2 × 2 average pooling. All convolutions (in the encoder and decoder) are zero-padded so that the input and output height and width are equal. The number of channels is doubled before each average pooling layer. Two more 3 × 3 convolutions are then performed, the last one without activation and the final output is used as the latent representation. All convolutional layers except for the final use a leaky ReLU nonlinearity BID25. For experiments on the synthetic "lines" task, the convolution-average pool blocks are repeated 4 times until we reach a latent dimensionality of 64.For subsequent experiments on real datasets (section 4), we repeat the blocks 3 times, ing in a latent dimensionality of 256.The decoder consists of blocks of two consecutive 3 × 3 convolutional layers with leaky ReLU nonlinearities followed by 2 × 2 nearest neighbor upsampling BID32. The number of channels is halved after each upsampling layer. These blocks are repeated until we reach the target resolution (32 × 32 in all experiments). Two more 3 × 3 convolutions are then performed, the last one without activation and with a number of channels equal to the number of desired colors. All parameters are initialized as zero-mean Gaussian random variables with a standard deviation of 1 / √ fan_in(1+0. In fig. 5, we show some samples from our VAE trained on the synthetic line benchmark. The VAE generally produces realistic samples and seems to cover the data distribution well, despite the fact that it does not produce high-quality interpolations ( fig. 3c). In this section, we provide a series of figures (figs. 6 to 11) showing interpolation behavior for the different autoencoders we studied. Further discussion of these is available in section 3.

Represent the short summary for finding the corresponding scientific article (long)

We propose a regularizer that improves interpolation and autoencoders and show that it also improves the learned representation for downstream tasks.

Represent the TLDR for finding the corresponding scientific article

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: We consider the problem of generating plausible and diverse video sequences, when we are only given a start and an end frame. This task is also known as inbetweening, and it belongs to the broader area of stochastic video generation, which is generally approached by means of recurrent neural networks (RNN). In this paper, we propose instead a fully convolutional model to generate video sequences directly in the pixel domain. We first obtain a latent video representation using a stochastic fusion mechanism that learns how to incorporate information from the start and end frames. Our model learns to produce such latent representation by progressively increasing the temporal resolution, and then decode in the spatiotemporal domain using 3D convolutions. The model is trained end-to-end by minimizing an adversarial loss. Experiments on several widely-used benchmark datasets show that it is able to generate meaningful and diverse in-between video sequences, according to both quantitative and qualitative evaluations. Imagine if we could teach an intelligent system to automatically turn comic books into animations. Being able to do so would undoubtedly revolutionize the animation industry. Although such an immensely labor-saving capability is still beyond the current state-of-the-art, advances in computer vision and machine learning are making it an increasingly more tangible goal. Situated at the heart of this challenge is video inbetweening, that is, the process of creating intermediate frames between two given key frames. Recent development in artificial neural network architectures and the emergence of generative adversarial networks (GAN) have led to rapid advancement in image and video synthesis (Aigner & Körner, 2018;). At the same time, the problem of inbetweening has received much less attention. The majority of the existing works focus on two different tasks: i) unconditional video generation, where the model learns the input data distribution during training and generates new plausible videos without receiving further input (; ;); and ii) video prediction, where the model is given a certain number of past frames and it learns to predict how the video evolves thereafter (; ; ;). In most cases, the generative process is modeled as a recurrent neural network (RNN) using either long-short term memory (LSTM) cells or gated recurrent units (GRU) . Indeed, it is generally assumed that some form of a recurrent model is necessary to capture long-term dependencies, when the goal is to generate videos over a length that cannot be handled by pure frame-interpolation methods based on optical flow. In this paper, we show that it is in fact possible to address the problem of video inbetweening using a stateless, fully convolutional model. A major advantage of this approach is its simplicity. The absence of recurrent components implies shorter gradient paths, hence allowing for deeper networks and more stable training. The model is also more easily parallelizable, due to the lack of sequential states. Moreover, in a convolutional model, it is straightforward to enforce temporal consistency with the start and end frames given as inputs. Motivated by these observations, we make the following contributions in this paper: • We propose a fully convolutional model to address the task of video inbetweening. The proposed model consists of three main components: i) a 2D-convolutional image encoder, which maps the input key frames to a latent space; ii) a 3D-convolutional latent representation generator, which learns how to incorporate the information contained in the input frames with progressively increasing temporal resolution; and iii) a video generator, which uses transposed 3D-convolutions to decode the latent representation into video frames. • Our key finding is that separating the generation of the latent representation from video decoding is of crucial importance to successfully address video inbetweening. Indeed, attempting to generate the final video directly from the encoded representations of the start and end frames tends to perform poorly, as further demonstrated in Section 4. To this end, we carefully design the latent representation generator to stochastically fuse the key frame representations and progressively increase the temporal resolution of the generated video. • We carried out extensive experiments on several widely used benchmark datasets, and demonstrate that the model is able to produce realistic video sequences, considering key frames that are well over a half second apart from each other. In addition, we show that it is possible to generate diverse sequences given the same start and end frames, by simply varying the input noise vector driving the generative process. The rest of the paper is organized as follows: We review the outstanding literature related to our work in Section 2. Section 3 describes our proposed model in details. Experimental , both quantitative and qualitative, are presented in Section 4, followed by our in Section 5. Recent advances based on deep networks have led to tremendous progress in three areas related to the current work: i) video prediction, ii) video generation and iii) video interpolation. Video prediction: Video prediction addresses the problem of producing future frames given one (or more) past frames of a video sequence. The methods that belong to this group are deterministic, in the sense that always produce the same output for the same input and they are trained to minimize the L2 loss between the ground truth and the predicted future frames. Most of the early works in this area adopted recurrent neural networks to model the temporal dynamics of video sequences. a LSTM encoder-decoder framework is used to learn video representations of image patches. The work in extends the prediction to video frames rather than patches, training a convolutional LSTM. The underlying idea is to compute the next frame by first predicting the motions of either individual pixels or image segments and then merge these predictions via masking. A multi-layer LSTM is also used in , progressively refining the prediction error. Some methods do not use recurrent networks to address the problem of video prediction. For example, a 3D convolutional neural network is adopted in. An adversarial loss is used in addition to the L2 loss to ensure that the predicted frames look realistic. More recently, Aigner & Körner proposed a similar approach, though in this case layers are added progressively to increase the image resolution during training . All the aforementioned methods aim at predicting the future frames in the pixel domain directly. An alternative approach is to first estimate local and global transformations (e.g., affine warping and local filters), and then apply them to each frame to predict the next, by locally warping the image content accordingly (; a; van). Video generation: Video generation differs from video prediction in that it aims at modelling future frames in a probabilistic manner, so as to generate diverse and plausible video sequences. To this end, methods based on generative adversarial networks (GAN) and variational autoencoder networks (VAN) are being currently explored in the literature. a GAN architecture is proposed, which consists of two generators (to produce, respectively, foreground and static pixels), and a discriminator to distinguish between real and generated video sequences. generates the whole output video sequence from a single latent vector, in a temporal generator is first used to produce a sequence of latent vectors that captures the temporal dynamics. Subsequently an image generator produces the output images from the latent vectors. Both the generators and the discriminator are based on CNNs. The model is also able to generate video sequences conditionally on an input label, as well as interpolating between frames by first linearly interpolating the temporal latent vectors. To address mode collapse in proposes to use a variational approach. Each frame is recursively generated combining the previous frame encoding with a latent vector. This is fed to a LSTM, whose output goes through a decoder. Similarly to this, samples a latent vector, which is then used as conditioning for the deterministic frame prediction network in. A variational approach is used to learn how to sample the latent vector, conditional on the past frames. Other methods do not attempt to predict the pixels of the future frame directly. Conversely, a variational autoencoder is trained to generate plausible differences between consecutive frames , or motion trajectories . Recently, proposed to use a loss function that combines a variational loss (to produce diverse videos) with an adversarial loss (to generate realistic frames) . Video sequences can be modelled as two distinct components: content and motion. the latent vector from which the video is generated is divided in two parts: content and motion. This leads to improved quality of the generated sequences when compared with previous approaches . A similar idea is explored in Villegas et al. (2017a), where two encoders, one for motion and one for content, are used to produce hidden representations that are then decoded to a video sequence. Also explicitly separates motion and content in two streams, which are generated by means of a variational network and then fused to produce the predicted sequence. An adversarial loss is then used to improve the realism of the generated videos. All of the aforementioned methods are able to predict or generate just a few video frames into the future. Long-term video prediction has been originally addressed in with the goal of predicting up to 100 future frames of an Atari game. The current frame is encoded using a CNN or LSTM, transformed conditionally on the player action, and decoded into the next frame. More recently, Villegas et al. (2017b) addressed a similar problem, but for the case of real-world video sequences. The key idea is to first estimate high-level structures from past frames (e.g., human poses). Then, a LSTM is used to predict a sequence of future structures, which are decoded to future frames. One shortcoming of Villegas et al. (2017b) is that it requires ground truth landmarks as supervision. This is addressed in , which proposes a fully unsupervised method that learns to predict a high-level encoding into the future. Then, a decoder with access to the first frame generates the future frames from the predicted high-level encoding. Video interpolation: Video interpolation is used to increase the temporal resolution of the input video sequence. This is addressed with different approaches: optical flow based interpolation (;, phase-based interpolation , and pixels motion transformation . These method typically target temporal super-resolution and the frame rate of the input sequence is often already sufficiently high. Interpolating frames becomes more difficult when the temporal distance between consecutive frames increases. Long-term video interpolation received far less attention in the past literature. Deterministic approaches have been explored using either block-based motion estimation/compensation , or convolutional LSTM models . Our work is closer to those using generative approaches. In Chen et al. (2017b) two convolutional encoders are used to generate hidden representations of both the first and last frame, which are then fed to a decoder to reconstruct all frames in between. A variational approach is presented in. A multi-layer convolutional LSTM is used to interpolate frames given a set of extended reference frames, with the goal of increasing the temporal resolution from 2 fps to 16 fps. In our experiments, we compare our method with those in , , and. The proposed model receives three inputs: a start frame x s, an end frame x e, and a Gaussian noise vector u ∈ R D. The output of the model is a video (x s,x 1, . . .,x T −2, x e), where different se- Up-sample (time-wise) Sigmoid Sigmoid × × Figure 1: Layout of the model used to generate the latent video representation z. The inputs are the encoded representations of the start and and frames E(x s) and E(x e), together with a noise vector u. quences of plausible in-between frames (x i) are generated by feeding different instantiations of the noise vector u. In the rest of this paper, we set T = 16 and D = 128. The model consists of three components: an image encoder, a latent representation generator and a video generator. In addition, a video discriminator and an image discriminator are added so that the whole model can be trained using adversarial learning to produce realistic video sequences. The image encoder E(x) receives as input a video frame of size H 0 × W 0 and produces a feature map of shape H × W × C, where C is the number of channels. The encoder architecture consists of six layers, alternating between 4 × 4 convolutions with stride-2 down-sampling and regular 3 × 3 convolutions, followed by a final layer to condense the feature map to the target depth C. This in spatial dimensions H = H 0 /8 and W = W 0 /8. We set C = 64 in all our experiments. The latent representation generator G Z (·) receives as input E(x s), E(x e) and u, and produces an output tensor of shape T × H × W × C. Its main function is to gradually fill in the video content between the start and end frames, working directly in the latent space defined by the image encoder. The model architecture is composed of a series of L residual blocks , each consisting of 3D convolutions and stochastic fusion with the encoded representations of x s and x e. This way, each block progressively learns a transformation that improves the video content generated by the previous block. The generic l-th block is represented by the inner rectangle in Figure 1. Note that the lengths of the intermediate representations can differ from the final video length T, due to the use of a coarse-to-fine scheme in the time dimension. To simplify the notation, we defer its description to the end of this section and omit the implied temporal up-sampling from the equations. Let T (l) denote the representation length within block l. First, we produce a layer-specific noise tensor of shape T (l) × C by applying a linear transformation to the input noise vector u: where (l) C, and reshaping the into a T (l) × C tensor u (l). This is used to drive two stochastic "gating" functions for the start and end frames, respectively: where * denotes convolution along the time dimension, k s, k e are kernels of width 3 and depth C, and σ(·) is the sigmoid activation function. The gating functions are used to progressively fuse the encoded representations of the start and end frames with the intermediate output of the previous layer z (l−1), as described by the following equation: where n n denotes an additional learned stochastic component added to stimulate diversity in the generative process. Note that z (l) in has shape T (l) × H × W × C. Therefore, to compute the component-wise multiplication ·, E(x s) and E(x e) (each of shape S × S × C) are broadcast (i.e., replicated uniformly) T (l) times along the time dimension, while g e and n (l) (each of shape T (l) × C) are broadcast H × W times over the spatial dimensions. The idea of the fusion step is similar to that of StyleGAN, albeit with different construction and purposes. Finally, the fused input is convolved spatially and temporally with 3 × 3 × 3 kernels k 2 in a residual unit : where h is the leaky ReLU activation function (with parameter α = 0.2). given E(x s) and E(x e), with A, k, b being its learnable parameters. The generation of the overall latent video representation z ∈ R T ×S×S×C can be expressed as: Coarse-to-fine generation: For computational efficiency, we adopt a coarse-to-fine scheme in the time dimension, represented by the outer dashed rectangle in Figure 1. More specifically we double the length of z (l) every L/3 generator blocks, i.e., z,..., z (L/3) have length T /4 = 4, z (L/3+1),..., z (2L/3) have T /2 = 8, and z (2L/3+1),..., z (L) have the full temporal resolution T = 16. We initialize z to (E(x s), E(x e)) (which becomes (E(x s), E(x s), E(x e), E(x e)) after the first up-sampling) and set L = 24, ing in 8 blocks per granularity level. The video generator G V produces the output video sequence (x s,x 1,x 2, . . ., x e) = G V (z) from the latent video representation z using spatially transposed 3D convolutions. The generator architecture alternates between 3 × 3 × 3 regular convolutions and transposed 3 × 4 × 4 convolutions with a stride of, hence applying only spatial (but not temporal) up-sampling. Note that it actually generates all T frames including the "reconstructed" start framex 0 and end framex T −1, though they are not used and are always replaced by the real x s and x e in the output. We train our model end-to-end by minimizing an adversarial loss function. To this end, we train two discriminators: a 3D convolutional video discriminator D V and a 2D convolutional image discriminator D I, following the approach of. The video discriminator has a similar architecture to , except that in our case we produce a single output for the entire video rather than for its sub-volumes ("patches"). For the image discriminator, we use a Resnet-based architecture instead of the DCGAN-based architecture used in. Let X = (x s, x 1, . . ., x T −2, x e) denote a real video andX = (x s,x 1, . . .,x T −2, x e) denote the corresponding generated video conditioned on x s and x e. Adopting the non-saturating log-loss, training amounts to optimizing the following adversarial objectives: During optimization we replace the average over the T − 2 intermediate frames with a single uniformly sampled frame to save computation, as is done in. This does not change the convergence properties of stochastic gradient descent, since the two quantities have the same expectation. We regularize the discriminators by penalizing the derivatives of the pre-sigmoid logits with respect to their input videos and images, as is proposed in to improve GAN stability and prevent mode collapse. In our case, instead of the adaptive scheme of , we opt for a constant coefficient of 0.1 for the gradient magnitude, which we found to be more reliable in our experiments. We use batch normalization on all 2D and 3D convolutional layers in the generator and layer normalization in the discriminators. 1D convolutions and fully-connected layers are not normalized. Architectural details of the encoder, decoder, and discriminators are further provided in Appendix A. We evaluated our approach on three well-known public datasets: BAIR robot pushing , KTH Action Database , and UCF101 Action Recognition Data Set . All video frames were down-sampled and cropped to 64×64, and subsequences of 16 frames were used in all the experiments, that is, 14 intermediate frames are generated. The videos in KTH and UCF101 datasets are 25 fps, translating to key frames 0.6 seconds apart. The frame rate of BAIR videos is not provided, though visually it appears to be much lower, hence longer time in between key frames. For all the datasets, we adopted the conventional train/test splits practised in the literature. A validation set held out from the training set was used for model checkpoint selection. More details on the exact splits are provided in Appendix B. We did not use any dataset-specific tuning of hyper-parameters, architectural choices, or training schemes. Our main objective is to generate plausible transition sequences with characteristics similar to real videos, rather than predicting the exact content of the original sequence from which the key frames were extracted. Therefore we use the recently proposed Fréchet video distance (FVD) as our primary evaluation metrics. The FVD is equivalent to the Fréchet Inception distance (FID) widely used for evaluating image generative models, but revisited in a way that it can be applied to evaluate videos, by adopting a deep neural network architecture that computes video embeddings taking the temporal dimension explicitly into account. The FVD is a more suitable metrics for evaluating video inbetweening than the widely used structural similarity index (SSIM) . The latter is suitable when evaluating prediction tasks, as it compares each synthetized frame with the original reference at the pixel level. Conversely, FVD compares the distributions of generated and ground-truth videos in an embedding space, thus measuring whether the synthesized video belongs to the distribution of realistic videos. Since the FVD was only recently proposed, we also report the SSIM to be able to compare with the previous literature. During testing, we ran the model 100 times for each pair of key frames, feeding different instances of the noise vector u to generate different sequences consistent with the given key frames, and computed the FVD for each of these stochastic generations. This entire procedure was repeated 10 121 [0.112, 0.129] times for each model variant and dataset to account for the randomness in training. We report the mean over all training runs and stochastic generations as well as the confidence intervals obtained by means of the bootstrap method . For training we used the ADAM optimizer with β 1 = 0.5, β 2 = 0.999, = 10 −8, and ran it on batches of 32 samples with a conservative learning rate of 5 × 10 − 5 for 500,000 steps. A checkpoint was saved every 5000 steps, ing in 100 checkpoints. Training took around 5 days on a single Nvidia Tesla V100 GPU. The checkpoint for evaluation was selected to be the one with the lowest FVD on the validation set. To assess the impact of the stochastic fusion mechanism as well the importance of having a separate latent video representation generator component, we compare the full model with baselines in which the corresponding components are omitted. • Baseline without fusion: The gating functions (Equation 2 and 3) are omitted and Equation 4 reduces to z • Naïve: The entire latent video representation generator described in Section 3.2 is omitted. Instead, decoding with transposed 3D convolution is performed directly on the (stacked) start/end frame encoded representations z = (E(x 1), E(x N)) (which has dimensionality 2×8×8), using a stride of 2 in both spatial and temporal dimensions when up-scaling, to eventually produce 16 64×64 frames. To maintain stochasticity in the generative process, a spatially uniform noise map is generated by sampling a Gaussian vector u, applying a (learned) linear transform, and adding the in the latent space before decoding. The in Table 1 shows that the dedicated latent video representation generator is indispensable, as the naïve baseline performs rather poorly. Moreover, stochastic fusion improves the quality of video generation. Note that the differences are statistically significant at 95% confidence level across all three datasets. To illustrate the generated videos, Figure 2 shows some exemplary outputs of our full model. The generated sequence is not expected (or even desired) to reproduce the ground truth, but only needs to be similar in style and consistent with the given start and end frames. The samples show that the model does well in this area. For stochastic generation, good models should produce samples that are not only high-quality but also diverse. Following the approach of, we measure diversity by means of the average pairwise cosine distance (i.e., 1 − cosine similarity) in the FVD embedding space among samples generated from the same start/end frames. The Table 2 shows that incorporating fusion increases sample diversity and the difference is statistically significant. A qualitative illustration of the diversity in the generated videos is further illustrated in Figure 3, where we take the average of 100 generated videos conditioned on the same start and end frames. If the robot arm has a very diverse set of trajectories, we should expect to see it "diffuse" into the due to averaging. Indeed this is the case, especially near the middle of the sequence. Finally we computed the average SSIM for our method for each dataset in order to compare our with those previously reported in the literature, before the FVD metrics was introduced. The are shown in Table 3 alongside several existing methods that are capable of video inbetweening, ranging from RNN-based video generation to optical flow-based interpolation 1. Note that the competing methods generate 7 frames 1 The numbers for these methods are cited directly from. Table 3: Average SSIM of our model using direct 3D convolution and alternative methods based on RNN, such as SDVI, or optical flow, such as SepConv and SuperSloMo . Higher is better. Note the difference in setup: our model spans a time base twice as long as the others. The SSIM for each test example is computed on the best sequence out of 100 stochastic generations, as per standard practice (;;. We report the mean and the 95%-confidence interval for our model over 10 training runs. and are conditioned on potentially multiple frames before and after. In contrast our model generates 14 frames, i.e., over a time base twice as long, and it is conditioned on only one frame before and after. Consequently, the SSIM figures are not directly comparable. However it is interesting to see that on UCF101, the most challenging dataset among the three, our model attains higher SSIM than all the other methods despite having to generate much longer sequences. This demonstrates the potential of the direct convolution approach to outperform existing methods, especially on difficult tasks. It is also worth noting from Table 3 that purely optical flow-based interpolation methods achieve essentially the same level of SSIM as the sophisticated RNN-based SDVI on BAIR and KTH, which suggests either that a 7-frame time base is insufficient in length to truly test video inbetweening models or that the SSIM is not an ideal metric for this task. We presented a method for video inbetweening using only direct 3D convolutions. Despite having no recurrent components, our model produces good performance on most widely-used benchmark datasets. The key to success for this approach is a dedicated component that learns a latent video representation, decoupled from the final video decoding phase. A stochastic gating mechanism is used to progressively fuse the information of the given key frames. The rather surprising fact that video inbetweening can be achieved over such a long time base without sophisticated recurrent models may provide a useful alternative perspective for future research on video generation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

This paper presents method for stochastically generating in-between video frames from given key frames, using direct 3D convolutions.

represent the short summary for finding the corresponding scientific article (long)

We propose a novel way to incorporate conditional image information into the discriminator of GANs using feature fusion that can be used for structured prediction tasks.


Question: Represent the article for finding a single-sentence TLDR

Question: Aligning knowledge graphs from different sources or languages, which aims to align both the entity and relation, is critical to a variety of applications such as knowledge graph construction and question answering. Existing methods of knowledge graph alignment usually rely on a large number of aligned knowledge triplets to train effective models. However, these aligned triplets may not be available or are expensive to obtain for many domains. Therefore, in this paper we study how to design fully-unsupervised methods or weakly-supervised methods, i.e., to align knowledge graphs without or with only a few aligned triplets. We propose an unsupervised framework based on adversarial training, which is able to map the entities and relations in a source knowledge graph to those in a target knowledge graph. This framework can be further seamlessly integrated with existing supervised methods, where only a limited number of aligned triplets are utilized as guidance. Experiments on real-world datasets prove the effectiveness of our proposed approach in both the weakly-supervised and unsupervised settings. Knowledge graphs represent a collection of knowledge facts and are quite popular in the real world. Each fact is represented as a triplet (h, r, t), meaning that the head entity h has the relation r with the tail entity t. Examples of real-world knowledge graphs include instances which contain knowledge facts from general domain in different languages (Freebase 1, DBPedia BID2, Yago BID19, WordNet 2) or facts from specific domains such as biomedical ontology (UMLS 3). Knowledge graphs are critical to a variety of applications such as question answering BID4 ) and semantic search BID13 ), which are attracting growing interest recently in both academia and industry communities. In practice, each knowledge graph is usually constructed from a single source or language, the coverage of which is limited. To enlarge the coverage and construct more unified knowledge graphs, a natural idea is to integrate multiple knowledge graphs from different sources or languages BID0 ). However, different knowledge graphs use distinct symbol systems to represent entities and relations, which are not compatible. As a , it is necessary to align entities and relations across different knowledge graphs (a.k.a., knowledge graph alignment) before integrating them. Indeed, there are some recent studies focusing on aligning entities and relations from a source knowledge graph to a target knowledge graph (BID23 ; BID6 ; BID7). These methods typically represent entities and relations in a low-dimensional space, and meanwhile learn a mapping function to align entities and relations from the source knowledge graph to the target one. However, these methods usually rely on a large number of aligned triplets as labeled data to train effective alignment models. In reality, the aligned triplets may not be available or can be expensive to obtain, making existing methods fail to achieve satisfactory . Therefore, we are seeking for an unsupervised or weakly-supervised approach, which is able to align knowledge graphs with a few or even without labeled data. In this paper, we propose an unsupervised approach for knowledge graph alignment with the adversarial training framework BID11. Our proposed approach aims to learn alignment functions, i.e., P e (e tgt |e src) and P r (r tgt |r src), to map the entities and relations (e src and r src) from the source knowledge graph to those (e tgt and r tgt) in the target graph, without any labeled data. Towards this goal, we notice that we can align each triplet in the source knowledge graph with one in the target knowledge graph by aligning the head/tail entities and relation respectively. Ideally, the optimal alignment functions would align all the source triplets to some valid triplets (i.e., triplets expressing true facts). Therefore, we can enhance the alignment functions by improving the plausibility of the aligned triplets. With this intuition, we train a triplet discriminator to distinguish between the real triplets in the target knowledge graph and those aligned from the source graph, which provides a reward function to measure the plausibility of a triplet. Meanwhile, the alignment functions are optimized to maximize the reward. The above process naturally forms an adversarial training procedure BID11 ). By alternatively optimizing the alignment functions and the discriminator, the discriminator can consistently enhance the alignment functions. However, the above approach may suffer from the problem of mode collapse BID17 ). Specifically, many entities in the source knowledge graph may be aligned to only a few entities in the target knowledge graph. This problem can be addressed if the aggregated posterior entity distribution e src P e (e tgt |e src)P (e src) derived by the alignment functions matches the prior entity distribution P (e tgt) in the target knowledge graph. Therefore, we match them with another adversarial training framework, which shares similar idea with adversarial auto-encoders BID16 ).The whole framework can also be seamlessly integrated with existing supervised methods, in which we can use a few aligned entities or relations as guidance, yielding a weakly-supervised approach. Our approach can be effectively optimized with stochastic gradient descent, where the gradient for the alignment functions is calculated by the REINFORCE algorithm . We conduct extensive experiments on several real-world knowledge graphs. Experimental prove the effectiveness of our proposed approach in both the weakly-supervised and unsupervised settings. Our work is related to knowledge graph embedding, that is, embedding knowledge graphs into lowdimensional spaces, in which each entity and relation is represented as a low-dimensional vector (a.k.a., embedding). A variety of knowledge graph embedding approaches have been proposed BID3; BID21 ), which can effectively preserve the semantic similarities of entities and relations into the learned embeddings. We treat these techniques as tools to learn entity and relation embeddings, which are further used as features for knowledge graph alignment. In literature, there are also some studies focusing on knowledge graph alignment. Most of them perform alignment by considering contextual features of entities and relations, such as their names BID15 ) or text descriptions BID8; BID20; BID21 ). However, such contextual features are not always available, and therefore these methods cannot generalize to most knowledge graphs. In this paper, we consider the most general case, in which only the triplets in knowledge graphs are used for alignment. The studies most related to ours are BID23 and BID6. Similar to our approach, they treat the entity and relation embeddings as features, and jointly train an alignment model. However, they totally rely on the labeled data (e.g., aligned entities or relations) to train the alignment model, whereas our approach incorporates additional signals by using adversarial training, and therefore achieves better in the weakly-supervised and unsupervised settings. More broadly, our work belongs to the family of domain alignment, which aims at mapping data from one domain to data in the other domain. With the success of generative adversarial networks BID11 ), many researchers have been bringing the idea to domain alignment, getting impressive in many applications, such as image-to-image translation BID24; BID25 ), word-to-word translation BID9 ) and text style transfer BID18 ). They typically train a domain discriminator to distinguish between data points from different domains, and then the alignment function is optimized by fooling the discriminator. Our approach shares similar idea, but is designed with some specific intuitions in knowledge graphs. Source Knowledge Graph Aligned Knowledge Graph Triplet Discriminator Figure 1: Framework overview. By applying the alignment functions to the triplets in the source knowledge graph, we obtain an aligned knowledge graph. The alignment functions are learned through two GANs. We expect all triplets in the aligned knowledge graph are valid, therefore we train a triplet discriminator to distinguish between valid and invalid triplets, and further use it to facilitate the alignment functions. FORMULA1 We also expect the entity distribution in the aligned knowledge graph matches the one in the target knowledge graph, which is achieved with another GAN. Definition 1 (KNOWLEDGE GRAPH.) A knowledge graph is denoted as G = (E, R, T), where E is a set of entities, R is a set of relations and T is a set of triplets. Each triplet (h, r, t) consists of a head entity h, a relation r and a tail entity t, meaning that entity h has relation r with entity t. In practice, the coverage of each individual knowledge graph is usually limited, since it is typically constructed from a single source or language. To construct knowledge graphs with broader coverage, a straightforward way is to integrate multiple knowledge graphs from different sources or languages. However, each knowledge graph uses a unique symbol system to represent entities and relations, which is not compatible with other knowledge graphs. Therefore, a prerequisite for knowledge graph integration is to align entities and relations across different knowledge graphs (a.k.a., knowledge graph alignment). In this paper, we study how to align entities and relations from a source knowledge graph to those in a target knowledge graph, and the problem is formally defined below:Definition 2 (KNOWLEDGE GRAPH ALIGNMENT.) Given a source knowledge graph G src = (E src, R src, T src) and a target knowledge graph G tgt = (E tgt, R tgt, T tgt), the problem aims at learning an entity alignment function P e and a relation alignment function P r. Given an entity e src in the source knowledge graph and an entity e tgt in the target knowledge graph, P e (e tgt |e src) gives the probability that e src aligns to e tgt. Similarly, for a source relation r src and a target relation r tgt, P r (r tgt |r src) gives the probability that r src aligns to r tgt. In this paper we propose an unsupervised approach to learning the alignment functions, i.e., P e (e tgt |e src) and P r (r tgt |r src), for knowledge graph alignment. To learn them without any supervision, we notice that we can align each triplet in the source knowledge graph with one in the target knowledge graph by aligning the head/tail entities and relation respectively. For an ideal alignment model, all the aligned triplets should be valid ones (i.e., triplets expressing true facts). As a , we can improve the alignment functions by raising the plausibility of the aligned triplets. With the intuition, our approach trains a triplet discriminator to distinguish between valid triplets and other ones. Then we build a reward function from the discriminator to facilitate the alignment functions. However, using the triplet discriminator alone may cause the problem of mode collapse. More specifically, many entities in the source knowledge graph are aligned to only a few entities in the target knowledge graph. This problem can be addressed if the aggregated posterior distribution of entities derived by the alignment functions matches the prior entity distribution from the target knowledge graph. Therefore, we follow the idea in adversarial auto-encoders BID16 ), and leverage another adversarial training framework to regularize the distribution. The above strategies yield an unsupervised approach. However, in many cases, the structures of the source and target knowledge graphs (e.g., entity and triplet distributions) can be very different, making our unsupervised approach unable to perform effective alignment. In such cases, we can integrate our approach with existing supervised methods, and use a few labeled data as guidance, which further yields a weakly-supervised approach. Formally, our approach starts by learning entity and relation embeddings with existing knowledge graph embedding techniques, which are denoted as {x e src} e src ∈E src, {x e tgt} e tgt ∈E tgt and {x r src} r src ∈R src, {x r tgt} r tgt ∈R tgt. The learned embeddings preserve the semantic correlations of entities and relations, hence we treat them as features and build our alignment functions on top of them. Specifically, we define the probability that a source entity e src or relation r src aligns to a target entity e tgt or relation r tgt as follows: DISPLAYFORM0 where γ is a temperature parameter, Z is a normalization term. W is a linear projection matrix, which maps an embedding in the source knowledge graph (e.g., x e src) to one in the target graph (e.g., W x e src), so that we can perform alignment by calculating the distance between the mapped source embeddings (e.g., W x e src) and the embeddings in the target graph (e.g., x e tgt). Note that W is the only parameter to be learned, and it is shared across the entity and relation alignment functions. We also try independent projection matrices or nonlinear projections, but get inferior . In the following chapters, we first briefly introduce the method for learning entity and relation embeddings (Section 4.1). Then, we introduce how we leverage the triplet discriminator (Section 4.2) and the regularization mechanism (Section 4.3) to facilitate training the alignment functions. Afterwards, we introduce a simple supervised method as an example, to illustrate how to incorporate labeled data (Section 4.4). Finally, we introduce our optimization algorithm (Section 4.5). In this paper, we leverage the TransE algorithm BID3 ) for entity and relation embedding learning, due to its simplicity and effectiveness in a wide range of datasets. In general, we can also use other knowledge graph embedding algorithms as well. Given a triplet t = (e h, r, e t), TransE defines its score as follows: DISPLAYFORM0 Then the model is trained by maximizing the margin between the scores of real triplets and random triplets, and the objective function is given below: DISPLAYFORM1 where T is the set of real triplets in the knowledge graph, T is the set of random triplets, and m is a parameter controlling the margin. By defining the alignment functions for entity and relation (Eqn. 1), we are able to align each triplet in the source knowledge graph to the target knowledge graph by aligning the entities and relation respectively. An ideal alignment function would align all the source triplets to some valid triplets. Therefore, we can enhance the alignment functions by raising the plausibility of the aligned triplets. Ideally, we would wish that all the aligned triplets are valid ones. Towards this goal, we train a triplet discriminator to distinguish between valid triplets and other ones. Then the discriminator is used to define different reward functions for guiding the alignment functions. In our approach, we train the discriminator by treating the real triplets in knowledge graphs as positive examples, and the aligned triplets generated by our approach as negative examples. Following existing studies BID11 ), we define the objective function below: DISPLAYFORM0 where t ∼ A(t src) is a triplet aligned from t src and A is defined in Eqn. 4. D t is the triplet discriminator, which concatenates the embeddings of the head/tail entities and relation in a triplet t, and further predicts the probability that t is a valid triplet. Based on the discriminator, we can construct a scalar-to-scalar reward function R to measure the plausibility of a triplet. Then the alignment functions can be trained by maximizing the reward: DISPLAYFORM1 There are several ways to define the reward function R, which essentially yields different adversarial training frameworks. For example, BID11 and BID14 Besides, we may also leverage R(x) = x, which is the first-order Taylor's expansion of − log(1−x) at x = 1 and has a limited range when x ∈. All different reward functions have the same optimal solution, i.e, the derived distribution of the aligned triplets matching the real triplet distribution in the target knowledge graph. In practice, these reward functions may have different variance, and we empirically compare them in the experiments TAB6 ).During optimization, the gradient with respect to the alignment functions cannot be calculated directly, as the triplets sampled from the alignment functions are discrete variables. Therefore, we leverage the REINFORCE algorithm , which calculates the gradient as follows: DISPLAYFORM2 where P (t|t src) = P e (e h |e src h)P r (r|r src)P e (e t |e src t) with t = (e h, r, e t), t src = (e src h, r src, e src t). Although the triplet discriminator provides effective reward to the alignment functions, many entities in the source knowledge graph can be aligned to only a few entities in the target knowledge graph. Such problems can be solved by constraining the aggregated posterior entity distribution derived by the alignment functions to match the prior entity distribution in the target knowledge graph. Formally, the aggregated posterior distribution of entities is given below: DISPLAYFORM0 where P (e src) is the entity distribution in the source knowledge graph. We expect this distribution to match the prior distribution P (e tgt), which is the entity distribution in the target knowledge graph. Following BID16, we regularize the distribution with another adversarial training framework BID11 ). During training, an entity discriminator D e is learned to distinguish between the posterior and prior distributions using the following objective function: DISPLAYFORM1 where D e takes the embedding of an entity as features to predict the probability that the entity is sampled from prior distribution P (e tgt). To enforce the posterior distribution to match the prior distribution, the entity alignment function is trained to fool the discriminator by maximizing the following objective: DISPLAYFORM2 where R is the same reward function as used in the triplet discriminator (Eqn. 6), and the gradient for the alignment functions can be similarly calculated with the REINFORCE algorithm. The above sections introduce an unsupervised approach to knowledge graph alignment. In many cases, the source and target knowledge graphs may have very different structures (e.g., entity or triplet distributions), making our approach fail to perform effective alignment. In these cases, we can integrate our approach with any supervised methods, and leverage a few labeled data (e.g., aligned entity or relation pairs) as guidance, which yields a weakly-supervised approach. In this section, we introduce a simple yet effective method to show how to utilize the labeled data. Suppose we are given some aligned entity pairs, and the aligned relation pairs can be handled in a similar way. We define our objective function as follows:O L = E (e src,e tgt)∈S log P e (e tgt |e src) − λH(P e (e tgt |e src)),where S is the set of aligned entity pairs, e src and e tgt are random variables of entities in the source and target knowledge graphs, H is the entropy of a distribution. The first term corresponds to a softmax classifier, which aims at maximizing the probability of aligning a source entity to the ground-truth target entity. The second term minimizes the entropy of the probability distribution calculated by the alignment function, which encourages the alignment function to make confident predictions. Such an entropy minimization strategy is used in many semi-supervised learning studies BID12 ). We leverage the stochastic gradient descent algorithm for optimization. In practice, we find that first pre-training the alignment functions with the given labeled data (Eqn. 11), then fine-tuning them with the triplet discriminator (Eqn. 6) and the regularization mechanism (Eqn. 8) leads to better performance, compared with jointly training all of them TAB7 ). Consequently, we adopt the pre-training and fine-tuning framework for optimization, which is summarized in Alg. 1. Update the triplet discriminator D t and the alignment functions with Eqn. 5 6. Update the entity discriminator D e and the alignment functions with Eqn. 9 10. 6: end while In experiment, we use four datasets for evaluation. In FB15k-1 and FB15k-2, the knowledge graphs have very different triplets, which can be seen as constructed from different sources; in WK15k(enfr) and WK15k(en-de), the knowledge graphs are from different languages. The statistics are summarized in TAB2. Following existing studies BID23; BID6 ), we consider the task of entity alignment, and three different settings are considered, including supervised, weakly-supervised and unsupervised settings. Hit@1 and Hit@10 are reported. • FB15k-1, FB15k-2: Following Zhu et al. (2017a), we construct two datasets from the FB15k dataset BID3 ). In FB15k-1, the two knowledge graphs share 50% triplets, and in FB15k-2 10% triplets are shared. According to the study, we use 5000 and 500 aligned entity pairs as labeled data in FB15k-1 and FB15k-2 respectively, and the rest for evaluation.• WK15k(en-fr): A bi-lingual (English and French) dataset in BID6. Some aligned triplets are provided as labeled data, and some aligned entity pairs as test data. The labeled data and test data have some overlaps, so we delete the overlapped pairs from labeled data.• WK15k(en-de): A bi-lingual (English and German) dataset used in BID6. The dataset is similar to WK15k(en-fr), so we perform preprocessing in the same way. Compared Algorithms FORMULA0 ): A supervised method for word translation, which learns the translation in a bootstrapping way. We apply the method on the pre-trained entity and relation embeddings to perform knowledge graph alignment. UWT BID9 ): An unsupervised word translation method, which leverages adversarial training and a refinement strategy. We apply the method to the entity and relation embeddings to perform alignment. KAGAN-sup: The supervised method introduced in Section 4.4, which is simple but effective, and can be easily integrated with our unsupervised approach. KAGAN: Our proposed Knowledgegraph Alignment GAN, which leverages the labeled data for pre-training, and then fine-tunes the model with the triplet discriminator and the regularization mechanism. KAGAN-t: A variant with only the triplet GAN, which first performs pre-training with the labeled data, and then performs fine-tuning with the triplet discriminator. KAGAN-e: A variant with only the entity GAN, which first pre-trains with the labeled data, and then fine-tunes with the regularization mechanism. The dimension of entity and relation embeddings is set as 512 for all compared methods. For our proposed approach, λ is set as 1, the temperature parameter γ is set as 1, and the reward function is set as x by default. SGD is used for optimization. The learning rate is set as 0.1 for pre-training and 0.001 for fine-tuning. 10% labeled pairs are treated as the validation set. The training process is terminated if the performance on the validation set drops. For the compared methods, the parameters are chosen according to the performance on the validation set. The main are presented in TAB3. In the supervised setting, our approach significantly outperforms all the compared methods. On the FB15k-1 and FB15k-2 datasets, without using any labeled data, our approach already achieves close as in the supervised setting. On the WK15k datasets under the weakly-supervised setting, our performance is comparable or even superior to the performance of other methods in the supervised setting, but with much fewer labeled data (about 13% in WK15k(en-fr) and 1% in WK15k(en-de)). Overall, our approach is quite effective in the weakly-supervised and unsupervised settings, outperforming all the baseline approaches. To understand the effect of each part in our approach, we further conduct some ablation studies. Table 3 presents the in the supervised setting. Both the triplet discriminator (KAGAN-t) and the regularization mechanism (KAGAN-e) improves the pre-trained alignment models (KAGAN-pre). Combining them (KAGAN) leads to even better . TAB5 gives the in the unsupervised (FB15k-2) and weakly-supervised (WK15k-fr2en, WK15k-de2en) settings. On the FB15k-2 dataset, using the regularization mechanism alone (KAGAN-e) already achieves impressive . This is because the source and target knowledge graphs in FB15k-2 share similar structures, and our regularization mechanism can effectively leverage such similarity for alignment. However, the performance of using only the triplet discriminator (KAGAN-t) is very poor, which is caused by the problem of mode collapse. The problem is effectively solved by integrating the approach with the regularization mechanism (KAGAN), which achieves the best in all cases. Comparison of the reward functions. In our approach, we can choose different reward functions, leading to different adversarial training frameworks. These frameworks have the same optimal solutions, but with different variance. In this section we compare them on the WK15k datasets, and the of Hit@1 are presented in TAB6. We notice that all reward functions lead to significant improvement compared with using no reward. Among them, x 1−x and x obtain the best . Comparison of the optimization methods. During training, our approach fixes the entity/relation embeddings, and uses a pre-training and fine-tuning framework for optimization. In this section, we compare the framework with some variants, and the of Hit@1 are presented in TAB7. We see that our framework (pre-training and fine-tuning) outperforms the joint training framework. Besides, fine-tuning entity/relation embeddings yields worse than fixing them during training. Case study. In this section, we present some visualization to intuitively show the effect of the triplet discriminator and regularization mechanism in our approach. We consider the unsupervised setting on the FB15k-2 dataset, and leverage the PCA algorithm to visualize certain embeddings. Figure 2 compares the entity embeddings obtained with and without the regularization mechanism, where red is for the mapped source entity embeddings (W x e src), and green for the target embeddings (x e tgt). We see that without the mechanism, many entities from the source knowledge graph are mapped to a small region (the red region), leading to the problem of mode collapse. The problem is addressed with the regularization mechanism. FIG2 compares the triplet embeddings obtained with and without the triplet discriminator, where the triplet embedding is obtained by concatenating the entity and relation embeddings in a triplet. Red color is for triplets aligned from the source knowledge graph, and green is for triplets in the target graph. Without the triplet discriminator, the aligned triplets look quite different from the real ones (under different distributions). With the triplet discriminator, the aligned triplets look like the real ones (under similar distributions). 6 This paper studied knowledge graph alignment. We proposed an unsupervised approach based on the adversarial training framework, which is able to align entities and relations from a source knowledge graph to those in a target knowledge graph. In practice, our approach can be seamlessly integrated with existing supervised methods, which enables us to leverage a few labeled data as guidance, leading to a weakly-supervised approach. Experimental on several real-world datasets proved the effectiveness of our approach in both the unsupervised and weakly-supervised settings. In the future, we plan to learn alignment functions from two directions (source to target and target to source) to further improve the , which is similar to CycleGAN BID24 ).

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper studies weakly-supervised knowledge graph alignment with adversarial training frameworks.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.