Question: Represent the scientific article to find a short TLDR

Question: Generative models with both discrete and continuous latent variables are highly motivated by the structure of many real-world data sets. They present, however, subtleties in training often manifesting in the discrete latent variable not being leveraged. In this paper, we show why such models struggle to train using traditional log-likelihood maximization, and that they are amenable to training using the Optimal Transport framework of Wasserstein Autoencoders. We find our discrete latent variable to be fully leveraged by the model when trained, without any modifications to the objective function or significant fine tuning. Our model generates comparable samples to other approaches while using relatively simple neural networks, since the discrete latent variable carries much of the descriptive burden. Furthermore, the discrete latent provides significant control over generation. Unsupervised learning using generative latent variable models provides a powerful and general approach to learning the underlying, low-dimensional structure from large, unlabeled datasets. Perhaps the two most common techniques for training such models are Variational Autoencoders (VAEs), and Generative Adversarial Networks (GANs) BID8. Both have advantages and disadvantages. VAEs provide a meaningful lower bound on the log likelihood that is stable under training, as well as an encoding distribution from the data into the latent. However, they generate blurry samples due to their objective being unable to handle deterministic decoders and tractability requiring simple priors BID12. On the other hand, GANs naturally enable deterministic generative models with sharply defined samples, but their training procedure is less stable.A relatively new approach to training generative models has emerged based on minimizing the Optimal Transport (OT) distance BID30 ) between the generative model distribution and that of the data. The OT approach provides a general framework for training generative models, which promises some of the best of both GANs and VAEs. Though interesting first have been given in; BID27;, the OT approach to generative modelling is still nascent. Our contributions are twofold: we seek to improve generative modelling capabilities with discrete and continuous latent variables, but importantly, we seek also to establish that training generative models with OT can be significantly more effective than the traditional VAE approach. Discrete latent-variable models are critical to the endeavor of unsupervised learning because of the ubiquity of discreteness in the natural world, and hence in the datasets that describe it. However, they are harder to train than their continuous counterparts. This has been tackled in a number of ways (e.g., directly mitigating high-variance discrete samples BID6 BID19, parametrizing discrete distributions using continuous ones BID14 BID22 BID29, deliberate model design leveraging conjugacy).However, even in the simple case where the number of mixtures is small enough that monte-carlo sampling from the discrete latent is avoidable, training can still be problematic. For example, in BID4 a Gaussian-mixture latent-variable model (GM-LVM) was studied, and the authors were unable to train their model on MNIST using variational inference without substantially modifying the VAE objective. What appears to happen is that the model quickly learns to "hack" the VAE objective function by collapsing the discrete latent variational distribution. This problem only occurs in the unsupervised setting, as are able to learn the discrete latent in the semi-supervised version of the same problem once they have labeled samples for the discrete latent to latch onto. This is discussed in more detail in Section 2.1.The OT approach to training generative models (in particular the Wasserstein distance, discussed in Section 2.2) induces a weaker topology on the space of distributions, enabling easier convergence of distributions than in the case of VAEs BID2. Thus, one might conjecture that the OT approach would enable easier training of GM-LVMs than the VAE approach. We provide evidence that this is indeed the case, showing that GM-LVMs can be trained in the unsupervised setting on MNIST, and motivating further the value of the OT approach to generative modelling. We consider a hierarchical generative model p G with two layers of latent variables, the highest one being discrete. Explicitly, if we denote the discrete latent k with density p D (D for discrete), and the continuous latent z with density p C (C for continuous), the generative model is given by: DISPLAYFORM0 In this work, we consider a GM-LVM with categorical distribution p D = Cat(K) and continuous DISPLAYFORM1 We refer to this GM-LVM as a GM-VAE when it is trained as a VAE or GM-WAE when trained as a Wasserstein Autoencoder ) (discussed in Section 2.2). Training GM-LVMs in the traditional VAE framework (GM-VAEs) involves maximizing the evidence lower bound (ELBO) averaged over the data. Such models are empirically hard to train BID4. This is likely due to the fact that the discrete latent variational distribution learns on a completely different scale from the generative distribution. Consequently, the discrete latent tends to instantly learn some unbalanced structure where its classes are meaningless in order to accommodate the untrained generative distribution. The generative model then learns around that structure, galvanizing the meaningless discrete distribution early in training. More explicitly, if we choose a variational distribution q(z, k|x) = q C (z|k, x) q D (k|x) to mirror the prior in Equation 1, the ELBO can be written as follows: DISPLAYFORM0 Both the first and the second term in Equation 2 depend on q D (k|x). However, the second term is much smaller than the first; it is bounded by log K for uniform p D over K classes, whereas the first term is unbounded from above (though we will initialize the modes of q C to match those of the priors making the continuous KL term initially small as well). As a consequence, q D (k|x) will immediately shut off the k values (i.e., q D (k|x) = 0 ∀x) with large reconstruction losses, DISPLAYFORM1. This is shown in the top row of Figure 1 where within the first 10 training steps the reconstruction loss has substantially decreased (Figure 1a) by simply shutting off 9 values of k in q D (k|x) (Figure 1b), ing in a drastic increase of the discrete KL term (Figure 1a). However, this increase in the discrete KL term is negligible since the term is multiple orders of magnitude smaller than the reconstruction term in the ELBO. All of this takes place in the first few training iterations; well before the generative model has learned to use its continuous latent (see Figure 1c). Subsequently, on a slower timescale, the generative model starts to learn to reconstruct from its continuous latent, causing q C (z|k, x) to shift away from its prior toward a more-useful distribution to the generative model. We see this in Figure 1d: the continuous KL curve grows concurrently with the downturn of the reconstruction loss term. Figure 1f shows that after this transition (taking a few thousands training steps), the reconstructions from the model start to look more like MNIST digits. DISPLAYFORM2 Figure 1: Top row shows a snapshot of the GM-VAE after 10 training steps. Loss curves are shown in (a), the discrete variational distribution in (b) with rows representing E {x|label(x)= } q D (k|x), and reconstructions are shown in (c). Bottom row shows the same snapshot after 6000 training steps. While the generative model learns to use the continuous latent, the discrete distribution q D (k|x) never revives the k values that it shut off. This is because the generative model would not know how to use the z ∼ q C (z|k, x) values for those ks, implying a significant penalty in the reconstruction term of the ELBO. This is evidenced in Figure 1d by the discrete KL staying flat, and in Figure 1e where the columns corresponding to the shut off k values never repopulate. We have discussed the difficulty of leveraging the structure of the latent variables in GM-VAEs using our specific implementation designed to mirror the GM-WAE of Section 2.2. Many other variants of this implementation performed similarly. Though the root cause of this difficulty has not been ascertained in generality, we expect it to be in part due to the per-data-point nature of the ELBO objective, in particular, the impact of the KL divergence term on learning the variational distribution. This point will be elaborated upon with more empirical justification in Section 3. The difficulty associated with training GM-VAEs may be interpreted as a problem of restricted convergence of a sequence of distributions, where the sequence is indexed by the training steps. If that were so, an objective function that induces a weaker topology (and therefore, allows sequences to converge more easily) might help GM-LVMs converge to a distribution that non-trivially uses its discrete latent variable. Hence, we are motivated to consider approaching the training of such models using the OT framework, and in particular the Wasserstein distance as our objective, as it is known to induce a weaker topology than that of maximum likelihood. Following the OT approach of, we would like to minimize the 2-Wasserstein distance between the underlying data distribution (from which we have samples) and our GM-LVM: DISPLAYFORM0 Under review as a conference paper at ICLR 2019 where P Z×K is the set of all joint distributions over z and k, such that q(z, k|x) = q C (z|k, x)q D (k|x) with q C and q D parametrized below. Any parametrization of q(z, k|x) reduces the search space of the infimum, so W † 2 is in fact an upper bound on the true 2-Wasserstein distance W 2. Note that W † 2 is only equal to the true 2-Wasserstein distance when p G (y|z) is deterministic, providing an upper bound in the case of random generative models. We choose to model the "variational" distribution q(z, k|x) deliberately to mirror the structure of the prior, which differs from, for example, BID23 who assume conditional independence between z|x and k|x. Since the constrained infimum is intractable, a relaxed version of W † 2 is introduced as follows: DISPLAYFORM1 which is equivalent to the original distance when λ → ∞. This equivalence requires only that D be a divergence. As in, we use the Maximum Mean Discrepancy (MMD) with a mixture of inverse multiquadratic (IMQ) kernels with various bandwidth C i. The MMD is a distance on the space of densities and has an unbiased U-estimator BID9. Explicitly, if k is a reproducing positive-definite kernel and is characteristic, then the MMD associated to k is given by DISPLAYFORM2 IMQ kernels have fatter tails than the classic radial basis function kernels, proving more useful early in training when the encoder has not yet learned to match the aggregated posterior with the prior. The choice of bandwidth for the kernel can be fickle, so we take a mixture of kernels with bandwidths DISPLAYFORM3.., 2}} reducing the sensitivity on any one choice (see BID5 ; BID10 ; BID20).Given the discrete latent in our model, we cannot directly use Equation 4 with the MMD. Instead we integrate out the discrete latent variable in Equation 3, arriving at our GM-WAE objective function: DISPLAYFORM4 This allows us to compute the MMD between two continuous distributions, where it is defined. As mentioned in Section 1, VAEs have the disadvantage that deterministic generative models cannot be used; this is not the case for the Wasserstein distance. Thus we parametrize the generative density p G (x|z) as a deterministic distribution x|z = g θ (z) where g θ is a mapping from the latent to the data space specified by a deep neural network with parameters θ. This parametrization allows the minimization the objective function using stochastic gradient descent with automatic differentiation. To enable gradient-based minimization for the infimum in Equation 6, we parametrize q(z, k|x) = q C (z|k, x) q D (k|x) with neural networks. We take q C (z|k, x) to be a Gaussian with diagonal covariance for each k, mirroring the prior, and use the reparameterization trick to compute gradients. In order to avoid back propagating through discrete variables, the expectation over the distribution q D (k|x) is computed exactly. It could be computed by sampling using standard techniques BID3 BID14; BID22 ).As previously mentioned, the weakness of the topology induced by the Wasserstein distance on the space of distributions may enable the GM-WAE to overcome the VAE training issues presented in Section 2.1. With the objective in hand, a more precise argument can be made to support this claim. Recall from Section 2.1 that the problem with the GM-VAE was that the objective function demands the various distributions to be optimized at the individual data-point level. For example, the DISPLAYFORM5 breaks off completely and becomes irrelevant due to its size. This causes the q D (k|x) distribution to shut off k values early, which becomes galvanized as the generative model learns. However, in posing the problem in terms of the most efficient way to move one distribution p G onto another p data, via the latent distribution q(z, k|x), the Wasserstein distance never demands the similarity of two distributions conditioned per data point. Indeed, the E pdata in Equation 6 is inside both the infimum and the divergence D. We expect that "aggregating" the posterior as such will allow q(z, k|x) (in particular, q D (k|x)) the flexibility to learn data-point specific information while still matching the prior on aggregate. Indeed, it is also found in BID23 that using an adversarial game to minimize the distance between an aggregated posterior and the prior is successful at unsupervised training on MNIST with a discrete-continuous latent-variable model. In this work we primarily seek to show the potential for OT techniques to enable the training of GM-LVMs. Thus, we use relatively simple neural network architectures and train on MNIST.We use a mixture of Gaussians for the prior, with 10 mixtures to represent the 10 digits in MNIST and a non-informative uniform prior over these mixtures. Namely, for each k ∈ {0, . . ., 9}: DISPLAYFORM0 where the µ 0 k and σ 0 k represent the mean and covariance of each mixture and are fixed before training. We found that choosing dim(z) = 9 worked well. We choose the µ 0 k to be equidistant and for each k, σ 0 k = σ 0 is chosen identically in order to admit ≈ 5% overlap between the 10 different modes of the prior (i.e., the distance between any pair of means µ 0 k1 and µ 0 k2 is 4σ 0).For the variational distribution, we take q(z, k|x) = q C (z|k, x) q D (k|x) with DISPLAYFORM1 where each component is parametrized by a neural network. For π k (x) a 3-layer DCGAN-style network BID25 is used with largest convolution layer composed of 64 filters. The Guassian networks µ k (x), σ k (x) are taken to be 32-unit two-hidden-layer dense networks. Finally, for the generative model, we take p θ G (x|z) to be deterministic with x|z = g θ (z), using a 3-layer DCGAN-style network with smallest transpose convolution layer composed of 128 filters. All the convolutional filters have size 5 × 5 except for the last layer which has size 1 × 1.We use batch normalisation BID13, ReLU activation functions BID7 after each hidden layer and Adam for optimization BID16 with a learning rate of 0.0005. We find that λ = 450 works well, although the value of λ does not impact performance appreciably as long as it is larger than a few hundred. The (µ k, σ k) networks are pretrained to match the prior moments, which accelerates training and improves stability (this was also done for GM-VAE in Section 2.1). Our implementation of GM-WAE is able to reconstruct MNIST digits from its latent variables well. In Figure 2a example data points from the held-out test set are shown on the odd rows, with their reconstructions on the respective rows below. The encoding of the input points is a two step process, first determining in which mode to encode the input via the discrete latent, and then drawing the continuous encoding from the corresponding mode. Samples from the GM-WAE are shown in Figure 2b and 2c. Since the discrete prior p D (k) is uniform, we can sample evenly across the ks in order from 0 through 9, while still displaying representative samples from p(z, k) = p C (z|k)p D (k). Again, this shows how the GM-WAE learns to leverage the structure of the prior, whereas the GM-VAE in the collapse of the several modes of the prior. GM-WAE has a smooth manifold structure in its latent variables. In Figure 3a the reconstructions of a linear interpolation with uniform step size in the continuous latent space is shown between pairs of data points. This compares similarly to other WAE and VAE approaches to MNIST. In Figure 3b a linear interpolation is performed between the prior mode µ 0 6, and the other nine prior modes µ 0 k =6. This not only shows the smoothness of the learned latent manifold in all directions around a single mode of the prior, but also shows that the variatonal distribution has learned to match the modes of the prior. As one would hope given the suitability of a 10-mode GM-LVM to MNIST, almost every mode of the prior now represents a different digit. This level of control built into the prior requires not only a multi-modal prior, but also a training procedure that actually leverages the structure in both the prior and variational distribution, which seems to not be the case for VAEs (see Section 2.1).The quality of samples from our GM-WAE is related to the ability of the encoder networks to match the prior distribution. Figure 2c and 3b demonstrate that the latent manifold learned is similar to the prior. Near the modes of the prior the samples are credible handwritten digits, with the encoder networks able to capture the structure within each mode of the data manifold (variation within each column) and clearly separate each different mode (variation between rows).We have argued that the VAE objective itself was responsible for the collapse of certain k values in the discrete variational distribution, and that the per-data-point nature of the KL played a significant role. To test this hypothesis, and to compare directly our trained WAE with the equivalent VAE discussed in Section 2.1, we initialize the VAE with the parameters of the final trained WAE, and train it according to the VAE objective. At initialization, the VAE with trained WAE parameters produces high quality samples and reconstructions FIG1. However, after a few hundred iterations, the reconstructions deteriorate significantly FIG1, and are not improved with further training. The learning curves over the period of training between FIG1 and 4b are shown in FIG1, where the cause of the performance deterioration is clear: the continuous KL term in the VAE objective is multiple orders of magnitude larger than the reconstruction term, causing optimization to sacrifice reconstruction in order to reduce this KL term. Of course, the approximate posterior aggregated over the data will not be far from the prior as that distance is minimized in the WAE objective. However, this is not enough to ensure that the continuous KL term is small for every data point individually. It is thus the per-data-point nature of the KL in the VAE objective that destroys the reconstructions. Indeed, in order to minimize the per-data-point KL term in the GM-VAE objective, q C (z|k, x) is forced toward the mean µ 0 k for every x, causing it to lose much of its x dependence. This can be seen in FIG1 where the reconstructions are less customized and blurrier. To compare the performance of GM-WAE against GM-VAE more quantitatively, we directly compare the reconstruction loss from the VAE objective (the first term on the right hand side of Equation 2). Strictly speaking, this quantity is ill-defined for the GM-WAE, as the generative model is chosen to be deterministic. Instead we simply use the values returned by the GM-WAE generative model as if they were the Bernoulli mean parameters of the GM-VAE. These reconstruction loss curves are shown FIG1. Also shown are the reconstruction losses for the GM-VAE with various rescaling factors β in front of the KL terms of Equation 2. This rescaled KL term is inspired by both BID11, which studies the impact of rescaling the KL term in VAEs, as well as by the WAE objective itself where λ plays the role of a regularization coefficient. While, the GM-WAE is not trained to minimized this reconstruction loss, it actually achieves the best . This shows that GM-WAE performs better at reconstructing MNIST digits than its VAE counterpart, as measured by the VAE's own reconstruction objective. We also show in FIG1 the reconstruction curve of a GM-VAE initialized with trained GM-WAE parameters. This echoes the previous discussion concerning the deterioration of the reconstructions in GM-VAEs due to the per-data-point KL term. In FIG1, the GM-VAE initialized with trained GM-WAE parameters uses a rescaling factor β = 10 for visualization purposes. The same phenomenological behavior is observed with no rescaling factor, just less visually pronounced. Overall, our for GM-WAE are qualitatively competitive with other approaches, despite a relatively low-complexity implementation. Furthermore, GM-WAE offers more control over generation and inference due to its latent-variable structure, which cannot be achieved with the GM-VAE objective. We have shown that the GM-WAE is able to both reconstruct data and generate new samples meaningfully from the prior distribution. We now turn to studying the variational distributions directly, including with how much fidelity a given class of digits is paired with a given discrete latent. Consider first the discrete distribution q D (k|x) shown in FIG2, where E {x|label(x)= } q D (k|x) is shown in row. From the staircase structure, it is clear that this distribution learns to approximately assign each discrete latent value k to a different class of digit. However, it does not do so perfectly. This is expected as the GM-WAE seeks only to reconstruct the data from its encoding, not to encode it in any particular way. This does not mean GM-WAE is failing to use its discrete latent effectively. Indeed, when comparing Figure 2c and FIG2, a meaningful source of overlap between different values of k and a single digit class can be seen. For example, in FIG2 the digit 5 is assigned partially to k = 3 and k = 5. In Figure 2c, 5s drawn with a big-round lower loop are similar to digit 3 and 5s with a small loop and long upper bar are assigned to another cluster corresponding to digit 5. A similar discussion holds for 8s and 9s. To assess the digit-class fidelity of the discrete encoder more quantitatively, we calculate the accuracy of the digit-class assignment according to q D (k|x). To assign a digit-class label to each k value, we follow a similar protocol to that of BID23: we assign the digit-class label to the k value that maximizes the average discrete latent for that class, in decreasing order of that maximum. FIG2 shows the ing accuracy throughout training. Our GM-WAE achieves an accuracy on the held-out test set just shy of 70%. The corresponding accuracies for the GM-VAE variations considered in FIG1 are also shown. The best performing GM-VAE with a scaling factor of β = 20 achieves approximately 30%. This shows again the difficulty of the GM-VAE to capture meaningful structure in the data. For reference, basic K-means clustering achieves 50-60%, and BID23 achieve 90% (using 16 discrete classes, and substantially different model and training procedure).Another way to study the latent variable structure of GM-WAE is to consider dimensionally reduced visualizations of the continuous latent z. In FIG2 such a visualization is shown using UMAP BID24. Distinct clusters can indeed be seen in the prior and in the samples from q C (z|k, x). Though the clusters of z ∼ q C (z|k, x) do not fully align with those from the prior z ∼ p D (z|k), they maintain significant overlap. Samples from q C (z|k, x) in FIG2 are colored according to the true digit labels, and show how GM-WAE learns to assign digits to the different clusters. In particular, the 7 / 9 cluster is clearly overlapping, as seen in FIG2 and 2c. We have see that the GM-WAE model is highly suited to the problem under study. It reconstructs data and provides meaningful samples, it effectively uses both discrete and continuous variational distributions, all while maintaining close proximity between the variational distribution and the prior. We studied an unsupervised generative model with a mixture-of-Gaussians latent variable structure, well suited to data containing discrete classes of objects with continuous variation within each class. We showed that such a simple and critical class of models fails to train using the VAE framework, in the sense that it immediately learns to discard its discrete-latent structure. We further exposed the root cause of this phenomenon with empirical . We then put to the test the abstract mathematical claim that the Wasserstein distance induces a weaker topology on the space of distributions by attempting to train the same mixture-of-Gaussians model in the WAE framework. We found the Wasserstein objective is successful at training this model to leverage its discrete-continuous latent structure fully. We provided promising on MNIST and demonstrated the additional control available to a highly structured model with both discrete and continuous latent variables. We hope this motivates further study of the exciting but nascent field of Optimal Transport in generative modeling.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

This paper shows that the Wasserstein distance objective enables the training of latent variable models with discrete latents in a case where the Variational Autoencoder objective fails to do so.

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO


Question: Represent the scientific article to find a short TLDR

Question: While machine learning models achieve human-comparable performance on sequential data, exploiting structured knowledge is still a challenging problem. Spatio-temporal graphs have been proved to be a useful tool to abstract interaction graphs and previous works exploits carefully designed feed-forward architecture to preserve such structure. We argue to scale such network design to real-world problem, a model needs to automatically learn a meaningful representation of the possible relations. Learning such interaction structure is not trivial: on the one hand, a model has to discover the hidden relations between different problem factors in an unsupervised way; on the other hand, the mined relations have to be interpretable. In this paper, we propose an attention module able to project a graph sub-structure in a fixed size embedding, preserving the influence that the neighbours exert on a given vertex. On a comprehensive evaluation done on real-world as well as toy task, we found our model competitive against strong baselines.

represent the tldr for finding the corresponding scientific article

A graph neural network able to automatically learn and leverage a dynamic interactive graph structure

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Represent the article for finding a single-sentence TLDR

Question: We introduce NAMSG, an adaptive first-order algorithm for training neural networks. The method is efficient in computation and memory, and is straightforward to implement. It computes the gradients at configurable remote observation points, in order to expedite the convergence by adjusting the step size for directions with different curvatures in the stochastic setting. It also scales the updating vector elementwise by a nonincreasing preconditioner to take the advantages of AMSGRAD. We analyze the convergence properties for both convex and nonconvex problems by modeling the training process as a dynamic system, and provide a strategy to select the observation factor without grid search. A data-dependent regret bound is proposed to guarantee the convergence in the convex setting. The method can further achieve a O(log(T)) regret bound for strongly convex functions. Experiments demonstrate that NAMSG works well in practical problems and compares favorably to popular adaptive methods, such as ADAM, NADAM, and AMSGRAD. Training deep neural networks (; ; ;) with large datasets costs a huge amount of time and computational resources. Efficient optimization methods are urgently required to accelerate the training process. First-order optimization methods (; ; ; ; ;) are currently the most popular for training neural networks. They are easy to implement since only first-order gradients are introduced as input. Besides, they require low computation overheads except for computing gradients, which is of the same computational complexity as just evaluating the function. Compared with second-order methods (; ;), they are more effective to handle gradient noise. show that the momentum is crucial to improve the performance of SGD. Momentum methods, such as , can amplify steps in low-curvature eigen-directions of the Hessian through accumulation, although careful tuning is required to ensure fine convergence along the high-curvature directions. also rewrite the Nesterov's Accelerated Gradient (NAG) in a momentum form, and show the performance improvement over HB. The method computes the gradient at a observation point ahead of the current point along the last updating direction. They illustrate that NAG suppresses the step along high curvature eigen-directions in order to prevent oscillations. However, all these approaches are approximation of their original forms derived for exact gradients, without fully study on gradient noise. show the insufficiency of HB and NAG in stochastic optimization, especially for small minibatches. They further present ASGD and show significant improvements. However, the method requires tuning of 3 parameters, leading to huge costs that impede its practical applications. Among variants of SGD methods, adaptive methods that scale the gradient elementwise by some form of averaging of the past gradients are particularly successful. ADAGRAD is the first popular method in this line. It is well-suited for sparse gradients since it uses all the past gradients to scale the update. Nevertheless, it suffers from rapid decay of step sizes, in cases of nonconvex loss functions or dense gradients. Subsequent adaptive methods, such as RMSPROP (Tieleman & Hinton., 2012), ADADELTA , ADAM , and NADAM , mitigate this problem by using the exponential moving averages of squared past gradients. show that ADAM does not converge to optimal solutions in some convex problems, and the analysis extends to RMSPROP, ADADELTA, and NADAM. They propose AMSGRAD, which fixes the problem and shows improvements in experiments. In this paper, we propose NAMSG, that is an efficient first-order method for training neural networks. The name is derived from combining a configurable NAG method (CNAG) and AMSGRAD. NAMSG computes the stochastic gradients at configurable observation points ahead of the current parameters along the last updating direction. Nevertheless, instead of approximating NAG for exact gradients, it adjusts the learning rates for eigen-directions with different curvatures to expedite convergence in the stochastic setting, by selecting the observation distance. It also scales the update vector elementwisely using the nonincreasing preconditioner of AMSGRAD. We analyze the convergence properties by modeling the training process as a dynamic system, reveal the benefits of remote gradient observations and provide a strategy to select the observation factor without grid search. A regret bound is introduced in the convex setting, and it is further improved for strongly convex functions. Finally, we present experiments to demonstrate the efficiency of NAMSG in real problems. 2 THE NAMSG SCHEME Before further description, we introduce the notations following , with slight abuse of notation. The letter t denotes iteration number, d denotes the dimension of vectors and matrices, denotes a predefined positive small value, and S d + denotes the set of all positive definite d × d matrix. For a vector a ∈ R d and a matrices M ∈ R d × R d, we use a/M to denote M −1 a, diag(a) to denote a square diagonal matrix with the elements of a on the main diagonal, M i to denote the i th row of M, and, we use √ a for elementwise square root, a 2 for elementwise square, a/b for elementwise division, and max(a, b) to denote elementwise maximum. For any vector θ i ∈ R d, θ i,j denotes its j th coordinate where j ∈ {1, 2, . . ., d}. We define F ⊂ R d as the feasible set of points. Assume that F has bounded diameter D ∞, i.e. x − y ≤ D ∞ for any x, y ∈ F, and ∇f t (x) ∞ ≤G ∞, ∇f t (x) 1 ≤G 1 for all x ∈ F. The projection operation is defined as Π F,A (y) = arg min x∈F A 1/2 (x − y) for A ∈ S d + and y ∈ R d. In the context of machine learning, we consider the minimization problem of a stochastic function, where x is a d dimensional vector consisting of the parameters of the model, and ξ is a random datum consisting of an input-output pair. Since the distribution of ξ is generally unavailable, the optimizing problem is approximated by minimizing the empirical risk on the training set {ζ 1, ζ 2, ..., ζ N}, as In order to save computation and avoid overfitting, it is common to estimate the objective function and its gradient with a minibatch of training data, as where the minibatch S t ⊂ {1, 2, ..., N}, and b = |S t | is the size of S t. Firstly, we propose a configurable NAG method (CNAG). Since the updating directions are partially maintained in momentum methods, gradients computed at observation points, which lie ahead of the current point along the last updating direction, contain the predictive information of the forthcoming update. The remote observation points are defined asẋ t = x t − η t u t−1 where u t−1 is the updating vector, andẋ 1 = x 1. By computing the gradient at a configurable observation pointẋ t, and substituting the gradient with the observation gradient in the HB update, we obtain the original form of CNAG, as where α t, β t, η t are configurable coefficients, and m 0 = 0. The observation distance η t can be configured to accommodate gradient noise, instead of η t = β t in NAG . Both x t andẋ t are required to update in. To make the method more efficient, we simplify the update by approximation. Assume that the coefficients α t, β 1t, and η t, change very slowly between adjacent iterations. Substituting x t byẋ t + η t−1 α t−1 m t−1, we obtain the concise form of CNAG, as where the observation factor µ t = η t (1 − β t)/β t, and we use x instead ofẋ for simplicity. In practical computation of CNAG, we further rearrange the update form as where only 3 scalar vector multiplications and 3 vector additions are required per iteration besides the gradient computation. Hereinafter, we still use for simplicity in expressions. Then, we study the relation of CNAG and ASGD, that guides the selection of the momentum coefficient. shows that ASGD improves on SGD in any information-theoretically admissible regime. By taking a long step as well as short step and an appropriate average of both of them, ASGD tries to make similar progress on different eigen-directions. It takes 3 hyper-parameters: short stepα, long step parameterκ, and statistical advantage parameterξ.α is the same as the step size in SGD. For convex functions,κ is an estimation of the condition number. The statistical advantage parameterξ ≤ √κ captures trade off between statistical and computational condition numbers, andξ √κ in high stochasticity regimes. These hyper-parameters vary in large ranges, and are difficult to estimate. The huge costs in tuning limits the application of ASGD. The appendix shows that CNAG is a more efficient equivalent form of ASGD. For CNAG with constant hyper-parameters, the momentum coefficient β t = β = (κ − 0.49ξ)/(κ + 0.7ξ). Since the condition number is generally large in real high dimension problems, and the statistical advantage parameterξ ≤ √κ, β is close to 1. To sum up, the equivalence of CNAG and ASGD shows that in order to narrow the gap between the step sizes on eigen-directions with different curvatures, the momentum coefficient β should be close to 1. Finally, we form NAMSG by equipping CNAG with the nonincreasing preconditioner of AMSGRAD, and project the parameter vector x into the feasible set F. Algorithm 1 shows the pseudo code of NAMSG. Compared with AMSGRAD, NAMSG requires low computation overheads, as a scalar vector multiplication and a vector addiction per iteration, which are much cheaper than the gradient estimation. Almost no more memory is needed if the vector operations are run by pipelines. In most cases, especially when weight decay is applied for regularization, which limits the norm of the parameter vectors, the projection can also be omitted in implementation to save computation. In Algorithm 1, the observation factor µ t is configurable to accelerate convergence. However, it is costly to select it by grid search. In this section we analyze the convergence rate in a local stochastic quadratic optimization setting by investigating the optimizing process as a dynamic system, and reveal the effect of remote gradient observation for both convex and non-convex problems. Based on the analysis, we provide the default values and a practical strategy to set the observation factor without grid search. The problem can be approximated locally as a stochastic quadratic optimization problem, as whereF is a local set of feasible parameter points. In the problem, the gradient observation is noisy as ∇f t (x) = ∇Φ(x) +ǵ t, whereǵ t is the gradient noise. 4: 5: 8: Consider the optimization process of NAMSG, and ignore the projections for simplicity. Sincev t varies slowly when t is large, we can ignore the change ofv t between recent iterations. The operation of dividing the update by √v t can be approximated by solving a preconditioned problem, as wherex, which is supposed to have improved condition number compared withĤ in the convex setting. Then, we model the optimization process as a dynamic system. Solving the quadratic problem by NAMSG is equal to solving the preconditioned problem by CNAG, aš where the preconditioned stochastic functionf t (x) = f t (V −1/4 tx), the initial momentumm 0 = 0, the coefficients α = (1 − β 1t)α t, β = β 1t, and µ = µ t are considered as constants. We use ν to denote a unit eigenvector of the Hessian H, and the corresponding eigenvalue is λ. We define the coefficients asṡ t = ν,x t,v t = ν,m t. According to, the coefficients are updated as where the gradient error coefficient δ t = V −1/4 tǵt, ν /λ. Substitutingv t byṽ t = αv t, and denote τ = αλ, we rewrite the update into a dynamic system as where A is the gain matrix. The eigenvalues of A are where ρ = 1 + β − τ (1 − β(1 − µ)). Denote the corresponding unit eigenvectors as w 1 and w 2, that are solved numerically since the expressions are too complicated. Define the coefficients c 1, c 2, d 1, d 2 satisfying From, and, we obtain Assume that δ t = σδ, where δ obeys the standard normal distribution, and σ is the standard deviation of δ t. From, we obtain E(ṡ t+1) = r According to the analysis in Section 2, we recommend the momentum factor β = 0.999. Figure 1 presents the gain factor g f ac = max(|r 1 |, |r 2 |) and the stand deviation limit lim t→+∞ Std(ṡ t) of CNAG. It is shown that compared with HB (µ = 0), a proper observation factor µ improves the convergence rate significantly, and also accelerates the divergence in nonconvex problems where τ = αλ < 0. When the step size α is constant, compared with large curvatures, a small curvature λ converges much slower, forming the bottleneck of the whole training process. The problem can be alleviated by using a large µ. However, the noise level also increases along with µ when α is constant, that prohibits too large µ. Consequently, we recommend µ = 0.1 to achieve fast convergence speed, and µ = 0.2 to improve generalization at the cost of more iterations, since higher noise level is beneficial for expanding the range of exploration. For NAMSG, experiments shows that when β 2 is close to 1, its variation does not effect the significantly. We recommend β 2 = 0.99. Only the step size α is left for grid search. Figure 1 also shows that a large β and a proper µ ensures a large convergence domain, while 0 < τ < 2 is required for convergence in SGD (β = 0). Since the range of eigenvalue λ is problemdependent, a large maximum τ (denoted by τ max) allows large step sizes. As shown in Figure 1 (a), µ does not effect g fac significantly for a tiny range of τ close to 0. Then, g fac decreases almost linearly according to τ to the minimum. Consequently, training with a large step size α and small µ is beneficial for both the convergence of tiny positive λ, and the divergence of tiny negative λ in nonconvex settings. While selecting a smaller µ and scaling α proportional to argmin τ g fac, the λ to minimize g fac is unchanged, and the convergence rate for 0 < λ < τ max /α is generally improved according to Figure 1. However, the noise level also increases, that prohibits too large α. We propose a hyper-parameter policy named observation boost (OBSB). The policy performs grid search for a small portion of iterations using a small µ to select an optimal initial α. In training, when the loss flattens, it doubles µ, and scales α proportional to argmin τ g fac. The recommend initial µ is 0.05. In this section, we provide a data dependent regret bound of NAMSG in the convex setting, and further improve the bound for strongly convex functions. Since the sequence of cost functions f t (x) are stochastic, we evaluate the convergence property of our algorithm by regret, which is the sum of all the previous difference between the online prediction f t (x t) and the best fixed point parameter f t (x *) for all the previous steps, defined as When the regret of an algorithm satisfies R T = o(T), the algorithm converges to the optimal parameters on average. The positive definiteness of Γ t in a nonincreasing step size and avoids the non-convergence of ADAM. , we derive the following key for NAMSG. Theorem 1. Let {x t}, {v t} and {v t} be the sequences obtained from Algorithm 1,, · · ·, T }, and x ∈ F. We have the following bound on the regret By compared with the regret bound of AMSGRAD , we find that the regret bounds of the two methods have the similar form. However, when β 1 and γ are close to 1, which is the typical situation, NAMSG has lower coefficients on all of the 3 terms. From Theorem 1, we can immediately obtain the following corollary. Corollary 1. Suppose β 1t = β 1 /t, then we have The bound in Corollary 1 is considerably better than O. For strongly convex functions, NAMSG further achieves a O(log(T)) regret bound with O(1/t) step size under certain assumptions., where λ is a positive constant. Let {x t}, {v t} and {v t} be the sequences obtained from Algorithm 1. The initial step size α ≥ max i∈{1,···,d} tv 1/2, · · ·, T }, and x ∈ F. We have the following bound on the regret When the gradients are sparse, satisfying The proof of theorems are given in the appendix. It should be noted that although the proof requires a decreasing schedule of α t and β 1t to ensure convergence, numerical experiments show that piecewise constant α t and constant β 1t provide fast convergence speed in practice. In this section, we present experiments to evaluate the performance of NAMSG and the OBSB policy for NAMSG, compared with SGD with momentum , CNAG, and popular adaptive stochastic optimization methods, such as ADAM , NADAM , and AMSGRAD 1 . We study logistic regression and neural networks for multiclass classification, representing convex and nonconvex settings, respectively. The experiments are carried out with MXNET . We compare the performance of SGD, ADAM, NADAM, CNAG, AMSGRAD, NAMSG and OBSB, for training logistic regression and neural network on the MNIST dataset . The dataset consists of 60k training images and 10k testing images in 10 classes. The image size is 28 × 28. Logistic regression:In the experiment, the minibatch size is 256. The hyper-parameters for all the methods except NAMSG and OBSB are chosen by grid search (see appendix), and the best in training are reported. In NAMSG and OBSB, only the step size α is chosen by grid search, and the other hyper-parameters are set according to the default values. We report the train and test in Figure 2, which are the average of 5 runs. It is observed that OBSB perform the best with respect to train loss, and NAMSG also converges faster than other methods. The test accuracy is roughly consistent with the train loss in the initial epochs, after which they fluctuate for overfitting. The experiment shows that NAMSG and OBSB achieves fast convergence in the convex setting. In the experiment, we train a simple convolutional neural network (CNN) for the multiclass classification problem on MNIST. The architecture has two 5 × 5 convolutional layers, with 20 and 50 outputs. Each convolutional layer is followed by Batch Normalization (BN) and a 2×2 max pooling. The network ends with a 500-way fully-connected layer with BN and ReLU , a 10-way fully-connected layer, and softmax. The hyper-parameters are set in a way similar to the previous experiment. The are also reported in Figure 2, which are the average of 5 runs. We can see that NAMSG has the lowest train loss, which translates to good generalization performance. OBSB also converges faster than other methods. The experiment shows that NAMSG and OBSB are efficient in non-convex problems. In the experiment, we train Resnet-20 on the CIFAR-10 dataset , that consists of 50k training images and 10k testing images in 10 classes. The image size is 32 × 32. The architecture of the network is as follows: In training, the network inputs are 28 × 28 images randomly cropped from the original images or their horizontal flips to save computation. The inputs are subtracted by the global mean and divided by the standard deviation. The first layer is 3 × 3 convolutions. Then we use a stack of 18 layers with 3 × 3 convolutions on the feature maps of sizes {28, 14, 7} respectively, with 6 layers for each feature map size. The numbers of filters are {16, 32, 64} respectively. A shortcut connection is added to each pair of 3×3 filters. The subsampling is performed by convolutions with a stride of 2. Batch normalization is adopted right after each convolution and before the ReLU activation. The network ends with a global average pooling, a 10-way fully-connected layer, and softmax. In testing, the original 32 × 32 images are used as inputs. We train Resnet-20 on CIFAR-10 using SGD, ADAM, NADAM, CNAG, AMSGRAD, NAMSG, and OBSB. The training for each network runs for 75 epochs. The hyper-parameters are selected in a way similar to the previous experiments, excepting that we divide the constant step size by 10 at the 12000 th iteration (in the 62 th epoch). A weight decay of 0.001 is used for regularization. Two group of hyper-parameters are obtained for each method, one of which minimizes the train loss before the dropping of step size, and the other maximizes the mean test accuracy of the last 5 epoches. Figure 3 shows the average of 5 runs. In experiments to achieve the fastest training speed (Figure 3 (a),(b) ), OBSB converges the fastest, and NAMSG is also faster than other methods. Compares with ADAM, OBSB is more than 1 time faster, and NAMSG is roughly 1 time faster to reach the train loss before the dropping of step size. OBSB has the best test accuracy, and NAMSG is better than other methods. CNAG achieves significant acceleration upon SGD, and is also faster than ADAM, NADAM, and AMSGRAD. In experiments to achieve the best generalization (Figure 3 (c), (d)), OBSB still converges the fastest, NAMSG and CNAG converge at almost the same speed, which is faster than other methods. The mean best generalization accuracy of SGD, ADAM, NADAM, CNAG, AMSGRAD, NAMSG, and OBSB are 0.9129, 0.9065, 0.9066, 0.9177, 0.9047, 0.9138, and 0.9132, respectively. CNAG achieves the highest test accuracy. OBSB, NAMSG, and SGD obtains almost the same final test accuracy, which is much higher than ADAM, NADAM, and AMSGRAD. It should be noted that CNAG achieves the best test accuracy at the cost of grid search for 3 parameters, while NAMSG and OBSB only search for the step size. The experiments show that in the machine learning problems tested, NAMSG and OBSB converges faster compared with other popular adaptive methods, such as ADAM, NADAM, and AMSGRAD. The acceleration is achieved with low computational overheads and almost no more memory. We present the NAMSG method, which computes the gradients at configurable remote observation points, and scales the update vector elementwise by a nonincreasing preconditioner. It is efficient in computation and memory, and is straightforward to implement. A data-dependent regret bound is proposed to guarantee the convergence in the convex setting. The bound is further improved to O(log(T)) for strongly convex functions. The analysis of the optimizing process provides a hyperparameter policy (OBSB) which leaves only the step size for grid search. Numerical experiments demonstrate that NAMSG and OBSB converge faster than ADAM, NADAM, and AMSGRAD, for the tested problems. A.1 PROOF OF THEOREM 1 In this proof, we use y i to denote the i th coordinate of a vector y. From Algorithm 1, we have Since 0 ≤ β 1t < 1, from the assumptions, Rearrange the inequity (A2), we obtain For simplicity, denote Because of the convexity of the objective function, the regret satisfies The first inequity follows from the convexity of function f t. The second inequality is due to (A4). We now bound the term We have In (A7), the second inequity is follows from the definition of v t, the fifth inequality is due to Cauchy-Schwarz inequality. The final inequality is due to the following bound on harmonic sum: From (A7), and Lemma A2, which bound, we further bound the term P 2 as The third inequity is due to β 1t ≥ β 1t+1 andv 1/2 t,i /α t ≥v 1/2 t−1,i /α t−1 by definition. We also have In (A9), the second inequity follows from the assumption β 1t < β 1t−1, the third and the last inequality is due tov 1/2 t,i /α t ≥v 1/2 t−1,i /α t−1 by definition and the assumption α t = α/ √ t. Combining (A6), (A8), and (A9), we obtain The proof is complete. The Lemmas used in the proof are as follows: Lemma A2. For the parameter settings and conditions assumed in Theorem 1, which is the same as Theorem 4 in , we have The proofs of Lemma A1 and A2 are described in. Because of the objective function is strongly convex, from (A3) and (A4) the regret satisfies We divide the righthand side of (A11) to three parts, as Firstly, we bound the term Q 1. ≤ 0 (A13) The first inequity follows from β t is nonincreasing, the second equity follows from α t = α/t. The last inequity is because of the assumption α ≥ max i∈{1,···,d} tv 1/2 Algorithm A1 ASGD Algorithm Input: initial parameter vector x 1, short stepα, long step parameterκ ≥ 1, statistical advantage parameterξ ≤ √κ, iteration number T Output: parameter vector x T 1: Setx 1 = x 1,β = 1 − 0.7 2ξ /κ. 2: for t = 1 to T − 1 do 3: g t = ∇f t (x t). 4:x t+1 =βx t + (1 −β) x t −κα 0.7 g t. 5: Finally, we bound the term Q 3. Both the first equity and the first inequity follow from the assumptions α t = α/t and β 1t = β 1 /t 2. The last inequity is due tov t,i is nondecreasing by definition. Combining (A11), (A13), (A17), and (A18), we obtain A.3 EQUIVALENCE OF CNAG AND ASGD The pseudo code of ASGD ) is shown in Algorithm A1. ASGD maintains two iterates: descent iterate x t and a running averagex t. The running average is a weighted average of the previous average and a long gradient step from the descent iterate, while the descent iterate is updated as a convex combination of short gradient step from the descent iterate and the running average. We rewrite the update of Algorithm A1 as Define the variable transform as m t x t =T x t x t,T = lkl 0 1, wherek arel are adjustable coefficients. Combining (A20) and (A21), we obtain m t+1 x t+1 =T m t x t +Tbg t,T =TÄT −1. In order to minimize the number of vector computations, we solve the adjustable coefficientsk andl by assigningT 1,2 = 0,T 2,1 = 1. We choose the solution as Combining (A22) and (A23), we obtain m t+1 x t+1 =T m t x t +Tbg t,T = 0.7β (1−β)+0.7 0 1 1. The update (A24) is identical to the practical form of CNAG update with constant hyperparameters. The momentum coefficient of CNAG is β t = β = 0.7β (1 −β) + 0.7 = (κ − 0.49ξ)/(κ + 0.7ξ), where the second equity follows from the definition ofβ in Algorithm A1. It should be noted that the computational overheads of ASGD besides the gradient computation is 6 scalar vector multiplications and 4 vector additions per iteration, while CNAG reduces the costs to 3 scalar vector multiplications and 3 vector additions. We use constant hyper-parameters in the experiments. For ADAM, NADAM, and AMSGRAD, the hyper-parameters (α, β 1, β 2) are selected from {0.0005, 0.001, 0.002, 0.005, 0.01, 0.02} × {0, 0.9, 0.99, 0.999, 0.9999} × {0.99, 0.999} by grid search. For SGD, the hyperparameters (α, β) are selected from {0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0} × {0, 0.9, 0.99, 0.999, 0.9999} by grid search. For CNAG, the hyper-parameters (α, β, µ) are selected from {0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0} × {0, 0.9, 0.99, 0.999, 0.9999} ×{0.001, 0.01, 0.05, 0.1, 0.2, 0.3, 0.5, 0.9} by grid search. For NAMSG and OBSB, the hyperparameters (α) is selected from {0.0005, 0.001, 0.002, 0.005, 0.01, 0.02} by grid search, (β 1, β 2, µ) are set according to the default values. In OBSB, the grid search runs for 5 epochs in the experiments on MNIST, and 20 epochs on CIFAR10. The average convergence rate is computed each 2 epoches on MNIST, and 10 epochs on CIFAR10. α and µ are scaled when the converging rate is halved to achieve fast convergence, and at the 50th epoch (when the loss flattens) to maximize generalization. The experiments are carried out on a workstation with an Intel Xeon E5-2680 v3 CPU and a NVIDIA K40 GPU. The source code of NAMSG can be downloaded at https://github.com/rationalspark/NAMSG/blob/master/Namsg.py, and the hyper-parameters can be downloaded at https://github.com/rationalspark/NAMSG/ blob/master/hyperparamters.txt. The simulation environment is MXNET, which can be downloaded at http://mxnet.incubator.apache.org. The MNIST dataset can be downloaded at http://yann.lecun.com/exdb/mnist; the CIFAR-10 dataset can be downloaded at http://www.cs.toronto.edu/~kriz/cifar.html.

Represent the TLDR for finding the corresponding scientific article

A new algorithm for training neural networks that compares favorably to popular adaptive methods.

Represent the short summary for finding the corresponding scientific article (long)

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question:
represent the scientific article to find a short tldr

Examples:

Given We propose a new anytime neural network which allows partial evaluation by subnetworks with different widths as well as depths. Compared to conventional anytime networks only with the depth controllability, the increased architectural diversity leads to higher resource utilization and consequent performance improvement under various and dynamic resource budgets. We highlight architectural features to make our scheme feasible as well as efficient, and show its effectiveness in image classification tasks. When we deploy deep neural network models on resource-constrained mobile devices or autonomous vehicles with a strict real-time latency requirement, it is essential to develop a model which makes the best of available resources. Although many network compaction techniques including distillation, pruning and quantization have been proposed BID7 BID12 BID4 BID3 BID9 BID6 BID13, this goal is still challenging because the resource availabilities are continuously changing over time while these resources are being shared with other program instances BID1, and multiple resources with different characteristics (e.g. computational capacity, memory usage) should be considered together. Anytime machine learning algorithms have addressed the first issue, how to get the optimal performance under dynamic resource budgets, by allowing us to activate only a part of the model with graceful output quality degradation BID14 BID2. Most anytime algorithms based on deep neural networks appear in the form of early termination of forward processing according to the current resource budget or the difficulty of the given task BID8 BID0 BID10. In other words, the conventional anytime networks are trained to embed many potential sub-networks with different effective depths so that the best one can be chosen according to the current budget. In this work, we propose a new type of the anytime neural network, doubly nested network, to solve the other issue, more efficient utilization of multiple heterogeneous resources. The proposed network can be sliced along the width as well as the depth to generate more diverse sub-networks than the conventional anytime networks allowing the sub-network extraction only along the depth-wise direction. As depicted in FIG0, the increased degree of freedom enables us to get higher resource utilization in the devices constrained by dynamically changing resource budget with multiple criteria. Causal convolution It is straightforward to form a sub-network along the depth by appending a separate output generation stage to the final convolution layer of the sub-network. Since one specific layer's output does not depend on the following (upper) layers' outputs, the jointly trained subnetwork does not suffer from the performance degradation even after the extraction. However, this approach does not work along the width because of the interdependence between two nodes at different horizontal locations (e.g. different channels). To address this issue, we propose a channelcausal convolution where i-th channel group in one layer is calculated only with activation values from the channel groups from the first to i-th channel group in the previous layer as shown in the right of FIG1. The circle indicates the feature map while the square indicates the classifier. Color refers each channel. Our network based on the causal convolution allows us to extract the sub-network easily along any directions by making both horizontal and vertical data flow unidirectionally. Output generation stage sharing fully-connected layers Rather than having a separate fullyconnected (FC) layer for one sub-network to generate the final output (e.g. a predicted class given image input), our network is designed to have the FC layers each of which takes only a part of activations from the preceding convolution layers and produce the final output for one sub-network by averaging multiple FC layers' outputs as depicted in FIG1. Sharing the FC layers between the sub-networks at the same depth helps us to have similar computational and memory costs of the FC layers in the depth-controllable anytime network BID8 even with much more possible output locations. We can obtain a loss function for each sub-network: DISPLAYFORM0 where L and C refer to the number of possible vertical and horizontal partitions and N is the number of classes. y i is a target label of class i andŷ DISPLAYFORM1 Experimental setup We evaluated the proposed method on the CIFAR-10 and the SVHN datasets. Similarly to the ResNet-32 model BID5, our full network architecture consists of one convolution layer fed by external input, the following 15 residual blocks and fully-connected layers for the final output generation. The network has 16 possible output locations along the depth from the first convolution layer and all residual blocks, and 22 locations along the width. Thus, we can extract 16×22 sub-networks with different widths and depths from the base network. Resource usages of the sub-networks As the selected sub-network gets deeper or wider, all computational and memory requirements such as the number of MAC (multiply-accumulate) operations, the number of parameters and the size of the largest feature map increase. However, their rates of the increase are different from each other as shown in FIG3. This means that our scheme can benefit from the larger diversity of resource usage compared to the conventional anytime methods. Comparison with other methods One of the key advantages of the proposed architecture is nontrivial nesting of sub-networks along the width direction. FIG4 shows that our scheme outperforms two straightforward vertical slicing schemes (Brute-force slicing, Fine-tuning) that can generate sub-networks with different widths to a large extent without significant performance degradation compared to the upper bound (Full training). We revealed that resource-constrained devices could benefit from the architectural diversity enriched by our anytime prediction scheme. Our future works include adding adaptive conditioning BID11 which modulates intermediate activations or weight parameters depending on the current sub-network configuration to improve the performance only with a small increase of conditioning parameters. it matches with We propose a new anytime neural network which allows partial evaluation by subnetworks with different widths as well as depths. but not with We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.

Question: Recent advances in Generative Adversarial Networks facilitated by improvements to the framework and successful application to various problems has ed in extensions to multiple domains. IRGAN attempts to leverage the framework for Information-Retrieval (IR), a task that can be described as modeling the correct conditional probability distribution p(d|q) over the documents (d), given the query (q). The work that proposes IRGAN claims that optimizing their minimax loss function will in a generator which can learn the distribution, but their setup and baseline term steer the model away from an exact adversarial formulation, and this work attempts to point out certain inaccuracies in their formulation. Analyzing their loss curves gives insight into possible mistakes in the loss functions and better performance can be obtained by using the co-training like setup we propose, where two models are trained in a co-operative rather than an adversarial fashion. Information-Retrieval (IR) involves providing a list of ranked documents {d 1, d 2, . . ., d k} in answer to a query q. This general formulation can be extended to various tasks like web-search, where the documents are web pages and information needs are queries, content-recommendation, where the documents are items/content to suggest and queries are users, and Question-Answering, where the documents are answers and queries are questions. The retrieved list can also be viewed as a probability distribution over candidates, one example being DISPLAYFORM0 where l is a hyperparameter. Even if the probability distribution is not explicit, it is desirable to retrieve a higher ranked document more often than a lower ranked document. GANs were proposed as alternatives to generative models and have been shown to be capable of modeling the true data well. High dimensional settings like images and word sequences have seen some success. Given that the generator in GANs tries to model the training data's distribution, adversarial setups seem like a natural fit for IR. The learned distribution can then be used to retrieve relevant documents for incoming queries. IRGAN is a framework proposed by, with the hope of giving Information-Retrieval, access to the large literature of GANs. IRGAN consists of a discriminator and a generator. Like in a typical setup, the discriminator learns to distinguish between documents produces by the real probability distribution or the real ranking and the generator's probability distribution. It increases the likelihood of the former and decreases it for the latter. The generator tries to bring its probability distribution closer to the real one so that it increases the likelihood of confusing the discriminator into believing that it is the true distribution. Ideally, equilibrium is achieved when the generator manages to rank the documents according to the true distribution. However, the formulation and implementation of the loss function in the work seems to have a few issues. Specifically, the use of the baseline term recommended in the work in pitting the loss functions of the discriminator and the generator directly against each other and this leads to issues that are conspicuous in the loss curves. The training starts off with a pre-trained discriminator and generator, and the performance of the generator decreases as the training proceeds, while you would actually expect the opposite. When pre-training is not used, the generator does not learn at all. This forces IRGAN to choose the generator or discriminator based on whichever has better performance, while it expected that the generator is chosen at equilibrium. Given the traction this paper has received since its inception (53 citations as of 27 th September 2018), it is important to critically analyze the work and attribute the claimed performance improvements correctly. To this end, we propose two models which outperform IRGAN on two of the three tasks and give a comparable performance on the third. They also serve as an ablation study by experimentally showing that the generator might not be playing a vital role during train or test time. The following contributions are made in this work• We propose a model motivated by Co-training which outperforms IRGANs • We point out inaccuracies in the minimax loss function used in IRGANs • We substantiate the same by drawing from the loss curves 2 RELATED WORK 2.1 GENERATIVE ADVERSARIAL NETWORKS Generative Adversarial Networks (GANs) BID11 ) were proposed as an alternative to generative models BID23 ) which used Markov Chains or other approximations to compute intractable probability distributions. In essence, the generator tries to model the real data distribution and the discriminator learns to differentiate between real data points and generated data points. GANs are notoriously unstable to train and works like DCGANs BID21 ) and Wasserstein GAN BID1 ) have successfully attempted to alleviate a few issues. Nonetheless, GANs have been widely applied to various problems like image generation, text generation, cross-modal retrieval and more niche ones like Interactive Image Generation BID29 ), Text to Image ), Image to Image style transfer BID12 ) and robotics BID4 ).While GANs allow generation based on a random variable z, Conditional GANs BID17 ) partition the sample variable into two parts (z and y). y is used to denote which part of the probability distribution the generator has to generate from, and z plays the same role played in Vanilla GANs BID11 ). Conditional GANs dovetail with IR because y can be used to represent the query or its embedding, and in theory, the model should be able to generate the required document. DISPLAYFORM1 We feel that an eventual adversarial formulation for IR will be similar to this in flavor. employed Sketch-GANs for the interesting task of retrieving similar merchant seals (images) based on an input image. DCGANs BID21 ) are used to generate an image, and post training, the last layer of the discriminator is popped off and the rest of it is used as an encoder. This model, however, is specifically for retrieving image responses. In the subsequent sections, D denotes the discriminator, G the generator, p true the real probability distribution over documents, φ the parameters of the discriminator, θ the parameters of the generator, d the document, q the query and r the rank of a document with respect to a query. The equations used to train the discriminator and generator in BID11 are the following respectively. DISPLAYFORM0 The discriminator minimizes the likelihood of a "generated" data point and maximizes it for a "real" data point, while the generator tries to generate data points which the discriminator thinks is "real". The two models are trained alternatively and the procedure culminates in a generator which is able to produce data which looks like the real data. This section elucidates the IRGAN formulation ). Comments by the authors are in italics (in this section alone), while normal typeface is a paraphrased version of IRGAN. IRGAN is motivated by the combination of two schools of thoughts, the generative retrieval model and the discriminative retrieval model. The generative retrieval model p θ (d|q, r) tries to sample relevant documents from a candidate pool with the aim of cloning the true probability distribution p true. The discriminative retrieval model f φ (q, d), which is a binary classifier, tries to discriminate between real and generated pairs (q, d).Two different loss functions IRGAN-Pointwise and IRGAN-Pairwise are proposed. This is called so because each data point is independently used to train, unlike in IRGAN-Pairwise where pairs of points are used. The dataset is expected to have some cue with respect to how often a document is correctly retrieved for a query, if at all. DISPLAYFORM0 Note that the generator G can alternately be written as p θ (d|q n, r), which denotes the modeled probability distribution, and D(d|q) = σ(f φ (d, q)) represents the discriminator's score. In some IR problems the training data may not be a set of relevant documents for each query, but rather a set of ordered document pairs DISPLAYFORM0, where d i d j means that the first document is more relevant for query q n than the second document. o represents a real pair DISPLAYFORM1 Note the similarity between this and the previous formula. The problem with this formula is that D(o|q) is actually supposed to denote the probability that the pair o is from the real data distribution and not the probability that the pair is correctly ranked, as mentioned in the paper. The generator samples documents from the candidate pool based on its belief (relevance score). This sampling has the downside that the gradients cannot be backpropagated, and policy gradients BID25 ) have to be used. As an intuition, the documents can be considered as the arms of a contextual multi-arm bandit BID2, BID16 ), and picking an arm can be viewed as analogous to choosing the document as relevant. The policy discovered gives us the relevance of each document and − log(1 − D(d|q)) is the reward for picking that action/document (d). Let J G represent the objective function of the generator that it has to maximize. The policy gradient (REINFORCE) can be written as the following. DISPLAYFORM0 To reduce the variance in REINFORCE, a standard trick is to use the advantage function instead of just the reward. This does not change the optimal parameters. DISPLAYFORM1 Another baseline term that is suggested for each query is f φ (d +, q), where d + represents the positive document. This is legal because the term does not depend on the document (action). This is motivated by the belief of a larger generator score if f φ (d +, q) is large and lower if f φ (d +, q) is low. This baseline term is used in two of their three tasks and causes the violation of adversarial formulation, as we show in the following section. Having shown that the generator can be optimized using REINFORCE, we focus on the loss function and show how the baseline term exacerbates training. We consider Stochastic Gradient Descent updates for ease of illustration. Consider a triple (q, d r, d g), where d r denotes the correct document according to the true distribution and d g denotes the generated document. The discriminator's updates are in the direction of ∇J D, with the following definition. DISPLAYFORM0 With the baseline term included, the generator's updates are in the direction of ∇J G, with the following definition. DISPLAYFORM1 Since maximizing log(1 − z) with respect to z is the same as maximizing − log z, we can write the following equivalent loss functions DISPLAYFORM2 Note that this substitution is similar in principle to the substitution in BID11, where the motivation is to allow easier flow of gradients. It is apparent that the discriminator and the generator are optimizing directly opposite loss functions and this detrimental to the performance of the models. We provide experimental proof later that the performance improvements shown in IRGAN are mainly because of the discriminator maximizing the likelihood of the real data and not because of the generator. We propose two models to compare and critically analyze performance gains facilitated by IRGAN and illustrate them in FIG0.The first model increases the likelihood of the training data and decreases the likelihood of documents which are not relevant to the query but have a high score according to its own parameters. It maximizes the following, where the sampling for the second term is from a candidate pool with only negative answers (denoted by p −). Not following this will lead to undesirable updates because sampling positive documents for the second term will in decreasing the likelihood of real data. ψ denotes the parameters of the only model. To alleviate the pernicious loss function of IRGAN, we propose a model which uses two discriminators in a co-operative setup influenced by Co-training BID3 ). Instead of using two different views (x 1, x 2) as mentioned in the work, we use the same views for both the discriminators but let them influence each other in a feedback loop. Training is similar to Model 1 with the only difference being that each discriminator decreases the likelihood of documents relevant to the other discriminator rather than itself, as shown in the equation below. This model achieves better performance than IRGAN. DISPLAYFORM0 DISPLAYFORM1 This section describes the datasets, the task and hyperparameters. We conduct experiments on three tasks, Web Search, Item Recommendation and Question Answering, using the same datasets mentioned in IRGAN. InsuranceQA by BID9 7.2 TASK In Web Search, the task is to retrieve the document which is most relevant to the query. Each query on average has around 5 positive documents. In Content Recommendation, users give ratings for movies and given a user, the task is to retrieve a movie that they would probably rate high. In IRGAN, any movie retrieved for which the user rating is greater than or equal to 4 (out of a scale of 5) is considered correct. Based on the dataset statistics, around 55% of the user-movie ratings are ≥ 5. This makes the problem easy to solve. In Question Answering, every query has just one relevant document in most cases. This is thus the hardest task. The hyperparameters for the proposed model are the same except for absence of G Epochs, for obvious reasons. Information about hyperparameter tuning is mentioned in the Appendix. We report only the P@5 and NDCG@5 values because all other metrics follow the same trend. TAB2 reports the performance of various models. As can be seen, both the Single Discriminator and the Co-training models outperform IRGAN models. The fact that each query is associated approximately with 5 positive documents provides evidence that the proposed models can perform well in sparse reward settings. This task, in contrast to the other two, has multiple relevant documents that can be retrieved for each query, making it slightly easier. Each user (query) rates a movie (document), and 55% of the entries in the train set and 56% in the test set are relevant pairs. It can be seen in TAB3 that the single discriminator model achieves only a slightly lower score, and given the small size of the dataset (943 users), it makes just 7 more mistakes when compared to IRGAN. This is not a statistically significant number, especially because the IRGAN generator is pre-initialized to a model which scores 0.34 but our model learns from scratch. After close correspondence with the authors of IRGAN, we obtained all the hyperparameters required for the models. Multiple random seeds were used in vain, the in the paper for Question-Answering tasks could not be replicated. We instead mention the best out of all random seeds. We believe that if there is some random seed which gives better performance for IR-GAN, it should do so for our model as well. The co-training model outperforms IRGAN-Pairwise. The loss curves in FIG1 picked from IRGAN's work show deteriorating performance of the generator, which is in contrast to what is observed in actual adversarial training. In the minimax setting, since the generator is expected to capture the real data distribution, its performance is supposed to improve and this can indirectly be seen in GANs and DCGANs where the samples generated look more and more like real-world data points. Further, a deteriorating generator implies that the discriminator's improvement in performance is only because of the first term of J D, which hints that our proposed models might be able to do better than IRGAN. The reason offered in the paper is that "A worse generator could be the of the sparsity of document distribution, i.e., each question usually has only one correct answer". But this reason does not seem plausible, given that DCGANs have been able to model very high dimensional data, where the probability distribution is only a tiny part of the real space. Further, the increase in performance of the discriminator in all cases is coupled with a deteriorating generator. This substantiates our claim that the discriminator and the generator are optimizing directly opposite loss functions. Item-recommendation task is a little different from the other two tasks at hand because of a large number of positive answers. When the loss curves are plotted, though the generator's performance improves, the discriminator's loss remains high and almost constant throughout the procedure, as shown in FIG2. This is another indication that the performance of IRGAN is not actually because of the adversarial setup, but because of the maximization of the likelihood of the real data. We have already shown in Section 2 that Conditional GANs are connected directly to Information Retrieval. The problem can also be viewed as a contextual multi-armed bandit problem , where each documents is an arm and the context x q,d can be used to determine the actionvalue function f θ (x q,d). In previous works BID14 ) f has been considered to be linear, but recent studies BID7 have modeled them as deep neural networks. In BID20, a parallel is drawn between Actor-Critic algorithms and GANs. This is directly related to our work because REINFORCE with a baseline can be connected to Actor-Critic algorithms when bootstrapping is used BID24 ). The work shows a restricted scenario which involves a stateless MDP, each action setting all the pixels of the image and cross-entropy loss instead of mean-squared Bellmann residual in which GANs are equivalent to Actor-Critic algorithms. But this equivalence holds only when the baseline term is not used so the formulation in IRGAN is not exactly equivalent to a GAN framework. Another study BID10 ) draws a parallel between Inverse Reinforcement Learning BID19 ) and GANs because both the methods try to "learn" the cost function to optimize for. The experiments performed show that IRGAN is by no means state-of-the-art on those datasets. Further, the performance does not justify the large training time of 4 hours per generator epoch and 1 hour of discriminator epoch as opposed to 2 hours per epoch of the co-training model (11 GB GPU and Question Answering task). The shaky mathematical formulation renders the generator useless after training, and any gains in performance can be attributed directly to the first term of J D, where the likelihood of the real data is increased. We showed that the discriminator and generator are optimizing directly opposite loss functions and this is the cause of deleterious training. The poor performance of IRGAN on Web-Search and Question Answering and only a satisfactory performance on Content-Recommendation (which has dense rewards) lead us to speculate that it does not work well in sparse reward scenarios. This is similar to a well-known problem called the Sparse Reward Reinforcement Learning. We think that a correct formulation along with established techniques from the former, like reward shaping BID18 ) may lead to better performance. Newer methods like Hindsight Experience Replay BID0 ) which allow models to learn both from mistakes and rewards may further ameliorate learning. We would also like to explore in the direction of learning correct adversarial frameworks for more complex tasks like Image Retrieval and Question Answering which will involve learning end-toend trainable models. With advances in modeling sequences, this could also involve generation of documents rather than sampling them. The hyperparameters are mentioned in tables 6, 7, 8, 9 and 10. The following were the ranges of hyperparameter tuning, along with the best value. Gradient Descent Optimizer was used so that the comparison with IRGAN is fair. For the co-training model, for every epoch, we optimize the two discriminators several times. We call these the outer and inner epochs in TAB5. TAB6 represents the number of candidates that are chosen before performing the softmax. This is done to make the procedure computationally tractable. We use the value suggested in IRGAN. The discriminator and the generator have the same architecture in all the tasks. For the Web-retrieval task, the model has a single hidden layer 46 units. For the content-recommendation task, the model converts users and movies to a 5 dimensional embedding. This can be though to be a single hidden layer which compresses a one-hot user embedding to a 5 dimensional embedding. For the Question-Answering task, each word is initialized to a 100 dimensional random vector. A Convolutional Neural Network is then used and the window size of the convolutional kernel is. A max-pooling-over-time strategy is then used and the output is a 100 dimensional vector because each feature map is pooled to a scalar. Note that this architecture is the same as the one used in the IRGAN paper. We refer the user to that for further description.

represent the tldr for finding the corresponding scientific article!

Points out problems in loss function used in IRGAN, a recently proposed GAN framework for Information Retrieval. Further, a model motivated by co-training is proposed, which achieves better performance.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.