Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The linear transformations in converged deep networks show fast eigenvalue decay. The distribution of eigenvalues looks like a Heavy-tail distribution, where the vast majority of eigenvalues is small, but not actually zero, and only a few spikes of large eigenvalues exist. We use a stochastic approximator to generate histograms of eigenvalues. This allows us to investigate layers with hundreds of thousands of dimensions. We show how the distributions change over the course of image net training, converging to a similar heavy-tail spectrum across all intermediate layers. The study of generalization in deep networks has shifted its focus from the skeleton structure of neural networks BID1 to the properties of the linear operators of the network layers BID3 BID4 BID12. Measures like matrix norms, including standard Frobenius, other p-Norms or spectral norm) or stable rank BID2 are important components of theoretical bounds on generalization. All of these measures rely on the singular values of the linear maps A, or equivalently on the eigenvalues of the operator times its transpose AA T. In order to visually inspect the eigenvalue spectrum of a matrix, it is useful to compute a histogram. Histograms allows us to roughly estimate the distribution of eigenvalues and detect properties like decay behavior or the largest eigenvalues. An example of such a histogram is FIG0, that shows the eigenvalues of a Convolution-Layer in a squeeze_net network that maps to a feature map of dimension 256 × 13 × 13 = 43, 264. It shows many interesting, but not wellunderstood characteristic properties of fully trained networks: A Heavy-tail eigenvalue distribution with the vast majority of eigenvalues being near zero, though none are actually zero, and only few spikes of large eigenvalues. Martin and Mahoney show this phenomenon in the linear layer of large pre-trained models BID10, we also show it in the convolution layers and follow its evolution over the course of optimization. Computing singular values exactly is a costly operation, particularly because the dimension in stateof-the art convolutional neural networks often exceeds 100,000. Fortunately when we are interested only in a histogram, we do not need to know the exact eigenvalues, but only the number of eigenvalues that fall into the bins of the histogram. Based on the decay property exhibited in deep networks, we propose an approach for estimating histograms of eigenvalues in deep networks based on two techniques: For estimating the few high eigenvalues, called spikes, and particularly the largest eigenvalue, we use ARPACK, a truncated eigenvalue decomposition method that does not require the matrix explicitly, but accesses it only via matrix-vector products. For estimating the remainder, called bulk, we use a method based on matrix Chebyshev approximations and Hutchinson's trace estimator BID11. Like ARPACK, it only accesses the matrix via matrix-vector products. In FIG0, we have colored the bins we computed exactly in red, the approximated are blue. We denote the number of eigenvalues of a symmetric linear operator A: DISPLAYFORM0 The highest-dimensional linear operators in deep networks used in production environments are probably convolution layers. These layers transform feature maps with number of raw-features in the input-and output feature often exceeding 100k. This is feasible because the convolution operator is not implemented as matrix-vector multiplication with dense weight matrices, but specialized and highly-optimized convolution routines are used. Really every reputable deep learning software framework provides these routines. We can use these same routines when we estimate the eigenvalues of the linear maps of network layers. We first make sure that the network layer does not add a bias term BID0. Now let H be the linear map of a neural network layer. The forward-pass of that layer computes Hx efficiently, whereas the backward-pass computes H T y with backward-flowing gradient information y. We are interested in the eigenvalues of HH T, hence to compute HH T y, we first pass y through the backward pass of H, and pass the ing gradient through the forward pass to obtain the ing vector. To estimate the spikes and particularly the largest eigenvalue, we use the implicitly restarted Lanczos method as implemented in the ARPACK software package. It computes a truncated eigenvalue decomposition for implicit matrices that are accessed via matrix-vector products BID9. We specify a number of spikes T > 0 and compute the first T eigenvalues with ARPACK. From the largest eigenvalue, we derive the equidistant histogram-binning over the range [0, λ 1]. We use a technique for stochastically estimating eigenvalue counts proposed by Napoli et al. BID11. It requires that all eigenvalues fall into the range [−1, 1]. Hence, we first transform the matrix via A → (2λ −1 1 A − I) since we already know λ 1 from the ARPACK-based spike estimator. We define the indicator function δ l,u (λ) that is 1 iff. l ≤ λ < u and notice that we can write the number of eigenvalues in [l, u) as DISPLAYFORM0 We can approximate δ l,u with Chebyshev polynomials of a fixed degree DISPLAYFORM1 is the kth Chebyshev basis and b k ∈ R its corresponding coefficient. These coefficients are DISPLAYFORM2 Compute coefficients b k for k = 0,..., K according to BID0. DISPLAYFORM3 known for the indicator function BID11: DISPLAYFORM4 Now we can rewrite the count as the trace of a polynomial matrix function 2 applied to our matrix of interest, as it holds that trf DISPLAYFORM5 where Φ k (A) is the kth Chebyshev base for matrix functions. This quantity in turn can be approximated using stochastic trace estimators, aka Hutchinson estimators BID7. It holds that trA = E x x T Ax where each component of x is drawn independently from a zero-mean distribution like standard normal or Rademacher. This expression lends itself to a simple sampling algorithm, where we draw S independent x 1,..., x S and estimate DISPLAYFORM6 We do not have to explicitly compute Φ k (A), as only the product Φ k (A)x i is required. Since Chebyshev polynomials by construction follow the recursion Φ k+1 (A) = 2AΦ k (A) − Φ k−1 (A), we derive Algorithm 1 to estimate the count. Our experiments are based on a pyTorch implementation of the proposed histogram estimator. We train a squeeze_net architecure BID8 on imagenet data. After 30 epochs of training, we reduce the learning rate to 10%, and repeat this after another 30 epochs. We train using plain stochastic gradient descent with mini-batches of size 128 and compute histograms for all convolution layers before the first and after every epoch. For the histogram computation, we use a budget of 1000 for the exact computation of eigenvalues and approximate the remainder using the stochastic estimator. We present some histograms in FIG2 for the first and last convolution layers BID2. The histograms of the other layers show similar behavior as the last layer, for instance FIG0 shows an intermediate layer after the first epoch of training quite similar to FIG2, but with less extreme decay. BID1 A real-valued function f (x): R → R has a corresponding matrix function f (A): R m×m → R m×m and the eigenvalues of f (A) are f (λ1),..., f (λm). For polynomials, we get this matrix function by replacing scalar multiplications with matrix multiplications and scalar additions with matrix additions. For other classes of functions and a comprehensive introduction to matrix functions see Highham's book BID6. BID2 Addional and animated histograms are available at https://whadup.github.io/Resultate/, however note that the website is not sufficiently anonymized for double-blind reviewing. Proceed with caution. Like Martin and Mahoney BID10 we identify different phases in the spectograms. Right after initialization, the matrix behaves almost as random matrix theory suggests given the element-wise independent random initialization with Gaussian random variables BID13. This can be observed in FIG2. However, we note that on the first layer Fig. 3a, there are some unexpected bumps in the histogram. We conjecture that this may be due to padding in the convolutions. As optimization commences, we start to see heavytail behavior. Already after one epoch of traning, the largest eigenvalues have seperated from the bulk, while the majority of eigenvalues remains in the same order of magnitude as before training. This can be seen for the first and large convolution layer in FIG2. The bumps in the first layer smooth out a little. Over the course of the first 30 epochs, the largest eigenvalues grow steadily as the tail of the spectrum grows further. Then as soon as the learning rate is reduced to 10%, the operator norms of the linear maps start to decrease as depicted in FIG1. Considering the importance of the operator norm in known generalization bounds for feed-forward networks, this suggests that some sort of regularization is happening. The bumps in the first layer smooth out further, but remain visible. The last layer for the most part keeps its shape in the last 60 epochs, beside the reduction of the norm we notice that the largest bar decreases in size from 4157 to 2853 and that the difference seems to move to the other bars in the blue portion of the histogram. Understanding the structure in the linear transformations might be an important aspect of understanding generalization in deep networks. To this end we have presented a stochastic approach that allows us to estimate the eigenvalue spectrum of these transformations. We show how the spectrum evolves during imagenet training using convolutional networks, more specifically squeeze_net networks. In the future we want to apply similar approaches to estimating the covariance structure of the intermediate feature representations and investigate the relations between covariance matrices and parameter matrices. Since the estimator we use is differentiable BID5 BID0, it may be interesting to investigate its usefulness for regularization.

represent the tldr for finding the corresponding scientific article

We investigate the eigenvalues of the linear layers in deep networks and show that the distributions develop heavy-tail behavior during training.

represent this text to find a much much longer article about it with say hundreds-thousands of words.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question:
represent the article for finding a single-sentence tldr
E.g. 'The variational autoencoder (VAE) has found success in modelling the manifold of natural images on certain datasets, allowing meaningful images to be generated while interpolating or extrapolating in the latent code space, but it is unclear whether similar capabilities are feasible for text considering its discrete nature. In this work, we investigate the reason why unsupervised learning of controllable representations fails for text. We find that traditional sequence VAEs can learn disentangled representations through their latent codes to some extent, but they often fail to properly decode when the latent factor is being manipulated, because the manipulated codes often land in holes or vacant regions in the aggregated posterior latent space, which the decoding network is not trained to process. Both as a validation of the explanation and as a fix to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method significantly outperforms unsupervised baselines and is competitive with strong supervised approaches on text style transfer. Furthermore, when switching the latent factor (e.g., topic) during a long sentence generation, our proposed framework can often complete the sentence in a seemingly natural way -- a capability that has never been attempted by previous methods. High-dimensional data, such as images and text, are often causally generated through the interaction of many complex factors, such as lighting and pose in images or style and content in texts. Recently, VAEs and other unsupervised generative models have found successes in modelling the manifold of natural images (; ;). These models often discover controllable latent factors that allow manipulation of the images through conditional generation from interpolated or extrapolated latent codes, often with impressive quality. On the other hand, while various attributes of text such as sentiment and topic can be discovered in an unsupervised way, manipulating the text by changing these learned factors have not been possible with unsupervised generative models to the best of our knowledge. Cífka et al.; observed that text manipulation is generally more challenging compared to images, and the successes of these models cannot be directly transferred to texts. Controllable text generation aims at generating realistic text with control over various attributes including sentiment, topic and other high-level properties. Besides being a scientific curiosity, the possibility of unsupervised controllable text generation could help in a wide range of application, e.g., dialogues systems . Existing promising progress (; ; ;) all relies on supervised learning from annotated attributes to generate the text in a controllable fashion. The high cost of labelling large training corpora with attributes of interest limits the usage of these models, as pre-existing annotations often do not align with some downstream goal. Even if cheap labels are available, for example, review scores as a proxy for sentiment, the control is limited to the variation defined by the attributes. In this work, we examine the obstacles that prevent sequence VAEs from performing well in unsupervised controllable text generation. We empirically discover that manipulating the latent factors for typical semantic variations often leads to latent codes that reside in some low-density region of the aggregated posterior distribution. In other words, there are vacant regions in the latent code space not being considered by the decoding network, at least not at convergence. As a , the decoding network is unable to process such manipulated latent codes, yielding unpredictable generation of low quality. In order to mitigate the latent vacancy problem, we propose to constrain the posterior mean to a learned probability simplex and only perform manipulation within the probability simplex. Two regularizers are added to the original objective of VAE. The first enforces an orthogonal structure of the learned probability simplex; the other encourages this simplex to be filled without holes. Besides confirming that latent vacancy is indeed a cause of failure in previous sequence VAEs', it is also the first successful attempt towards unsupervised learning of controllable representations for text to the best of our knowledge. Experimental on text style transfer show that our approach significantly outperforms unsupervised baselines, and is competitive with strong supervised approaches across a wide range of evaluation metrics. Our proposed framework also enables finer-grained and more flexible control over text generation. In particular, we can switch the topic in the middle of sentence generation, and the model will often still find a way to complete the sentence in a natural way. The variational autoencoder (VAE) is a generative model defined by a prior p(z z z) and a conditional distribution p θ θ θ (x x x|z z z). The VAE is trained to optimize a tractable variational lower bound of log p θ θ θ (x x x): where q φ φ φ (z z z|x x x) is a variational distribution parameterized by an encoding network with parameters φ φ φ, and p θ θ θ (x x x|z z z) denotes the decoding network with parameters θ θ θ. This objective tries to minimize the reconstruction error to generate the data, and at the same time regularizes q φ φ φ (z z z|x x x) towards the prior p(z z z). In this paper, p(z z z) is chosen as N (0, I 0, I 0, I). For text modelling, the input x x x is some observed text. Both the encoding and decoding network are usually recurrent neural networks. Note that during learning, the decoding network p θ θ θ only learns to decode conditioned on z z z that are sampled from q φ φ φ (z z z|x x x). In other words, the decoding network only learns to process z z z sampled from the aggregated posterior distribution q φ φ φ (z z z) = E x x x∼p d (x x x) q φ φ φ (z z z|x x x), where p d (x x x) is the data distribution. If q φ φ φ (z z z) has regions of low density, there is no guarantee that p θ θ θ would decode well in such regions. This is an important intuition that will become central to our analysis in Sec. 3. In this section, we take a deeper look into the aggregated posterior latent space of sequence VAE trained on text, and provide justification for the alternative solution we propose in Section 4. As pointed out by , one of the motivations to apply VAEs on text is to allow generation of the sentences conditioned on extrinsic features by controlling the latent codes. Without annotated labels, no previous methods have successfully learned controllable latent factors as mentioned in Sec. 1. To understand what is missing, we conduct exploratory experiments to use VAE for unsupervised sentiment manipulation. We use the Yelp restaurant reviews dataset and the same data split following. We train a β-VAE 1 with a latent space of 80 dimensions, an LSTM encoder, and an LSTM decoder. Details about this experiment are described in Appendix A.1. By inspecting the accuracy on the validation set, we find that there exists one dimension of latent code achieving higher than 90% sentiment classification accuracy by its value alone, while other latent codes get accuracy around 50%. Further details can be found in Appendix A.2. It means that this latent dimension is an effective sentiment indicator. Similar phenomena have been observed in large-scale language models . However, the direct influence on the generative process of the model observed in does not apply on the VAE. When we try to perform sentiment manipulation by modifying this latent dimension 2, the decoding network fails to generate the desired outputs most of the time, as evidenced by the poor quantitative evaluation in Table. 1, and poor samples shown in Appendix A.3. One possible reason for the failure is that the decoding network is never trained on codes like the manipulated ones. This is the case if the aggregated posterior has holes or regions of low density, and the manipulated codes fall into such vacant regions. Supposing the aggregated posterior latent space possesses a shape as shown in Fig. 1, the direct manipulated latent codes will fall out of the aggregated posterior latent space for most input samples. Such latent codes are never seen by the model during training and possess a low density under the aggregated posterior distribution, leading to unpredictable behaviours during decoding. To verify our hypothesis demonstrated in Fig. 1, we empirically estimate the density of sentimentmanipulated codes under the aggregated posterior distribution of our trained VAE. Here, we approximate the data distribution p d (x x x) with the empirical distribution over all the training samples. As a , the estimated aggregated posterior distribution is a large mixture of Gaussian distribution. For all 1000 test samples, we move the dimension of code capturing sentiment from µ − 2σ to µ + 2σ where µ and σ are the mean and the standard deviation estimated on all the training samples and measure the averaged negative log-likelihood (NLL) under the aggregated posterior distribution. As depicted in Fig. 3 (a), the NLL plotted in blue dot curve rises sharply when moving away from µ even if there is only one dimension of code is changing, indicating the existence of the vacancy in the aggregated posterior latent space. In addition, we draw the histogram of all the test samples' NLL considering their original latent codes and modified ones in Fig. 3 (b). The histogram shows that there is a large divergence in NLL between the original latent codes and the modified ones. Also, the modified latent codes have two separate modes, confirming the irregular shape of the aggregated posterior latent space. In order to resolve this issue, the approach proposed in this work is to constrain the posterior in a way that the manipulation only happens in a learned simplex, as depicted in Fig. 2. In this constrained subspace, the phenomenom of low density holes of aggregated posterior is significantly reduced, as Fig. 3 (a) and (c) empirically show that there is little change in NLL of original versus modified codes. The details of our method is presented in the next section. The experiments conducted in Sec. 3 validates the existence of vacancy in the aggregated posterior latent space. One potential way to resolve the problem is to better match the aggregated posterior with the prior (; ;). However, in terms of unsupervised learning of controllable representation for text, these previous methods have not shown successes; only attempted supervised text style transfer, and also reported negative from the AAE . Another way to resolve the vacancy issue is to directly enforce that the aggregated posterior itself has no vacant region anywhere where we would like to perform latent code manipulation. We propose to map the posterior Gaussian mean to a constrained space, more specifically a learned probability simplex, where we can encourage the constrained latent space to be filled without vacancy, and perform manipulation to be within this simplex. As illustrated in Fig. 2, we add an additional mapping function as part of the encoding network which maps the mean of the Gaussian posterior to a constrained space. Two regularization terms are introduced later to ensure the learned simplex is not degenerate and that this subspace is well filled. In addition, we separately model the relevant factors that we wish to control and the irrelevant factors by splitting z z z into two parts, z z z and z z z, following prior work . The first part captures the relevant factors that are dominant in the data without an inductive bias from external signals, while the second part learns to encode the remaining local information that is useful for reconstructing the source sentences. As a , q φ φ φ (z z z|x x x) is decomposed into q φ φ φ1 (z z z |x x x)q φ φ φ2 (z z z |x x x) where φ φ φ = φ φ φ 1 ∪ φ φ φ 2. With diagonal covariances the KL divergence term in Eq. 1 splits into two separate KL terms. In practice, we use a MLP encoding network to parametrize z z z with some sentence representations as the input (e.g., averaging GloVe embeddings over the input tokens) and a LSTM encoding network to parametrize z z z. We only constrain the posterior of z z z and z z z is optimized the same way as the traditional VAE. We now describe how to map the mean µ µ µ of the Gaussian posterior for z z z ∈ R N to a constrained latent space. We would like to constrain the mean µ µ µ to have a structure as follows: e e e i, e e e j = 0, where e e e i are vectors representing the relevant factors, p i is the proportion of ith relevant factor encoded in z z z and K is a hyperparameter indicating the number of relevant factors to discover. In other words, the mean of the Gaussian posterior of z z z is constrained to be inside a K-dimension probability simplex in R N whose vertices are represented by the orthogonal basis vectors e e e i, i = 1,..., K. Given the outputs of the MLP encoder h h h and log σ σ σ 2, we learn an additional mapping function π which maps h h h to the constrained posterior space, which can be treated as part of the encoding network: where E E E = [e e e 1, . . ., e e e K] is a learnable embedding matrix representing the bases, W W W is the learnable weight matrix, and b b b is the learnable bias vector. As a , the constrained posterior is parametrized by µ µ µ and log σ σ σ 2 as a Gaussian distribution N (µ µ µ, diag(σ σ σ 2)). With the mapping function alone, the proposed VAE suffers from posterior collapse , a well-known problem where the model ignores the latent code z z z during the training. Further complicating matters is the fact that there is an abundance of signals for predicting the next token in the text, but the signals indicating high-level semantics are quite sparse. It is thus unlikely that the VAEs can capture useful relevant factors from raw text without collapse. For these reasons, we enforce orthogonality in the learnt basis vectors as defined in Eq. 2, which introduces a natural recipe to prevent posterior collapse for z z z. Note that the KL divergence between q φ φ φ1 (z z z |x x x) and p(z z z ) is With orthogonality in the basis vectors, the first term in the above equation can be factorized into i e e e i e e e i. To encourage orthogonality in the basis vectors, a regularization term is added to the objective function: where I I I is the identity matrix and α is a hyperparamter. When L REG = 0, e e e i e e e i = α. In this case, µ µ µ µ µ µ = α i p 2 i reaches its minimum α K when p p p is a uniform distribution. The proof can be found in Appendix C. In practice, L REG will quickly decrease to around 0, ensuring that the KL term will never fully collapse with the structural constraint. When it comes to controlled generation, one can choose a vertex or any desired point in the probability simplex, as illustrated in Fig. 2. Note that the constrained posterior also means that the aggregated posterior can never match the isotropic Gaussian prior. In other word, we achieve good controlled text generation potentially at the cost of poor uncontrolled generation from the prior, but such is not the focus of this current work, and could potentially be resolved by selecting or learning a better prior as in. Constraining the posterior inside a certain space does not guarantee that this space will be filled after training. In order to prevent this, we want the probability distribution over the relevant factors p p p to cover as much of the constrained latent space as possible. We introduce a reconstruction error of the structured latent code in order to push p p p away from a uniform distribution. For each input sentence, we randomly sample m sentences from the training data as negative samples. By applying the same encoding process, we get the structured latent code µ µ µ (−) i for each negative sample. Our goal is to make the raw latent code h h h similar to the restructured latent code µ µ µ while different from latent codes µ µ µ (−) i of the negative samples, so that p p p is generally different for each input sample. The structured reconstruction loss is formulated as a margin loss as follows: Our final objective function is defined as follows: 5 RELATED WORK Learning disentangled representations is an important step towards better representation learning which can be useful for (semi-)supervised learning of downstream tasks, transfer and few-shot learning . VAEs have achieved promising for unsupervised learning of disentangled representations. Several variations of VAEs have been proposed to achieve better disentanglement (; ; ;). However, most recent progress in this direction has been restricted to the domain of images. In order to perform controllable text generation, previous methods either assume annotated attributes or multiple text datasets with different known styles (; ; ; ; ; ; ;). The requirement of labelled data largely restricts the capabilities and the applications of these models. Instead, all our proposed framework needs is raw text without any annotated attribute. The dominant underlying relevant factors in the given corpus will be discovered and disentangled by our unsupervised method, which can in turn be used for controlled generation. To demonstrate the effectiveness of our approach, we compare it to unsupervised baselines with traditional VAEs, considering the density under the aggregated posterior distribution and the performance on sentiment manipulation. Following evaluation protocols in text style transfer, we also compare our method to strong supervised approaches. Furthermore, we showcase the ability of finer-grained style discovery and transition possessed by our system, which has not been attempted in the literature. In this section, our proposed framework is referred as CP-VAE (Constrained Posterior VAE). Detailed configurations including the hyperparameters, model architecture, training regimes, and decoding strategy are found in Appendix B. Experimental setup: We use the same experimental setting and dataset as mentioned in Sec. 3. The 80D latent code is split into 16 and 64 dimensions for z z z and z z z respectively. The sentence representations used for z z z is the averaged GloVe embeddings over the input tokens and K is chosen as 3. To decide which basis vector corresponds to which sentiment, we sample 10 positive and 10 negative sentences respectively in the development set, pass them to the encoder, and choose the basis vector with the highest average p i in p p p = softmax(W W Wh h h + b b b), yielding v p as the positive basis and v n as the negative basis. If v p and v n are chosen to be the same vector, we choose the index with the second highest p i for v p. To perform sentiment manipulation, we fix z z z to be the chosen basis vector; that is, v p or v n. Comparisons on density under the aggregated posterior distribution: First, we do linear interpolation between the two discovered basis vectors v p and v n and estimate the averaged NLL under the aggregated posterior distribution the same way as introduced in Sec. 3. The green solid curve in Fig. 3 (a) shows that the NLL of CP-VAE is relatively stable for the whole range of the interpolation. In Fig. 3 (c), the original latent codes and the modified ones largely overlap with each other. Both observations validate the effectiveness of CP-VAE in resolving the latent vacancy problem, leading to significant improvements on unsupervised sentiment manipulation, as seen later. Comparsions with metrics on text style transfer: For quantitative evaluation, we adopt automatic evaluation metrics used in text style transfer including classification accuracy (AC), BLEU score (BL), GLEU score (GL) and language model perplexity (PL), whose definitions are elaborated in the next section. We also report D KL (q φ φ φ1 (z z z |x x x) p(z z z)) (KL) for z z z of CP-VAE. As shown in Tab. 1, CP-VAE performs significantly better than β-VAE in terms of accuracy, BLEU and GLEU. The lower perplexity of β-VAE is due to mode collapse, which produces very short pivot sentences such as "great!". The match our observations from the experiments on density under the aggregated posterior distribution, confirming that latent vacancy prevents effective manipulation of the latent codes. We also conduct an ablation study by removing L REG and L S-REC from the objective. The demonstrate that both terms are crucial to the success of CP-VAE. Without L REG, CP-VAE experiences posterior collapse for z z z. As a , v p and v n collide with each other, leading to failure in disentangled representation learning. Since we choose K as 3, it is convenient to visualize the samples during training with p p p in the learnt probability simplex, as shown in Fig. 4. We can see that the whole simplex is mostly covered with samples with the help of L S-REC. Without L S-REC, the decoding network fails to recognize the basis vectors due to the poor coverage of the probability simplex, causing the model to lose most of its transferring ability. , denoted as CP-G(loVe) and CP-B(ert) respectively. The remaining settings are as described in the above section. On the two datasets, we compare to three adversarially trained models: StyleEmbedding (SE) , MultiDecoder (MD) , CrossAligned (CA) and two state-of-the-art models based on a "delete, transform, and generate" framework: DeleteAndRetrieve (D&R) and Blind-GenerativeStyleTransformer (B-GST) . Evaluation protocols: Four different automatic evaluation metrics are used to measure the different perspectives of the transferring quality, following. To measure transferring ability, we use pre-trained CNN based classifiers achieving 98% and 84% accuracies on the test sets of Yelp and Amazon respectively. To measure content preservation, we use the BLEU score between the transferred sentences and the source sentences. To measure fluency, we finetune OpenAI GPT-2 with 345 million parameters on the same trainingdev-test split to obtain the perplexity of generated sentences. The fine-tuned language models achieve perplexities of 26.6 and 34.5 on the test sets of Yelp and Amazon respectively. In addition, argued that the Generalized Language Evaluation Understanding Metric (GLEU) has a better correlation with the human judgement. Here, we use the implementation of GLEU 3 provided by to calculate the GLEU score. Result Analysis: As observed by and , accuracy, BLEU score and perplexity do not correlate well with human evaluations. Therefore, it is important to not consider them in isolation. Tab. 2 shows that our proposed approaches get similar scores on these metrics with human reference sentences on the second row, indicating that the generated sentences of our proposed approaches is reasonable considering the combination of these metrics. As seen by and verified in Sec. 6.1, GLEU strike a balance between target style match and content retention and correlate well with the human evaluations. From Tab. 2, CP-VAE consistently outperforms the three adversarially trained models on GLEU by a noticeable margin and achieve competitive as compared to the recent state-of-the-art models. By checking the samples generated from the models as shown in Tab. 3, B-GST, the current state-of-the-art, is more they are cheerios, afterall, and we love the original kind. HUMAN because it is made of cast iron, scorching is maximized. they are cheerios, and we love them. B-GST because it s cheaply made of cast iron, is useless. they are sturdy, afterall, sturdy and we love the original. because it s made of cast iron, vomitting. they are ripe, tastier, and we love them. because it s made of cast iron, limp. they are divine, fluffier, and we love them. consistent to the source sentence, which can be expected, since it only makes necessary edits to flip the sentiment. CP-VAE tends to generate more diverse contents which may not be relevant sometimes, but the overall quality is reasonable considering it is trained without the label information. More samples can be found in Appendix E. Table 4: Two pairs of samples generated without and with topic transition. The first sentence in the pair is generated with a topic fixed throughout the generation; while the second sentence is generated with topic transition, the generated outputs after switching are marked as bold. World throughout A federal judge on Friday ordered a federal appeals court to overturn a federal appeals court ruling that the Visa and MasterCard credit card associations violated federal antitrust law by barring the names of the state. A federal judge on Friday ordered a federal appeals court to overturn a decision by the Supreme Court to overturn a decision by the Federal Communications Commission to block the company's antitrust case against Microsoft Corp. Sports throughout NEW YORK (Reuters) -Roger Federer, the world's No. 1 player, will miss the rest of the season because of a sore quadriceps. Sports to Business NEW YORK (Reuters) -Roger Federer, the world's No. 1 player, will miss the rest of the year because of a bid-rigging scandal. To further explore the potential of CP-VAE, we conduct the following exploratory experiments. We use the AG news dataset constructed by , which contains four topic categories which are World, Sports, Business and Sci/Tech, with the title and description fields. Here, we drop the title and just use the description field to train CP-VAE and set K = 10. All four topics are automatically discovered by CP-VAE and identified as described in Sec. 6.1. We also compare the of our identified topics to standard baselines for unsupervised topic modelling, the details can be found in Appendix D. We choose a basis vector discovered by our model and generate a few tokens. Then, we switch the basis vector and continue the generation until the end-of-seq token is generated. Generated samples are shown in Table 4. We see that our model learns to transition from one topic to another in a natural and fluent way within the same sentence. Several observations can be made based on these samples: it is good at detecting name entities and replacing them with the name entities related to the chosen topic; there is no hard restriction on when to switch the topic; the model will determine an appropriate way to do the transition by itself. Such observations confirm that CP-VAE possesses a filled constrained latent space which make the latent code robust to manipulation across different time steps, which can be effectively reflected in the generation process. Due to space limitations, we put more samples in Appendix F. In this work, we investigate latent vacancy as an important problem in unsupervised learning of controllable representations when modelling text with VAEs. To mitigate this, we propose to constrain the posterior within a learned probability simplex, achieving the first success towards controlled text generation without supervision. For the β-VAE we used for the exploratory experiments, we use a LSTM encoding network and a LSTM decoding network. For the encoding network, the input size is 256, and the hidden size is 1,024. For the decoding network, the input size is 256, the hidden size is 1,024, and dropouts with probability 0.5 are applied on after the embedding layer and the LSTM layer in the decoding network. β is chosen as 0.35, the dimension for the latent code is 80, and the batch size is 32. We use SGD with learning rate 1.0 to update the parameters for both the encoding and the decoding network. We train the model until the reconstruction loss stops decreasing. First, we normalize the value of each latent code by subtracting the mean estimated over all the training samples. Then we use the polarity of each latent code to classify the sentiment in the validation set. The one with the highest accuracy is identified as the latent factor indicating the sentiment. A.3 SAMPLES GENERATED FROM β-VAE Table 5: Samples of generated sentences from β-VAE on Yelp. Positive to Negative Negative to Positive SRC this place is super yummy! but it probably sucks too! β-VAE this place is perfect for all of us or so long and over priced! thank you! SRC i will be going back and enjoying this great place there is definitely not enough room in that part of the venue. β-VAE i will be going back and recommending this place to anyone who lives in the valley! there is great. Following manipulation strategies have been attempted: fixing the relevant factor to µ + 2σ and µ − 2σ; fixing the relevant factor to µ − σ and µ − σ; fixing the relevant factor to the maximum value and the minimum value of the relevant factor appearing in the training samples; calculating a latent vector based on 10 manually constructed parallel sentences with opposite sentiment while keeping other factors unchanged. However, none of these four strategies is effective considering the generation . We report the with the first strategy in the paper, since it performs the best considering the accuracy and the BLEU score. Across all the datasets, we use Adam with learning rate 0.001 to update the parameters for the encoding network, while SGD with learning rate 1.0 to update the parameters for the decoding network. The batch size is chosen to be 32. Dropouts with drop probability 0.5 are applied on applied on after the embedding layer and the LSTM layer in the decoding network. We train the model until the reconstruction loss stops decreasing. For the structured part z z z, we use β-VAE setting β as 0.2 across all the datasets. For the unstructured part z z z, different strategies are employed for each dataset: • Yelp: β-VAE setting β as 0.35. • Amazon: β-VAE setting β as 0.35. • AG-News: KL annealing, from 0.1 to 1.0 in 10 epochs. The hyperparameters are chosen by checking L VAE, KL, and the generated outputs on the development set for Yelp and AG-News. Amazon follows the same setting as Yelp without extra tuning. For decoding, we use beam search with a beam size of 5. C PROOF OF MINIMALIZATION OF EQ. 5 The problem can be formulated as an optimization problem as follows: By introducing a Lagrange multiplier λ, the Lagrange function is defined as In order to find the optimal point, we require that which shows that all p i are equal. By using the constraint i p i = 1, we find p i = Experimental setup: We use the AG news dataset for this task constructed by . It contains four topic categories which are World, Sports, Business and Sci/Tech, with the title and description fields. For each category, there are 30, 000 training samples and 1, 900 test samples. In this paper, we drop the title and just use the description field. We compare our approach to two standard baselines for unsupervised topic modelling: LDA , a standard implementation of LDA is used for this baseline 4; k-means. To show the power of our approach beyond the pre-trained sentence representations, we perform k-means clustering directly on the sentence representations. Following , we assign each inferred topic to one of the gold-standard topics with the optimal mapping and report the precision (a.k.a. purity), recall (a.k.a. collocation) and F 1 score. The number of topics is chosen to be 10. The reported for the baselines and our model are the average over 10 runs. Quantitative : The are shown in Table 7. We can see that our approach achieves comparable to LDA while significantly outperforming k-means in all four categories, indicating that our approach can go beyond just clustering on pre-trained sentence representations. Table 8: Sentiment manipulation from positive to negative SRC this was the best i have ever had! B-GST this was the worst place i have ever had! CP-G this was the worst pizza i have ever had! CP-B this was the worst i have ever had! SRC friendly and welcoming with a fun atmosphere and terrific food. B-GST the hummus is ridiculously bland and bland. CP-G rude and unorganized with a terrible atmosphere and coffee. CP-B the hummus is ridiculously greasy and tasteless. SRC i ordered the carne asada steak and it was cooked perfectly! B-GST i ordered the carne asada steak and it was just as bad! CP-G i ordered the carne asada steak and it was n't cooked and it was lacking. CP-B i ordered the carne asada burrito and it was mediocre. SRC the owner is a hoot and the facility is very accommodating. B-GST the owner is a jerk and the facility is very outdated. CP-G the owner is a hoot and the facility is empty and the layout is empty. the owner is a riot and the facility is very clean. SRC i will be going back and enjoying this great place! B-GST i wo n't be going back and this place is horrible! CP-G i will be going back and eat this pizza hut elsewhere. CP-B i will be going back and hated the worst dining experience. Table 9: Sentiment manipulation from negative to positive SRC there is definitely not enough room in that part of the venue. B-GST there is plenty enough seating in that part of the venue. there is definitely an authentic dinner in that part. there is definitely a nice theatre in that part. SRC but it probably sucks too! B-GST but it tastes great too! CP-G but it's truly fun and insanely delicious. CP-B but it's probably wonderful when u! SRC always rude in their tone and always have shitty customer service! B-GST always in tune with their tone and have great customer service. CP-G always great with their birthdays and always excellent music. CP-B always accommodating and my dog is always on family. SRC i was very sick the night after. B-GST i was very happy the night after. CP-G i was very pleased with the night. CP-B i was very happy with the night. SRC this is a horrible venue. B-GST this is a wonderful venue. CP-G this is a great place for celebrating friends. this is a great place for beginners. Table 10: Sentiment manipulation from positive to negative SRC most pizza wheels that i ve seen are much smaller. B-GST most pizza dough that i ve seen are much better. CP-G most pizza wheels that i ve seen are much more good and are much quality. CP-B most pizza wheels that i ve seen are much better than are much better SRC however, this is an example of how rosle got it right. B-GST however, this game is an example of how rosle loves it. CP-G however, this is an example of how toxic... sad... obviously. however, this is an example of how cheap. similar. cheap advice. cheap advice. similar. SRC auto shut off after num num hours, which is a good feature. B-GST auto shuts off after num num hours, which is a shame. CP-G whipped mask off after num num hours, which is slimy, which is disgusting. auto shut off after num num hours, which is a stupid idea, which seems to be bad. SRC that said, the mic did pic up everything it could. B-GST that said, the game took up everything it could. CP-G that said, the shampoo did nt smell him well. stopped cleaning everything. ended up smelling sick CP-B that said, the mic did not fit everything on well, let me down it weren t cleaning SRC i also prefered tha blade weight and thickness of the wustof! B-GST i also like the blade weight and of the wustof. CP-G i also disliked the blade weight and thickness of the materials. i also slammed the blade weight and thickness of the wide. Table 11: Sentiment manipulation from negative to positive SRC the quality is declined quickly by heat exposure. B-GST the water is quickly drained by head exposure. CP-G the quality is utilitarian so grinding or sandwiches. the quality is priceless quickly by heat rises. SRC the directions were easy to follow but the quality of the easel was pathetic. B-GST the directions were easy to follow but the quality of the product was excellent. CP-G the directions were easy to follow but the quality is good for the quality and is CP-B the directions were easy to follow but the quality is what the quality is like the best quality of SRC multiplayer is just as bad, though thankfully not worse. B-GST quality is just as good, though thankfully not perfect. besides it is just good, though. those.. usually... usually... multiplayer is just as bad, though somebody s also so far not so far but no problem. SRC another energy product that simply wastes our money. B-GST another energy product that simply saves our money. another energy product that simply glides your pasta. another energy product that simply wastes this money. SRC i received the wrong color and it shreds easily. B-GST i received the color and it works easily. CP-G i low the new color and it closes easily. CP-B i received the wrong color and it pours easily from dishwasher and dries easily on garlic easily. said on Tuesday it had developed a prototype micro fuel cell aimed at protecting the world's largest oil producer.' == 'why previous VAEs on text cannot learn controllable latent representation as on images, as well as a fix to enable the first success towards controlled text generation without supervision' != 'We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.'

Question: Capturing long-range feature relations has been a central issue on convolutional neural networks(CNNs). To tackle this, attempts to integrate end-to-end trainable attention module on CNNs are widespread. Main goal of these works is to adjust feature maps considering spatial-channel correlation inside a convolution layer. In this paper, we focus on modeling relationships among layers and propose a novel structure,'Recurrent Layer Attention network,' which stores the hierarchy of features into recurrent neural networks(RNNs) that concurrently propagating with CNN and adaptively scales feature volumes of all layers. We further introduce several structural derivatives for demonstrating the compatibility on recent attention modules and the expandability of proposed network. For semantic understanding on learned features, we also visualize intermediate layers and plot the curve of layer scaling coefficients(i.e., layer attention). Recurrent Layer Attention network achieves significant performance enhancement requiring a slight increase on parameters in an image classification task with CIFAR and ImageNet-1K 2012 dataset and an object detection task with Microsoft COCO 2014 dataset. Concatenating all features in the order of layers in convolutional neural network (CNN) provides new interpretation, features form a sequence, consisting of features with small receptive fields to large receptive fields. Interestingly, recurrent neural network (RNN) is one of the representatives for modeling the sequential information. . On the other hands, Recent attempts to utilize attention mechanism for empowering CNN a better representational power are prevalent (b;). Motivated by intrinsic characteristics of CNN and RNN, and recent attention works on computer vision, we present the Recurrent Layer Attention Network (RLA network), which is differentiable and light-weight while improving the representational power of CNN by a slightly different way from other attention works in computer vision. The main goal of our work is applying global weights balance among layers by inheriting the feature hierarchy from previous CNN layers. We accomplish the goal by two main structural designs: employing our inter-layer attention mechanism to make the network re-adjust features, and utilizing RNN to memorize the feature hierarchy with concurrently propagating parallel with CNN. We hypothesize that RLA network gains additional class discriminability through inheriting the informative feature hierarchy, such as repetitive appearances of important features or relevant features with different receptive fields. For example, our network raises the activation of neurons that is responsible to the whole body of zebra, using a history of features of relatively smaller receptive field. We demonstrate our hypothesis through the Grad-CAM visualization of intermediate features and corresponding layer attention value (i.e., the importance of layer.) We evaluate RLA network on image classification and object detection tasks using benchmark datasets: CIFAR, ImageNet-1K, and Microsoft COCO. On both tasks, RLA network gets comparable with the state-of-the-arts, Squeeze-and-Excitation network (b), and superior than original ResNet architecture. Moreover, we suggest the compatibility of the RLA network by introducing the expansion of RLA network combining our inter-layer attention mechanism toward recent attention works (b) on ablation study. Incorporating RLA network to recent attention works (Call these networks as intra-layer attention networks), further variations of RLA network can be recognized as the generalized attention network considering correlations among both inside and outside a layer. We summarize our contributions as follows: • We propose two new concepts: the weight balancing of CNN features along layers (call it inter-layer attention), and the connection from shallow CNN layers to deep CNN layers through concurrently propagating RNN. • We demonstrate the effectiveness of proposed RLA network for an image classification task and an object detection task on benchmark datasets. RLA network achieves similar or superior compared with leading deep network while requiring small model parameters increase. • Ablation studies investigate the effectiveness of two proposed concepts and show the compatibility towards existing intra-attention models, and the further expandability by tuning architectural designs. We show and discuss how RLA network is learned to interpret images by visualizing intermediate layers and plotting layer attention values. • RLA network is easy to implement requiring basic operations of modern deep learning frameworks. Attention mechanism in computer vision. Attention mechanism can be interpreted as a methodology to bias the allocation of available neurons to the most informative components of input signal (b). Recent main application of attention mechanism on computer vision is integrating end-to-end trainable attention module for deep CNNs. These attention modules are divided into two categories; spatial attention module, and channel-wise attention module. Spatial attention module learns spatial masks on feature maps for regulating the activations of neurons (; a). Channel-wise attention module earns channel-wise distribution and then utilizes it to refine feature maps (b; a). Several architectural designs both benefitting spatial and channel-wise attention are also stressed out (; b; a; b). However, all these spatial and channel-wise attention modules, as we call intra-layer attention, only handles interaction inside a CNN layer. Therefore, currently suggested modules structurally lack to model relation among visual features captured in different CNN layers. Therefore, we suggest inter-layer attention mechanism. Through global weight balancing along CNN layers, our module models complex relationships among visual features of different receptive fields. RNN in computer vision. Numbers of computer vision papers utilize RNN for various tasks containing visual question answering (a;), image captioning , multi-object classification (a;), and video description . They share common ground on the way that exploiting RNN; extracting visual features from images using pre-trained CNNs, then employ RNN to model sequential procedures of each task. For example, obtained CNN features per subsequent time frame series are becoming inputs to belonging RNN units, and RNN conduct a prediction over video recognition and description tasks . However, there are only few works to exploit RNN for directly enhancing Figure 2: The structure of Recurrent Layer Attention network (* = 1), RLA-vector (* = C l), and IA+RLA (including a dotted line). ⊗ denotes scale operations belonging to intra-layer attention. the performance of deep neural networks on vision tasks that do not require sequential decisions such as image classification or single object detection. One trial propose the deep network that employs RNN and region attending mechanism, but is non-differentiable. In contrast to earlier works, we manipulate CNN and RNN to simultaneously affect to each other at every layer of CNN so that enhancing the representational power of CNN. To the best of our knowledge, our concept of conveying information from shallow CNN layer to deep CNN layer through the use of concurrently propagating memory units is the first attempt. We start by defining several expressions to reduce unnecessary ambiguity. In our works, local patterns denote class-specific characteristics in images such as a head of a cat, which are collected as a response of the convolution operation. G denotes a convolution operation alone, or the arbitrary combinations of convolution operations and other mapping functions, such as residual block or inception Module . Features are the of convolution operation G. RLA network consists of three operations: summarizing features, storing summarized features in LSTM, and inferring the layer attention. Right after the convolution operation G, RLA module "summarizes" the feature volume into a statistic, which is a bag of the local patterns in images. Then it "stores" the context in LSTM hidden units, and finally "infers" the layer attention of how much to concentrate on feature volume from G. The concept figure describing this procedure is in Figure 1. The intuition of utilzing RNN is derived from recognizing concatenated feature volumes along CNN layers as a sequence. Because each element of the feature sequence is a prior information implying which local patterns emerged, we hypothesize that LSTM hidden units which is carrying the hierarchy of features can provide meaningful information to CNN such as the importance of latter layer. Therefore, we utilize the attention mechanism to empower CNN reducing the activation of overlapped local patterns, or emphasizing the activation of important local patterns which highly contribute to the class discriminability. Following subsections are organized as follows. The formulation of RLA network is described in Subsection 3.1, several architectural derivatives for showing the expandability and the compatibility toward intra-attention modules are introduced in Subsection 3.2, and the light-weightness of the RLA network is described in Subsection 3.3. We present our RLA network by describing one forward-passing procedure of l-th layer. The feature volume A l is yielded by passing G l: Summarizing Feature. Feature volume A l is summarized as the context. Here, the context S l ∈ R N signifies a simplified representation of local patterns in the image, and is computed at every layer of CNN through: where F context is a combination of feature volume summarizing operation and consequent downsampling (or up-sampling) operation. We investigate various types of feature summarizing operations for finding proper statistics that embeds the feature volume on Section 3.2. In order to feed S l into LSTM cell, down-sampling and up-sampling techniques must be adaptively applied per layers, because RNN requires a fixed size of input to each cell. Note here, all the same feature summarizing operation is used on every layer. Storing Context. The context captured in each earlier layer of CNN is recurrently inserted into LSTM hidden units h l ∈ R M, and selectively embedded in h l through the updating functions of LSTM with h l−1: Here, F LST M is a LSTM cell updating operation described in . The LSTM hidden states and cell states of the first stage, h 1 is initialized with zero. Inferring Layer Attention. Considering the sequence of contexts previously fed into LSTM through the cascades of convolution operations and memory mechanism of LSTM until l-th layer, h l infers the layer attention, α l ∈, the scalar value for scaling the feature volume A l: where F att is composed of two fully-connected layers with r. δ and σ refer to the ReLU and sigmoid operations, respectively. Reduction ratio r denotes compressing ratio of the first fully-connected layer. Finally, the scaled feature volumeÃ l is computed by element-wise multiplication F scale between α l and all elements of A l: We introduce several derivatives of RLA network based on two key design variables, the attention structure and the context. All instances are experimented and discussed on Subsection 4.1. Adoption of intra-layer attention mechanism. We show the compatibility of RLA network toward intra-layer attention mechanism. Adopting the intra-layer attention module, the scale operation for c-th channel of feature volume on l-th layer is given by: where A c l denotes a c-th feature map of, and F scale denotes an arbitrary intra-layer attention mechanism which calibrate a feature volume inside the layer. Figure 2 depicts the integration of RLA with intra-layer attention module. We call this structure as Intra-layer Attention + RLA network (IA+RLA). To evaluate the performance of IA+RLA, we select Squeeze-and-Excitation block (b) as IA module. Distilling advantages from feature hierarchy. RLA network inherits two advantages: utilizing feature hierarchy to affect on subsequent features using RNN, and adaptively scaling features layer by layer. Because both two factors are latently contributing on guiding CNN to have better representational power, it's hard to discriminate effects of them. To evaluate different effectiveness of two factors, we additionally design a structure named RLA-vector. Between previously mentioned two concepts, RLA-vector only adopt the concept of exploiting feature hierarchy with concurrently propagating RNN, while not applying inter-layer attention mechanism. Utilizing the intra-layer attention mechanism in Squeeze-and-Excitation module (SE module) (b), RLA-vector adjust feature volumes channel-wisely. In the other words, LSTM hidden units of RLA-vector predict a channel-sized vector, not a layer attention scalar. Through the comparison between Squeeze-andExcitation block that does not inherits a feature hierarchy and RLA-vector, we can examine a pure impact on the usage of co-propagating RNN. Context: simplified representation of feature volume. We focus on finding the context, a summarized feature volume, that assures preserving the information of local patterns during feature summarization procedure. Here, we concentrate on employing statistics that only summarize in spatial (H × W) dimensions of the feature volume, while not exploiting the channel-wise summary of feature volume. This intuition comes from the existence information of local patterns is less related to it's location. First, we adopt global average pooled feature and global max pooled feature introduced in (b;) as contexts. Correspondingly, they summarize the spatial dimensions of the feature volume using average pooling and max pooling. We call these statistics as global average pooled features (GAP) and global max pooled features (GMP). The meaning of exploiting GAP for our works is considering the size or multiple appearances of local patterns in spatial dimensions of feature volume. On the other hand, GMP is used for catching the most salient part of local patterns. Second, we additionally apply non-linear operations on GAP and GMP by attaching two fully connected layers, and call them as GAP+MLP and GMP+MLP. These statistics are for modeling the multiple appearance of similar local patterns at different channels. Seemingly, these statistics have same structure with SE module (b). However, the goal for exploiting non-linearity is different. We use non-linearity for producing more informative context, while SE network is for predicting channel-wise scaling coefficients of the feature volume. Additional model parameters required for utilizing RLA module is given by: where M is number of neurons in LSTM cell, N is a dimension of context, L is a number of layers in CNN, and r denotes the reduction ratio. As the formula implies, the additional model complexity caused by exploiting RLA module doesn't depend on the depth of backbone CNN. This is by the virtue of weight-sharing property from RNN. In contrast to ResNet-56 with SE module causes 1.51% parameters increase, ResNet-56 with our module (RLA) only increases 0.14% parameters compared to original ResNet-56. In this section, we conduct ablation experiments for analyzing proposed instances and introduce experimental on CIFAR and ImageNet-1K datasets for an image classification task, and Microsoft COCO datasets for an image detection task. We adopt ResNet as a backbone CNN architecture for all experiments. Ensuring a fair comparison with our works, we re-implement the of ResNet and Squeeze-and-Excitation network. Considering the readibility, implementation details over experiments are described on Appendix A and the parametric study for examining the effects of hyper-parameters (i.e., M, N, r) are reported on Appendix B. The main goal of ablation study is demonstrating the expandability of RLA network. Key concepts illustrated in this part are as follows: the compatibility of RLA toward intra-layer attention mechanism, the impacts of inheriting feature hierarchy using RNN for empowering CNN to have better representational power, the effectiveness of introduced statistics for context. We adopt ResNet-56 as backbone structure for the ablation study. Adopting Channel Attention. Table 1 shows every instances outperform ResNet-56 on CIFAR-10 and CIFAR-100 dataset. Interestingly, we find out a IA+RLA, the integrated structure of intra-layer attention and our approach, works worse than SE and RLA alone, but still better than ResNet. Note here, these might have different tendency belonging to the backbone structure and dataset. The reason for that is two designs are on the complementary positions. Specifically, intra-layer attention mechanism adjusts features with similar size of receptive fields, while inter-layer attention mechanism focus on calibrating features with different size of receptive fields. We leave further experiments and analysis on this observation as future works. Distillation of Feature Hierarchy. Supporting a key design concept of RLA network, the exploitation of concurrently propagating RNN, we observe RLA-vector significantly works well compared to SE network. These experimental claims that utilizing the feature hierarchy through RNN alone also remarkably aids CNN to interpret an image better. This is interesting since it suggests the potential that the proposed concept of inheriting feature hierarchy through RNN can be solely applied to other neural network architectural designs. Utilizing Various Context. Various feature summary statistics for the context differently affects on performance. Similar to the reported in (b;), GMP reports higher error than GAP. Interestingly, the on GAP+MLP and GMP+MLP are different depending on the dataset. Through this observation, we could recognize considering multiple appearances of local patterns on different channels in feature volume has negligible impact. We confirm GAP is the most appropriate context considering the performance increase and the model complexity. CIFAR dataset is consist of 50K training images and 10K validation images. We train networks for 160 epochs with the learning rate is initialized with 0.1 and divided by 10 when reached 80 and 120 epochs. The training and validation curve over CIFAR dataset are depicted in Figure 3. Through the optimization schedule, we observe that RLA network achieves lower training/validation error. ImageNet-1K 2012 dataset is comprised of around 1.28M training images and 50K validation images. We train networks for 100 epochs on ImageNet-1K 2012 dataset with the learning rate is set to 0.1 and divided by 10 every 30 epochs. Data augmentation and optimization details follows the methods that ResNet adopts. More implementation details are discussed on Appendix A. Table 2 (top-right) describes the experimental . We notice that RLA network outperforms ResNet, but ranks below the SE network on performance. We interpret these come from structural limitation of RLA module, when applied to backbone CNNs with large channels. Giving an example of ResNet-50 as a backbone architecture, RLA scale the network only using 16 scale coefficients (total numbers of residual blocks), while SE module adjusts the network using 15104 scale coefficients (total numbers of adjusting channels over all residual blocks). As mentioned in ablation study, we expect the integration of intra-layer mechanism and RLA network produces better . Microsoft COCO 2014 dataset contains 83K training images and 41K validation images. We adopt Faster R-CNN (b) as a detection method and replace the baseline network from ResNet to RLA network. We train networks for 490K iterations with ImageNet-1K 2012 pre-trained ResNet and RLA networks. More implementation details are discussed on Appendix A. Table 2 (bottom-right) summarizes the experimental . Here, we observe the improvements from original ResNet baseline, verifying a generalization performance on the object detection task. Grad-CAM visualziation. We hypothesized that RLA network gains additional class discriminability by considering repetitive appearances of unimportant features or relevant features with different receptive fields. For demonstrating above hypothesis and semantically understanding the way how RLA network learned, we applied Grad-CAM visualization over intermediate CNN layers. This visualization technique is of interest to our work since it provides a measure of importance of each pixel in a feature map towards the overall decision of the CNN . Grad-CAM visualization is generally applied for the last CNN layer because the last CNN layer is normally recognized as the most semantically salient layer. Instead, we apply the Grad-CAM technique for the layers which are located directly before scale operation with layer attention We select three target class having local patterns of different visual characteristics; racer, ostrich, freight car. Here, different visual characteristics signify different receptive fields of seemingly salient local patterns for discriminating target class. Sorting in ascending order, we recognize racer class has local patterns of smallest receptive fields such as a wing, wheels, ostrich class has local pattern of medium receptive field, head. Third, freight car class seems to have no specific local pattern or parts by itself. As , we observe following interesting facts. First, RLA network learns to enhance the layer attention value on layers that catch more seemingly salient features for each target class. The layers marked with red border denote layers that achieve the highest layer attention value. The first layer that captures simple pattern or small parts of racer class, the fifth layer that focus to catch head part of ostrich class, the seventh layer that captures global information or relations among locally ambiguous features for freight car class belongs that category. Through these observation, we hypothesize that RLA network emphasize the layer that catches class-specific local patterns for each target class. Second, deep layers of RLA network tend to learn class-agnostic features after acquiring class-specific or semantically salient features in previous layers. The layers marked with yellow border denote a representative that less focus on class-specific local patterns in the images. Giving an example of ostrich class, the last layer of RLA network tend to learn class-agnostic local patterns, . More visualizations on different target classes are on Appendix E. Comparing with original ResNet keeps allocating more class-specific neurons until deep layers, these observations are quite interesting. We additionally note that distinctly different highlighting pattern between close layers is because the residual architecture learns the'residual' while keeping previously obtained features in earlier layers. Furthermore, This observation accords with the investigation about the class selectivity on Gather-Excite network (a); the intra-attention mechanism applied CNN has lower class selectivity in deeper layers compared to original backbone. We understand this finding is identically applied to our inter-attention mechanism (i.e., RLA network) and our support the of previous work. Mean layer attention curve. Figure 5 shows a mean layer attention curve along RLA network layers. Investigation on layer attention values over RLA network enable deeper understanding on the residual architecture, and traditional debates on CNN. First, layer attention values are relatively high in shallow layers and rather lower values in middle layers. We recognize this observation could be another way for further understanding residual architecture. The intuition behind the residual learning is to let the layers learn the perturbations with reference to an identity function, and original paper supports their intuition via displaying the standard deviation of feature responses along layers. Here, feature responses are defined as the outcomes (batch-normalized) of residual blocks. Comparing with that, we notice that our plot provides similar information but more straightforward. Through the observation on layer attention values decrease along layers, we could reason the intuition of the residual learning as latter layers learn less important layers. Second, the variance of layer attention values increase along the layers except the last residual block, and layers in last residual block of RLA network has the biggest layer attention values that close to 1. We interpret traditional debates on CNN, shallow layers in CNN learn simple or "low-level" features while deep layers in CNN learn powerful, semantic, or "high-level" features supports these observation. Supported by previous investigation, we think it's natural that RLA network learns large layer attention close to the latest CNN layers which are semantically salient, and layer attention values drastically vary on deep layers that contains abstract features. In this paper, we first propose inter-layer attention mechansim to enhance the representational power of CNN. We structurized our mechanism as'Recurrent Layer Attention network' by utilizing two new concepts: the weight balancing of CNN features along layers and the link from shallow CNN layers to deep CNN layers via RNN for directly conveying the feature hierarchy. We introduce structural derivatives of RLA network:'IA+RLA' for proving an applicability of our work toward recent intra-layer attention mechanism, and'RLA-vector' for distilling the impacts of proposed two new concepts. We also precisely select statistics for the context by focusing local patterns preserved in feature summarization procedure. We evaluate RLA network using CIFAR and ImageNet-1k 2012 datasets for an image classification task, and also verify it's generalization ability toward an object detection task via experiments on Microsoft COCO dataset. For demonstrating our hypothesis that RLA network gains additional class discriminability and semantically understanding how RLA network induce their model parameters to be learned, we visualize RLA network utilizing Grad-CAM visualization, plot the layer attention value curve, and report several interesting findings. For future works, we are planning to integrate our inter-layer attention mechanism to intra-layer attention mechanism with heavier experiments first, and to utilize the concept of making any kinds of arbitrary connection from earlier layer to latter layer through RNN in other domains. Image recognition task. Data augmentation and optimization details follow. We exploit scale augmentation, random-size cropping, and random horizontal flipping, not using the color augmentation. For optimization, we use nesterov SGD with momentum 0.9 with weight decay of 0.0001, and initialize weights by the strategy described in. Object detection task. For fast implementation on object detection tasks, we utilize ImageNet-1K pretrained ResNet, SE networt, and RLA network. Implementation details follows the works investigating the fast implementation of faster ). We present hyperparameters for designs of RLA such as a reduction ratio r of layer attention, a dimension of context N, the number of neurons M in LSTM cell. Since hyper-parameters are highly related to model complexity, it's a crucial issue to select them. In this section, we explain the standard to select hyper-parameters and demonstrate effects on the performance of the network. Dimension of Context. Because feature volumes over the layers of CNN have different number of channels, summarized feature volume must be downsampled or upsampled for being inserted into LSTM. However, these operations cause an unintended lack or lost of information. Therefore, we select a dimension of context as a minimum number of downsampling or upsampling to be executed. Size of LSTM Units. In our works, the size of LSTM hidden units, M, highly affects model and computational complexity. Following other works to find proper RNN hidden units size by rule-ofthumb or conducting experiments , we perform experiments for finding the trade-off among the performance and complexities. Table 3 shows experimental on varying M ∈. Surprisingly, the performance of RLA does not drop as M decrease, rather increase. We deduce the reason of this observation is because the distributions of contexts are clearly separable, only small numbers of LSTM hidden units are enough for reducing considerable bias. Reduction Ratio. Neither the reduction ratio, r, has golden rule as hidden units. Accordingly, we conduct experiments on the reduction ratio r ∈, and found out there exists only slight differences on top-1%-error contrast to experimental with varying hidden units. Therefore, we compress model complexity of RLA network by choosing small M which also affects model performance, and further regulate the complexity using reduction ratio. Exploiting big M on RLA-vector. Following interesting tendency that smaller M induce performance increase, we conduct an ablation experiments with M = 4. However, we found that RLA-vector with small M yields prominent performance decrease. We interpret this as, reduced model capacity by decreasing M can not scale channel-sized vector of each layer in CNN. For the fair comparison with SE network, we set M = 32 considering that the average size of global average pooled feature volumes also equals to 32 in SE-ResNet-56. hyperparameter selection in ImageNet-1K experiments. We observe the tendency on performance by changing hyperparameter in parametric study. However, This tendency does not always fit in other dataset too. Considering the data diversity of ImageNet-1K 2012 datasets, we exploit commonly used RNN hidden units size for ResNet-56, M = 512, as exploited in previous CNN-RNN paper . We note that further consideration on differing LSTM hidden units size M possibly reports better performance even with smaller parameters, as shown in the parametric study on CIFAR dataset. About reduction ratio, we apply r = 8 for RLA-18/50 and r = 16 for RLA-101. We describe the location of connection between CNN and RNN in RLA-50 network. When exploiting vanila CNN, applying RLA to backbone CNN is straightforward, making connections with concur-rently propagating RNN just after every convolution layer of the network. However, it's confusing to apply RLA when using residual architecture. Table 4 shows the structure of backbone CNN using ResNet-50 and ResNet-50 + RLA. Note here, RNN that concurrently propagating with CNN is not depicted for ResNet-50 + RLA. In short, RLA network stores the context and infers the layer attention for scaling features per each residual block. For aiding semantic understanding of how RLA network induces their model paramters to be learned, we provide Grad-CAM visualization of the intermediate feature maps over the CNN layers. We could find out similar observations debated on the main article on other target class; the tendency to enhance the layer attention value of layers such that catches the most semantically important visual features, and another tendency to learn non class-specific features in deep layers. Figure 6 depicts visualization on the target class of junco, achidna, killer whales, leonberg, tiger cat, sleeping bags.

represent the short summary for finding the corresponding scientific article (long)

We propose a new type of end-to-end trainable attention module, which applies global weight balances among layers by utilizing co-propagating RNN with CNN.

Represent the TLDR for finding the corresponding scientific article

We propose to structure the generator of a GAN to consider objects and their relations explicitly, and generate images by means of composition


Question: represent the article for finding a single-sentence tldr

Question: We seek to auto-generate stronger input features for ML methods faced with limited training data. Biological neural nets (BNNs) excel at fast learning, implying that they extract highly informative features. In particular, the insect olfactory network learns new odors very rapidly, by means of three key elements: A competitive inhibition layer; randomized, sparse connectivity into a high-dimensional sparse plastic layer; and Hebbian updates of synaptic weights. In this work we deploy MothNet, a computational model of the moth olfactory network, as an automatic feature generator. Attached as a front-end pre-processor, MothNet's readout neurons provide new features, derived from the original features, for use by standard ML classifiers. These ``insect cyborgs'' (part BNN and part ML method) have significantly better performance than baseline ML methods alone on vectorized MNIST and Omniglot data sets, reducing test set error averages 20% to 55%. The MothNet feature generator also substantially out-performs other feature generating methods including PCA, PLS, and NNs. These highlight the potential value of BNN-inspired feature generators in the ML context. Machine learning (ML) methods, especially neural nets (NNs) with backprop, often require large amounts of training data to attain their high performance. This creates bottlenecks to deployment, and constrains the types of problems that can be addressed. The limited-data constraint is common for ML targets that use medical, scientific, or field-collected data, as well as AI efforts focused on rapid learning. We seek to improve ML methods' ability to learn from limited data by means of an architecure that automatically generates, from existing features, a new set of class-separating features. Biological neural nets (BNNs) are able to learn rapidly, even from just a few samples. Assuming that rapid learning requires effective ways to separate classes, we may look to BNNs for effective feature-generators. One of the simplest BNNs that can learn is the insect olfactory network, containing the Antennal Lobe (AL) and Mushroom Body(MB), which can learn a new odor given just a few exposures. This simple but effective feedforward network contains three key elements that are ubiquitous in BNN designs: Competitive inhibition, high-dimensional sparse layers [7; 8], and a Hebbian update mechanism. Synaptic connections are largely random. MothNet is a computational model of the M. sexta moth AL-MB that demonstrated rapid learning of vectorized MNIST digits, with performance superior to standard ML methods given N ≤ 10 training samples per class. The MothNet model includes three key elements, as follows. (i) Competitive inhibition in the AL: Each neural unit in the AL receives input from one feature, and outputs not only a feedforward excitatory signal to the MB, but also an inhibitory signal to other neural units in the AL that tries to dampen other features' presence in the sample's output AL signature. (ii) Sparsity in the MB, of two types: The projections from the AL to the MB are non-dense (≈ 15% non-zero), and the MB neurons fire sparsely in the sense that only the strongest 5% to 15% of the total population are allowed to fire (through a mechanism of global inhibition). (iii) Weight updates affect only MB→Readout connections (AL connections are not plastic). Hebbian updates occur as: ∆w ij = αf i f j if f i f j > 0 (growth), and ∆w ij = −δw ij if f i f j = 0 (decay), where f i, f j are two neural firing rates (f i ∈ MB, f j ∈ Readouts) with connection weight w ij. In this work we tested whether the MothNet architecture can usefully serve as a front-end feature generator for an ML classifier (our thanks to Blake Richards for this suggestion). We combined MothNet with a downstream ML module, so that the Readouts of the trained AL-MB model were fed into the ML module as additional features. From the ML perspective, the AL-MB acted as an automatic feature generator; from the biological perspective, the ML module stood in for the downstream processing in more complex BNNs. Our Test Case was a non-spatial, 85-feature, 10-class task derived from the downsampled, vectorized MNIST data set (hereafter "vMNIST"). On this non-spatial dataset, CNNs or other spatial methods were not applicable. The trained Mothnet Readouts, used as features, significantly improved the accuracies of ML methods (NN, SVM, and Nearest Neighbors) on the test set in almost every case. That is, the original input features (pixels) contained class-relevant information unavailable to the ML methods alone, but which the AL-MB network encoded in a form that enabled the ML methods to access it. MothNet-generated features also significantly out-performed features generated by PCA (Principal Components Analysis), PLS (Partial Least Squares), NNs, and transfer learning (weight pretraining) in terms of their ability to improve ML accuracy. These indicate that the insect-derived network generated significantly stronger features than these other methods. To generate vMNIST, we downsampled, preprocessed, and vectorized the MNIST data set to give samples with 85 pixels-as-features. vMNIST has the advantage that our baseline ML methods (Nearest Neighbors, SVM, and Neural Net) do not attain full accuracy at low N. Trained accuracy of baseline ML methods was controlled by restricting training data. Full network architecture details of the AL-MB model (MothNet) are given in. Full Matlab code for these cyborg experiments including comparison methods, all details re ML methods and hyperparameters, and code for MothNet simulations, can be found at. MothNet instances were generated randomly from templates that specified connectivity parameters. We ran two sets of experiments: Cyborg vs baseline ML methods on vMNIST Experiments were structured as follows: 1. A random set of N training samples per class was drawn from vMNIST. 2. The ML methods trained on these samples, to provide a baseline. 3. MothNet was trained on these same samples, using time-evolved stochastic differential equation simulations and Hebbian updates, as in. 4. The ML methods were then retrained from scratch, with the Readout Neuron outputs from the trained MothNet instance fed in as additional features. These were the "insect cyborgs", i.e. an AL-MB feature generator joined to a ML classifier. 5. Trained ML accuracies of the baselines and cyborgs were compared to assess gains. To compare the effectiveness of MothNet features vs features generated by conventional ML methods, we ran vMNIST experiments structured as as above, but with MothNet replaced by one of the following feature generators: 1. PCA applied to the vMNIST training samples. The new features were the projections onto each of the top 10 modes. 2. PLS applied to the vMNIST training samples. The new features were the projections onto each of the top 10 modes. Since PLS incorporates class information, we expected it to out-perform PCA. 3. NN pre-trained on the vMNIST training samples. The new features were the (logs of the) 10 output units. This feature generator was used as a front end to SVM and Nearest Neighbors only. Since vMNIST has no spatial content, CNNs were not used. 4. NN with weights initialized by training on an 85-feature vectorized Omniglot data set, then trained on the vMNIST data as usual (transfer learning, applied to the NN baseline only). Omniglot is an MNIST-like thumbnail collection of 1623 characters with 20 samples each. (5.) For the baseline NN method, we used one hidden layer. Including two hidden layers did not improve baseline performance. This was an implicit control, showing that MothNet features were not equivalent to just adding an extra layer to a NN. MothNet readouts as features significantly improved accuracy of ML methods, demonstrating that the MothNet architecture effectively captured new class-relevant features. We also tested a non-spatial, 10-class task derived from the Omniglot data set and found similar gains. MothNet-generated features were also far more effective than the comparison feature generators (PCA, PLS, and NN). Gains due to MothNet features on vMNIST ML baseline test set accuracies ranged from 10% to 88%, depending on method and on N (we stopped our sweep at N = 100). This baseline accuracy is marked by the lower colored circles in Fig 1. Cyborg test set accuracy is marked by the upper colored circles in Fig 1, and the raw gains in accuracy due to MothNet features are marked by thick vertical bars. MothNet features increased raw accuracy across all ML models. Relative reduction in test set error, as a percentage of baseline error, was 20% to 55%, with high baseline accuracies seeing the most benefit (Fig 2). NN models saw the greatest benefits, with 40% to 55% relative reduction in test error. Remarkably, a MothNet front-end improved ML accuracy even in cases where the ML baseline already exceeded the ≈ 75% accuracy ceiling of MothNet (e.g. NNs at N = 15 to 100 samples per class): the MothNet readouts contained clustering information which ML methods leveraged more effectively than MothNet itself. Gains were significant in almost all cases with N > 3. Table 1 gives p-values of the gains due to MothNet. Table 1 ). We ran the cyborg framework on vMNIST using PCA (projections onto top 10 modes), PLS (projection onto top 10 modes), and NN (logs of the 10 output units) as feature generators. Each feature generator was trained (e.g. PCA projections were defined) using the training samples. Table 2 gives the relative increase in mean accuracy due to the various feature generators (or to pre-training) for NN models (13 runs per data point). Results for Nearest Neighbors and SVM were similar. MothNet features were far more effective than these other methods. Effect of pass-through AL The MothNet architecture has two main layers: a competitive inhibition layer (AL) and a highdimensional, sparse layer (MB). To test the effectiveness the MB alone, we ran the vMNIST experiments, but using a pass-through (identity) AL layer for MothNet. Cyborgs with a pass-through AL still posted significant improvements in accuracy over baseline ML methods. The gains of cyborgs with pass-through ALs were generally between 60% and 100% of the gains posted by cyborgs with normal ALs (see Table 3), suggesting that the high-dimensional, trainable layer (the MB) was most important. However, the competitive inhibition of the AL layer clearly added value in terms of generating strong features, up to 40% of the total gain. NNs benefitted most from the AL layer. We deployed an automated feature generator based on a very simple BNN, containing three key elements rare in engineered NNs but endemic in BNNs of all complexity levels: (i) competitive inhibition; (ii) sparse projection into a high-dimensional sparse layer; and (iii) Hebbian weight updates for training. This bio-mimetic feature generator significantly improved the learning abilities of standard ML methods on both vMNIST and vOmniglot. Class-relevant information in the raw feature distributions, not extracted by the ML methods alone, was evidently made accessible by MothNet's pre-processing. In addition, MothNet features were consistently much more useful than features generated by standard methods such as PCA, PLS, NNs, and pre-training. The competitive inhibition layer may enhance classification by creating several attractor basins for inputs, each focused on the features that present most strongly for a given class. This may push otherwise similar samples (of different classes) away from each other, towards their respective class attractors, increasing the effective distance between the samples. The sparse connectivity from AL to MB has been analysed as an additive function, which has computational and anti-noise benefits. The insect MB brings to mind sparse autoencoders (SAs) e.g.. However, there are several differences: MBs do not seek to match the identity function; the sparse layers of SAs have fewer active neurons than the input dimension, while in the MB the number of active neurons is much greater than the input dimension; MBs have no pre-training step; and the MB needs very few samples to bake in structure that improves classification. The MB differs from Reservoir Networks in that MB neurons have no recurrent connections. Finally, the Hebbian update mechanism appears to be quite distinct from backprop. It has no objective function or output-based loss that is pushed back through the network, and Hebbian weight updates, either growth or decay, occur on a local "use it or lose it" basis. We suspect that the dissimilarity of the optimizers (MothNet vs ML) was an asset in terms of increasing total encoded information.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Features auto-generated by the bio-mimetic MothNet model significantly improve the test accuracy of standard ML methods on vectorized MNIST. The MothNet-generated features also outperform standard feature generators.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.