Question: Represent the scientific article to find a short TLDR

Question: Interpretability and small labelled datasets are key issues in the practical application of deep learning, particularly in areas such as medicine. In this paper, we present a semi-supervised technique that addresses both these issues simultaneously. We learn dense representations from large unlabelled image datasets, then use those representations to both learn classifiers from small labeled sets and generate visual rationales explaining the predictions. Using chest radiography diagnosis as a motivating application, we show our method has good generalization ability by learning to represent our chest radiography dataset while training a classifier on an separate set from a different institution. Our method identifies heart failure and other thoracic diseases. For each prediction, we generate visual rationales for positive classifications by optimizing a latent representation to minimize the probability of disease while constrained by a similarity measure in image space. Decoding the ant latent representation produces an image without apparent disease. The difference between the original and the altered image forms an interpretable visual rationale for the algorithm's prediction. Our method simultaneously produces visual rationales that compare favourably to previous techniques and a classifier that outperforms the current state-of-the-art. Deep learning as applied to medicine has attracted much interest in recent years as a potential solution to many difficult problems in medicine, such as the recognition of diseases on pathology slides or radiology images. However, adoption of machine learning algorithms in fields such as medicine relies on the end user being able to understand and trust the algorithm, as incorrect implementation and errors may have significant consequences. Hence, there has recently been much interest in interpretability in machine learning as this is a key aspect of implementing machine learning algorithms in practice. We propose a novel method of creating visual rationales to help explain individual predictions and explore a specific application to classifying chest radiographs. There are several well-known techniques in the literature for generating visual heatmaps. Gradient based methods were first proposed in 2013 described as a saliency map in BID11, where the derivative of the final class predictions is computed with respect to the input pixels, generating a map of which pixels are considered important. However, these saliency maps are often unintelligible as convolutional neural networks tend to be sensitive to almost imperceptible changes in pixel intensities, as demonstrated by recent work in adversarial examples. In fact, obtaining the saliency map is often the first step in generating adversarial examples as in BID3. Other recent developments in gradient based methods such as Integrated Gradients from BID12 have introduced fundamental axioms, including the idea of sensitivity which helps focus gradients on relevant features. Occlusion sensitivity proposed by is another method which covers parts of the image with a grey box, mapping the ant change in prediction. This produces a heatmap where features important to the final prediction are highlighted as they are occluded. Another wellknown method of generating visual heatmaps is global average pooling. Using fully convolutional neural networks with a global average pooling layer as described in BID15, we can examine the class activation map for the final convolutional output prior to pooling, providing a low resolution heatmap for activations pertinent to that class. A novel analysis method by BID10 known as locally interpretable model-agnostic explanations (LIME) attempts to explain individual predictions by simulating model predictions in the local neighbourhood around this example. Gradient based methods and occlusion sensitivity can also be viewed in this light -attempting to explain each classification by changing individual input pixels or occluding square areas. However, sampling the neighbourhood surrounding an example in raw feature space can often be tricky, especially for image data. Image data is extremely complex and high-dimensional -hence real examples are sparsely distributed in pixel space. Sampling randomly in all directions around pixel space is likely to produce non-realistic images. LIME's solution to this is to use superpixel based algorithms to oversegment images, and to perturb the image by replacing each superpixel by its average value, or a fixed pre-determined value. While this produces more plausible looking images as opposed to occlusion or changing individual pixels, it is still sensitive to the parameters and the type of oversegmentation used -as features larger than a superpixel and differences in global statistics may not be represented in the set of perturbed images. This difficulty in producing high resolution visual rationales using existing techniques motivates our current research. We introduce a novel method utilizing recent developments in generative adversarial networks (GANs) to generate high resolution visual rationales. We demonstrate the use of this method on a large dataset of frontal chest radiographs by training a classifier to recognize heart failure on chest radiographs, a common task for doctors. Our method comprises of three main steps -we first use generative adversarial networks to train a generator on an unlabelled dataset. Secondly, we use the trained generator as the decoder section of an autoencoder. This enables us to encode and decode, to and from the latent space while still producing high resolution images. Lastly, we train simple supervised classifiers on the encoded representations of a smaller, labelled dataset. We optimize over the latent space surrounding each encoded instance with the objective of changing the instance's predicted class while penalizing differences in the ant decoded image and the original reconstructed image. This enables us to visualize what that instance would appear as if it belonged in a different class. Firstly, we use the Wasserstein GAN formulation by and find that the addition of the gradient penalty term helps to stabilize training as introduced by BID4. Our unlabelled dataset comprises of a set of 98,900 chest radiograph images, which are scaled to 128 by 128 pixels while maintaining their original aspect ratio through letterboxing, and then randomly translated by up to 8 pixels. We use a 100 dimensional latent space. Our discriminator and generator both use the DCGAN architecture while excluding the batch normalization layers and using Scaled Exponential Linear Units described in BID7 as activations except for the final layer of the generator which utilized a Tanh layer. We train the critic for 4 steps for each generator training step. The GAN training process was run for 200k generator iterations before visually acceptable generated images were produced. ADAM was used as the optimizer with the generator and discriminator learning rates both set to 5 x 10 -5.In the next step, we use the trained generator as the decoder for the autoencoder. We fix the weights of the decoder during training and train our autoencoder to reproduce each of the images from the unlabelled dataset. The unlabelled dataset was split by patient in a 15 to 1 ratio into a training and validation set. We minimize the Laplacian loss between the input and the output, inspired by BID1. Minimal overfitting was observed during the training process even when the autoencoder was trained for over 1000 epochs, as demonstrated in 2.We then train a classifier on a smaller labelled dataset consisting of 7,391 chest radiograph images paired with a B-type natriuretic peptide (BNP) blood test that is correlated with heart failure. This test is measured in nanograms per litre, and higher readings indicate heart failure. This is a task of real-world medical interest as BNP test readings are not often available immediately and offered at all laboratories. Furthermore, the reading of chest radiograph images can be complex, as suggested by the widely varying levels of accuracy amongst doctors of different seniority levels reported by BID6. We perform a natural logarithm on the actual BNP value and divide the ant number by 10 to scale these readings to between 0 and 1. This task can be viewed as either a regression or classification task, as a cut-off value is often chosen as a diagnostic threshold. In this paper, we train our network to predict the actual BNP value but evaluate its AUC at the threshold of 100ng/L. We choose AUC at this threshold as this is the cut-off suggested by BID9, and AUC is a widely used metric of comparison in the medical literature. We augment each labelled image and encode it into the latent space using our previously trained autoencoder. To prevent contamination, we separate our images by patient into a training and testing set with a ratio of 4 to 1 prior to augmentation and encoding. We demonstrate the success of simple classifiers upon this latent representation, including a 2 layer multilayer perceptron with 256 hidden units as well as a linear regressor. To obtain image specific rationales, we optimize over the latent space starting with the latent representation of the given example. We fix the weights of the entire model and apply the ADAM optimizer on a composite objective comprising of the output value of the original predicted class and a linearly weighted mean squared error term between the decoded latent representation and the decoded original representation. We cap the maximum number of iterations at 5000 and set our learning rate at 0.1. We stop the iteration process early if the cutoff value for that class is achieved. The full algorithm is described in Algorithm 1. This generates a latent representation with a different prediction from the initial representation. The difference between the decoded generated representation and the decoded original representation is scaled and overlaid over the original image to create the visual rationale for that image. We use gradient descent to optimize the following objective: DISPLAYFORM0 DISPLAYFORM1 Where X is the reconstructed input image (having been passed through the autoencoder); X target and z target are the output image and its latent representation. G is our trained generator neural network. γ is a coefficient that trades-off the classification and reconstruction objectives. L target is a target objective which can be a class probability or a regression target. The critical difference between our objective and the one used for adversarial example generation is that optimization is performed in the latent space, not the image space. Algorithm 1 Visual rationale generation Require: α, learning rate γ, image similarity penalty ρ, cutoff value Require: x, the initial input f: x → z, a function approximating the mapping between image and latent space DISPLAYFORM2 We also apply our method to external datasets and demostrate good cross-dataset generalization, in particular the National Institutes of Health (NIH) ChestX-ray8 dataset comprising of 108,948 frontal chest radiographs, recently released by BID13. We downsize the provided images to work with our autoencoder and split this by patient into a training, validation and testing set in the 7:1:2 ratio used by the dataset's authors. We encode these images into the latent space and apply a 6 layer fully connected neural network with 100 hidden units in each layer utilizing residual connections. We train this with a batch size of 2048. This architecture is fully described in figure 1. To evaluate the usefulness of the generated visual rationales, we conduct an experiment where we compare visual rationales generated by a classifier to one which is contaminated. We train the classifier directly on the testing examples and over train until almost perfect accuracy on this set is achieved. We reason that the contaminated classifier will simply memorize the testing examples and hence will not be able to produce useful rationales. We also apply our method to the well known MNIST dataset and apply a linear classifier with a 10 way softmax. In order to generate our visual rationales we select an initial class and a target classwe have chosen to transform the digit 9 to the digit 4 as these bear physical resemblance. We alter our optimization objective by adding a negatively weighted term for the predicted probability of the target class as described in Algorithm 2.Algorithm 2 Visual rationale generation for multiclass predictors Require: α, learning rate γ, image similarity penalty ρ, cutoff value β, target class weighting t, target class Require: x, the initial input f: x → z, a function approximating the mapping between image and latent space g: z → x h c (z) → P (c|z), classifier predicting class probability from DISPLAYFORM3 To illustrate the fidelity of our autoencoder we reconstruct each image in a smaller labelled set which has not been seen during training. The reconstructed images are show in Fig. 3 In FIG4 we demonstrate an example of the algorithm's reconstruction of a chest radiograph from a patient with heart failure, as well as the visualization of the same patient's chest radiograph without heart failure. We subtract the visualization of the radiograph without heart failure from the original reconstructed image and superimpose this as a heatmap on the original image to demonstrate the visual rationale for this prediction. For the same image, we apply the saliency map method, integrated gradients, the occlusion sensitivity method with a window size of 8, as well as LIME to obtain Fig. 6 for comparison. All of these methods yield noisy and potentially irrelevant features as compared to our method of generating visual rationales. Figure 6: Comparison of other methods -top left to bottom right: Saliency map, saliency map overlaid on original image, heatmap generated via occlusion sensitivity method, Integrated gradients, integrated gradients overlaid on original image, LIME output We apply our classifier as described above to the chest radiograph dataset released by the NIH recently and achieve similar to or exceeding that of the baseline reported in the original dataset. ROC curves are demonstrated in FIG5 Comparison AUC are reported in TAB0. We show that even without repeating the autoencoder or GAN training process on the new dataset, we are able to classify encoded representations of these chest radiographs with an accuracy comparable to or exceeding the performance of the published baseline network, which utilizes various state of the art network architectures as well as higher resolution images. We apply our method to the MNIST dataset and demonstrate class switching between digits from 9 to 4 and 3 to 2. FIG6. demonstrates the visual rationales for why each digit has been classified as a 9 rather than a 4, as well as the transformed versions of each digit. As expected, the top horizontal line in the digit 9 is removed to make each digit appear as a 4. Interestingly, the algorithm failed to convert several digits into a 4 and instead converts them into other digits which are presumably more similar to that instance, despite the addition of the weighted term encouraging the latent representation to prefer the target class. This type of failure is observed more in digits that are less similar to each other, such as from converting from the digits 3 to 2, as simply removing the lower curve of the digit may not always in a centered "two" digit. This precludes the simple interpretation that we are able to attribute to the 9 to 4 task. This behaviour is not noted in our chest radiograph dataset as we are able to convert every image from the predicted class to the converse, which is presumably due to the smaller differences between chest X-rays with and without heart failure. Similarly, the time taken to generate a visual rationale depends on the confidence of the classifier in its prediction, as the algorithm runs until the input has been altered sufficiently or a maximum number of steps (in our case 500) have been completed. In the case of converting digit 9s to 4s -we were able to generate 1000 visual rationales in 1 minute and 58 seconds. Lastly, we contaminate our heart failure classifier as described in the methods section and compare visual rationales generated by the contaminated classifier with those generated previously. FIG0 demonstrates images where both classifiers predict the presence of heart failure. The rationales from the contaminated classifier focus on small unique aspects of the image and largely do not correspond to our notion of what makes a chest radiograph more likely to represent heart failure, namely enlarged hearts and congested lung fields. To demonstrate this we present 100 images classified as having a BNP level of over 100ng/L to two expert reviewers, equally split between a contaminated or a normally trained classifier. Each image and the associated visual rationale was presented to the reviewers who were blinded to the origin of the classifier. Reviewers were tasked in selecting features from a provided list which they felt corresponded with the visual rationales. Each reviewer rated each image twice. Aggregated from experts are presented in TAB2. This clearly shows that the contaminated classifier indeed produces less interpretable visual rationales. (b) Contaminated classifier. In Reviewer A's first round, 22 out of 50 visual rationales generated were identified as having features of cardiomegaly, and 3 had features of effusions. Each visual rationale can contain multiple features. We show in this work that using the generator of a GAN as the decoder of an autoencoder is viable and produces high quality autoencoders. The constraints of adversarial training force the generator to produce realistic radiographs for a given latent space, in this case a 100-dimensional space normally distributed around 0 with a standard deviation of 1.This method bears resemblance to previous work done on inverting GANS done by BID2, although we are not as concerned with recovering the exact latent representation but rather the ability to recreate images from our dataset. It is suggested in previous work in BID8 that directly training a encoder to reverse the mapping learnt by the generator in a decoupled fashion does not yield good as the encoder never sees any real images during training. By training upon the loss between the real input and generated output images we overcome this. We further establish the utility of this encoder by using encoded latent representations to predict outcomes on unseen datasets, including one not from our institution. We achieve this without retraining our encoder on these unseen datasets, suggesting that the encoder has learnt useful features about chest radiographs in general. Our primary contribution in this paper however is not the inversion of the generator but rather the ability to generate useful visual rationales. For each prediction of the model we generate a corresponding visual rationale with a target class different to the original prediction. We display some examples of the rationales this method produces and inspect these manually to check if these are similar to our understanding of how to interpret these images. The ability to autoencode inputs is essential to our rationale generation although we have not explored in-depth in this paper the effect of different autoencoding algorithms (for instance variational autoencoders) upon the quality of the generated rationales, as our initial experiments with variational and vanilla autoencoders were not able to reconstruct the level of detail required. For chest radiographs, common signs of heart failure are an enlarged heart or congested lung fields, which appear as increased opacities in the parts of the image corresponding to the lungs. The rationales generated by the normally trained classifier in FIG0 to be consistent with features described in the medical literature while the contaminated classifier is unable to generate these rationales. We also demonstrate the generation of rationales with the MNIST dataset where the digit 9 is transformed into 4 while retaining the appearance of the original digit. We can see that the transformation generally removes the upper horizontal line of the 9 to convert this into a 4. Interestingly, some digits are not successfully converted. Even with different permutations of delta and gamma weights in Algorithm 2 some digits remain resistant to conversion. We hypothesize that this may be due to the relative difficulty of the chest radiograph dataset compared to MNIST -leading to the extreme confidence of the MNIST model that some digits are not the target class. This may cause vanishingly small gradients in the target class prediction, preventing gradient descent from achieving the target class. We compare the visual rationale generated by our method to various other methods including integrated gradients, saliency maps, occlusion sensitivity as well as LIME in Fig. 6.All of these methods share similarities in that they attempt to perturb the original image to examine the impact of changes in the image on the final prediction, thereby identifying the most salient elements. In the saliency map approach, each individual pixel is perturbed, while in the occlusion sensitivity method, squares of the image are perturbed. LIME changes individual superpixels in an image by changing all the pixels in a given superpixel to the average value. This approach fails on images where the superpixel classification is too coarse, or where the classification is not dependent on high resolution details within the superpixel. To paraphrase BID12, attribution or explanation for humans relies upon counterfactual intuition -or altering the image to remove the cause of the predicted outcome. Model agnostic methods such as gradient based methods, while fulfilling the sensitivity and implementation invariance axioms, do not acknowledge the natural structure of the inputs. For instance, this often leads to noisy pixel-wise attribution as seen in Fig. 6. This does not fit well with our human intuition as for many images, large continuous objects dominate our perception and we often do not expect attributions to differ drastically between neighbouring pixels. Fundamentally these other approaches suffer from their inability to perturb the image in a realistic fashion, whereas our approach perturbs the image's latent representation, enabling each perturbed image to look realistic as enforced by the GAN's constraints. Under the manifold hypothesis, natural images lie on a low dimensional manifold embedded in pixel space. Our learned latent space serves as a approximate but useful coordinate system for the manifold of natural images. More specifically the image (pardon the pun) of the generator G [R d] is approximately the set of'natural images' (in this case radiographs) and small displacements in latent space around a point z closely map into the tangent space of natural images around G(z). Performing optimization in latent space is implicitly constraining the solutions to lie on the manifold of natural images, which is why our output images remain realistic while being modified under almost the same objective used for adversarial image generation. Hence, our method differs from these previously described methods as it generates high resolution rationales by switching the predicted class of an input image while observing the constraints of the input structure. This can be targeted at particular classes, enabling us answer the question posed to our trained model -'Why does this image represent Class A rather than Class B?'There are obvious limitations in this paper in that we do not have a rigorous definition of what interpretability entails, as pointed out by BID12. An intuitive understanding of the meaning of interpretability can be obtained from its colloquial usage -as when a teacher attempts to teach by example, an interpretation or explanation for each image helps the student to learn faster and generalize broadly without needing specific examples. Future work could focus on the measurement of interpretability by judging how much data a second model requires when learning from the predictions and interpretations provided by another pretrained model. Maximizing the interpretability of a model may be related to the ability of models to transfer information between each other, facilitating learning without resorting to the use of large scale datasets. Such an approach could help evaluate non-image based visual explanations such as sentences, as described in BID5.Other technical limitations include the difficulty of training a GAN capable of generating realistic images larger than 128 by 128 pixels. This limits the performance of subsequent classifiers in identifying small features. This can be seen in the poor performance of our model in detecting nodules, a relatively small feature, compared to the baseline implementation in the NIH dataset. In , we describe a method of semi-supervised learning and apply this to chest radiographs, using local data as well as recent datasets. We show that this method can be leveraged to generate visual rationales and demonstrate these qualitatively on chest radiographs as well as the well known MNIST set.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a method of using GANs to generate high quality visual rationales to help explain model predictions.

represent the short summary for finding the corresponding scientific article (long)

Given We propose an unsupervised way to learn multiple embeddings for sentences and phrases , a positive would be Most deep learning for NLP represents each word with a single point or single-mode region in semantic space, while the existing multi-mode word embeddings cannot represent longer word sequences like phrases or sentences. We introduce a phrase representation (also applicable to sentences) where each phrase has a distinct set of multi-mode codebook embeddings to capture different semantic facets of the phrase's meaning. The codebook embeddings can be viewed as the cluster centers which summarize the distribution of possibly co-occurring words in a pre-trained word embedding space. We propose an end-to-end trainable neural model that directly predicts the set of cluster centers from the input text sequence (e.g., a phrase or a sentence) during test time. We find that the per-phrase/sentence codebook embeddings not only provide a more interpretable semantic representation but also outperform strong baselines (by a large margin in some tasks) on benchmark datasets for unsupervised phrase similarity, sentence similarity, hypernym detection, and extractive summarization. Many widely-applicable NLP models learn a representation from only co-occurrence statistics in the raw text without any supervision. Examples include word embedding like word2vec or GloVe , sentence embeddings like skip-thoughts , and contextualized word embedding like ELMo and BERT . Most of these models use a single embedding to represent one sentence or one phrase and can only provide symmetric similarity measurement when no annotation is available. However, a word or phrase might have multiple senses, and a sentence can involve multiple topics, which are hard to analyze based on a single embedding without supervision. To address the issue, word sense induction methods and recent multi-mode word embeddings (; ;) represent each target word as multiple points or regions in a distributional semantic space by (explicitly or implicitly) clustering all the words appearing beside the target word. In Figure 1, the multi-mode representation of real property is illustrated as an example. Real property can be observed in legal documents where it usually means a real estate, while a real property can also mean a true characteristic in philosophic discussions. The previous approaches discover those senses by clustering observed neighboring words (e.g., company and tax). In contrast with topic modeling like LDA , the approaches need to solve a distinct clustering problem for every target word while topic modeling finds a single set of clusters by clustering all the words in the corpus. Extending these multi-mode representations to arbitrary sequences like phrases or sentences is difficult due to two efficiency challenges. First, there are usually many more unique phrases and sentences in a corpus than there are words, while the number of parameters for clustering-based approaches is O(|V | × |K| × |E|), where |V | is number of unique sequences, |K| is number of modes/clusters, and |E| is the number of embedding dimensions. Estimating and storing such a large number of parameters take time and space. More important, many unique sequences imply much fewer co-occurring words to be clustered for each sequence, especially for long sequences Figure 1: The target phrase real property is represented by four clustering centers. The previous work discovers the four modes by finding clustering centers which well compress the embedding of observed co-occurring words. Instead, our compositional model learns to predict the embeddings of cluster centers from the sequence of words in the target phrase so as to reconstruct the (unseen) co-occurring distribution well. like sentences, so an effective model needs to overcome this sample efficient challenge (i.e., sparseness in the co-occurring statistics). However, clustering approaches often have too many parameters to learn the compositional meaning of each sequence without overfitting. Nevertheless, the sentences (or phrases) sharing multiple words tend to have similar cluster centers, so we should be able to compress many redundant parameters in these local clustering problems to circumvent the challenges. In this work, we adopt a neural encoder and decoder to achieve the goal. As shown in Figure 1, instead of clustering co-occurring words beside a target sequence at test time as in previous approaches, we learn a mapping between the target sequence (i.e., phrases or sentences) and the corresponding cluster centers during training so that we can directly predict those cluster centers using a single forward pass of the neural network for an arbitrary unseen input sequences during testing. To allow the neural network to generate the cluster centers in an arbitrary order, we use a nonnegative and sparse coefficient matrix to dynamically match the sequence of predicted cluster centers and the observed set of co-occurring word embeddings during training. After the coefficient matrix is estimated for each input sequence, the gradients are back-propagated to cluster centers (i.e., codebook embeddings) and weights of decoder and encoder, which allows us to train the whole model jointly and end-to-end. In experiments, we show that the proposed model captures the compositional meanings of words in unsupervised phrase similarity tasks much better than averaging their (contextualized) word embeddings, strong baselines that are widely used in practice. In addition to similarity, our model can also measure asymmetric relations like hypernymy without any supervision. Furthermore, the multimode representation is shown to outperform the single-mode alternatives in sentence representation, especially as demonstrated in our extractive summarization experiment. In this section, we first formalize our training setup in Section 2.1. Next, our objective function and the architecture of our prediction mode will be described in Section 2.2 and Section 2.3, respectively. The approach is summarized in Figure 2 using an example sentence. Figure 2: Our model for sentence representation. We represent each sentence as multiple codebook embeddings (i.e., clustering centers) predicted by our sequence to embeddings model. Our loss encourages the model to generate codebook embeddings whose linear combination can well reconstruct the embeddings of co-occurring words (e.g., Music), while not able to reconstruct the negatively sampled words (i.e., the co-occurring words from other sentences) to avoid predicting common topics which co-occur with every sentence (e.g., is in this example). We express tth sequence of words in the corpus as I t = w xt...w yt <eos>, where x t and y t are the start and end position of the target sequence, respectively, and <eos> is the end of sequence symbol. We assume neighboring words beside each target phrase or sentence are related to some aspects of the sequence, so given I t as input, our training signal is to reconstruct a set of neighboring words, is a fixed window size. For sentence representation, N t is the set of all words in the previous and the next sentence. Notice that the training signal for sentences and phrases are different, which means we need to train one model for phrase and one model for sentence if both representations are desired. Since there are not many co-occurring words for a long sequence (none are observed for unseen testing sequences), our goal is to cluster the set of words which could "possibly" occur beside the sequence instead of the actual occurring words in the training corpus (e.g., the hidden co-occurring distribution instead of green and underlined words in Figure 2). Although most of the possibly co-occurring words are not observed in the corpus, we can still learn to predict them by observing co-occurring words from similar sequences. To focus on the semantics rather than syntax, we view the co-occurring words as a set rather than a sequence as in skip-thoughts . Notice that our model considers the word order information in the input sequence I t, but ignores the order of the co-occurring words N t. In this work, we model the distribution of co-occurring words in a pre-trained word embedding space. The embeddings of co-occurring words N t are arranged into a matrix W (N t) = [w t j] j=1...|Nt| with size |E| × |N t |, where |E| is the dimension of pre-trained word embedding, and each of its column w t j is a normalized word embedding whose 2-norm is 1. The normalization makes the cosine distance between two words become half of their Euclidean distance. Similarly, we denote the predicted cluster centers c t k of the input sequence I t as a |E| × K matrix where F is our neural network model and K is the number of clusters. We fix the number of clusters K in this work to simplify the design of our prediction model and how it is applied to downstream tasks. The effect of different cluster numbers will be discussed in the experimental section. The reconstruction loss of k-means clustering in the word embedding space can be written as, where M k,j = 1 if the jth word belongs to the k cluster and 0 otherwise. That is, M is a permutation matrix which matches the cluster centers and co-occurring words and allow the neural network to generate the centers in an arbitrary order. Non-negative sparse coding (NNSC) relaxes the constraints by allowing the coefficient M k,j to be a positive value but encouraging it to be 0. In this work, we adopt the relaxation because for all neural architectures (including transformers and LSTMs) we tried, the models using a NNSC loss learn to generates diverse K cluster centers while the predicted cluster centers using the kmeans loss collapse to much fewer modes which cannot well capture the conditional co-occurrence distribution. We hypothesize that it is because a NNSC loss is smoother and easier to be optimized for a neural network, while finding the nearest cluster center in the kmeans loss cannot stably encourage predicted clusters to play different roles for reconstructing the embeddings of observed co-occurring words. Using NNSC, we define our reconstruction error as where λ is a hyper-parameter controlling the sparsity of M. We force the coefficient value M k,j ≤ 1 to avoid the neural network learning to predict centers with small magnitudes which makes the optimal values of M k,j large and unstable. Having multiple outputs and estimating the permutation between the prediction and ground truth words is often computationally expensive . However, the proposed loss is efficient because we minimize the L2 distance in a pre-trained embedding space as in rather than using softmax, and M Ot can be efficiently estimated on the fly using convex optimization (we use RMSprop in our implementation). After M Ot is estimated, we can treat it as a constant and back-propagate the gradients to our neural network to achieve end-to-end training. To prevent the neural network from predicting the same global topics regardless of the input, our loss function for tth sequence is defined as where N rt is a set of co-occurring words of a randomly sampled sequence I rt. In our experiment, we use SGD to solve F = arg min F t L t (F). Our method could be viewed as a generalization of Word2Vec that can encode the compositional meaning of the words and decode multiple embeddings. Our neural network architecture is similar to transformation-based sequence to sequence (seq2seq) model . We use the same encoder T E(I t), which transforms the input sequence into a contextualized embeddings where the goal of the encoder is to map the sentences which are likely to have similar co-occuring word distribution closer together. Different from the typical seq2seq model , our decoder does not need to make discrete decisions because our outputs are a sequence of embeddings instead of words. This allows us to predict all the codebook embeddings in a single forward pass while still well capturing the dependency between output without the need of auto-regressive decoding. Similar to BERT, we treat the embedding of <eos> as the sentence representation. To make different codebook embeddings capture different aspects, we pass the embeddings of <eos> to different linear layers L k before becoming the input of the decoder T D. Specifically, the codebook embeddings We find that removing the attention on the e xt...e yt, contextualized word embeddings from the encoder, significantly increases our validation loss for sentence representation because there are often too many facets to be compressed into a single embedding. On the other hand, the attention does not change the performance of phrase representation too much, and we remove the attention connection between encoder and decoder (i.e., encoder and decoder have the same architecture) when evaluating our phrase representation. Notice that the framework is flexible. We can replace the encoder and decoder with other architectures. Besides transformers, we also try (bi-)LSTMs in our experiments. This flexibility also allows us to incorporate other input features (e.g., the author who writes the sentence). We first visualize the cluster centers predicted by our model in Table 1 (like we visualize the meaning of the red cluster center in Figure 2 using the word song or Music). The centers summarize the target sequence well and more codebook embeddings capture more semantic facets of a phrase or a sentence. Due to the difficulty of evaluating the topics conditioned on the input sequence using the classic metrics for global topic modeling, we show that the codebook embeddings can be used to improve the performances of various unsupervised semantic tasks, which indirectly measures the quality of the generated topics. We use the cased version (840B) of pre-trained GloVe embedding for sentence representation and use the uncased version (42B) for phrase representation. 2 Our model is trained on Wikipedia 2016 while the stop words are removed from the set of co-occurring words. In the phrase experiments, we only consider noun phrases, and their boundaries are extracted by applying simple regular expression rules to POS tags before training. The sentence boundaries and POS tags are detected using spaCy. 3 Our models do not need resources such as PPDB or other multi-lingual resources, so our models are compared with the baselines that only use the raw text and sentence/phrase boundaries. This setting is particularly practical for the domains with low resources such as scientific literature. To control the effect of embedding size, we set the number of dimensions in our transformers as the GloVe embedding size. Limited by computational resources, we train all the models using one modern GPU within a week. Because of the relatively small model size, we find that our models underfit the data after a week (i.e., the training loss is very close to the validation loss). It is hard to make a fair comparison with BERT . BERT is trained on a masked language modeling loss, which preserves more syntax information and can produce an effective pretrained embedding for many supervised downstream tasks. We report the performance of the BERT base model, which is still trained using more parameters, more output dimensions, larger corpus, and more computational resources during training compared with our models. Furthermore, BERT uses a word piece model to alleviate the out-of-vocabulary problem. Nevertheless, we still provide its unsupervised performances based on cosine similarity as a reference. Semeval 2013 task 5(a) English and Turney 2012 are two standard benchmarks for evaluating phrase similarity . BiRD and WikiSRS , which are recently collected, contain ground truth phrase similarities derived from human annotations. The task of Semeval 2013 is to distinguish similar phrase pairs from dissimilar phrase pairs. In Turney, given each query bigram, the goal is to identify the unigram that is most similar to the query bigram among 5 unigram candidates, 5 and Turney adds 5 more negative phrase pairs by pairing the reverse of bigram with the 5 unigrams. BiRD and WikiSRS-Rel measure the relatedness of phrases and WikiSRS-Sim measures the similarity of phrases. For our model, we evaluate two scoring functions that measure phrase similarity. The first way averages the contextualized word embeddings from our transformer encoder as the phrase embedding and computes the cosine similarity between two phrase embeddings. We label the method as Ours Emb. The similar phrases should have similar multi-facet embeddings, so we compute the reconstruction error from the set of normalized codebook embeddings of one phrase S 1 q to the embeddings of the other phrase S 2 q, vice versa, and add them together to become a symmetric distance SC: where When ranking for retrieving similar phrases, we use the negative distance to represent similarity. We compare our performance with 5 baselines. GloVe Avg and Word2Vec Avg compute the cosine similarity between two averaged word embeddings, which has been shown to be a strong baseline . BERT CLS and BERT Avg are the cosine similarities between CLS embeddings and between the averages of contextualized word embeddings from BERT , respectively. FCT LM Emb learns the weights of linearly combining word embeddings based on several linguistic features. The performances are presented in Table 2. SemEval 2013 and Turney have training and testing split and the performances in test sets are reported. Our models significantly outperform all baselines in the 4 datasets. Furthermore, our strong performances in Turney verify that our encoder incorporates the word order information when producing phrase embeddings. The indicate the effectiveness of non-linearly composing word embeddings (unlike GloVe, Word2Vec, and FCT baselines) in order to predict the set of co-occurring word embeddings (unlike the BERT baselines). The performance of Ours (K=1) is usually slightly better than Ours (K=10). This supports the finding of that multi-mode embeddings may not improve the performance in word similarity benchmarks even if they capture more senses or aspects of polysemies. Even though being slightly worse, the performances of Ours (K=10) remain strong compared with baselines. This indicates that the similarity performance is not sensitive to the number of clusters, which alleviates the problem of selecting K in practice. STS benchmark is a widely used sentence similarity task. Each model predicts a semantic similarity score between each sentence pair, and the scores are compared with average similarity annotations using the Pearson correlation coefficient. Intuitively, when two sentences are less similar to each other, humans tend to judge the similarity based on how they are similar in each aspect. Thus, we also compare the performances on lower half the datasets where their ground truth similarities are less than the median similarity score, and we call this benchmark STSB Low. In addition to BERT CLS, BERT Avg, and GloVe Avg, we compare our method with word mover's distance (WMD) and cosine similarity between skip-thought embeddings (ST Cos) . propose to weight the word w in each sentence according to α α+p(w), where α is a constant and p(w) is the probability of seeing the word w in the corpus. Following its recommendation, we set α to be 10 −4 in STS benchmark. After the weighting, we adopt the post-processing method from to remove the first principal component that is estimated using the training distribution and denote the method as GloVe SIF. The post-processing is not desired in some applications , so we also report the performance before removing principal components, which is called GloVe Prob_avg. The strong performance of (weighted) average embedding suggests that we should consider the embeddings of words in the sentence in addition to the sentence embedding(s) when measuring the sentence similarity. This is hard to be achieved in other sentence representation methods because their sentence embeddings and word embeddings are in different semantic spaces. Since our multi-facet embeddings are in the same space of word embeddings, we can use the multifacet embeddings to estimate the word importance (in terms of predicting possible co-occurring words beside the sentence). To compute the importance of a word in the sentence, we first compute the cosine similarity between the word and all predicted codebook embeddings, truncate the negative similarity to 0, and sum all similarity. Specifically, our simple importance/attention weighting for all the words in the query sentence S q is defined by where 1 is an all-one vector. The importance weighting is multiplied with the original weighting vectors on GloVe Avg (uniform weight), GloVe Prob_avg, and GloVe SIF to generate the of Our Avg, Our Prob_avg, and Our SIF, respectively. We compare all the in the development set and test set in Table 3. Ours SC, which matches between two sets of topics outperforms WMD, which matches between two sets of words in the sentence, and also outperforms BERT Avg, especially in STSB Low. All the scores in Ours (K=10) are significantly better than Ours (K=1),which demonstrates the benefits of multi-mode representation. Multiplying the proposed attention weighting boosts the performance from (weighted) averagingbased methods especially in STSB Low and when we do not rely on the generalization assumption of our training distribution. We also test a variant of our method which uses a bi-LSTM as the encoder and a LSTM as the decoder. Its average validation loss (-0.1289) in equation 2 is significantly higher than that of the transformer alternative (-0.1439), and it performances on Table 3 are also worse. Notice the architecture of this variant is similar to skip-thoughts except that skip-thoughts decodes a sequence instead of a set. The variant significantly outperforms ST Cos, which further justifies our approach of ignoring the order of co-occurring words in our NNSC loss. We apply our model to HypeNet , an unsupervised hypernymy detection dataset, based on an assumption that the co-occurring words of a phrase are often less related to some of its hyponyms. For instance, fly is a co-occurring word of animal which is less related to brown dog. Thus, the predicted codebook embeddings of a hyponym S hypo q (e.g., brown dog), which cluster the embeddings of co-occurring words (e.g., eats), often reconstruct the embeddings of its hypernym S hyper q (e.g., animal) better than the other way around (i.e., Er( Based on the assumption, our asymmetric scoring function is defined as The AUC of detecting hypernym among other relations and accuracy of detecting the hypernym direction are compared in Table 4 . Our methods outperform baselines, which only provide symmetric similarity measurement, and Ours (K=1) performs similarly compared with Ours (K=10). A good summary should cover multiple aspects that well represent all topics/concepts in the whole document. The objective can be quantified as discovering a summary A with a set of normalized embeddings C(A) which best reconstructs the distribution of normalized word embedding w in the document D . That is, where γ w is the importance of the word w, which is set as α α+p(w) as we did in Section 3.2. We further assume that the summary A consists of T sentences A 1... A T and the embedding set of the summary is the union of the embedding sets of the sentences C(A) = ∪ T t=1 C(A t), and we greedily select sentences to optimize equation 8 as did in. This extractive summarization method provides us a way to evaluate the embedding(s) of sentence. Our model can generate multiple codebook embeddings, which capture its different aspects as we see in Table 1, to represent each sentence in the document, so we let C(A t) = {F u (A t)}, a set of column vectors in the matrix F u (A t). We compare our approach with other alternative ways of modeling the aspects of sentences. For example, we can compute average word embeddings as a single-aspect sentence embedding. This baseline is labeled as Sent Emb. We can also use the embedding of all the words in the sentences as different aspects of the sentences. Since longer sentences have more words, we normalize the gain of the reconstruction loss by the sentence length. The method is denoted as W Emb. In contrast, the fixed number of codebook embeddings in our method avoids the problem. We also test the baselines of selecting random sentences (Rnd) and first n sentences (Lead) in the document. The on the testing set of CNN/Daily Mail are compared using F1 of ROUGE in Table 5. R-1, R-2, and Len mean ROUGE-1, ROUGE-2, and average summary length, respectively. All methods choose 3 sentences by following the setting in. Unsup, No Order means the methods do not use the sentence position information in the documents. In CNN/Daily Mail or other English news corpora, the sentence order information is a very strong signal. For example, the unsupervised methods such as Lead-3 are very strong baselines with performances similar to supervised methods such as RL , a state-of-the-art approach in this evaluation. To evaluate the quality of unsupervised sentence embeddings, we focus on comparing the unsupervised methods which do not assume the first few sentences form a good summary. In Table 5, predicting more aspects (i.e., using higher cluster numbers K) yields better , and setting K = 100 gives us the best performance after selecting 3 sentences. This demonstrates that larger cluster numbers K is desired in this application. Our method allows us to set K to be a relatively large number because of greatly alleviating the computational and sample efficiency challenges. Topic modeling has been extensively studied and widely applied due to its interpretability and flexibility of incorporating different forms of input features .; demonstrate that neural networks could be applied to discover semantically coherent topics. However, instead of optimizing a global topic model, our goal is to jointly and efficiently discovering different sets of topics/clusters on the small subsets of words that co-occur with target phrases or sentences. Sparse coding on word embedding space is used to model the multiple aspects of a word , and parameterizing word embeddings using neural networks is used to test hypothesis and save storage space . Besides, to capture asymmetric relations such as entailment, words are represented as single or multiple regions in Gaussian embeddings rather than a single point. However, the challenges of extending these methods to longer sequences are not addressed in these studies. One of our main challenges is to design a neural decoder for a set rather a sequence while modeling the dependency between the elements. This requires a matching step between two sets and compute the distance loss after the matching . One popular loss is called Chamfer distance, which is widely adopted in the auto-encoder models for point clouds , while more sophisticated matching loss options are also proposed . The goal of the studies focus on measuring symmetric distances between the ground truth set and predicted set (usually with the equal size), while our set decoder tries to reconstruct a set using a set of much fewer bases. Other ways to achieving the permutation invariants loss for neural networks includes removing the elements in the ground truth set which have been predicted , beam search , or predicting the permutation using a CNN , a transformer or reinforcement learning . In contrast, our goal is to efficiently predict a set of clustering centers that can well reconstruct the set of observed instances instead of predicting the set or sequence of the observed instances. In this work, we overcome the computational and sampling efficiency challenges of learning the multi-mode representation for long sequences like phrases or sentences. We use a neural encoder to model the compositional meaning of the target sequence and use a neural decoder to predict a set of codebook embeddings as the representation of the sentences or phrases. During training, we use a non-negative sparse coefficient matrix to dynamically match the predicted codebook embeddings to a set of observed co-occurring words and allow the neural decoder to predict the clustering centers with an arbitrary permutation. We demonstrate that the proposed models can learn to predict interpretable clustering centers conditioned on an (unseen) sequence, and the representation outperforms widely-used baselines such as BERT, skip-thoughts and various approaches based on GloVe in several unsupervised benchmarks. The experimental also suggest that multi-facet embeddings perform the best when the input sequence (e.g., a sentence) involves many aspects, while multi-facet and single-facet embeddings perform similarly good when the input sequence (e.g., a phrase) usually involves only one aspect. In the future, we would like to train a single model which could generate multi-facet embeddings for both phrases and sentences, and evaluate the method as a pre-trained embedding approach for supervised or semi-supervised settings. Furthermore, we plan to apply this method to other unsupervised learning tasks that heavily rely on co-occurrence statistics such as graph embedding or recommendation. Given the computational resource constraints, we keep our model simple enough to have a nearly converged training loss after 1 or 2 epoch(s). Since training takes a long time, we do not fine-tune the hyper-parameters in our models. We use a much smaller model compared with BERT but the architecture details in our transformer and most of its hyper-parameters are the same as the ones used in BERT. The sparsity penalty weights on coefficient matrix λ is set to be 0.4. The maximal sentence size is set to be 50 and we ignore the sentences longer than that. The maximal number of co-occurring words is set to be 30 (after removing the stop words), and we sub-sample the words if there are more words in the previous and next sentence. The number of dimensions in transformers is set to be 300. For sentence representation, the number of transformer layers on the decoder side is 5 and dropout on attention is 0.1 for K = 10 and the number of transformer layer on the decoder side is set to be 1 for K = 1 because we do not need to model the dependency of output basis. For phrase representation, the number of transformer layers on the decoder side is 2 and the dropout on attention is 0.5. The window size d t is set to be 5. We will release the code to reveal more hyper-parameter setting details. All the architecture and hyperparameters (except the number of codebook embeddings) in our models are determined by the validation loss of the self-supervised co-occurring word reconstruction task. The number of codebook embeddings K is chosen by the performance of training data in each task, but we observe that the performances are usually not sensitive to the numbers as long as K is large enough. We also suspect that the slight performance drops of models with too large K might just be caused by the fact that larger K needs longer training time and 1 week of training is insufficient to make the model converges. For skip-thoughts, the hidden embedding of is set to be 600. To make the comparison fair, we retrain the skip-thoughts in Wikipedia 2016 for 2 weeks. As mentioned in Section 3, our model has fewer parameters than the BERT base model and uses much less computational resources for training, so we only present the BERT base performance in the experiment sections. Nevertheless, we still wonder how well BERT large can perform in these unsupervised semantic tasks, so we compare our method with BERT Large in Table 6, Table 7, and Table 8. As we can see, BERT large is usually better than BERT base in the similarity tasks, but perform worse in the hypernym detection task. The performance gains of BERT in similarity tasks might imply that training a larger version of our model might be a promising future direction. Although increasing the model size boosts the performance of BERT, our method is still much better in most of the cases, especially in the phrase similarity tasks. One of the main reasons we hypothesize is that BERT is trained by predicting the masked words in the input sequence and the objective function might not be good if sequences are short (like phrases). C SUMMARIZATION COMPARISON GIVEN THE SAME SUMMARY LENGTH In Section 3.4, we compare our methods with other baselines when all the methods choose the same number of sentences. We suspect that the bad performances for W Emb (*) methods (i.e., representing each sentence using the embedding of words in the sentence) might come from the tendency of selecting shorter sentences. 6 To verify the hypothesis, we plot the R-1 performance of different unsupervised summarization methods that do not use sentence order information versus the sentence length in Figure 3. In the figure, we first observe that Ours (K=100) significantly outperforms W Emb (GloVe) and Sent Emb (GloVe) when summaries have similar length. In addition, we find that W Emb (*) actually usually outperform Sent Emb (*) when comparing the summaries with a similar length. Notice that this comparison might not be fair because W Emb (*) are allowed to select more sentences given the same length of summary and it might be easier to cover more topics in the document using more sentences. In practice, preventing choosing many short sentences might be preferable in an extractive summarization if fluency is an important factor. Nevertheless, if our goal is simply to maximize the ROUGE F1 score given a fixed length of summary without accessing the ground truth summary and sentence order information, the figure indicates that Ours (K=100) is the best choice when the summary length is less than around 50 words and W Emb (BERT) becomes the best method for a longer summary. The BERT in this figure is the BERT base model. The mixed suggest that combining our method with BERT in a way might be a promising direction to get the best performance in this task (e.g., use contextualized word embedding from BERT as our pre-trained word embedding space). We visualize predicted embeddings from 10 randomly selected sentences in our validation set. The format of the file is similar to Table 1. The first line of an example is always the prepossessed input sentence, where <unk> means an out-of-vocabulary placeholder. The embedding in each row is visualized by the nearest five neighbors in a GloVe embedding space and their cosine similarities to the vector.

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.