Question: Represent the text to find a very short summary (1-2 sentences max):

Question: Vision-Language Navigation (VLN) is the task where an agent is commanded to navigate in photo-realistic unknown environments with natural language instructions. Previous research on VLN is primarily conducted on the Room-to-Room (R2R) dataset with only English instructions. The ultimate goal of VLN, however, is to serve people speaking arbitrary languages. Towards multilingual VLN with numerous languages, we collect a cross-lingual R2R dataset, which extends the original benchmark with corresponding Chinese instructions. But it is time-consuming and expensive to collect large-scale human instructions for every existing language. Based on the newly introduced dataset, we propose a general cross-lingual VLN framework to enable instruction-following navigation for different languages. We first explore the possibility of building a cross-lingual agent when no training data of the target language is available. The cross-lingual agent is equipped with a meta-learner to aggregate cross-lingual representations and a visually grounded cross-lingual alignment module to align textual representations of different languages. Under the zero-shot learning scenario, our model shows competitive even compared to a model trained with all target language instructions. In addition, we introduce an adversarial domain adaption loss to improve the transferring ability of our model when given a certain amount of target language data. Our methods and dataset demonstrate the potentials of building a cross-lingual agent to serve speakers with different languages. Recently, the Vision-Language Navigation (VLN) task , which requires the agent to follow natural language instructions and navigate in houses, has drawn significant attention. In contrast to some existing navigation tasks , where the agent has an explicit representation of the target to know if the goal has been reached or not, an agent in the VLN task can only infer the target from natural language instructions. Therefore, in addition to normal visual challenges in navigation tasks, language understanding and cross-modal alignment are essential to complete the VLN task. However, existing benchmarks for the VLN task are all monolingual in that they only contain English instructions. Specifically, the navigation agents are trained and tested with only English corpus and thus unable to serve non-English speakers. To fill this gap, one can collect the corresponding instructions in the language that the agent is expected to execute. But it is not scalable and practical as there are thousands of languages on this planet and collecting large-scale data for each language would be very expensive and time-consuming. Therefore, in this paper, we study the task of cross-lingual VLN to endow an agent the ability to understand multiple languages. First, can we learn a model that has been trained on existing English instructions but is still able to perform reasonably well on a different language (e.g. Chinese)? This is indeed a zero-shot learning scenario where no training data of target language is available. An intuitive approach is to train the agent with English data, and at test time, use a machine translation system to translate the target language instructions to English, which are then fed into the agent for testing (see the above part of Figure 1). The inverse solution is also rational: we can translate all English instructions into the target language and train the agent on the translated data, so it can be directly tested with target language instructions (see the below part of Figure 1). The former agent is tested on translated instructions while the latter is trained on translated instructions. Both solutions suffer from translation errors and deviation from the corresponding human-annotated instructions. But meanwhile, the former is trained on human-annotated English instructions (which we view as "golden" data) and the latter is tested on "golden" target language instructions. Motivated by this fact, we design a cross-lingual VLN framework that learns to benefit from both solutions. As shown in Figure 1, we combine these two principles and introduce a meta-learner, which learns to produce beliefs for human-annotated instruction and its translation pair and dynamically fuse the cross-lingual representations for better navigation. In this case, however, the training and inference are mismatched. During training, the agent takes source human language and target machine translation (MT) data as input, while during inference, it needs to navigate with target human instructions and source MT data. To better align the source and target languages, we propose a visually grounded cross-lingual alignment module to align the paired instructions via the same visual feature because they are describing the same demonstration path. The cross-lingual alignment loss can also implicitly alleviate the translation errors by aligning the human language and its MT pair in the latent visual space. After obtaining an efficient zero-shot agent, we investigate the question that, given a certain amount of data for the target language, can we learn a better adaptation model to improve source-to-target knowledge transfer? The meta-learner and visually grounded cross-lingual alignment module provide a foundation for solving the circumstances that the agent has access to the source language and (partial) target language instructions for training. To further leverage the fact that the agent has access to the target language training data, we introduce an adversarial domain adaption loss to alleviate the domain shifting issue between human-annotated and MT data, thus enhancing the model's transferring ability. To validate our methods, we collect a cross-lingual VLN dataset (XL-R2R) by extending complimentary Chinese instructions for the English instructions in the R2R dataset. Overall, our contributions are four-fold: We collect the first cross-lingual VLN dataset to facilitate navigation models towards accomplishing instructions of various languages such as English and Chinese, and conduct analysis between English and Chinese corpus. We introduce the task of cross-lingual visionlanguage navigation and propose a principled meta-learning method that dynamically utilizes the augmented MT data for zero-shot cross-lingual VLN. We propose a visually grounded crosslingual alignment module for better cross-lingual knowledge transfer. We investigate how to transfer knowledge between human-annotated and MT data and introduce an adversarial domain adaption loss to improve the navigation performance given a certain amount of human-annotated target language data. The cross-lingual vision-language navigation task is defined as follows: we consider an embodied agent that learns to follow natural language instructions and navigate from a starting pose to a goal location in photo-realistic 3D indoor environments. Formally, given an environment E, an initial pose p 1 = (v 1, φ 1, θ 1) (spatial position, heading, elevation angles) and natural language instructions x 1:N, the agent takes a sequence of actions a 1:T to finally reach the goal G. Thus the VLN dataset D is defined as {(E, p 1, x 1:N, G)} |D |. Note that we eliminate the footscript here for simplicity. At each time step t, the agent at pose p t receives a new observation I t = E(p t), which is a raw RGB image pictured by the mounted camera. Then it takes an action a t and leads to a new pose p t+1 = (v t+1, φ t+1, θ t+1). Taking actions sequentially, the agent stops when a stop action is taken. A cross-lingual VLN agent learns to understand multiple languages and navigate to the goal. Without loss of generality, we consider a bilingual situation coined as cross-lingual VLN. For this specific task, we built the XL-VLN dataset D, which extends the VLN dataset and includes a bilingual version of instructions. Specifically, where S and T indicate source and target language domains separately. The source language domain S contains instructions in the source language covering the full VLN dataset D (including training and testing splits), while the target language domain T consists of a fully annotated testing set and a training set in the target language that covers a varying percentage of trajectories of the training set in D (may range from 0% to 100%). The agent is allowed to leverage both source and target language training sets and expected to perform navigation given an instruction from either the source or target language testing sets. In this study, we first focus on a more challenging setting where no human-annotated target language data are available for training (= 0%), i.e., with no training data for the target language but the only access to the source language training set, the agent is required to follow a target language instruction x T 1:N to navigate in houses. Then we investigate the agent's transferring ability by gradually increasing the percentage of human-annotated target language instructions for training (= 0%, 10%, ..., 100%). We build a Cross-Lingual Room-to-Room (XL-R2R) dataset 1, the first cross-lingual dataset for the vision-language navigation task. The XL-R2R dataset includes 4,675/340/783 trajectories for train/validation seen/validation unseen sets, preserving the same split as in the R2R dataset. The official testing set of R2R is unavailable because the testing trajectories are held for challenge use. Each trajectory is described with 3 English and 3 Chinese instructions independently annotated by different workers. Data Collection. We keep the English instructions of the R2R dataset and collect Chinese instructions via a public Chinese crowdsourcing platform. The Chinese instructions are annotated by native speakers through an interactive 3D WebGL environment, following the R2R dataset guidance . More details can be found in the Appendix. Data Analysis. XL-R2R dataset includes 5,798 trajectories in total and 17,394 instructions for both languages. The bilingual instruction part here is compared with each other from four perspectives for a broad understanding, including vocabulary, instruction length, sub-instruction number per instruction and part-of-speech tags. Removing words with less than 5 frequency, we obtain an English vocabulary with 1,583 words and a Chinese one with 1,134 words. The Chinese instructions are relatively shorter than English ones and less likely to be long sentences (Figure 2a). The instructions usually consist of several sub-instructions separated by punctuation tokens, and the number of sub-instructions per instruction distributes similar across language (Figure 2b). Figure 2c and Figure 2d show that nouns and verbs, which often refer to landmarks and actions respectively, are more frequent in Chinese dataset (32.9% and 29.0%) than in English one (24.3% and 13.7%) 2. We present a general cross-lingual VLN framework in Figure 3. It is based on an encoder-decoder architecture and composed of three novel modules: a cross-lingual meta-learner (Meta), a visually grounded cross-lingual alignment module (txt2img), and an adversarial domain adaptation module. Particularly, as shown in Figure 3, both English and Chinese instructions are encoded by a shared encoder. Then the shared decoder takes the encoded contextual embeddings c In addition, the txt2img module is introduced to align h en t and h zh t in the visual space with the visual feature I t as an anchor point, improving the cross-lingual knowledge transfer via a visually grounded cross-lingual alignment loss. The adversarial domain adaptation module is particularly designed for the transfer setting where human-annotated target language instructions are also provided. We employ a domain discriminator that is trying to distinguish human-annotated instructions from machine translation (MT) instructions, and a gradient reversal layer to reverse the gradients back-propagated to the encoder so that the encoder is indeed trying to generate indistinguishable representations for both human-annotated and MT data and align the distributions of the two domains. We employ the sequence-to-sequence architecture in for both languages. Receiving a pair of natural language instruction x L 1:N, L ∈ {S, T}, the agent encodes it with an embedding matrix followed by an LSTM encoder to obtain contextual word representations c L 1:N. The decoder LSTM takes the concatenation of current image feature I t and previous action embedding a t−1 as input, and updates the hidden state s t−1 to s t aware of the historical trajectory: An attention mechanism is used to compute a weighted context representation, grounded on the instruction c To bridge the gap between source and target languages, we leverage a machine translation (MT) system to translate the source language in the training data into the target language. During testing, the MT system will translate the target language instruction into the source language. The MT data serves as augmented data for zero-shot or low-resource settings as well as associates two different human languages in general. We take two instructions (the human language instruction and its MT pair) as input for both training and testing. We observed that, even if one instruction is a direct translation from the other, when the paired instructions are fed into the same encoder and decoder, the two instructions will often generate different predictions when executing. At each time step, when the agent observed the local visual environment, with two instructions at hand but lead to different next positions. It remains a challenging question which language representation the agent shall trust more. Therefore, we propose a cross-lingual meta-learner that tries to help the agent make the judgment. At each time step, we let the cross-lingual meta-learner decide which language representation we should have more faith in, i.e., "learning to trust". The meta-learner is a SoftMax layer which takes the concatenation of two hidden states h S t and h T t as input, and produces a probability α t representing the belief of the source language representation. The final hidden vector used for predicting actions is defined as a mixture of the representations in two languages: Finally, the predicted action distribution for the next time step is computed as: To better ground and align two languages to the images they describe, we map h S t and h T t into the latent space of image representations such that their similarity is maximized. In other words, we use the image space as an anchor point to align cross-lingual representations. Let I t be the latent representation of the local visual environment on the target trajectory at time step t (e.g. the final layer of a ResNet), the loss function is formulated as: where ψ denotes a non-linearity activation such as ReLU or tanh, W I, W S and W T are the projection matrices. L2 distance is used to measure the similarity between contextual word and image features in the same vector space. The intuition behind such an aligning mechanism is that, since the human instruction and the MT instruction are both describing the same trajectory, their representation should be close to the visual environment in some way (a projected latent space in our case). This module would ensure the consistency among cross-lingual representations and the visual inputs. During training, we have human-annotated data for the source language and the machine-translated data for the target language, while an opposite situation for testing. To bridge the gap between training and inference, we leverage an adversarial domain adaption loss to make the context representations indistinguishable across domains for the transfer setting, where a certain amount of human-annotated instructions for the target language is available. A sentence vector c is computed as the mean of the context vector c 1:N to a single vector, then forwarded to a domain discriminator through the gradient reversal layer. With the gradient reversal layer, the gradients minimizing domain classification errors are passed back with opposed sign to the language encoder, which adversarially encourages the encoder to be domain-agnostic. We then minimize the following domain adaptation loss: where y H is the domain label of an instruction indicating whether it is from human annotation or machine translation, andŷ H is the approximation by the domain discriminator. Figure 1. meta+txt2img equips the meta-learner with txt2img module. The first three models are all for zero-shot learning. The last one, train w/ AN is to train the agent with 100% Chinese human annotated data. All models are tested with Chinese human instructions. , measures the fidelity to the described path, unlike previous metrics which are mostly based on goal completion. Model PL NE ↓ SR ↑ SPL ↑ CLS ↑ PL NE ↓ SR ↑ SPL ↑ CLS ↑ train We first report for the zero-shot setting, to show the effectiveness of our two components, meta-learner, and txt2img. We compare with two models, a seq2seq model trained with humanannotated Chinese instructions (collected in XL-R2R dataset), and that trained with MT Chinese instructions translated from English. Results are shown in Table 1. First, there is a clear gap between training with human-annotated and MT data, indicating the insufficiency of using only an MT system for zero-shot learning. Second, our meta-learner can successfully aggregate the information of the annotated data and MT data, which enables efficient zero-shot learning. Third, the txt2img module further improves vision-language alignment, which especially helps the agent generalize better on the unseen data. Besides, even though the agent does not have access to the target annotation data, it achieves competitive compared to training with 100% annotated data. To investigate the potential of transferring knowledge from English to Chinese, we draw learning curves by utilizing varying percentages of Chinese annotations for training (see Figure 4). The starting point is our zero-shot setting, where one has no access to the human-annotated data for the target language, and the endpoint is when one has 100% training data of the target language. The figures demonstrate that the proposed adversarial domain adaption module provides consistent improvement over other methods, for both seen and unseen environments. The approach works for both low-resource and high-resource settings and is capable of transferring knowledge steadily as the size of target data grows. Besides, our transfer method trained with 40% Chinese human data can achieve similar performance as trained with 100% Chinese human data. This demonstrates the potential of building a functioning cross-lingual VLN agent by collecting a large-scale dataset for a certain language (i.e., English), and a small amount of data for other languages. One can also observe that pretraining with English data and MT Chinese data help the model learn useful encoding that is especially valuable when only limited Chinese training data are available. To enable cross-lingual VLN, we examine four models (see Figure 5) equipped with our metalearner and txt2img modules: Base-Bi, which has two separate encoder-decoder. Shared Enc, which has a shared language encoder. Shared Dec, which has a shared policy decoder. Share Enc-Dec, which shares both the encoder and the decoder, with different word embeddings for different languages. These models take English and Chinese natural language instructions as input, for both training and testing. They are also compared with Base-mono, which is a single encoder-decoder model trained and tested with Chinese human instructions only. Table 2: Performance comparison for cross-lingual VLN models. All models are trained and tested with English and Chinese annotation data. Results are averaged over 3 runs. Table 2 shows the of four architectures on the validation seen and unseen part. First, the performance of multi-lingual models are consistently improved over the monolingual model (Basemono), indicating the potential of cross-lingual learning for improving navigation . Second, sharing parameters can further boost navigation performance. Finally, Shared Enc and Shared EncDec produce similar , which motivates us to use a Shared Enc-Dec design since it yields a competitive good with fewer parameters required. For a more intuitive understanding of the meta-learner, we visualize the confidences assigned to each language in Figure 6. In this case, the meta-learner trusts more on the human-annotated Chinese instruction which is of better quality. More specifically, at time step 10, when the meta-learner has the highest faith in the Chinese instruction, we visualize the textual attention on the whole instruction at this time step. Evidently, the corresponding textual attention on the Chinese command makes more sense than the machine-translated English command. The agent is supposed to keep turning left and then move forward to the green plant. The attentions on Chinese instruction assigns 0.25 to "turn left", and nearly zero attention to "head towards the door" which is already completed by previous actions. While the attention on English is more uniform and less accurate than on Chinese. Textual attention at time step 10: Turn left Go to Figure 6: Case Study. We choose a succeeded instruction from the validation set for illustration. Vision and Language Grounding. Over the past years, deep learning approaches have boosted the performance of computer vision and natural language processing tasks (; ; ;). A large body of benchmarks are proposed to facilitate the research, including Image and Video caption , VQA , and visual dialog . These tasks require grounding on both visual and textual modalities, but mostly limited to a fixed visual input. Thus, we focus on the task of vision-language navigation (VLN) , where an agent needs to actively interact with the visual environment following language instructions. Vision-Language Navigation. Several approaches have been proposed for the VLN task on the R2R dataset. For example, presented a planned-ahead module combining modelfree and model-based reinforcement learning methods, introduced a speaker which can synthesize new instructions and implement pragmatic reasoning. Subsequent methods extend the speaker-follower model with Reinforced Cross-modal Matching (a), self-monitoring , back-translation (etc. Previous works mainly improve navigation performance by data augmentation or leveraging efficient searching methods. In this paper, we address the task from a cross-lingual perspective, aiming at building an agent to execute instructions for different languages. Cross-lingual Language Understanding. Learning cross-lingual representations is a crucial step to make natural language tasks scalable to all the world's languages. Recently, cross-lingual studies on typical NLP tasks has achieved success, such as Part-of-Speech tagging), sentiment classification and Named Entity Recognition These studies successfully disentangle the linguistic knowledge into languagecommon and language-specific parts and learn both knowledges with individual modules. Moreover, cross-lingual image and video captioning (; b) aim to bridge vision and language towards a deeper understanding, by learning a cross-lingual model grounded on visual inputs. Our dataset and method address the cross-lingual representation learning for the vision-language navigation task. To our knowledge, we are the first to study the cross-lingual learning in a dynamic visual environment, where the agent needs to interact with its surroundings and take a sequence of actions. In this paper, we introduce a new task, namely cross-lingual vision-language navigation, to study cross-lingual representation learning situated in the navigation task where cross-modal interaction with the real world is involved. We collect a cross-lingual R2R dataset and conduct pivot studies towards solving this challenging but practical task. The proposed cross-lingual VLN framework is proven effective in cross-lingual knowledge transfer. There are still lots of promising future directions for this task and dataset, e.g. to incorporate recent advances in VLN and greatly improve the model capacity. It would also be valuable to extend the cross-lingual setting to support numerous different languages in addition to English and Chinese. We follow the same preprocessing procedure as in previous work. A ResNet-152 pretrained on ImageNet is used to extract image features, which are 2, 048-d vectors. Instructions are clipped with a maximum length of 80. Words are embedded into a 256-d vector space, and the action embedding is 32. The hidden size for the encoder and decoder LSTM is 512. The dropout ratio is 0.5. The meta-learner is a single fully connected layer. The dimension of vision-language alignment vector space is set to 1, 024. Each episode consists of no more that 40 actions. The network is optimized via the ADAM optimizer with an initial learning rate of 0.001, a weight decay of 0.0005, and a batch size of 100. The learning rate of domain adaptation loss is scheduled with an adaption factor : where γ is set to 10 and p is learning steps. We use 0.2λ p to train the domain discriminator. We run each model 30, 000 iterations and report the iteration with the highest SPL, and evaluate the models every 500 iterations. For evaluating our proposed approach on the unseen test set, we participate in the Vision and Language Navigation challenge and submitted our to the test server. Here we treat Chinese as the source language and English as the target language. Hence for zeroshot learning, the agent has 100% Chinese annotated data but no English annotated data. The agent is commanded to follow English human instructions during testing. Results are shown in Table 3. For zero-shot learning, our method (meta+txt2img) improves over the model trained with MT data only. For transfer learning, our method can efficiently transfer knowledge between Chinese and English data. The are coherent with the reported on the validation set. (See Table 1 and Figure 4). Table 3: Performance comparison on the English test set. The first two rows are for zero-shot learning, the last two rows are trained with access to 100% target training data (i.e. English annotated instructions). Meta-learner To validate the effectiveness of the meta-learner, we compare it with a simple ensemble, which assigns equal confidence to two languages at all time steps, without any learnable parameters. The are summarized in Table 4. Our meta-leaner has higher performance on the validation unseen set, suggests that "learning to trust" is important for cross-lingual vision-language navigation. Adversarial Domain Adaptation Loss To demonstrate the domain adaptation loss indeed enhance knowledge transfer between two languages, we compare it with our vanilla zero-shot model, a metalearner equipped with a txt2img module. Table 5 shows that, as the size of target training data grows, although the vanilla model can also benefit from the augmented data, the performance stops growing as the data size reaches 40% or 60%. Meanwhile, domain adaptation loss provides a more consistent Table 4: Ablation study for the meta-learner. Reported are averages of 5 individual runs. ensemble is to assign equal weight to each language. meta-learner is our basic framework in Figure 1. Both are reported on Chinese human instructions. and steady improvement. At the endpoint (100%), the SPL is 22.14 vs 21.50, proves its efficiency and the potential of transferring knowledge between different languages. We compare the statistics of the Chinese annotated dataset with a machine-translated one. The annotated instructions are more likely to contain fewer words as well as fewer instructions. Besides, nouns and verbs, which usually represent landmarks and actions in VLN task, are more frequent in annotated instructions than machine-translated ones.

Represent the short summary for finding the corresponding scientific article (long):

We introduce a new task and dataset on cross-lingual vision-language navigation, and propose a general cross-lingual VLN framework for the task.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We systematically examine why knowledge distillation is crucial to the training of non-autoregressive translation (NAT) models, and propose methods to further improve the distilled data to best match the capacity of an NAT model.


Question: Represent the scientific article to find a short TLDR

Question: Deep generative models have advanced the state-of-the-art in semi-supervised classification, however their capacity for deriving useful discriminative features in a completely unsupervised fashion for classification in difficult real-world data sets, where adequate manifold separation is required has not been adequately explored. Most methods rely on defining a pipeline of deriving features via generative modeling and then applying clustering algorithms, separating the modeling and discriminative processes. We propose a deep hierarchical generative model which uses a mixture of discrete and continuous distributions to learn to effectively separate the different data manifolds and is trainable end-to-end. We show that by specifying the form of the discrete variable distribution we are imposing a specific structure on the model's latent representations. We test our model's discriminative performance on the task of CLL diagnosis against baselines from the field of computational FC, as well as the Variational Autoencoder literature. Variational Autoencoders (VAEs) have recently shown remarkable performance in unsupervised generative modeling of high-dimensional data generated by complex distributions BID19, as well as semi-supervised classification where only a small subset of the data set is labeled. While the interaction between the generative and the classification capabilities of semi-supervised models has been recently explored in literature, there has been little investigation of the discriminative capabilities of a purely unsupervised framework with most works focusing on the task of unsupervised clustering BID17 BID36 BID16. Furthermore, most of these works have been evaluated on mostly benchmark data sets which do not capture the difficulties that are often encountered on real-world data. For instance, there has been no investigation of the performance of these methods on data sets with significant class imbalance. The question that is, then, posed is whether deep generative models can be used effectively as unsupervised classifiers, which can, in essence, be cast into a question of what type of features and architectural choices are required to achieve good classification performance in an unsupervised manner. To examine the aforementioned questions we propose a deep hierarchical generative model and evaluate its performance on a difficult real world data set. In principle, we train our model in a completely unsupervised fashion, however in our experiments we rely on labeled data to measure our model's performance using suitable metrics for the problem domain, as well as derive a stopping criterion for training. Our model outperforms established state-of-the-art baselines used in the field of the problem domain. Our contributions are summarized in the following:• A framework which utilizes a hierarchy of continuous representations which conclude in a discrete variable explicitly representing categories, ing in complex, expressive, invariant and interpretable representations BID1, which are crucial in separating widely overlapping manifolds and achieve good classification in significantly imbalanced data sets.• Controllable representation structure through specification of the form of the aforementioned discrete variable which better suits the task at hand given a problem scenario. In this section we review techniques and frameworks that provide a useful backdrop for the derivation of our own model. Variational Autoencoders (VAEs) as presented in BID19 use at most two layers of stochastic latent variables, however generalizations to multiple layers of latent variables have since been introduced BID6.Generation in a deep generative model is achieved by a top-down stochastic pass through the model defined as: DISPLAYFORM0 where L is the number of stochastic hidden layers and z L denotes the number of latent variables in the layer. As in a standard VAE the dependence of each layer on the previous one is considered to be nonlinear and is modeled by multi-layered perceptrons (MLPs). Similarly, inference is carried out by a bottom-up stochastic pass through the model's layers: DISPLAYFORM1 Optimization of a deep generative model is akin to that of a standard VAE. Namely, the reparameterization trick BID19 ) is being applied to each layer of stochastic latent variables. Until recently, models trained with the variational objective have been employing mainly Gaussian latent stochastic variables, optimizing indirectly through discrete variables wherever they were used (e.g. integrate over the discrete variable). This was due to the inability of backpropagating through discrete variables because of their discontinuous nature. BID15 and BID26 independently developed a continuous relaxation of discrete random variables. The ing distribution was presented as the Gumbel-Softmax distribution and the Concrete distribution respectively but essentially has the same functional form. From this point on, for the sake of clarity we will adopt the latter name to refer to this distribution. A simple way to sample from a discrete distribution is to employ the Gumbel-Max trick BID11 BID28 BID25. The Gumbel distribution produces samples according to − log(− log(u)) where u ∼ Uniform. Given parameters α 1,..., α k and samples g i from Gumbel, samples z from the Categorical distribution can be drawn according to: DISPLAYFORM0 where z is represented as a one-hot vector. Samples from the Concrete distribution are produced by replacing the argmax operation with the softmax function: DISPLAYFORM1 Crucially, the reparameterization trick can now be applied in a similar manner to Gaussian samples. The probability density function of the Concrete distribution is the following: DISPLAYFORM2 As the temperature parameter τ approaches 0, samples from the Concrete distribution more accurately approximate one-hot samples from a Categorical distribution and in the limit, the two distributions become identical. We refer the reader to BID15 BID26 for more details. In this section we present the problem setting and our proposed model. Additionally, we motivate our model design choices and modifications to the variational objective. Chronic lymphocytic leukemia (CLL) is the most common form of leukemia in adults in Western countries BID12. CLL is usually diagnosed during routine blood tests and Flow Cytometry (FC) is one of the examination procedures used for confirming the diagnosis BID12. Flow cytometry (FC) is a powerful technique for single cell analysis, which is routinely used for the diagnosis of haematological malignancies BID5, however it is heavily dependent on the experience of the expert performing it, oftentimes ing in serious discrepancies across experts. During or after treatment of CLL, the term Minimal Residual Disease (MRD) is used to define the small amount of leukemic cells detected in patients (typically 10s to a few 100s of leukemic cells in samples of 500,000 cells or more). Particularly for CLL, the limit for MRD positive diagnosis is set at 1 leukemic cell per 10,000 white blood cells in blood or bone marrow BID2. The problem, from a manifold learning and generative modeling point of view is to adequately separate the two data manifolds of healthy and leukemic cells. This problem is difficult not just because of the sheer size of the healthy cell population, which leads to significant manifold overlap, but also because there are other manifolds present in the data, e.g. representing different types of cell populations in the blood sample, with many different factors of variation, some of which are known (cell granulation, cell size, etc.) and some of which may be unknown. Most clustering algorithms, which are traditionally used in computational FC, particularly those that "infer" the number of clusters automatically, are unable to separate the manifolds of interest given a particular problem, because they act directly on the input/data space without making any assumptions about the latent structure present in a data set. As a they are sensitive to noise and ultimately one has to resort to merging clusters. Furthermore, for the clustering to be interpretable, significant amounts of hyperparameter tuning are necessary during the clustering proper, the cluster merging phase or both (e.g. number of nearest neighbors to be taken into account, appropriate distance metric, linkage, etc) ing in an impractical, computationally expensive overall solution. An alternative to clustering algorithms would be to learn a low-dimensional feature mapping as in traditional deterministic autoencoders and then perform clustering in feature space, optimizing the two tasks either separately or jointly BID36. While this is a viable strategy, possibly alleviating some of the problems discussed above, depending on the problem domain, the deterministic mapping could potentially lack expressiveness due to the low-dimensionality requirement, as it would need to compress information to avoid the curse of dimensionality. Furthermore, these methods are sensitive to noise and so, they overfit the data, being unable to disentangle the different factors of variation BID1.A practical and principled solution in this problem domain should be able to adequately separate the multiple manifolds underlying the data and model the signal of interest, remaining invariant to factors of variation. Furthermore, the candidate model should be able to learn feature representations that enforce a structure that facilitates the above and as a be able to correctly classify each cell in an unsupervised fashion. Ideally it should be trained and optimized end-to-end without resorting to clustering algorithms that are separated from the feature representation learning process. Latent variable models like VAEs and deep generative models seem like an ideal fit to the above description. They address the problem of generation of new samples with the help of a recognition/inference model which learns the global structure of the training data. In theory, by employing this encoder, such a model could learn representations that would prove useful in unsupervised discriminative tasks. In practice, however, this process is a lot more complicated. First of all, recognition models with inadequate expressiveness are known to encode only local information, because they model well only the regions of the posterior distribution that lie near the training examples, being unable to generalize to the full posterior manifold BID37. When one uses powerful and expressive generative models as in Bowman et al. FORMULA0; Serban et al. FORMULA0; BID9, this is further exacerbated to the point where generation on the one hand and representation learning on the other, become competing tasks with the model preferring to encode mainly local information using the generative/decoding model p(x|z) and ignore the latent code. From a representation learning standpoint which is what we're interested in for this particular problem, the generative model is now perceived as the regularizer and we need to make sure that the recognition model is expressive enough to be able to model our true posterior as closely as possible, as well as provide an interpretable way of making a diagnosis. As such we propose a framework that addresses both these needs jointly. To address the issues discussed in 3.2 we introduce a deep generative model composed of L layers of continuous stochastic latent variables. Beyond alleviating the aforementioned issues, we also aim to model a set of latent factors affecting the cell measurements and eventually, the diagnosis. We introduce a hierarchy of continuous latent variables to express these latent factors. The diagnosis is itself represented as a single (relaxed) binary variable after this cascade of layers. For the sake of brevity we will refrain from explicitly stating the continuous relaxation on this discrete variable for the remainder of this text, except when necessary. To highlight the close relationship between the discrete diagnosis and the continuous latent factors and for practical reasons we denote the mixture of the discrete and continuous variables by h. The generative model assumes the following form: DISPLAYFORM0 with DISPLAYFORM1 where DISPLAYFORM2 Parameters µ θ (·), Σ θ (·) are non-linear functions of the next layer in the hierarchy computed by feedforward neural networks and θ are the generative parameters. The prior on the discrete variable p(y) is set to be the discrete uniform probability. The observation model is defined as: DISPLAYFORM3 and describes the observations conditioned on the latent variables. The inference model is described below: DISPLAYFORM4 where Figure 1: PCA plots of latent representations for different number of classes K of discrete variable y. Increasing K, imposes a different structure on the model, encouraging it to learn and separate different manifolds. E.g. setting K = 4 the model successfully separates the 4 different cell populations present in the data (lymphocytes, monocytes, granulocytes and destroyed red blood cells). DISPLAYFORM5 DISPLAYFORM6 where similar to the generative model, parameters DISPLAYFORM7 are functions that describe non-linear relationships between stochastic layers and are computed by feedforward neural networks with φ being the variational parameters. The temperature parameter τ governs the degree of relaxation of the discrete variable and can either be constant, annealed during training or learned. In the case of the first stochastic layer z 1, layer z 0 refers to the observed data x. The function α φ (z L) models the relationship between the last Gaussian layer and the discrete variable y. By conditioning on the discrete variable y, we are enforcing a tight coupling between the latent factors of variation (represented by z) and categorical features (represented by y). As a the activity of the discrete variable y and continuous variables z 1...z L is highly correlated. More than simply capturing the modes of continuous variable z L, the discrete variable y imposes a particular structure that is propagated through all the continuous layers z 1...z L of the feature hierarchy. This allows us a degree of control with respect to the structure of the latent variables, which we can exploit depending on the task at hand. Figure 1 illustrates such an example. By increasing K we facilitate multiple manifold learning. Both in the inference and the generative models, covariances assume a diagonal structure. Our model's overall architecture can be seen in FIG2.Employing a relaxed discrete variable allows us to avoid marginalization over all its possible values. At the same time, however, we are using a continuous surrogate loss in place of the original discrete one BID32 BID26. DISPLAYFORM8 To induce more distributed and thus expressive latent codes we adopt deterministic warm-up per Sønderby et al. FORMULA0, where we begin training our model with a deterministic autoencoder and then gradually introduce the KL term, to prevent the approximate posterior from collapsing onto the prior early in training and disconnecting latent units. Thus we introduce a λ term in expression 11, which we linearly anneal during training from 0 to 1: The gradient estimations of the lower bound will be biased with respect to the original discrete loss, but unbiased and low-variance with respect to the surrogate loss BID26. DISPLAYFORM9 To obtain low variance updates to our parameters through stochastic gradient optimization of the bound, we use the reparameterization trick for both discrete and continuous variables concurrently: DISPLAYFORM10 where h is expressed as a deterministic function of the form presented in BID19 and section 2.2:h DISPLAYFORM11 with DISPLAYFORM12 We note that q φ (h|x) is used to denote the Concrete density and the KL term is computed according to eq. 20 in BID26. Finally, we also make use of batch normalization introduced by BID14, since it has been shown to both improve convergence time and allow for the training of the upper layers in deep generative models. In our experiments we use two real world data sets of deidentified patient data, which correspond to flow cytometric measurements of two different blood samples 1 We investigate our model's ability to learn useful representations via its performance as an "unsupervised classifier". Detailed explanations of the data sets can be found in Appendix A. In our experiments we compare our model's predictive capacity with that of popular baselines from the field of computational FC, as well as generative models. Because in this particular problem one population (i.e. class) vastly outnumbers the other, one cannot rely on accuracy to correctly estimate the models' performance, as a model that would predict only the over-represented class would consistently get high accuracy scores, but in reality would have limited predictive power. Instead, we make use of confusion matrix-based metrics, which are popular in medical applications. More specifically, we use true positive rate (TPR -also called sensitivity or recall) and true negative rate (TNR -also called specificity).Because we found that training the model is difficult, as far as retaining optimal representations is concerned throughout the training procedure, we use early stopping, where we check the model's performance on the test set with a fixed frequency. To derive a useful criterion we also turn to a confusion matrix-based quantity, Matthews correlation coefficient (MCC), introduced by BID27. As a correlation coefficient, MCC takes values in [-1,1] with 1 suggesting that predictions and ground truth are in complete accordance, -1 that they are in complete discord and 0 that predictions are random. MCC is regarded as a good measure for estimating classifier performance on imbalanced data sets BID3. Formally, it is defined as follows: DISPLAYFORM0 We check MCC in the test set during training in fixed intervals and save the model parameters once it has crossed a set threshold. In general, we noticed in our experiments that a threshold of 0.5 yields good with very small variance in performance. In our experiments we investigate our model's predictive performance, i.e. its capacity to correctly classify cell measurements 2. Our model's predictions are based on hard assignment of the probabilities obtained by the predictive distribution q φ (y|z). I.e. given a minibatch of observations DISPLAYFORM0, probabilities of samples drawn from q φ (y|z) are thresholded at 0.5. Probabilities that lie above this threshold are considered positive cell measurements. As baselines we choose the 3 best-performing algorithms from who compare a wide set of clustering algorithm implementations used in computational FC in two different tasks -multiple population identification and rare population detection. We are interested in the latter task which is closer to our own. The first baseline, X-shift BID31 ) is based on k-Nearest Neighbors (kNN) density estimation. Local density maxima become centroids and the remaining points are connected to those centroids via ascending density paths. Rclusterpp BID22 is the second baseline and is an efficient implementation (with respect to memory requirements) of hierarchical clustering. The last baseline, flowMEANS BID0 ) is based on k-means clustering but can identify concave cell populations using multiple clusters. We argue that to successfully learn and separate the two manifolds of healthy and pathological cells, more expressive and distributed representations are necessary, with explicit steps taken during training to enforce these characteristics. Additionally, the proposed model must be a mixture of continuous and discrete distributions to appropriately represent different cell attributes and category respectively. To illustrate the merit in these points, we also include 2 methods from the VAE literature, a vanilla VAE BID19, followed by a linear support vector machine (denoted by VAE+SVM in tables 1 and 2) and a version of our own model but within the β-VAE framework BID13. The motivation for VAE+SVM is the fact that given adequate manifold separation, a linear classifier such as a support vector machine should, in principle, be able to correctly classify observations. The β-VAE framework introduces a hyperparameter β to the variational objective, similar to λ, which we are using for deterministic warm-up. For β > 1, the model is forced to learn more compressed and disentangled latent representations, which is what BID13 argue for. Learning disentangled representations is certainly closely related to successfully separating multiple data manifolds and we consider a representation to be efficient and disentangled in this scenario if it leads to good predictive performance, implying adequate manifold separation. In short we are treating "unsupervised classification" as a proxy for manifold separation. Finally to illustrate the merits of generative modeling in the task of separating widely overlapping data manifolds we also include a deterministic 2-layer MLP classifier which is trained in a supervised way, i.e. using cross entropy as a cost function. The of our experiments can be shown in tables 1 and 2, where we denote our model by HCDVAE (Hierarchical Continuous-Discrete VAE). The baselines developed for the domain of FC are in essence clustering algorithms and as such they suffer from issues clustering algorithms traditionally suffer from, the most important of which, are sensitivity to parameter configurations and dependence on initialization schemes. Concordantly, they exhibited high variance across multiple runs so we present the best across 30 runs for each algorithm. Most of the baselines were able to achieve good predictive performance for the healthy cell population across most runs, which is to be expected since it is overrepresented in the data set, but average and erratic performance, for pathological cells. With the exception of Rclusterpp which achieved good overall performance in the second data set, all algorithms in both data sets exhibited a tendency to either be sensitive (high TPR) sacrificing healthy cell predictive accuracy or be specific (high TNR) and sacrifice pathological cell predictive accuracy. The supervised MLP classifier is clearly unable to separate the two manifolds, "overfitting" the healthy cell population. This suggests that at least for binary problems in which classes are severely imbalanced, training supervised classifiers (i.e. training on log-likelihood objectives with explicit labeled information) in subpar performance compared to generative modeling. This is not surprising as appropriate generative models incorporate more information about the data in their representations than just its labels, ing in greater discriminative power. The performance of VAE+SVM suggests that the topmost latent layer of vanilla VAE is not able to separate the healthy/pathological manifolds. Presumably this is the case because the approximate posterior has collapsed early on the prior and rendered most units inactive, making the overall representation uninformative with respect to the global structure of the data. Deep VAE architectures are also known to be difficult to train, with top layers being slow to learn useful representations or even unable to learn at all. This is not the case, however with β-VAE and HCDVAE. The predictive performance of both approaches is similar, however opting for a more distributed representation seems to yield marginally better predictive capacity. Visually, the two approaches seem to in similar latent representations as can be seen in FIG4, where they collapse the majority of healthy cells into a compact cluster and separate the pathological cell manifold, remaining largely invariant to other factors present in the data. A possible explanation for the marginally better performance of HCDVAE is that its denser representation captures the activity of more explanatory factors in the data set, which β-VAE's representations miss due to excessive compression. We further hypoth- esize that the mixture of discrete and continuous variables, which HCDVAE and β-VAE share in our experiments provides a powerful posterior which benefits both approaches, ing in similar performance. The discrete variable y at the top of the feature hierarchy representing cell state (or alternatively category) is largely invariant to perturbations in the factors modeled by the lower latent layers. Overall, HCDVAE achieves near human-level performance in both data sets, especially in the second case. We note that that the ground-truth diagnosis was crosschecked by 2 different human experts. Models that use mixtures of discrete and continuous variables and are trained with the variational objective have previously been discussed in literature. proposed a model for semi-supervised classification (M2) based on a discrete class variable y and a continuous "style" variable z. Optimization is performed using the reparameterization trick for z and marginalizing over y. This model is similar in structure to our own and also raise the point of learning better discriminative features which are more clearly separable, making classification easier (M1 model). Another similarity is that the distribution q φ (y|x) can be used as a classifier, performing classification through inference. A crucial difference is that we are using a relaxed discrete variable, removing the need to marginalize over its possible values. Additionally, we are enforcing dependence between the continuous and discrete variables. BID24 presented a method that also employs discrete and continuous variables also in the context of semi-supervised learning where the continuous variables model the formation of natural clusters in the data and the discrete variables representing class information refine this clustering scheme. BID17 use both discrete and continuous latent variables to construct a structured VAE that uses conjugate priors to create more flexible approximating posteriors in the context of switching linear dynam-ical systems. Works that attempt to alleviate the non-informativeness of the prior have also been presented in literature. BID8 present a VAE variant for the task of unsupervised clustering, introducing a Gaussian Mixture prior to enforce multi-modality on the inference distribution while employing the minimum information constraint of to avoid cluster degeneracy. BID10 develop tree-based priors, which they integrate into VAEs and train them jointly for learning representations in videos with the overall model being able to learn a hierarchical representation of the data set. presented a model which combines a discrete component which consists of a bipartite Boltzmann machine with binary units on the one hand and a continuous component consisting of multiple continuous layers on the other. To perform optimization using the reparameterization trick, the binary variables are marginalized out. While the proposed "discrete VAE" learns the class of objects in images, it is significantly more complex in its architecture and still relies on marginalization of the discrete variables. A DATA SETS Both data sets represent measurements on blood samples taken from patients and comprise 500,000 cells. Each cell measurement is represented by a 12-dimensional vector, corresponding to 10 different markers for each cell, the time stamp of the measurement and a label indicating healthy/pathological state. The first dimension corresponds to the time stamp, while the next 2 correspond to the forward and side scatter of the light. The remaining dimensions correspond to the protein markers used to make a diagnosis. The data sets' last column represents the label for each measurement. The data set used in the first experiment contains 107 pathological cells, while the one used in the second experiment contains 103 pathological cells. All patient identifiers are removed and the data sets are anonymized. For our experiments we shuffle and then split the data sets to training and test sets with a 75/25 split. To speed up training, we ensure that there is at least one example of a pathological cell in every minibatch and reshuffle the training set every 20 iterations. We used early stopping to stabilize training, with the stopping criterion being a threshold score of 0.5 for MCC, as reported in the main text. For these experiments we used 3 layers of latent variables of 128, 64 and 32 units respectively. The discrete variable was chosen to be a Bernoulli variable and p θ (x|h) is a Gaussian. In the inference model, every Gaussian latent layer is parameterized by feedforward neural networks with 2 hidden layers of 256 units each. The non-linearity used is the rectifier function, while the mean and variance parameters are given by linear and softplus layers respectively. The mean parameter of the relaxed Bernoulli distribution is also given by a feedforward neural network with the same architecture as above. For the generative model, every Gaussian latent layer is also parameterized by feedforward neural networks with 2 hidden layers with 128 rectifier units each. The mean and variance parameters are computed as before. All networks are optimized with minibatch gradient descent using the Adam optimizer BID18 with an initial learning rate of 10 −4 which we decay exponentially for 3500 steps with a basis of 0.99. The minibatch size was set at 100. The relaxation parameter τ of the relaxed Bernoulli distribution was fixed at 0.66 per BID26. The λ term was linearly annealed from 0 to 1 with a step of 0.002. For the baselines, all parameter values were chosen according to BID35. The above architecture is shared between the model we denote by β-VAE and HCDVAE in tables 1 and 2. The VAE part of the model we denote by VAE+SVM has the same Gaussian layer architecture.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Unsupervised classification via deep generative modeling with controllable feature learning evaluated in a difficult real world task

Represent the short summary for finding the corresponding scientific article (long).

Filter level sparsity emerges implicitly in CNNs trained with adaptive gradient descent approaches due to various phenomena, and the extent of sparsity can be inadvertently affected by different seemingly unrelated hyperparameters.