Question:
represent the article for finding a single-sentence tldr
Fewshots:
'Deep neural networks (DNNs) are inspired from the human brain and the interconnection between the two has been widely studied in the literature. However, it is still an open question whether DNNs are able to make decisions like the brain. Previous work has demonstrated that DNNs, trained by matching the neural responses from inferior temporal (IT) cortex in monkey's brain, is able to achieve human-level performance on the image object recognition tasks. This indicates that neural dynamics can provide informative knowledge to help DNNs accomplish specific tasks. In this paper, we introduce the concept of a neuro-AI interface, which aims to use human's neural responses as supervised information for helping AI systems solve a task that is difficult when using traditional machine learning strategies. In order to deliver the idea of neuro-AI interfaces, we focus on deploying it to one of the fundamental problems in generative adversarial networks (GANs): designing a proper evaluation metric to evaluate the quality of images produced by GANs. Deep neural networks (DNNs) have successfully been applied to a number of different areas such as computer vision and natural language processing where they have demonstrated state-of-the-art , often matching and even sometimes surpassing a human's ability. Moreover, DNNs have been studied with respect to how similar processing is carried out in the human brain, where identifying these overlaps and interconnections has been a focus of study and investigation in the literature BID5 BID4 BID11 BID18 BID30 BID1 BID35 BID17 BID15. In this research area, convolutional neural networks (CNNs) are widely studied to be compared with the visual system in human's brain because of following reasons: CNNs and human's visual system are both hierarchical system; Steps of processing input between CNNs and human's visual system are similar to each other e.g., in a object recognition task, both CNNs and human recognize a object based on their its shape, edge, color etc.. Work BID35 outlines the use of CNNs approach for delving even more deeply into understanding the development and organization of sensory cortical processing. It has been demonstrated that CNNs are able to reflect the spatio-temporal neural dynamics in human's brain visual area BID5 BID30 BID18. Despite lots of work is carried out to reveal the similarity between CNNs and brain system, research on interacting between CNNs and neural dynamics is less discussed in the literature as understanding of neural dynamics in the neuroscience area is still limited. There is a growing interest in studying generative adversarial networks (GANs) in the deep learning community BID10. Specifically, GANs have been widely applied to various domains such as computer vision BID14, natural language processing BID7 and speech synthesis BID6. Compared with other deep generative models (e.g. variational autoencoders (VAEs)), GANs are favored for effectively handling sharp estimated density functions, efficiently generating desired samples and eliminating deterministic bias. Due to these properties GANs have successfully contributed to plausible image generation BID14, image to image translation BID38, image super-resolution BID19, image completion BID37 etc.. However, three main challenges still exist currently in the research of GANs: Mode collapse -the model cannot learn the distribution of the full dataset well, which leads to poor generalization ability; Difficult to trainit is non-trivial for discriminator and generator to achieve Nash equilibrium during the training; Hard to evaluate -the evaluation of GANs can be considered as an effort to measure the dissimilarity between real distribution p r and generated distribution p g. Unfortunately, the accurate estimation of p r is intractable. Thus, it is challenging to have a good estimation of the correspondence between p r and p g. Aspects and are more concerned with computational aspects where much research has been carried out to mitigate these issues BID20; BID0. Aspect is similarly fundamental, however, limited literature is available and most of the current metrics only focus on measuring the dissimilarity between training and generated images. A more meaning-ful GANs evaluation metric that is consistent with human perceptions is paramount in helping researchers to further refine and design better GANs. Although some evaluation metrics, e.g., Inception Score (IS), Kernel Maximum Mean Discrepancy (MMD) and Fréchet Inception Distance (FID), have already been proposed (; BID13 BID2, their limitations are obvious: These metrics do not agree with human perceptual judgments and human rankings of GAN models. A small artefact on images can have a large effect on the decision made by a machine learning system BID16, whilst the intrinsic image content does not change. In this aspect, we consider human perception to be more robust to adversarial images samples when compared to a machine learning system; These metrics require large sample sizes for evaluation ). Large-scale samples for evaluation sometimes are not realistic in real-world applications since it is time-consuming; and They are not able to rank individual GAN-generated images by their quality i.e., the metrics are generated on a collection of images rather than on a single image basis. The within GAN variances are crucial because it can provide the insight on the variability of that GAN.Work BID36 demonstrates that CNN matched with neural data recorded from inferior temporal cortex BID3 has high performance in object recognition tasks. Given the evidence above that a CNN is able to predict the neural response in the brain, we describe a neuro-AI interface system, where human being's neural response is used as supervised information to help the AI system (CNNs used in this work) solve more difficult problems in real-world. As a starting point for exploiting the idea of neuro-AI interface, we focus on utilizing it to solve one of the fundamental problems in GANs: designing a proper evaluation metric. Neural response Stimulus Figure 1. Schematic of neuro-AI interface. Stimuli (image stimuli used in this work) are simultaneously presented to an AI system and participants. Participants' neural responses are transferred to the AI system as supervised information for assisting the AI system make decision. In this paper, we first demonstrate the ability of a brainproduced score (we call it Neuroscore), generated from human being's electroencephalography (EEG) signals, in terms of the quality evaluation on GANs. Secondly, we demonstrate and validate a neural-AI interface (as seen in Fig. 1), which uses neural responses as supervised information to train a CNN. The trained CNN model is able to predict Neuroscore for images without corresponding neural responses. We test this framework via three models: Shallow convolutional neural network, Mobilenet V2 BID26 and Inception V3 BID29.In detail, Neuroscore is calculated via measurement of the P300, an event-related potential (ERP) BID23 present in EEG, via a rapid serial visual presentation (RSVP) paradigm BID32. P300 and RSVP paradigm are mature techniques in the brain-computer interface (BCI) community and have been applied in a wider variety of tasks such as image search BID9, information retrieval BID22, and etc. The unique benefit of Neuroscore is that it more directly reflects the human perceptual judgment of images, which is intuitively more reliable compared to the conventional metrics in the literature BID2. Current literature has demonstrated that CNNs are able to predict neural responses in inferior temporal cortex in image recognition task BID36 BID35 via invasive BCI techniques BID31. Evidence shows that neural responses in inferior temporal cortex directly link the information processing during the image recognition task. Therefore, a CNN trained by predicting neural responses in inferior temporal cortex also achieves the good performance during the image recognition BID36. Comparing the traditional end-toend machine learning system, use of DNNs for predicting neural responses in the brain favors following benefits: It enables the information processing of DNNs closer to human being's brain system; For some difficult tasks in real-world e.g, evaluation of image quality demonstrated in this paper, it is still challenging to design the machine learning algorithms, which teach DNNs to process the information like humans; and Neural signals directly reflect the human perception and interfacing between neural responses and DNNs can be more efficient than the traditional methods regarding the area of human and AI.The investigation of using CNNs to predict neural response from non-invasive BCI aspect is still blank in the literature. Comparing to invasively measured neural dynamics, EEG favors pros such as simple measurement, unpainful experience during recording, free to ethic argument and more easily generalized to real-world applications. However, EEG suffers challenges such as low signal quality (i.e TAB1 Neuro-AI Interface low SNR), low spatial resolution (interested neural activities span all scalp and difficult to be localized), which makes the prediction for EEG response still challenging. With advanced machine leaching technologies applied to non-invasive BCI area, source localization and reconstruction are feasible for EEG signals today. Previously work BID33 a) have demonstrated the efficacy of using spatial filtering approaches for reconstructing P300 source ERP signals. The low SNR issue can be remedied by averaging the EEG trials. Based on this evidence, we explore the use of DNNs to predict Neuroscore when neural information is available. We propose a neuro-AI interface in order to generalize the use of Neuroscore. This kind of framework interfaces between neural responses and AI systems (CNN used in this study), which uses neural responses as supervised information to train a CNN. The trained CNN is then used for predicting Neuroscore given images generated by one of the popular GAN models. Figure 2 demonstrates the schematic of neuro-AI interface used in this work 1. Flow 1 shows that the image processed by human being's brain and produces single trial P300 source signal for each input image. Flow 2 in Fig. 2 demonstrates a CNN with including EEG signals during training stage. The convolutional and pooling layers process the image similarly as retina done BID21. Fully connected layers (FC) 1-3 aim to emulate the brain's functionality that produces EEG signal. Yellow dense layer in the architecture aims to predict the single trial P300 source signal in 400-600 ms response from each image input. In order to help model make a more accurate prediction for the single trial P300 amplitude for the output, the single trial P300 source signal in 400-600 ms is fed to the yellow dense layer to learn parameters for the previous layers in the training step. The model was then trained to predict the single trial P300 source amplitude (red point shown in signal trail P300 source signal of Fig. 2). Mobilenet V2, Inception V3 and Shallow network were explored in this work, where in flow 2 we use these three network bones: such as Conv1-pooling layers. For Mobilenet V2 and Inception V3. We used pretrained parameters from up to the FC 1 shown in Fig. 2. We trained parameters from FC 1 to FC 4 for Mobilenet V2 and Inception V3. θ 1 is 1 We understand that human being's brain system is much more complex than what we demonstrated in this work and the flow in the brain is not one-directional BID27 BID2. Our framework can be further extended to be more biologically plausible. Add windowed single trial P300 source signal Figure 2. A neuro-AI interface and training details with adding EEG information. Our training strategy includes two stages: Learning from image to P300 source signal; and Learning from P300 source signal to P300 amplitude. loss1 is the L2 distance between the yellow layer and the single trial P300 source signal in the 400 -600 ms corresponding to the single input image. loss2 is the mean square error between model prediction and the single trial P300 amplitude. loss1 and loss2 will be introduced in section 3.2.used to denote the parameters from FC 1 to FC 3 and θ 2 indicates the parameters in FC 4. For the Shallow model, we trained all parameters from scratch. We added EEG to the model because we first want to find a function f (χ) → s that maps the images space χ to the corresponding single trial P300 source signal s. This prior knowledge can help us to predict the single trial P300 amplitude in the second learning stage. We compared the performance of the models with and without EEG for training. We defined two stage loss function (loss 1 for single trial P300 source signal in the 400 -600 ms time window and loss 2 for single trial P300 amplitude) as DISPLAYFORM0 where S true i ∈ R 1×T is the single trial P300 signal in the 400 -600 ms time window to the presented image, and y i refers to the single trial P300 amplitude to each image. The training of the models without using EEG is straightforward, models were trained directly to minimize loss 2 (θ 1, θ 2) by feeding images and the corresponding single trial P300 amplitude. Training with EEG information is explained in Algorithm 1 and visualized in the "Flow 2" of Fig. 2 with two stages. Stage 1 learns parameters θ 1 to predict P300 source signal while stage 2 learns parameters θ 2 to predict single trial P300 amplitude with θ 1 fixed. Update θ 2 by descending its stochastic gradient: and without EEG. All models with EEG perform better than models without EEG, with much smaller errors and variances. Statistic tests between model with EEG and without EEG are also carried out to verify the significance of including EEG information during the training phase. One-way ANOVA tests (P-value) for each model with EEG and without EEG are stated as: P Shallow = 0.003, P M obilenet = 0.012 and P Inception = 5.980e − 05. Results here demonstrate that including EEG during the training stage helps all three CNNs improve the performance on predicting the Neuroscore. The performance of models with EEG is ranked as follows: Inception-EEG, Mobilenet-EEG, and Shallow-EEG, which indicates that deeper neural networks may achieve better performance in this task. We used the randomized EEG signal here as a baseline to see the efficacy of adding EEG to produce better Neuroscore output. When randomizing the EEG, it shows that the error for each three model increases significantly. For Mobilenet and Inception, the error of the randomized EEG is even higher than those without EEG in the training stage, demonstrating that the EEG information in the training stage is crucial to each model. Figure 3 shows that the models with EEG information have a stronger correlation between predicted Neuroscore and real Neuroscore. The cluster (blue, orange, and green circles) for each category of the model trained with EEG (left column) is more separable than the cluster produced by model without EEG (right column). This conveys with EEG for training models: Neuroscore is more accurate; and Neuroscore is able to rank the performances of different GANs, which cannot be achieved by other metrics BID2. Figure 3. Scatter plot of predicted and real Neuroscore of 6 models (Shallow, Mobilenet, Inception with and without EEG for training) cross participants by 20 times repeated shuffling training and testing set. Each circle represents the cluster for a specific category. Small triangle markers inside each cluster correspond to each shuffling process. The dot at the center of each cluster is the mean. In this paper, we introduce a neuro-AI interface that interacts CNNs with neural signals. We demonstrate the use of neuro-AI interface by introducing a challenge in the area of GANs i.e., evaluate the quality of images produced by GANs. Three deep network architectures are explored and the demonstrate that including neural responses during the training phase of the neuro-AI interface improves its accuracy even when neural measurements are absent when evaluating on the test set. More details of the performance of Neuroscore can be referred in Appendix. FIG1 shows the averaged reconstructed P300 signal across all participants (using LDA beamformer) in the RSVP experiment. It should be noted here that the averaged reconstructed P300 signal is calculated as the difference between averaged target trials and averaged standard trials after applying the LDA beamformer method. The solid lines in FIG1 are the means of the averaged reconstructed P300 signals for each image category (across 12 participants) while the shaded areas represent the standard deviations (across participants). It can be seen that the averaged reconstructed P300 (across participants) clearly distinguishes between different image categories. In order to statistically measure this correlative relationship, we calculated the Pearson correlation coefficient and p-value (two-tailed) between Neuroscore and BE accuracy and found (r = −0.767, p = 2.089e − 10). We also did the Pearson statistical test and bootstrap on the correlation between Neuroscore and BE accuracy (human judgment performance) only for GANs i.e., DCGAN, BEGAN and PROGAN. Pearson statistic is (r=-0.827, p=4.766e-10) and the bootstrapped p ≤ 0.0001. Three traditional methods are also employed to evaluate the GANs used in this study. score indicates better GAN performance), we use 1/Neuroscore for comparison. It can be seen that all three methods are consistent with each other and they rank the GANs in the same order of PROGAN, DCGAN and BEGAN from high to low performance. By comparing the three traditional evaluation metrics to the human, it can be seen that they are not consistent with human judgment of GAN performance. It should be remembered that Inception Score is able to measure the quality of the generated images while the other two methods cannot do so. However, Inception Score still rates DCGAN as outperforming BE-GAN. Our proposed Neuroscore is consistent with human judgment. Another property of using Neuroscore is the ability to track the quality of an individual image. Traditional evaluation metrics are unable to score each individual image for two reasons: They need large-scale samples for evaluation; Most methods (e.g. MMD and FID) evaluate GANs based on the dissimilarity between real images and generated images so they are not able to score the generated image one by one. For our proposed method, the score of each single image can also be evaluated as a single trial P300 amplitude. We demonstrate that using the predicted single trial P300 amplitude to observe the single image quality in Fig. 6. This property provides Neuroscore with a novel capability that can observe the variations within a typical GAN. Although Neuroscore and IS are generated from deep neural networks. Neuroscore is more suitable than IS for evaluating GANs in that: It is more explainable than IS as it is a direct reflection of human perception; Much smaller sample size is required for evaluation; Higher Neuroscore exactly indicates better image quality while IS Figure 6. P300 for each single image predicted by the proposed neuro-AI interface in our paper. Higher predicted P300 indicates the better image quality.does not. We also included the RFACE images in our generalization test. FIG3 (c) demonstrates that the predicted Neuroscore is still correlated with the real Neuroscore when adding the RFACE images and the model ranks the types of images as: PROGAN>RFACE>BEGAN>DCGAN, which is consistent with the Neuroscore that has been measured directly from participants shown in FIG3.Compared to traditional evaluation metrics, Neuroscore is able to score the GAN based on very few image samples, relatively. Recording EEG in the training stage could be the limitation of generalizing Neuroscore to evaluate a new GAN. However, the use of dry electrode EEG recording system BID8 can accelerate and simplify the data acquisition significantly. Moreover, GANs enable the possibility of synthesizing the EEG BID12, which has wide applications in brain-machine interface research.' == 'Describe a neuro-AI interface technique to evaluate generative adversarial networks' != 'Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise in the knowledge distillation framework and studied their effect on generalization and robustness.'

Question: Voice Conversion (VC) is a task of converting perceived speaker identity from a source speaker to a particular target speaker. Earlier approaches in the literature primarily find a mapping between the given source-target speaker-pairs. Developing mapping techniques for many-to-many VC using non-parallel data, including zero-shot learning remains less explored areas in VC. Most of the many-to-many VC architectures require training data from all the target speakers for whom we want to convert the voices. In this paper, we propose a novel style transfer architecture, which can also be extended to generate voices even for target speakers whose data were not used in the training (i.e., case of zero-shot learning). In particular, propose Adaptive Generative Adversarial Network (AdaGAN), new architectural training procedure help in learning normalized speaker-independent latent representation, which will be used to generate speech with different speaking styles in the context of VC. We compare our with the state-of-the-art StarGAN-VC architecture. In particular, the AdaGAN achieves 31.73%, and 10.37% relative improvement compared to the StarGAN in MOS tests for speech quality and speaker similarity, respectively. The key strength of the proposed architectures is that it yields these with less computational complexity. AdaGAN is 88.6% less complex than StarGAN-VC in terms of FLoating Operation Per Second (FLOPS), and 85.46% less complex in terms of trainable parameters. Language is the core of civilization, and speech is the most powerful and natural form of communication. Human voice mimicry has always been considered as one of the most difficult tasks since it involves understanding of the sophisticated human speech production mechanism and challenging concepts of prosodic transfer . In the literature, this is achieved using Voice Conversion (VC) technique . Recently, VC has gained more attention due to its fascinating real-world applications in privacy and identity protection, military operations, generating new voices for animated and fictional movies, voice repair in medical-domain, voice assistants, etc. Voice Conversion (VC) technique converts source speaker's voice in such a way as if it were spoken by the target speaker. This is primarily achieved by modifying spectral and prosodic features while retaining the linguistic information in the given speech signal . In addition, Voice cloning is one of the closely related task to VC . However, in this research work we only focus to advance the Voice Conversion. With the emergence of deep learning techniques, VC has become more efficient. Deep learningbased techniques have made remarkable progress in parallel VC. However, it is difficult to get parallel data, and such data needs alignment (which is a arduous process) to get better . Building a VC system from non-parallel data is highly challenging, at the same time valuable for practical application scenarios. Recently, many deep learning-based style transfer algorithms have been applied for non-parallel VC task. Hence, this problem can be formulated as a style transfer problem, where one speaker's style is converted into another while preserving the linguistic content as it is. In particular, Conditional Variational AutoEncoders (CVAEs), Generative Adversarial Networks (GANs) (proposed by), and its variants have gained significant attention in non-parallel VC. However, it is known that the training task for GAN is hard, and the convergence property of GAN is fragile . There is no substantial evidence that the gen-erated speech is perceptually good. Moreover, CVAEs alone do not guarantee distribution matching and suffers from the issue of over smoothing of the converted features. Although, there are few GAN-based systems that produced state-of-the-art for non-parallel VC. Among these algorithms, even fewer can be applied for many-to-many VC tasks. At last, there is the only system available for zero-shot VC proposed by. Zero-shot conversion is a technique to convert source speaker's voice into an unseen target speaker's speaker via looking at a few utterances of that speaker. As known, solutions to a challenging problem comes with trade-offs. Despite the , architectures have become more complex, which is not desirable in real-world scenarios because the quality of algorithms or architectures is also measured by the training time and computational complexity of learning trainable parameters ). Motivated by this, we propose computationally less expensive Adaptive GAN (AdaGAN), a new style transfer framework, and a new architectural training procedure that we apply to the GAN-based framework. In AdaGAN, the generator encapsulates Adaptive Instance Normalization (AdaIN) for style transfer, and the discriminator is responsible for adversarial training. Recently, StarGAN-VC (proposed by) is a state-of-the-art method among all the GAN-based frameworks for non-parallel many-to-many VC. AdaGAN is also GAN-based framework. Therefore, we compare AdaGAN with StarGAN-VC for non-parallel many-to-many VC in terms of naturalness, speaker similarity, and computational complexity. We observe that AdaGAN yields state-of-the-art for this with almost 88.6% less computational complexity. Recently proposed AutoVC (by) is the only framework for zero-shot VC. Inspired by this, we propose AdaGAN for zero-shot VC as an independent study, which is the first GAN-based framework to perform zeroshot VC. We reported initial for zero-shot VC using AdaGAN.The main contributions of this work are as follows: • We introduce the concept of latent representation based many-to-many VC using GAN for the first time in literature. • We show that in the latent space content of the speech can be represented as the distribution and the properties of this distribution will represent the speaking style of the speaker. • Although AdaGAN has much lesser computation complexity, AdaGAN shows much better in terms of naturalness and speaker similarity compared to the baseline. Developing a non-parallel VC framework is challenging task because of the problems associated with the training conditions using non-parallel data in deep learning architectures. However, attempts have been made to develop many non-parallel VC frameworks in the past decade. For example, Maximum Likelihood (ML)-based approach proposed by , speaker adaptation technique by , GMM-based VC method using Maximum a posteriori (MAP) adaptation technique by , iterative alignment method by , Automatic Speech Recognition (ASR)-based method by , speaker verification-based method using i-vectors by , and many other frameworks (; ; ; ; ; Saito et al. (2018a);; Shah et al. (2018b; c);; ). Recently, a method using Conditional Variational Autoencoders (CVAEs) was proposed for non-parallel VC by (; Saito et al. (2018a) ). Recently, VAE based method for VC was proposed, which also uses AdaIN to transfer the speaking style . One powerful framework that can potentially overcome the weakness of VAEs involves GANs. While GAN-based methods were originally applied for image translation problems, these methods have also been employed with noteworthy success for various speech technology-related applications, we can see via architectures proposed by (; Saito et al. (2018b); Shah et al. (2018a) ), and many others. In GANs-based methods, Cycle-consistent Adversarial Network (CycleGAN)-VC is one of the state-of-the-art methods in the non-parallel VC task proposed by . Among these non-parallel algorithms, a few can produce good for non-parallel many-tomany VC. Recently, StarGAN-VC is a state-of-the-art method for the nonparallel many-to-many VC among all the GAN-based frameworks. Past attempts have been made to achieve conversion using style transfer algorithms (; ;). The most recent framework is the AutoVC (proposed by) using style transfer scheme, the first and the only framework in VC literature which achieved state-of-the-art in zero-shot VC. The traditional VC problem is being reformulated as a style transfer problem. Here, we assume Z is a set of n speakers denoted by Z = {Z 1, Z 2, ..., Z n}, where Z i is the i th speaker, and U is the set of m speech utterances denoted by U = {U 1, U 2, ..., U m}, where U i is the i th speech utterance. Now, probability density function (pdf) is generated for given Z i, and U i denoted by p X (.|Z i, U i) via the stochastic process of random sampling from the distributions Z i and U i. Here, X i ∼ p X (.|Z i, U i) can be referred as features of given U i with speaking style of Z i. The key idea is to transfer the speaking style of one speaker into another in order to achieve VC. For this, let us consider a set of random variables (Z 1, U 1) corresponding to a source speaker, and (Z 2, U 2) corresponding to a target speaker. Here, U 1 and U 2 are spoken by Z 1 and Z 2, respectively. Our goal is to achieve pX (.|Z 2, U 1). Now, we want to learn a mapping function to achieve our goal for VC. Our mapping function is able to generate the distribution denoted byX Z1→Z2 with speaking style of Z 2 while retaining the linguistic content of U 1. Formally, we want to generate the pdf (i.e., pX (.|Z 1, U 1, Z 2, U 2)) to be close or equal to the pX (.|Z 2, U 1). Accurately, our mapping function will achieve this property, as shown in eq. 1. pX Intuitively, we want to transfer the speaking style of Z 2 to the Z 1 while preserving the linguistic content of U 1. Therefore, converted voice is perceptually sound as if utterance U 1 were spoken by Z 2. With this, AdaGAN is also designed to achieve zero-shot VC. During zero-shot conversion, U 1 and U 2 can be seen or unseen utterances, and Z 1 and Z 2 can be seen or unseen speakers. Our key idea for style transfer in VC revolves around the AdaIN. First, AdaIN was introduced for arbitrary style transfer in image-to-image translation tasks by. In this paper, AdaIN helps us to capture the speaking style and linguistic content into a single feature representation. AdaIN takes features of a source speaker's speech (i.e., X) and sample features of the target speaker's speech (i.e., Y). Here, x is a feature from the set X related to the linguistic content of source speech, and Y is features related to the speaking style of the target speaker. AdaIN will map the mean and standard deviation of X (i.e., µ X and σ x) in such a way that it will match with mean, and standard deviation of Y (i.e., µ Y and σ Y). Mathematical equation of AdaIN is defined as : From eq., we can infer that AdaIN first normalizes x, and scales back based on mean and standard deviations of y. Intuitively, let's assume that we have one latent space which represents the linguistic content in the distribution and also contains speaking style in terms of the mean and standard deviation of the same distribution. To transfer the speaking style, we have adopted the distribution properties (i.e., its mean and standard deviation) of the target speaker. As a , the output produced by AdaIN has the high average activation for the features which are responsible for style (y) while preserving linguistic content. AdaIN does not have any learning parameters. Hence, it will not affect the computational complexity of the framework. In this Section, we discuss our proposed AdaGAN architecture in detail. we show that AdaIN helps the generator make speaking style transfer easy and efficient, and can achieve zero-shot VC. We present an intuitive and theoretical analysis for the proposed framework. The AdaGAN framework consists of an encoder En, a decoder De, and a discriminator Dis. Here, En encodes the input features of speech to the latent space, De generates the features of speech from the given latent space, and Dis ensures adversarial training. The style transfer scheme and training procedure are shown in Fig Features of source speaker's speech (i.e., x), and any sample features of target speaker's speech (i.e., y), is taken as input to En to get the required latent space representations S x and S y as given in eq. 3. Now, AdaIN is used to transfer distribution properties (i.e., its mean and standard deviation) of S y to S x, and generate the single feature representation denoted by t as per eq. 3. In the next step, we have used De to generate features of speech (i.e., x Z1→Z2) from t. This entire process is illustrated via Fig. 1 (a). This generated features x Z1→Z2 contains the speaking style of target speaker via retaining the linguistic content of source speaker speech. We have encapsulated this style transfer algorithm into the generator of AdaGAN in order to improve the quality of x Z1→Z2 via adversarial training. We have applied a new training methodology in GAN-based framework. We have designed a training procedure based on non-parallel data in order to learn the mapping function for many-to-many as well as zero-shot VC. We know that the idea of transitivity as a way to regularize structured data has a long history. People have extended this concept into the training methodologies of deep learning architectures . In this paper, we have encapsulated the idea of transitivity via introducing the reconstruction loss along with adversarial training. The entire training procedure is illustrated in Fig. 1(b). First, we randomly select the two speakers Z 1 and Z 2. Formally, we have two sets of random variables, (Z 1, U 1, X) and (Z 2, U 2, Y) corresponding to the source and target speaker, respectively. After this, we randomly select x 1, x 2 ∈ p X (.|Z 1, U 1), and y 1, y 2 ∈ p Y (.|Z 2, U 2). During the training, VC is done from the source speaker (Z 1) to target speaker (Z 2) via style transfer scheme illustrated in Fig. 1(a). Using x 1, y 1, we transfer speaking style of speaker Z 2 to Z 1. From eq., we can describe this procedure as shown in eq.. Now, using another sample of source speech (i.e., x 2), we have reconstructed the source speech features (i.e., x Z1→Z2→Z1) from the features of converted speech (x Z1→Z2) in order to achieve better conversion efficiency. This procedure is described in eq.. Now, the same cycle process is again applied to transfer the speaking style of Z 2 to Z 1, we get following equations: During testing, we gave features of the source speaker's speech along with the sample features of target speaker to the encoder. AdaGAN requires 3 s to 5 s of sample speech features of the target speaker in order to transfer speaking style of target speaker to source speaker. This sample speech will be used to estimate the mean and standard deviation of the target speaker's distribution in its respective latent space. After this, the speaking style will be transferred in latent space of source speaker using AdaIN. Next, the decoder will generate speech back from the converted latent representation of the source speaker. Briefly, the decoder will generate the speech with speaking style of the target speaker. Now, the generator of AdaGAN is consist of Encoder and Decoder. Hence, we can say that the generator of AdaGAN will generate the speech with speaking style of the target speaker for a given source speaker's speech along with the sample of target speaker during testing. The training procedure of AdaGAN is formally presented in Algorithm 1. sample 4 minibatches of cepstral features {x 1, x 2} ∈ p X (.|Z 1, U 1), and {y 1, y 2} ∈ p Y (.|Z 2, U 2). First column shows the process of transferring speaking style of speaker Z 2 to Z 1. Second column shows the process of transferring speaking style of speaker Z 1 to Z 2. Comment ends */ 9: 10: S Y1 ← En(y 1); 11: 12: x ← De(t 1); y ← De(t 1); 13: 14: Update the generator by descending its stochastic gradient: Update the discriminator by descending its stochastic gradient: 21: end for 22: return To achieve many-to-many and zero-shot VC, AdaGAN uses four different loss functions: Adversarial loss, reconstruction loss, content preserve loss, and style transfer loss. Adversarial loss: This loss measures how distinguishable the converted data is from the normal speech data. The smaller the loss is, the converted data distribution is more closer to normal speech distribution. Hence, we want to minimize objective function given in eq. against an adversary Dis that tries to maximize it. Here, this loss is used to make the generated or converted speech indistinguishable from the original speech, and can be mathematically formulated as: Reconstruction Loss: By using only adversarial loss, we may loose linguistic information in the converted voice. This loss helps the encoder and decoder to retain the linguistic information in converted voice. We have used L 1 norm as a reconstruction loss, and can be described as: Content Preserve Loss: To preserve the linguistic content of the input speech during AdaIN. This loss also ensure that our encoder and decoder are noise free. We have used following L 1 norm for this loss, i.e., Style transfer Loss: This loss function is at the heart of the AdaGAN. This loss plays a vital role in achieving many-to-many and zero-shot VC using AdaGAN. This loss helps AdaGAN to create a latent space with the speaking style features in terms of mean and standard deviation of the distribution while preserving the linguistic content in the same distribution. We have used L 1 norm as style transfer loss, i.e., Final Objective Function: The overall objective function of AdaGAN can be defined as: where λ 1, λ 2, λ 3, λ 4, and λ 5 are the hyperparameters. These parameters controls the relative importance of each loss w.r.t. each other. We have used λ 1 = 10, λ 2 = 2, λ 3 = 2, λ 4 = 3, and λ 5 = 3 during the experiments. We theoretically proved that how these simple loss functions are the key idea behind the performance of AdaGAN in the next Section. We optimized these loss functions according to the Algorithm 1. AdaGAN framework contains a Generator and a Discriminator. In this Section, we provide detailed information about each component of the AdaGAN framework. As shown in Fig. 1, Generator of AdaGAN consists of mainly 2 modules: Encoder and Decoder. AdaGAN uses the same encoder to extract the features from the source and target speakers' speech. Input of encoder is a vector of 40 Mel cepstral features, which it converts to a latent space of size 1x512. The decoder takes normalized feature vector of size 1x512 as input and converts it to 1x40 target speech features. In encoder and decoder, all layers are fully-connected layers. In encoder, the input and output layer has 40 and 512 cell size, respectively. In decoder, input and output layer have 512 and 40 cell size, respectively. All the hidden layers in encoder and decoder consist 512 cell size. All the layers are followed by Rectified Linear Unit (ReLU) activation function except output layer. In AdaGAN, main goal of the discriminator is similar to traditional GAN training. Accurately, it will discriminate whether the input is generated (x Z1→Z2) or from the original distribution. Same as Encoder and Decoder, structure of discriminator follows the stacked fully-connected layers. It consists of an input layer, 3 hidden layers and, an output layer with 40, 512, and 1 cell size, respectively. In discriminator, each layer followed by the ReLU activation function and output layer followed by a sigmoid activation function. In this Section, we show the theoretical correctness and intuitive explanation of AdaGAN. The key idea of the AdaGAN is to learn the latent space, where we can represent our features as per our requirements. Consider the training procedure of AdaGAN described in Section 4.2. Let us take two latent space features S x1 and S x2 corresponding to two different sample features, x 1 and x 2, respectively, of the same speaker Z 1. We are also going to take S y1 from latent space of another speaker Z 2, where y 1 is a sample feature of that speaker, and Z 1 = Z 2. After training of AdaGAN for a large number of iteration of τ, where theoretically τ → ∞, let us assume the following: 1. In the latent space, mean and standard deviation of the same speaker are constant irrespective of the linguistic content. Formally, we have µ Sx 1 = µ Sx 2, and σ Sx 1 = σ Sx 2. 2. If we have different speakers, then mean and standard deviation of respective latent representations are different. Accurately, µ Sx 1 = µ Sy 1, and σ Sx 1 = σ Sy 1. Theorem 1: Given these assumptions, ∃ a latent space where normalized latent representation of input features will be the same irrespective of speaking style. Here, we take input features of same utterance U 1. Hence, where KL(·|·) is the KL-divergence, and p N (.|Z i, U i) is pdf of normalized latent representation of input feature U i, with speaking style of speaker Z i. This is the fundamental theorem that lies behind the concept of AdaGAN. Intuitively, from this theorem, we can observe that the normalized latent representation of the same utterance spoken by different speakers is the same. This fact leads to the that linguistic content of speech is captured by the distribution of normalized latent space, and speaking style of a speaker is being captured by mean and standard deviation of the same distribution. Theorem 2: By optimization of min En,De L C X→Y + L sty X→Y, the assumptions made in Theorem 1 can be satisfied. The proof of both the theorems are given in Appendix A. Both the theorems conclude that AdaIN made style transfer easy and efficient via only using the mean and standard deviation of the distribution. In Appendix B, we provided the t-SNE visualization of the features in latent space to give the empirical proof. In this Section, we show a comparison between AdaGAN and StarGAN-VC in terms of computational complexity. Table 2, G StarGAN, D StarGAN, and Cls are generator, discriminator, and classifier of StarGAN, respectively. All these three modules contain convolution layers. In StarGAN, there is weight sharing between the 5 convolution layers of discriminator and classifier. Here, we remove the FLOPS and trainable parameters of shared layers from the Cls. Hence, we consider it once in the calculation of total FLOPS and trainable parameters. We can observe that AdaGAN is 88.6% less complex than StarGAN in terms of FLOPS, and 85.46% less complex in terms of trainable parameters. Moreover, StarGAN uses a one-hot encoding to get the information about the target speaker. However, AdaGAN requires any sample of 3 s -5 s from the target speaker. In this Section, we will show experimental setup, and subjective evaluation (or ) of AdaGAN. Samples of converted audio files are provided here 3. The experiments are performed on the VCTK corpus , which contains 44 hours of data for 109 speakers. The statistics of the database are given. The database is designed to provide non-parallel data for VC. From this database, AdaGAN system was developed on data of 20 speakers (10 males and 10 females). Out of this, we have used 80% data for training and 20% data for testing for each speaker. Particularly, we have used 6.27 and 1.45 hours of data for the training and testing, respectively. The 40-dimensional (dim) Mel Cepstral Coefficients (MCCs) (including the 0 th coefficient) and 1-dimensional F 0 are extracted from the speech of source, and the target speakers with 25 ms window and 5 ms frame-shift. For analysis-synthesis, we have used AHOCODER . Mean-variance transformation method has been applied for fundamental frequency F 0 conversion. To evaluate AdaGAN empirically, we performed two subjective tests for evaluating naturalness and speaker similarity. In particular, Mean Opinion Score (MOS) test have been conducted, where subjects have been asked to rate the randomly played converted speech on 5-point scale for naturalness, where 1 means converted voice is very robotic, and 5 means converted voice is very natural. In the second test, subjects have been asked to rate how similar the converted voice given the reference target speech in terms of speaker similarity. Subjects rated converted voices for speaker similarity on the 5-point scale, where 1 means dissimilar with high confidence and 5 means similar with high confidence w.r.t. the given target speaker. Total 15 subjects (6 females and 9 males with no known hearing impairments with age varies between 18 to 31 years) took part in the subjective evaluations. Randomly 2 male and 2 female speakers have been selected from the testing dataset for subjective evaluations. We evaluated four different conversion systems, i.e., male-male (M2M), female-female (F2F), male-female (M2F), and female-male (F2M) developed using proposed AdaGAN and Star-GAN. From each system, two converted audio files have been selected. Hence, 8 audio files from AdaGAN and another 8 audio files from the StarGAN have been taken for subjective evaluations. We kept the same source-target speaker-pairs for fair comparison. Fig. 2 shows the comparison of MOS scores between AdaGAN and the baseline StarGAN-VC. Total of 15 subjects (6 females and 9 males) between 18-30 years of age and with no known hearing impairments took part in the subjective test. For statistically significant analysis, are shown in different conversion possibilities with 95% confidence interval. In addition, for our subjective tests, we obtain p-value 0.013, which is much lesser then 0.05. Therefore, it clearly shows the statistical significance of the . From Fig. 2, it is clear that there is 31.73 % relative improvement (on an average) in MOS score for the AdaGAN compared to the baseline StarGAN. In terms of speaker similarity, AdaGAN yields on an average 10.37% relative improvement in speaker similarity compare to baseline (as shown in Fig. 3). Although AdaGAN outperforms StarGAN, both the methods are not able to achieve good score in the similarity test. The main reason is due to the F 0 conversion and errors in statistical vocoder (i.e., AHOCODER and WORLD-vocoder). However, neural network-based Wavenet-vocoder shows very promising on speech synthesis. Although they are very accurate, they are data-driven approaches. In summary, AdaGAN achieves better performance in MOS tests compared to the StarGAN-VC for naturalness and speaker similarity. In traditional many-to-many VC, all the target speakers are seen while training the architecture. Hence, traditional algorithms are not able to do VC for an unseen speaker (i.e., for the cases of zeroshot VC). Along with many-to-many VC, we extended our study of AdaGAN for zero-shot VC. Zero-shot conversion is the task of transferring the speaking style of seen/unseen source speaker to seen/unseen target speaker. In simple terms, conversion can be done between any speaker whether their data were present in the corpus or not at the time of training. StarGAN-VC uses a one-hot vector for target speaker reference during conversion. In the case of an unseen target speaker, it will not be able to perform the zero-shot conversion. However, AdaGAN maps the input to the required latent space (as proved in Appendix A). Therefore, AdaGAN will be able to learn more promised latent space for even unseen speakers. Here, we show our experimental for the zero-shot VC task. We performed subjective tests in a similar manner as performed in many-to-many VC. We have used AdaGAN trained on 20 speakers (10 males and 10 females). Later on, we selected randomly 1 seen, and 1 unseen male speakers and 1 seen, and 1 unseen female speakers. And we applied the permutations on these different speakers to get all the different conversion samples, such as seen-seen (S2S), seen-unseen (S2U), unseen-seen (U2S), and unseen-unseen (U2U). Fig. 4, and Fig. 5 shows the MOS scores for naturalness, and speaker similarity, respectively. Recently, AutoVC has been proposed, which is the only framework for zero-shot conversion. To the best of authors' knowledge, this is the first GAN-based framework to achieve zero-shot VC. To do the zero-shot conversion, AutoVC requires few samples (20 s) of possible target speakers. However, AdaGAN requires only 3s to 5s of sample speech of the seen or unseen target speaker to extract latent representation for the target speaker in order to generate voices that sound perceptually similar to the target speaker. Moreover, trained AdaGAN architecture can work on any source or target speaker. In this paper, we proposed novel AdaGAN primarily for non-parallel many-to-many VC task. Moreover, we analyzed our proposed architecture w.r.t. current GAN-based state-of-the-art StarGAN-VC method for the same task. We know that the main aim of VC is to convert the source speaker's voice into the target speaker's voice while preserving linguistic content. To achieve this, we have used the style transfer algorithm along with the adversarial training. AdaGAN transfers the style of the target speaker into the voice of a source speaker without using any feature-based mapping between the linguistic content of the source speaker's speech. For this task, AdaGAN uses only one generator and one discriminator, which leads to less complexity. AdaGAN is almost 88.6% computationally less complex than the StarGAN-VC. We have performed subjective analysis on the VCTK corpus to show the efficiency of the proposed method. We can clearly see that AdaGAN gives superior in the subjective evaluations compared to StarGAN-VC. Motivated by the work of AutoVC, we also extended the concept of AdaGAN for the zero-shot conversion as an independent study and reported . AdaGAN is the first GAN-based framework for zero-shot VC. In the future, we plan to explore high-quality vocoders, namely, WaveNet, for further improvement in voice quality. The perceptual difference observed between the estimated and the ground truth indicates the need for exploring better objective function that can perceptually optimize the network parameters of GAN-based architectures, which also forms our immediate future work. At τ → ∞, the assumptions that made in Section 5.1 are true. Hence, from eq., we can conclude that there exists a latent space where normalized latent representation of input features will be the same irrespective of speaking style. Theorem 2: By optimization of min En,De L C X→Y + L sty X→Y, the assumptions made in Theorem 1 can be satisfied. Proof: Our objective function is the following: Iterate step by step to calculate the term (t 2) used in loss function L sty X→Y. Consider, we have the latent representations S x1 and S y1 corresponding to the source and target speech, respectively. Step 1: S x1 (τ) − µ 1 (τ) σ 1 (τ) σ 2 (τ) + µ 2 (τ) (Representation of t 1), Step 2&3: En De S x1 (τ) − µ 1 (τ) σ 1 (τ) σ 2 (τ) + µ 2 (τ). After applying decoder and encoder sequentially on latent representation, we will again get back to the same representation. This is ensured by the loss function L C X→Y. Formally, we want to make L C X→Y → 0. Therefore, we can write step 4 as: Step 4: S x1 (τ) − µ 1 (τ) σ 1 (τ) σ 2 (τ) + µ 2 (τ) (i.e., reconstructed t 1), Step 5: 1σ 2 (τ) S x1 (τ) − µ 1 (τ) σ 1 (τ) ¨σ 2 (τ) +¨μ 2 (τ) −¨μ 2 (τ) (Normalization with its own (i.e., latent representation in Step 4) µ and σ during AdaIN ), Step 6: S x1 (τ) − µ 1 (τ) σ 1 (τ) (Final output of Step 5), Step 7: S x1 (τ) − µ 1 (τ) σ 1 (τ) σ 1 (τ) + µ 1 (τ) (Output after de-normalization in AdaIN . Representation of t 2), where µ 1 and σ 1 are the mean and standard deviations of the another input source speech, x 2. Now, using the mathematical representation of t 2, we can write loss function L sty X→Y as: According to eq., we want to minimize the loss function L sty X→Y. Formally, L sty X→Y → 0. Therefore, we will get µ 1 = µ 1, and σ 1 = σ 1 to achieve our goal. Hence, mean and standard deviation of the same speaker are constant, and different for different speakers irrespective of the linguistic content. We come to the that our loss function satisfies the necessary constraints (assumptions) required in proof of Theorem 1. Figure 6: t-SNE visualization of latent representation of two speakers' speech and its normalized form, where, each point denotes a feature extracted from the 25 ms of speech segment. As we know, Neural Networks (NNs) are hard to train and optimize. Even if everything has been proven in terms of theoretical proofs, statistical and empirical analysis is required. For this analysis, we have adopted t-SNE visualization. Here, we randomly selected few utterances from two different speakers from the VCTK corpus. Latent representations are extracted for the speech of that speakers, and features are reduced to 2-D using t-SNE. The scatter plot shown in Fig. 6 shows that data points are clustered based on the speaking style. After normalized with their respective means and standard deviations, these distribution overlapped. This shows that the distribution of normalized latent representation captures linguistic information-based features irrespective of speaking style as proved in Theorem 1. Therefore, we can say that AdaGAN, and its losses are efficient for practical purposes.

Represent the TLDR for finding the corresponding scientific article

Novel adaptive instance normalization based GAN framework for non parallel many-to-many and zero-shot VC.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose to explicitly model deep feature distributions of source and target data as Gaussian mixture distributions for Unsupervised Domain Adaptation (UDA) and achieve superior results in multiple UDA tasks than state-of-the-art methods.


Question: Represent the following document.

Question: Self-attention-based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks. Self attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context. To tackle the problem, we propose a novel model called Sparse Transformer. Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments. Extensive experimental on a series of natural language processing tasks, including neural machine translation, image captioning, and language modeling, all demonstrate the advantages of Sparse Transformer in model performance. Sparse Transformer reaches the state-of-the-art performances in the IWSLT 2015 English-to-Vietnamese translation and IWSLT 2014 German-to-English translation. In addition, we conduct qualitative analysis to account for Sparse Transformer's superior performance. Understanding natural language requires the ability to pay attention to the most relevant information. For example, people tend to focus on the most relevant segments to search for the answers to their questions in mind during reading. However, retrieving problems may occur if irrelevant segments impose negative impacts on reading comprehension. Such distraction hinders the understanding process, which calls for an effective attention. This principle is also applicable to the computation systems for natural language. Attention has been a vital component of the models for natural language understanding and natural language generation. proposed Transformer, a model based on the attention mechanism for Neural Machine Translation(NMT). Transformer has shown outstanding performance in natural language generation tasks. More recently, the success of BERT in natural language processing shows the great usefulness of both the attention mechanism and the framework of Transformer. However, the attention in vanilla Transformer has a obvious drawback, as the Transformer assigns credits to all components of the context. This causes a lack of focus. As illustrated in Figure 1, the attention in vanilla Transformer assigns high credits to many irrelevant words, while in Explicit Sparse Transformer, it concentrates on the most relevant k words. For the word "tim", the most related words should be "heart" and the immediate words. Yet the attention in vanilla Transformer does not focus on them but gives credits to some irrelevant words such as "him". Recent works have studied applying sparse attention in Transformer model. However, they either add local attention constraints which break long term dependency or hurt the time efficiency . Inspired by which introduce sparse credit assignment to the LSTM model, we propose a novel model called Explicit Sparse Transformer which is equipped with our sparse attention mechanism. We implement an explicit selection method based on top-k selection. Unlike vanilla Transformer, Explicit Sparse Transformer only pays attention to the k most contributive states. Thus Explicit Sparse Transformer can perform more concentrated attention than vanilla Transformer. Figure 1: Illustration of self-attention in the models. The orange bar denotes the attention score of our proposed model while the blue bar denotes the attention scores of the vanilla Transformer. The orange line denotes the attention between the target word "tim" and the selected top-k positions in the sequence. In the attention of vanilla Transformer, "tim" assigns too many non-zero attention scores to the irrelevant words. But for the proposal, the top-k largest attention scores removes the distraction from irrelevant words and the attention becomes concentrated. We first validate our methods on three tasks. For further investigation, we compare our methods with previous sparse attention methods and experimentally answer how to choose k in a series of qualitative analyses. We are surprised to find that the proposed sparse attention method can also help with training as a regularization method. Visual analysis shows that Explicit Sparse Transformer exhibits a higher potential in performing a high-quality alignment. The contributions of this paper are presented below: • We propose a novel model called Explicit Sparse Transformer, which enhances the concentration of the Transformer's attention through explicit selection. • We conducted extensive experiments on three natural language processing tasks, including Neural Machine Translation, Image Captioning and Language Modeling. Compared with vanilla Transformer, Explicit Sparse Transformer demonstrates better performances in the above three tasks. Specifically, our model reaches the state-of-the-art performances in the IWSLT 2015 English-to-Vietnamese translation. • Compared to previous sparse attention methods for transformers, our methods are much faster in training and testing, and achieves better . The review to the attention mechanism and the attention-based framework of Transformer can be found in Appendix A.1. Lack of concentration in the attention can lead to the failure of relevant information extraction. To this end, we propose a novel model, Explicit Sparse Transformer, which enables the focus on only a few elements through explicit selection. Compared with the conventional attention, no credit will be assigned to the value that is not highly correlated to the query. We provide a comparison between the attention of vanilla Transformer and that of Explicit Sparse Transformer in Figure 2. Explicit Sparse Transformer is still based on the Transformer framework. The difference is in the implementation of self-attention. The attention is degenerated to the sparse attention through top-k selection. In this way, the most contributive components for attention are reserved and the other irrelevant information are removed. This selective method is effective in preserving important information and removing noise. The attention can be much more concentrated on the most contributive elements of value. In the following, we first introduce the sparsification in self-attention and then extend it to context attention. In the unihead self-attention, the key components, the query are the linear transformation of the source context, namely the input of each layer, where Q = W Q x, K = W K x and V = W V x. Explicit Sparse Transformer first generates the attention scores P as demonstrated below: Then the model evaluates the values of the scores P based on the hypothesis that scores with larger values demonstrate higher relevance. The sparse attention masking operation M(·) is implemented upon P in order to select the top-k contributive elements. Specifically, we select the k largest element of each row in P and record their positions in the position matrix (i, j), where k is a hyperparameter. To be specific, say the k-th largest value of row i is t i, if the value of the j-th component is larger than t i, the position (i, j) is recorded. We concatenate the threshold value of each row to form a The masking functions M(·, ·) is illustrated as follows: With the top-k selection, the high attention scores are selected through an explicit way. This is different from dropout which randomly abandons the scores. Such explicit selection can not only guarantee the preservation of important components, but also simplify the model since k is usually a small number such as 8, detailed analysis can be found in 5.2. The next step after top-k selection is normalization: where A refers to the normalized scores. As the scores that are smaller than the top k largest scores are assigned with negative infinity by the masking function M(·, ·), their normalized scores, namely the probabilities, approximate 0. We show the back-propagation process of Top-k selection in A.3. The output representation of self-attention C can be computed as below: ConvS2S 25.2 --Actor-Critic --28.5 NPMT+LM ( 28.4 --RNMT 28.5 --Fixup 29.3 -34.5 Weighted Transformer 28.9 --Universal Transformer 28.9 --Layer-wise Coordination 29.1 --Transformer(relative position) 29.2 --Transformer 29.3 --DynamicConv 29. The output is the expectation of the value following the sparsified distribution A. Following the distribution of the selected components, the attention in the Explicit Sparse Transformer model can obtain more focused attention. Also, such sparse attention can extend to context attention. Resembling but different from the self-attention mechanism, the Q is no longer the linear transformation of the source context but the decoding states s. In the implementation, we replace Q with W Q s, where W Q is still learnable matrix. In brief, the attention in our proposed Explicit Sparse Transformer sparsifies the attention weights. The attention can then become focused on the most contributive elements, and it is compatible to both self-attention and context attention. The simple implementation of this method is in the Appendix A.4. We conducted a series of experiments on three natural language processing tasks, including neural machine translation, image captioning and language modeling. Detailed experimental settings are in Appendix A.2. Dataset To evaluate the performance of Explicit Sparse Transformer in NMT, we conducted experiments on three NMT tasks, English-to-German translation (En-De) with a large dataset, English-to-Vietnamese (En-Vi) translation and German-to-English translation (De-En) with two datasets of medium size. For En-De, we trained Explicit Sparse Transformer on the standard dataset for WMT 2014 En-De translation. The dataset consists of around 4.5 million sentence pairs. The source and target languages share a vocabulary of 32K sub-word units. We used the newstest 2013 for validation and the newstest 2014 as our test set. We report the on the test set. For En-Vi, we trained our model on the dataset in IWSLT 2015 . The dataset consists of around 133K sentence pairs from translated TED talks. The vocabulary size for source language is around 17,200 and that for target language is around 7,800. We used tst2012 for validation, and tst2013 for testing and report the testing . For De-En, we used the dataset in IWSLT 2014. The training set contains 160K sentence pairs and the validation set contains 7K sentences., we used the same test set with around 7K sentences. The data were preprocessed with byte-pair encoding . The vocabulary size is 14,000. Result Dataset We evaluated our approach on the image captioning task. Image captioning is a task that combines image understanding and language generation. We conducted experiments on the Microsoft COCO 2014 dataset (a). It contains 123,287 images, each of which is paired 5 with descriptive sentences. We report the and evaluate the image captioning model on the MSCOCO 2014 test set for image captioning. We used the publicly-available splits provided by. The validation set and test set both contain 5,000 images. Result Table 2 shows the of the baseline models and Explicit Sparse Transformer on the COCO Karpathy test split. Transformer outperforms the mentioned baseline models. Explicit Sparse Transformer outperforms the implemented Transformer by +0.4 in terms of BLEU-4, +0.3 in terms of METEOR, +0.7 in terms of CIDEr., which consistently proves its effectiveness in Image Captioning. 2 is large-scale dataset for character-level language modeling. It contains 100M bytes of unprocessed Wikipedia texts. The inputs include Latin alphabets, non-Latin alphabets, XML markups and special characters. The vocabulary size 205 tokens, including one for unknown characters. We used the same preprocessing method following. The training set contains 90M bytes of data, and the validation set and the test set contains 5M respectively. Result Table 3 shows the of the baseline models and Explicit Sparse Transformer-XL on the test set of enwiki8. Compared with the other strong baselines, Transformer-XL can reach a better performance, and Explicit Sparse Transformer outperforms Transformer-XL with an advantage. Table 4: In the Transformer model, the proposed method, top-k selection before softmax is faster than previous sparse attention methods and is comparable in terms of BLEU scores. In this section, we performed several analyses for further discussion of Explicit Sparse Transformer. First, we compare the proposed method of topk selection before softmax with previous sparse attention method including various variants of sparsemax (; ;). Second, we discuss about the selection of the value of k. Third, we demonstrate that the top-k sparse attention method helps training. In the end, we conducted a series of qualitative analyses to visualize proposed sparse attention in Transformer. We compare the performance and speed of our method with the previous sparse attention methods 3 on the basis of strong implemented transformer baseline. The training and inference speed are reported on the platform of Pytorch and IWSLT 2014 De-En translation dataset, the batch size for inference is set to 128 in terms of sentence and half precision training(FP-16) is applied. As we can see from Table 4, the proposed sparse attention method achieve the comparable as previous sparse attention methods, but the training and testing speed is 2x faster than sparsemax and 10x faster than Entmax-alpha during the inference. This is due to the fact that our method does not introduce too much computation for calculating sparse attention scores. The other group of sparse attention methods of adding local attention constraints into attention , do not show performance on neural machine translation, so we do not compare them in Table 4. Base T T&P En-Vi (BLEU) 27.4 27.7 27.8 Table 5: Results of the ablation study of the sparsification at different phases on the En-Vi test set. "Base" denotes vanilla Transformer. "T" denotes only adding the sparsification in the training phase, and "T&P" denotes adding it at both phases as the implementation of Explicit Sparse Transformer does. The natural question of how to choose the optimal k comes with the proposed method. We compare the effect of the value of k at exponential scales. We perform experiments on En-Vi and De-En from 3 different initializations for each value of K, and report the mean BLEU scores on the valid set. The figure 3 shows that regardless of the value of 16 on the En-Vi dataset, the model performance generally rises first and then falls as k increases. Under the setting of the k ∈ {4, 8, 16, 32}, setting the value of k to 8 achieves consistent improvements over the We are surprised to find that only adding the sparsification in the training phase can also bring an improvement in the performance. We experiment this idea on IWSLT En-Vi and report the on the valid set in Table 5,. The improvement of 0.3 BLEU scores shows that vanilla Transformer may be overparameterized and the sparsification encourages the simplification of the model. To perform a thorough evaluation of our Explicit Sparse Transformer, we conducted a case study and visualize the attention distributions of our model and the baseline for further comparison. Specifically, we conducted the analysis on the test set of En-Vi, and randomly selected a sample pair of attention visualization of both models. The visualization of the context attention of the decoder's bottom layer in Figure 4 (a). The attention distribution of the left figure is fairly disperse. On the contrary, the right figure shows that the sparse attention can choose to focus only on several positions so that the model can be forced to stay focused. For example, when generating the phrase "for thinking about my heart"(Word-to-word translation from Vietnamese), the generated word cannot be aligned to the corresponding words. As to Explicit Sparse Transformer, when generating the phrase "with all my heart", the attention can focus on the corresponding positions with strong confidence. The visualization of the decoder's top layer is shown in Figure 4 (b). From the figure, the context attention at the top layer of the vanilla Transformer decoder suffers from focusing on the last source token. This is a common behavior of the attention in vanilla Transformer. Such attention with wrong alignment cannot sufficiently extract enough relevant source-side information for the generation. In contrast, Explicit Sparse Transformer, with simple modification on the vanilla version, does not suffer from this problem, but instead focuses on the relevant sections of the source context. The figure on the right demonstrating the attention distribution of Explicit Sparse Transformer shows that our proposed attention in the model is able to perform accurate alignment. Attention mechanism has demonstrated outstanding performances in a number of neural-networkbased methods, and it has been a focus in the NLP studies . A number of studies are proposed to enhance the effects of attention mechanism (; ;). propose local attention and propose local attention for self-attention. propose hard attention that pays discrete attention in image captioning. propose a combination soft attention with hard attention to construct hierarchical memory network. propose a temperature mechanism to change the softness of attention distribution. propose an attention which can select a small proportion for focusing. It is trained by reinforcement learning algorithms . In terms of memory networks, propose to sparse access memory recently propose to use local attention and block attention to sparsify the transformer. Our approach differs from them in that our method does not need to block sentences and still capture long distance dependencies. Besides, we demonstrate the importance of Explicit Sparse Transformer in sequence to sequence learning. Although the variants of sparsemax (; ;) improve in machine translation tasks, we empirically demonstrate in 5.1 that our method introduces less computation in the standard transformer and is much faster than those sparse attention methods on GPUs. first introduced the attention mechanism to learn the alignment between the target-side context and the source-side context, and formulated several versions for local and global attention. In general, the attention mechanism maps a query and a key-value pair to an output. The attention score function and softmax normalization can turn the query Q and the key K into a distribution α. Following the distribution α, the attention mechanism computes the expectation of the value V and finally generates the output C. Take the original attention mechanism in NMT as an example. Both key K ∈ R n×d and value V ∈ R n×d are the sequence of output states from the encoder. Query Q ∈ R m×d is the sequence of output states from the decoder, where m is the length of Q, n is the length of K and V, and d is the dimension of the states. Thus, the attention mechanism is formulated as: where f refers to the attention score computation. Transformer , which is fully based on the attention mechanism, demonstrates the state-of-the-art performances in a series of natural language generation tasks. Specifically, we focus on self-attention and multi-head attention. The ideology of self-attention is, as the name implies, the attention over the context itself. In the implementation, the query Q, key K and value V are the linear transformation of the input x, so that Q = W Q x, K = W K x and V = W V x where W Q, W K and W V are learnable parameters. Therefore, the computation can be formulated as below: where d refers to the dimension of the states. The aforementioned mechanism can be regarded as the unihead attention. As to the multi-head attention, the attention computation is separated into g heads (namely 8 for basic model and 16 for large model in the common practice). Thus multiple parts of the inputs can be computed individually. For the i-th head, the output can be computed as in the following formula: where C (i) refers to the output of the head, Q (i), K (i) and V (i) are the query, key and value of the head, and d k refers to the size of each head (d k = d/g). Finally, the output of each head are concatenated for the output: In common practice, C is sent through a linear transformation with weight matrix W c for the final output of multi-head attention. However, soft attention can assign weights to a lot more words that are less relevent to the query. Therefore, in order to improve concentration in attention for effective information extraction, we study the problem of sparse attention in Transformer and propose our model Explicit Sparse Transformer. We use the default setting in for the implementation of our proposed Explicit Sparse Transformer. The hyper parameters including beam size and training steps are tuned on the valid set. Neural Machine Translation Training For En-Vi translation, we use default scripts and hyperparameter setting of tensor2tensor 4 v1.11.0 to preprocess, train and evaluate our model. We use the default scripts of fairseq 5 v0.6.1 to preprocess the De-En and En-De dataset. We train the model on the En-Vi dataset for 35K steps with batch size of 4K. For IWSLT 2015 De-En dataset, batch size is also set to 4K, we update the model every 4 steps and train the model for 90epochs. For WMT 2014 En-De dataset, we train the model for 72 epochs on 4 GPUs with update frequency of 32 and batch size of 3584. We train all models on a single RTX2080TI for two small IWSLT datasets and on a single machine of 4 RTX TITAN for WMT14 En-De. In order to reduce the impact of random initialization, we perform experiments with three different initializations for all models and report the highest for small datasets. Evaluation We use case-sensitive tokenized BLEU score for the evaluation of WMT14 En-De, and we use case-insensitive BLEU for that of IWSLT 2015 En-Vi and IWSLT 2014 De-En following. Same as , compound splitting is used for WMT 14 En-De. For WMT 14 En-De and IWSLT 2014 De-En, we save checkpoints every epoch and average last 10 checkpoints every 5 epochs, We select the averaged checkpoint with best valid BLEU and report its BLEU score on the test set. For IWSLT 2015 En-Vi, we save checkpoints every 600 seconds and average last 20 checkpoints. Image Captioning We still use the default setting of Transformer for training our proposed Explicit Sparse Transformer. We report the standard automatic evaluation metrics with the help of the COCO captioning evaluation toolkit 6 (b), which includes the commonly-used evaluation metrics, BLEU-4 Papineni et al. Language Models We follow and use their implementation for our Explicit Sparse Transformer. Following the previous work , we use BPC (E[log 2 P (xt + 1|ht)]), standing for the average number of Bits-Per-Character, for evaluation. Lower BPC refers to better performance. As to the model implementation, we implement Explicit Sparse Transformer-XL, which is based on the base version of Transformer-XL. 7 Transformer-XL is a model based on Transformer but has better capability of representing long sequences. The masking function M(·, ·) is illustrated as follow: M(P, k) ij = P ij if P ij ≥ t i (k-th largest value of row i) −∞ if P ij < t i (k-th largest value of row i) Denote M = M(P, k). We regard t i as constants. When back-propagating, ∂M ij ∂P ij = 1 if P ij ≥ t i (k-th largest value of row i) 0 if P ij < t i (k-th largest value of row i) The next step after top-k selection is normalization: where A refers to the normalized scores. When backpropagating, =    ∂A ij ∂M kl if P ij ≥ t i (k-th largest value of row i) 0 if P ij < t i (k-th largest value of row i) The softmax function is evidently differentiable, therefore, we have calculated the gradient involved in top-k selection. A.4 IMPLEMENTATION Figure 5 shows the code for the idea in case of single head self-attention, the proposed method is easy to implement and plug in the successful Transformer model.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

This work propose Sparse Transformer to improve the concentration of attention on the global context through an explicit selection of the most relevant segments for sequence to sequence learning.

Represent the short summary for finding the corresponding scientific article (long)

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.