Question: Represent the scientific article to find a short TLDR!

Question: Most deep learning for NLP represents each word with a single point or single-mode region in semantic space, while the existing multi-mode word embeddings cannot represent longer word sequences like phrases or sentences. We introduce a phrase representation (also applicable to sentences) where each phrase has a distinct set of multi-mode codebook embeddings to capture different semantic facets of the phrase's meaning. The codebook embeddings can be viewed as the cluster centers which summarize the distribution of possibly co-occurring words in a pre-trained word embedding space. We propose an end-to-end trainable neural model that directly predicts the set of cluster centers from the input text sequence (e.g., a phrase or a sentence) during test time. We find that the per-phrase/sentence codebook embeddings not only provide a more interpretable semantic representation but also outperform strong baselines (by a large margin in some tasks) on benchmark datasets for unsupervised phrase similarity, sentence similarity, hypernym detection, and extractive summarization. Many widely-applicable NLP models learn a representation from only co-occurrence statistics in the raw text without any supervision. Examples include word embedding like word2vec or GloVe , sentence embeddings like skip-thoughts , and contextualized word embedding like ELMo and BERT . Most of these models use a single embedding to represent one sentence or one phrase and can only provide symmetric similarity measurement when no annotation is available. However, a word or phrase might have multiple senses, and a sentence can involve multiple topics, which are hard to analyze based on a single embedding without supervision. To address the issue, word sense induction methods and recent multi-mode word embeddings (; ;) represent each target word as multiple points or regions in a distributional semantic space by (explicitly or implicitly) clustering all the words appearing beside the target word. In Figure 1, the multi-mode representation of real property is illustrated as an example. Real property can be observed in legal documents where it usually means a real estate, while a real property can also mean a true characteristic in philosophic discussions. The previous approaches discover those senses by clustering observed neighboring words (e.g., company and tax). In contrast with topic modeling like LDA , the approaches need to solve a distinct clustering problem for every target word while topic modeling finds a single set of clusters by clustering all the words in the corpus. Extending these multi-mode representations to arbitrary sequences like phrases or sentences is difficult due to two efficiency challenges. First, there are usually many more unique phrases and sentences in a corpus than there are words, while the number of parameters for clustering-based approaches is O(|V | × |K| × |E|), where |V | is number of unique sequences, |K| is number of modes/clusters, and |E| is the number of embedding dimensions. Estimating and storing such a large number of parameters take time and space. More important, many unique sequences imply much fewer co-occurring words to be clustered for each sequence, especially for long sequences Figure 1: The target phrase real property is represented by four clustering centers. The previous work discovers the four modes by finding clustering centers which well compress the embedding of observed co-occurring words. Instead, our compositional model learns to predict the embeddings of cluster centers from the sequence of words in the target phrase so as to reconstruct the (unseen) co-occurring distribution well. like sentences, so an effective model needs to overcome this sample efficient challenge (i.e., sparseness in the co-occurring statistics). However, clustering approaches often have too many parameters to learn the compositional meaning of each sequence without overfitting. Nevertheless, the sentences (or phrases) sharing multiple words tend to have similar cluster centers, so we should be able to compress many redundant parameters in these local clustering problems to circumvent the challenges. In this work, we adopt a neural encoder and decoder to achieve the goal. As shown in Figure 1, instead of clustering co-occurring words beside a target sequence at test time as in previous approaches, we learn a mapping between the target sequence (i.e., phrases or sentences) and the corresponding cluster centers during training so that we can directly predict those cluster centers using a single forward pass of the neural network for an arbitrary unseen input sequences during testing. To allow the neural network to generate the cluster centers in an arbitrary order, we use a nonnegative and sparse coefficient matrix to dynamically match the sequence of predicted cluster centers and the observed set of co-occurring word embeddings during training. After the coefficient matrix is estimated for each input sequence, the gradients are back-propagated to cluster centers (i.e., codebook embeddings) and weights of decoder and encoder, which allows us to train the whole model jointly and end-to-end. In experiments, we show that the proposed model captures the compositional meanings of words in unsupervised phrase similarity tasks much better than averaging their (contextualized) word embeddings, strong baselines that are widely used in practice. In addition to similarity, our model can also measure asymmetric relations like hypernymy without any supervision. Furthermore, the multimode representation is shown to outperform the single-mode alternatives in sentence representation, especially as demonstrated in our extractive summarization experiment. In this section, we first formalize our training setup in Section 2.1. Next, our objective function and the architecture of our prediction mode will be described in Section 2.2 and Section 2.3, respectively. The approach is summarized in Figure 2 using an example sentence. Figure 2: Our model for sentence representation. We represent each sentence as multiple codebook embeddings (i.e., clustering centers) predicted by our sequence to embeddings model. Our loss encourages the model to generate codebook embeddings whose linear combination can well reconstruct the embeddings of co-occurring words (e.g., Music), while not able to reconstruct the negatively sampled words (i.e., the co-occurring words from other sentences) to avoid predicting common topics which co-occur with every sentence (e.g., is in this example). We express tth sequence of words in the corpus as I t = w xt...w yt <eos>, where x t and y t are the start and end position of the target sequence, respectively, and <eos> is the end of sequence symbol. We assume neighboring words beside each target phrase or sentence are related to some aspects of the sequence, so given I t as input, our training signal is to reconstruct a set of neighboring words, is a fixed window size. For sentence representation, N t is the set of all words in the previous and the next sentence. Notice that the training signal for sentences and phrases are different, which means we need to train one model for phrase and one model for sentence if both representations are desired. Since there are not many co-occurring words for a long sequence (none are observed for unseen testing sequences), our goal is to cluster the set of words which could "possibly" occur beside the sequence instead of the actual occurring words in the training corpus (e.g., the hidden co-occurring distribution instead of green and underlined words in Figure 2). Although most of the possibly co-occurring words are not observed in the corpus, we can still learn to predict them by observing co-occurring words from similar sequences. To focus on the semantics rather than syntax, we view the co-occurring words as a set rather than a sequence as in skip-thoughts . Notice that our model considers the word order information in the input sequence I t, but ignores the order of the co-occurring words N t. In this work, we model the distribution of co-occurring words in a pre-trained word embedding space. The embeddings of co-occurring words N t are arranged into a matrix W (N t) = [w t j] j=1...|Nt| with size |E| × |N t |, where |E| is the dimension of pre-trained word embedding, and each of its column w t j is a normalized word embedding whose 2-norm is 1. The normalization makes the cosine distance between two words become half of their Euclidean distance. Similarly, we denote the predicted cluster centers c t k of the input sequence I t as a |E| × K matrix where F is our neural network model and K is the number of clusters. We fix the number of clusters K in this work to simplify the design of our prediction model and how it is applied to downstream tasks. The effect of different cluster numbers will be discussed in the experimental section. The reconstruction loss of k-means clustering in the word embedding space can be written as, where M k,j = 1 if the jth word belongs to the k cluster and 0 otherwise. That is, M is a permutation matrix which matches the cluster centers and co-occurring words and allow the neural network to generate the centers in an arbitrary order. Non-negative sparse coding (NNSC) relaxes the constraints by allowing the coefficient M k,j to be a positive value but encouraging it to be 0. In this work, we adopt the relaxation because for all neural architectures (including transformers and LSTMs) we tried, the models using a NNSC loss learn to generates diverse K cluster centers while the predicted cluster centers using the kmeans loss collapse to much fewer modes which cannot well capture the conditional co-occurrence distribution. We hypothesize that it is because a NNSC loss is smoother and easier to be optimized for a neural network, while finding the nearest cluster center in the kmeans loss cannot stably encourage predicted clusters to play different roles for reconstructing the embeddings of observed co-occurring words. Using NNSC, we define our reconstruction error as where λ is a hyper-parameter controlling the sparsity of M. We force the coefficient value M k,j ≤ 1 to avoid the neural network learning to predict centers with small magnitudes which makes the optimal values of M k,j large and unstable. Having multiple outputs and estimating the permutation between the prediction and ground truth words is often computationally expensive . However, the proposed loss is efficient because we minimize the L2 distance in a pre-trained embedding space as in rather than using softmax, and M Ot can be efficiently estimated on the fly using convex optimization (we use RMSprop in our implementation). After M Ot is estimated, we can treat it as a constant and back-propagate the gradients to our neural network to achieve end-to-end training. To prevent the neural network from predicting the same global topics regardless of the input, our loss function for tth sequence is defined as where N rt is a set of co-occurring words of a randomly sampled sequence I rt. In our experiment, we use SGD to solve F = arg min F t L t (F). Our method could be viewed as a generalization of Word2Vec that can encode the compositional meaning of the words and decode multiple embeddings. Our neural network architecture is similar to transformation-based sequence to sequence (seq2seq) model . We use the same encoder T E(I t), which transforms the input sequence into a contextualized embeddings where the goal of the encoder is to map the sentences which are likely to have similar co-occuring word distribution closer together. Different from the typical seq2seq model , our decoder does not need to make discrete decisions because our outputs are a sequence of embeddings instead of words. This allows us to predict all the codebook embeddings in a single forward pass while still well capturing the dependency between output without the need of auto-regressive decoding. Similar to BERT, we treat the embedding of <eos> as the sentence representation. To make different codebook embeddings capture different aspects, we pass the embeddings of <eos> to different linear layers L k before becoming the input of the decoder T D. Specifically, the codebook embeddings We find that removing the attention on the e xt...e yt, contextualized word embeddings from the encoder, significantly increases our validation loss for sentence representation because there are often too many facets to be compressed into a single embedding. On the other hand, the attention does not change the performance of phrase representation too much, and we remove the attention connection between encoder and decoder (i.e., encoder and decoder have the same architecture) when evaluating our phrase representation. Notice that the framework is flexible. We can replace the encoder and decoder with other architectures. Besides transformers, we also try (bi-)LSTMs in our experiments. This flexibility also allows us to incorporate other input features (e.g., the author who writes the sentence). We first visualize the cluster centers predicted by our model in Table 1 (like we visualize the meaning of the red cluster center in Figure 2 using the word song or Music). The centers summarize the target sequence well and more codebook embeddings capture more semantic facets of a phrase or a sentence. Due to the difficulty of evaluating the topics conditioned on the input sequence using the classic metrics for global topic modeling, we show that the codebook embeddings can be used to improve the performances of various unsupervised semantic tasks, which indirectly measures the quality of the generated topics. We use the cased version (840B) of pre-trained GloVe embedding for sentence representation and use the uncased version (42B) for phrase representation. 2 Our model is trained on Wikipedia 2016 while the stop words are removed from the set of co-occurring words. In the phrase experiments, we only consider noun phrases, and their boundaries are extracted by applying simple regular expression rules to POS tags before training. The sentence boundaries and POS tags are detected using spaCy. 3 Our models do not need resources such as PPDB or other multi-lingual resources, so our models are compared with the baselines that only use the raw text and sentence/phrase boundaries. This setting is particularly practical for the domains with low resources such as scientific literature. To control the effect of embedding size, we set the number of dimensions in our transformers as the GloVe embedding size. Limited by computational resources, we train all the models using one modern GPU within a week. Because of the relatively small model size, we find that our models underfit the data after a week (i.e., the training loss is very close to the validation loss). It is hard to make a fair comparison with BERT . BERT is trained on a masked language modeling loss, which preserves more syntax information and can produce an effective pretrained embedding for many supervised downstream tasks. We report the performance of the BERT base model, which is still trained using more parameters, more output dimensions, larger corpus, and more computational resources during training compared with our models. Furthermore, BERT uses a word piece model to alleviate the out-of-vocabulary problem. Nevertheless, we still provide its unsupervised performances based on cosine similarity as a reference. Semeval 2013 task 5(a) English and Turney 2012 are two standard benchmarks for evaluating phrase similarity . BiRD and WikiSRS , which are recently collected, contain ground truth phrase similarities derived from human annotations. The task of Semeval 2013 is to distinguish similar phrase pairs from dissimilar phrase pairs. In Turney, given each query bigram, the goal is to identify the unigram that is most similar to the query bigram among 5 unigram candidates, 5 and Turney adds 5 more negative phrase pairs by pairing the reverse of bigram with the 5 unigrams. BiRD and WikiSRS-Rel measure the relatedness of phrases and WikiSRS-Sim measures the similarity of phrases. For our model, we evaluate two scoring functions that measure phrase similarity. The first way averages the contextualized word embeddings from our transformer encoder as the phrase embedding and computes the cosine similarity between two phrase embeddings. We label the method as Ours Emb. The similar phrases should have similar multi-facet embeddings, so we compute the reconstruction error from the set of normalized codebook embeddings of one phrase S 1 q to the embeddings of the other phrase S 2 q, vice versa, and add them together to become a symmetric distance SC: where When ranking for retrieving similar phrases, we use the negative distance to represent similarity. We compare our performance with 5 baselines. GloVe Avg and Word2Vec Avg compute the cosine similarity between two averaged word embeddings, which has been shown to be a strong baseline . BERT CLS and BERT Avg are the cosine similarities between CLS embeddings and between the averages of contextualized word embeddings from BERT , respectively. FCT LM Emb learns the weights of linearly combining word embeddings based on several linguistic features. The performances are presented in Table 2. SemEval 2013 and Turney have training and testing split and the performances in test sets are reported. Our models significantly outperform all baselines in the 4 datasets. Furthermore, our strong performances in Turney verify that our encoder incorporates the word order information when producing phrase embeddings. The indicate the effectiveness of non-linearly composing word embeddings (unlike GloVe, Word2Vec, and FCT baselines) in order to predict the set of co-occurring word embeddings (unlike the BERT baselines). The performance of Ours (K=1) is usually slightly better than Ours (K=10). This supports the finding of that multi-mode embeddings may not improve the performance in word similarity benchmarks even if they capture more senses or aspects of polysemies. Even though being slightly worse, the performances of Ours (K=10) remain strong compared with baselines. This indicates that the similarity performance is not sensitive to the number of clusters, which alleviates the problem of selecting K in practice. STS benchmark is a widely used sentence similarity task. Each model predicts a semantic similarity score between each sentence pair, and the scores are compared with average similarity annotations using the Pearson correlation coefficient. Intuitively, when two sentences are less similar to each other, humans tend to judge the similarity based on how they are similar in each aspect. Thus, we also compare the performances on lower half the datasets where their ground truth similarities are less than the median similarity score, and we call this benchmark STSB Low. In addition to BERT CLS, BERT Avg, and GloVe Avg, we compare our method with word mover's distance (WMD) and cosine similarity between skip-thought embeddings (ST Cos) . propose to weight the word w in each sentence according to α α+p(w), where α is a constant and p(w) is the probability of seeing the word w in the corpus. Following its recommendation, we set α to be 10 −4 in STS benchmark. After the weighting, we adopt the post-processing method from to remove the first principal component that is estimated using the training distribution and denote the method as GloVe SIF. The post-processing is not desired in some applications , so we also report the performance before removing principal components, which is called GloVe Prob_avg. The strong performance of (weighted) average embedding suggests that we should consider the embeddings of words in the sentence in addition to the sentence embedding(s) when measuring the sentence similarity. This is hard to be achieved in other sentence representation methods because their sentence embeddings and word embeddings are in different semantic spaces. Since our multi-facet embeddings are in the same space of word embeddings, we can use the multifacet embeddings to estimate the word importance (in terms of predicting possible co-occurring words beside the sentence). To compute the importance of a word in the sentence, we first compute the cosine similarity between the word and all predicted codebook embeddings, truncate the negative similarity to 0, and sum all similarity. Specifically, our simple importance/attention weighting for all the words in the query sentence S q is defined by where 1 is an all-one vector. The importance weighting is multiplied with the original weighting vectors on GloVe Avg (uniform weight), GloVe Prob_avg, and GloVe SIF to generate the of Our Avg, Our Prob_avg, and Our SIF, respectively. We compare all the in the development set and test set in Table 3. Ours SC, which matches between two sets of topics outperforms WMD, which matches between two sets of words in the sentence, and also outperforms BERT Avg, especially in STSB Low. All the scores in Ours (K=10) are significantly better than Ours (K=1),which demonstrates the benefits of multi-mode representation. Multiplying the proposed attention weighting boosts the performance from (weighted) averagingbased methods especially in STSB Low and when we do not rely on the generalization assumption of our training distribution. We also test a variant of our method which uses a bi-LSTM as the encoder and a LSTM as the decoder. Its average validation loss (-0.1289) in equation 2 is significantly higher than that of the transformer alternative (-0.1439), and it performances on Table 3 are also worse. Notice the architecture of this variant is similar to skip-thoughts except that skip-thoughts decodes a sequence instead of a set. The variant significantly outperforms ST Cos, which further justifies our approach of ignoring the order of co-occurring words in our NNSC loss. We apply our model to HypeNet , an unsupervised hypernymy detection dataset, based on an assumption that the co-occurring words of a phrase are often less related to some of its hyponyms. For instance, fly is a co-occurring word of animal which is less related to brown dog. Thus, the predicted codebook embeddings of a hyponym S hypo q (e.g., brown dog), which cluster the embeddings of co-occurring words (e.g., eats), often reconstruct the embeddings of its hypernym S hyper q (e.g., animal) better than the other way around (i.e., Er( Based on the assumption, our asymmetric scoring function is defined as The AUC of detecting hypernym among other relations and accuracy of detecting the hypernym direction are compared in Table 4 . Our methods outperform baselines, which only provide symmetric similarity measurement, and Ours (K=1) performs similarly compared with Ours (K=10). A good summary should cover multiple aspects that well represent all topics/concepts in the whole document. The objective can be quantified as discovering a summary A with a set of normalized embeddings C(A) which best reconstructs the distribution of normalized word embedding w in the document D . That is, where γ w is the importance of the word w, which is set as α α+p(w) as we did in Section 3.2. We further assume that the summary A consists of T sentences A 1... A T and the embedding set of the summary is the union of the embedding sets of the sentences C(A) = ∪ T t=1 C(A t), and we greedily select sentences to optimize equation 8 as did in. This extractive summarization method provides us a way to evaluate the embedding(s) of sentence. Our model can generate multiple codebook embeddings, which capture its different aspects as we see in Table 1, to represent each sentence in the document, so we let C(A t) = {F u (A t)}, a set of column vectors in the matrix F u (A t). We compare our approach with other alternative ways of modeling the aspects of sentences. For example, we can compute average word embeddings as a single-aspect sentence embedding. This baseline is labeled as Sent Emb. We can also use the embedding of all the words in the sentences as different aspects of the sentences. Since longer sentences have more words, we normalize the gain of the reconstruction loss by the sentence length. The method is denoted as W Emb. In contrast, the fixed number of codebook embeddings in our method avoids the problem. We also test the baselines of selecting random sentences (Rnd) and first n sentences (Lead) in the document. The on the testing set of CNN/Daily Mail are compared using F1 of ROUGE in Table 5. R-1, R-2, and Len mean ROUGE-1, ROUGE-2, and average summary length, respectively. All methods choose 3 sentences by following the setting in. Unsup, No Order means the methods do not use the sentence position information in the documents. In CNN/Daily Mail or other English news corpora, the sentence order information is a very strong signal. For example, the unsupervised methods such as Lead-3 are very strong baselines with performances similar to supervised methods such as RL , a state-of-the-art approach in this evaluation. To evaluate the quality of unsupervised sentence embeddings, we focus on comparing the unsupervised methods which do not assume the first few sentences form a good summary. In Table 5, predicting more aspects (i.e., using higher cluster numbers K) yields better , and setting K = 100 gives us the best performance after selecting 3 sentences. This demonstrates that larger cluster numbers K is desired in this application. Our method allows us to set K to be a relatively large number because of greatly alleviating the computational and sample efficiency challenges. Topic modeling has been extensively studied and widely applied due to its interpretability and flexibility of incorporating different forms of input features .; demonstrate that neural networks could be applied to discover semantically coherent topics. However, instead of optimizing a global topic model, our goal is to jointly and efficiently discovering different sets of topics/clusters on the small subsets of words that co-occur with target phrases or sentences. Sparse coding on word embedding space is used to model the multiple aspects of a word , and parameterizing word embeddings using neural networks is used to test hypothesis and save storage space . Besides, to capture asymmetric relations such as entailment, words are represented as single or multiple regions in Gaussian embeddings rather than a single point. However, the challenges of extending these methods to longer sequences are not addressed in these studies. One of our main challenges is to design a neural decoder for a set rather a sequence while modeling the dependency between the elements. This requires a matching step between two sets and compute the distance loss after the matching . One popular loss is called Chamfer distance, which is widely adopted in the auto-encoder models for point clouds , while more sophisticated matching loss options are also proposed . The goal of the studies focus on measuring symmetric distances between the ground truth set and predicted set (usually with the equal size), while our set decoder tries to reconstruct a set using a set of much fewer bases. Other ways to achieving the permutation invariants loss for neural networks includes removing the elements in the ground truth set which have been predicted , beam search , or predicting the permutation using a CNN , a transformer or reinforcement learning . In contrast, our goal is to efficiently predict a set of clustering centers that can well reconstruct the set of observed instances instead of predicting the set or sequence of the observed instances. In this work, we overcome the computational and sampling efficiency challenges of learning the multi-mode representation for long sequences like phrases or sentences. We use a neural encoder to model the compositional meaning of the target sequence and use a neural decoder to predict a set of codebook embeddings as the representation of the sentences or phrases. During training, we use a non-negative sparse coefficient matrix to dynamically match the predicted codebook embeddings to a set of observed co-occurring words and allow the neural decoder to predict the clustering centers with an arbitrary permutation. We demonstrate that the proposed models can learn to predict interpretable clustering centers conditioned on an (unseen) sequence, and the representation outperforms widely-used baselines such as BERT, skip-thoughts and various approaches based on GloVe in several unsupervised benchmarks. The experimental also suggest that multi-facet embeddings perform the best when the input sequence (e.g., a sentence) involves many aspects, while multi-facet and single-facet embeddings perform similarly good when the input sequence (e.g., a phrase) usually involves only one aspect. In the future, we would like to train a single model which could generate multi-facet embeddings for both phrases and sentences, and evaluate the method as a pre-trained embedding approach for supervised or semi-supervised settings. Furthermore, we plan to apply this method to other unsupervised learning tasks that heavily rely on co-occurrence statistics such as graph embedding or recommendation. Given the computational resource constraints, we keep our model simple enough to have a nearly converged training loss after 1 or 2 epoch(s). Since training takes a long time, we do not fine-tune the hyper-parameters in our models. We use a much smaller model compared with BERT but the architecture details in our transformer and most of its hyper-parameters are the same as the ones used in BERT. The sparsity penalty weights on coefficient matrix λ is set to be 0.4. The maximal sentence size is set to be 50 and we ignore the sentences longer than that. The maximal number of co-occurring words is set to be 30 (after removing the stop words), and we sub-sample the words if there are more words in the previous and next sentence. The number of dimensions in transformers is set to be 300. For sentence representation, the number of transformer layers on the decoder side is 5 and dropout on attention is 0.1 for K = 10 and the number of transformer layer on the decoder side is set to be 1 for K = 1 because we do not need to model the dependency of output basis. For phrase representation, the number of transformer layers on the decoder side is 2 and the dropout on attention is 0.5. The window size d t is set to be 5. We will release the code to reveal more hyper-parameter setting details. All the architecture and hyperparameters (except the number of codebook embeddings) in our models are determined by the validation loss of the self-supervised co-occurring word reconstruction task. The number of codebook embeddings K is chosen by the performance of training data in each task, but we observe that the performances are usually not sensitive to the numbers as long as K is large enough. We also suspect that the slight performance drops of models with too large K might just be caused by the fact that larger K needs longer training time and 1 week of training is insufficient to make the model converges. For skip-thoughts, the hidden embedding of is set to be 600. To make the comparison fair, we retrain the skip-thoughts in Wikipedia 2016 for 2 weeks. As mentioned in Section 3, our model has fewer parameters than the BERT base model and uses much less computational resources for training, so we only present the BERT base performance in the experiment sections. Nevertheless, we still wonder how well BERT large can perform in these unsupervised semantic tasks, so we compare our method with BERT Large in Table 6, Table 7, and Table 8. As we can see, BERT large is usually better than BERT base in the similarity tasks, but perform worse in the hypernym detection task. The performance gains of BERT in similarity tasks might imply that training a larger version of our model might be a promising future direction. Although increasing the model size boosts the performance of BERT, our method is still much better in most of the cases, especially in the phrase similarity tasks. One of the main reasons we hypothesize is that BERT is trained by predicting the masked words in the input sequence and the objective function might not be good if sequences are short (like phrases). C SUMMARIZATION COMPARISON GIVEN THE SAME SUMMARY LENGTH In Section 3.4, we compare our methods with other baselines when all the methods choose the same number of sentences. We suspect that the bad performances for W Emb (*) methods (i.e., representing each sentence using the embedding of words in the sentence) might come from the tendency of selecting shorter sentences. 6 To verify the hypothesis, we plot the R-1 performance of different unsupervised summarization methods that do not use sentence order information versus the sentence length in Figure 3. In the figure, we first observe that Ours (K=100) significantly outperforms W Emb (GloVe) and Sent Emb (GloVe) when summaries have similar length. In addition, we find that W Emb (*) actually usually outperform Sent Emb (*) when comparing the summaries with a similar length. Notice that this comparison might not be fair because W Emb (*) are allowed to select more sentences given the same length of summary and it might be easier to cover more topics in the document using more sentences. In practice, preventing choosing many short sentences might be preferable in an extractive summarization if fluency is an important factor. Nevertheless, if our goal is simply to maximize the ROUGE F1 score given a fixed length of summary without accessing the ground truth summary and sentence order information, the figure indicates that Ours (K=100) is the best choice when the summary length is less than around 50 words and W Emb (BERT) becomes the best method for a longer summary. The BERT in this figure is the BERT base model. The mixed suggest that combining our method with BERT in a way might be a promising direction to get the best performance in this task (e.g., use contextualized word embedding from BERT as our pre-trained word embedding space). We visualize predicted embeddings from 10 randomly selected sentences in our validation set. The format of the file is similar to Table 1. The first line of an example is always the prepossessed input sentence, where <unk> means an out-of-vocabulary placeholder. The embedding in each row is visualized by the nearest five neighbors in a GloVe embedding space and their cosine similarities to the vector.

represent this text to find a much much longer article about it with say hundreds-thousands of words.

We propose an unsupervised way to learn multiple embeddings for sentences and phrases

Represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We present a meta-learning approach for adaptive text-to-speech (TTS) with few data. During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker. The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system. Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers. We introduce and benchmark three strategies: (i) learning the speaker embedding while keeping the WaveNet core fixed, (ii) fine-tuning the entire architecture with stochastic gradient descent, and (iii) predicting the speaker embedding with a trained neural network encoder. The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers. Training a large model with lots of data and subsequently deploying this model to carry out classification or regression is an important and common methodology in machine learning. It has been particularly successful in speech recognition, machine translation BID1 and image recognition BID2 BID3. In this textto-speech (TTS) work, we are instead interested in few-shot meta-learning. Here the objective of training with many data is not to learn a fixed-parameter classifier, but rather to learn a "prior" neural network. This prior TTS network can be adapted rapidly, using few data, to produce TTS systems for new speakers at deployment time. That is, the intention is not to learn a fixed final model, but rather to learn a model prior that harnesses few data at deployment time to learn new behaviours rapidly. The output of training is not longer a fixed model, but rather a fast learner. Biology provides motivation for this line of research. It may be argued that evolution is a slow adaptation process that has ed in biological machines with the ability to adapt rapidly to new data during their lifetimes. These machines are born with strong priors that facilitate rapid learning. We consider a meta-learning approach where the model has two types of parameters: task-dependent parameters and task-independent parameters. During training, we learn all of these parameters but discard the task-dependent parameters for deployment. The goal is to use few data to learn the task-dependent parameters for new tasks rapidly. Task-dependent parameters play a similar role to latent variables in classical probabilistic graphical models. Intuitively, these variables introduce flexibility, thus making it easier to learn the taskindependent parameters. For example, in classical HMMs, knowing the latent variables in a simple learning problem of estimating the parameters of an exponential-family distribution. In neural networks, this approach also facilitates learning when there is clear data diversity and categorization. We show this for adaptive TTS BID4 BID5. In this setting, speakers correspond to tasks. During training we have many speakers, and it is therefore helpful to have task-dependent parameters to capture speaker-specific voice styles. At the same time, it is useful to have a large model with shared parameters to capture the generic process of mapping text to speech. To this end, we employ the WaveNet model. WaveNet BID6 is an autoregressive generative model for audio waveforms that has yielded state-of-art performance in speech synthesis. This model was later modified for real-time speech generation via probability density distillation into a feed-forward model BID7. A fundamental limitation of WaveNet is the need for hours of training data for each speaker. In this paper we describe a new WaveNet training procedure that facilitates adaptation to new speakers, allowing the synthesis of new voices from no more than 10 minutes of data with high sample quality. We propose several extensions of WaveNet for sample-efficient adaptive TTS. First, we present two non-parametric adaptation methods that involve fine-tuning either the speaker embeddings only or all the model parameters given few data from a new speaker. Second, we present a parametric textindependent approach whereby an auxiliary network is trained to predict new speaker embeddings. The experiments will show that all the proposed approaches, when provided with just a few seconds or minutes of recording, can generate high-fidelity utterances that closely resemble the vocal tract characteristics of a demonstration speaker, particularly when the entire model is fine-tuned end-to-end. When fine-tuning by first estimating the speaker embedding and subsequently fine-tuning the entire model, we achieve state-of-the-art in terms of sample naturalness and voice similarity to target speakers. These are robust across speech datasets recorded under different conditions and, moreover, we demonstrate that the generated samples are capable of confusing the state-of-the-art text-independent speaker verification system BID8.TTS techniques require hours of high-quality recordings, collected in controlled environments, for each new voice style. Given this high cost, reducing the length of the training dataset could be valuable. For example, it is likely to be very beneficial when attempting to restore the voices of patients who suffer from voice-impairing medical conditions. In these cases, long high quality recordings are scarce. WaveNet is an autoregressive model that factorizes the joint probability distribution of a waveform, x = {x 1, . . ., x T}, into a product of conditional distributions using the probabilistic chain rule: DISPLAYFORM0 p(x t |x 1:t−1, h; w), where x t is the t-th timestep sample, and h and w are respectively the conditioning inputs and parameters of the model. To train a multi-speaker WaveNet, the conditioning inputs h consist of the speaker identity s, the linguistic features l, and the logarithmic fundamental frequency f 0 values. l encodes the sequence of phonemes derived from the input text, and f 0 controls the dynamics of the pitch in the generated utterance. Given the speaker identity s for each utterance in the dataset, the Figure 2: Training (slow, lots of data), adaptation (fast, few data) and inference stages for the SEA-ALL architecture. The components with bold pink outlines are fine-tuned during the adaptation phase. The purpose of training is to produce a prior. This prior is combined with few data during adaptation to solve a new task. This adapted model is then deployed in the final inference stage. model is expressed as: DISPLAYFORM1 where a table of speaker embedding vectors e s (Embedding in FIG0) is learned alongside the standard WaveNet parameters. These vectors capture salient voice characteristics across individual speakers, and provide a convenient mechanism for generalizing WaveNet to the few-shot adaptation setting in this paper. The linguistic features l and fundamental frequency values f 0 are both time-series with a lower sampling frequency than the waveform. Thus, to be used as local conditioning variables they are upsampled by a transposed convolutional network. During training, l and f 0 are extracted by signal processing methods from pairs of training utterance and transcript, and during testing, those values are predicted from text by existing models. In recent years, a large body of literature uses large datasets to train models to learn an input-output mapping that is then used for inference. In contrast, few-shot meta-learning introduces an additional step, adaptation. In this meta-learning setting, the purpose of training becomes to learn a prior. During adaptation, this prior is combined with few data to rapidly learn a new skill; in this case adapting to a new speakers' voice style. Finally, the new skill is deployed, which in this paper we are referring to as inference. These three stages -training, adaptation and inference -are illustrated in Figure 2.We present two multi-speaker WaveNet extensions for few-shot voice adaptation. First, we introduce a non-parametric model fine-tuning approach, which involves adapting either the speaker embeddings or all the model parameters using held-aside demonstration data. Second, and for comparison purposes, we use a parametric approach whereby an auxiliary network is trained to predict the embedding vector of a new speaker using the demonstration data. Inspired by few-shot learning we first pre-train a multi-speaker conditional WaveNet model on a large and diverse dataset, as described in Section 2. Subsequently, we fine-tune the model parameters by retraining with respect to held-aside adaptation data. Training this WaveNet model to maximize the conditional log-likelihood of the generated audio jointly optimizes both the set of speaker parameters {e s} and the shared WaveNet core parameters w. Next, we extend this method to a new speaker by extracting the l and f 0 features from their adaptation data and randomly initializing a new embedding vector e. We then optimize e such that the demonstration waveforms, {x DISPLAYFORM0 0,demo)}, are likely under the model with w fixed (SEA-EMB): DISPLAYFORM1 Alternatively, all of the model parameters may be additionally fine-tuned (SEA-ALL): DISPLAYFORM2 0,demo; e, w).Both methods are non-parametric approaches to few-shot voice adaptation as the number of embedding vectors scales with the number of speakers. However, the training processes are slightly different. Because the SEA-EMB method optimizes only a low-dimensional vector, it is far less prone to overfitting, and we are therefore able to retrain the model to convergence even with mere seconds of adaptation data. By contrast, the SEA-ALL has many more parameters that might overfit to the adaptation data. We therefore hold out 10% of our demonstration data for calculating a standard early termination criterion. We also initialize e with the optimal value from the SEA-EMB method, and we find this initialization significantly improves the generalization performance even with a few seconds of adaptation data. In contrast to the non-parametric approach, whereby a different embedding vector is fitted for each speaker, one can train an auxiliary encoder network to predict an embedding vector for a new speaker given their demonstration data. Specifically, we model: DISPLAYFORM0 where for each training example, we include a randomly selected demonstration utterance from that speaker in addition to the regular conditioning inputs. The full WaveNet model and the encoder network e(·) are trained together from scratch. We refer the reader to the Appendix for further architectural details. This approach (SEA-ENC) exhibits the advantage of being trained in a transcriptindependent setting given only the input waveform, e(x demo), and requires negligible computation at adaptation time. However, the learned encoder can also introduce bias when fitting an embedding due to its limited network capacity. As an example, demonstrated a typical scenario whereby speaker identity information can be very quickly extracted with deep models from audio signals. Nonetheless, that the model is less capable of effectively leveraging additional training than approaches based on statistical methods. The linguistic features and fundamental frequencies which are used as inputs contain information specific to an individual speaker. As an example, the average voice pitch in the fundamental frequency sequence is highly speaker-dependent. Instead, we would like these features to be as speaker-independent as possible such that identity is modeled via global conditioning on the speaker embedding. To achieve this, we normalize the fundamental frequency values to have zero mean and unit variance separately for each speaker during training, denoted asf 0: DISPLAYFORM0 As mentioned earlier, at test time, we use an existing model to predict (l,f 0). Few-shot learning to build models, where one can rapidly learn using only a small amount of available data, is one of the most important open challenges in machine learning. Recent studies have attempted to address the problem of few-shot learning by using deep neural networks, and they have shown promising on classification tasks in vision BID11 BID12 and language. Few-shot learning can also be leveraged in reinforcement learning, such as by imitating human Atari gameplay from a single recorded action sequence BID14 or online video BID15.Meta-learning offers a sound framework for addressing few-shot learning. Here, an expensive learning process in machines with the ability to learn rapidly from few data. Meta-learning has a long history BID16 BID17, and recent studies include efforts to learn optimization processes BID18 that have been shown to extend naturally to the few-shot setting BID20 ). An alternative approach is model-agnostic meta learning (MAML) BID21, which differs by using a fixed optimizer and learning a set of base parameters that can be adapted to minimize any task loss by few steps of gradient descent. This method has shown promise in robotics BID22 BID23.In generative modeling, few-shot learning has been addressed from several perspectives, including matching networks BID24 and variable inference for memory addressing BID25. BID26 developed a sequential generative model that extended the Deep Recurrent Attention Writer (DRAW) model BID27, and BID28 extended PixelCNN with neural attention for few-shot auto-regressive density modeling. BID30 presented a gated linear model able to model complex densities from a single pass of a limited dataset. Early attempts of few-shot adaptation involved the attention models of BID28 and MAML BID21 ), but we found both of these strategies failed to learn informative speaker embedding in our preliminary experiments. There is growing interest in developing neural TTS models that can be trained end-to-end without the need for hand-crafted representations. In this study we focus on extending the autoregressive WaveNet model BID6 to the few-shot learning setting to adapt to speakers that were not presented at training time. Other recent neural TTS models include Tacotron 2 (building on) which uses WaveNet as a vocoder to invert mel-spectrograms generated by an attentive sequence-to-sequence model. DeepVoice 2 ) (building on BID34) introduced a multi-speaker variation of Tacotron that learns a low-dimensional embedding for each speaker, which was further extended in DeepVoice 3 to a 2,400 multi-speaker scenario. Unlike WaveNet and DeepVoice, the Char2Wav BID36 and VoiceLoop ) models produce World Vocoder Features BID38 instead of generating raw audio signals. Although many of these systems have produced high-quality samples for speakers present in the training set, generalizing to new speakers given only a few seconds of audio remains a challenge. There have been several concurrent works to address this few-shot learning problem. The VoiceLoop model introduced a novel memory-based architecture that was extended by to few-shot voice style adaptation, by introducing an auxiliary fitting network that predicts the embedding of a new speaker. BID40 extended the Tacotron model for one-shot speaker adaptation by conditioning on a speaker embedding vector extracted from a pretrained speaker identity model of BID8. The most similar approached to our work was proposed by for the DeepVoice 3 model. They considered both predicting the embedding with an encoding network and fitting the embedding based on a small amount of adaptation data, but the adaptation was applied to a prediction model for mel-spectrograms with a fixed vocoder. In this section, we evaluate the quality of samples of SEA-ALL, SEA-EMB and SEA-ENC. We first measure the naturalness of the generated utterances using the standard Mean Opinion Score (MOS) procedure. Then, we evaluate the similarity of generated and real samples using the subjective MOS test and objectively using a speaker verification system BID8. Finally, we study these varying the size of the adaptation dataset. We train a WaveNet model for each of our three methods using the same dataset, which combines the high-quality LibriSpeech audiobook corpus BID42 3.61 ± 0.06 3.56 ± 0.06 3.65 ± 0.06 3.58 ± 0.06 Table 1: Naturalness of the adapted voices using a 5-scale MOS score (higher is better) with 95% confidence interval on the LibriSpeech and VCTK held-out adaptation datasets. Numbers in bold are the best few-shot learning on each dataset without statistically significant difference. van den was trained with 24-hour production quality data, Our few-shot model performance is evaluated using two hold-out datasets. First, the LibriSpeech test corpus consists of 39 speakers, with an average of approximately 52 utterances and 5 minutes of audio per speaker. For every test speaker, we randomly split their demonstration utterances into an adaptation set for adapting our WaveNet models and a test set for evaluation. The subset of utterances used for early termination in Section 3.1 is chosen from the adaptation set. There are about 4.2 utterances on average per speaker in the test set and the rest in the adaptation set. Second, we consider a subset of the CSTR VCTK corpus BID43 consisting of 21 American English speakers, with approximately 368 utterances and 12 minutes of audio per speaker. We also apply the adaptation/test split with 10 utterances per speaker for test. We emphasize that no data from VCTK was presented to the model at training time. Since our underlying WaveNet model was trained on data largely from LibriSpeech (which was recorded under noisier conditions than VCTK), one might expect that the generated samples on the VCTK dataset contain characteristic artifacts that make generated samples easier to distinguish from real utterances. However, our evaluation using VCTK indicates that our model generalizes effectively and that such artifacts are not detectable. Synthetic utterances are provided on our demo webpage 1.It is worth mentioning, that SEA-ENC requires no adaptation time. Where for SEA-EMB, it takes 5 ∼ 10k optimizing steps to fit the embedding vector, and an additional 100 ∼ 200 steps to fine-tune the entire model using early stopping for SEA-ALL. We measure the quality of the generated samples by conducting a MOS test, whereby subjects are asked to rate the naturalness of generated utterances on a five-point Likert Scale (1: Bad, 2: Poor, 3: Fair, 4: Good, 5: Excellent). Furthermore, we compare with other published few-shot TTS systems systems, that were developed in parallel to this work. However, the literature uses varying combinations of training data and evaluation splits making comparison difficult. The presented are from the closest experimental setups to ours. Table 1 presents MOS for the adaptation models compared to real utterances. Two different adaptation dataset sizes are considered; T = 10 seconds, and T ≤ 5 minutes for LibriSpeech (T ≤ 10 minutes for VCTK). For reference on 16 kHz data, WaveNet trained on a 24-hour production quality speech dataset (van den) achieves a score of 4.21, while for LibriSpeech our best few-shot model attains an MOS score of 4.13 using only 5 minutes of data given a pre-trained multi-speaker model. We note that both fine-tuning models produce overall "good" samples for both the LibriSpeech and VCTK test sets, with SEA-ALL outperforming SEA-EMB in all cases. SEA-ALL is on par 3.41 ± 0.10 3.75 ± 0.09 3.51 ± 0.10 3.97 ± 0.09 SEA-EMB (ours)3.42 ± 0.10 3.56 ± 0.10 3.07 ± 0.10 3.18 ± 0.10 SEA-ENC (ours)2.47 ± 0.09 2.59 ± 0.09 2.07 ± 0.08 2.19 ± 0.09 Table 2: Voice similarity of generated voices using a 5-scale MOS score (higher is better) with 95% confidence interval on the LibriSpeech and VCTK held-out adaptation datasets.with the state-of-the-art performance on both datasets. The addition of extra adaptation data beyond 10 seconds of audio helps performance on LibriSpeech but not VCTK, and the gap between our best model and the real utterance is also wider on VCTK, possibly due to the different recording conditions. Beside naturalness, we also measure the similarity of the generated and real voices. The quality of similarity is the main evaluation metric for the voice adaptation problem. We first follow the experiment setup of BID40 to run a MOS test for a subjective assessment and then use a speaker verification model for objective evaluation in the next section. In every trial of this test a subject is presented with a pair of utterances consisting of a real utterance and another real or generated utterance from the same speaker, and is asked to rate the similarity in voice identity using a five-scale score (1: Not at all similar, 2: Slightly similar, 3: Moderately similar, 4: Very similar, 5: Extremely similar). Table 2 shows the MOS for real utterances and all the adaptation models under two adaptation data time settings on both datasets. Again, the SEA-ALL model outperforms the other two models, and the improvement over SEA-EMB scales with the amount of adaptation data. Particularly, the learned voices on the VCTK dataset achieve an average score of 3.97, demonstrating the generalization performance on a different dataset. As a rough comparisson, because of varying training setups, the state of the art system of BID40 achieves scores of 3.03 for LibriSpeech and 2.77 for VCTK when trained on LibriSpeech. Their model computes the embedding based on the d-vector, similar to our SEA-ENC approach, and performs competitively for the one-shot learning setting, but its performance saturates with 5 seconds of adaptation data, as explained in Section 3.2. We note the gap of similarity scores between SEA-ALL and real utterances, which suggests that although the generated samples sound similar to the target speakers, humans can still tell the difference from real utterances. We also apply the state-of-the-art text independent speaker verification (TI-SV) model of BID8 to objectively assess whether the generated samples preserve the acoustic features of the speakers. We calculate the TI-SV d-vector embeddings for generated and real voices. In Figure 3, we visualize the 2-dimensional projection of the d-vectors for a SEA-ALL model trained on T ≤ 5 minutes of data on the LibriSpeech dataset, and T ≤ 10 minutes on VCTK. There are clear clusters on both datasets, with a strikingly large inter-cluster distance and low intra-cluster separation. This shows both an ease of correctly identifying the speaker associated with a given generated utterance, and the difficulty in differentiating real from synthetic samples. A similar figure is presented in BID40, but there the generated and real samples do not overlap. This indicates that the method presented in this paper generates voices that are more indistinguishable from real ones, when measured with the same verification system. In the following subsections, we further analyze these . BID8. The utterances were generated using T ≤ 5 and T ≤ 10 minute samples from LibriSpeech and VCTK respectively. EER is marked with a dot. We first quantify whether generated utterances are attributed to the correct speaker. Following common practice in speaker verification BID8, we select the hold-out test set of real utterances from test speakers as the enrollment set and compute the centroid of the d-vectors for each speaker c i. We then use the adaptation set of test speakers as the verification set. For every verification utterance, we compute the cosine similarity between its d-vector v and a randomly chosen centroid c i. The utterance is accepted as one from speaker i if the similarity is exceeds a given threshold. We repeat the experiments with the same enrollment set and replace the verification set with samples generated by each adaptation method under different data size settings. Figure 5: Cosine similarity of real and generated utterances to the real enrollment set. In our setup we fix the enrollment set together with the speaker verification model from BID8, and study the performance of different verification sets that are either from real utterances or generated by a TTS system. TAB4 lists the equal error rate (EER) of the verification model with real and generated verification utterances, and Figure 4 shows the detection error trade-off (DET) curves for a more thorough inspection. Figure 4 only shows the adaptation models with the maximum data size setting (T ≤ 5 minutes for LibriSpeech and ≤ 10 minutes for VCTK). The for other data sizes are provided in Appendix B.We find that SEA-ALL outperforms the other two approaches, and the error rate decreases clearly with the size of demonstration data. Noticeably, the EER of SEA-ALL is even lower than the real utterance on the LibriSpeech dataset with sufficient adaptation data. A possible explanation is that the generated samples might be concentrated closer to the centroid of a speaker's embeddings than real speech with larger variance across utterances. Our SEA-EMB model performs better than SEA-ENC. Additionally, the benefit of more demonstration data is less significant than for SEA-ALL in both of these models. In this section, we compare the generated samples and the real utterances of the speaker being imitated. Figure 5 shows the box-plot of the cosine similarity between the embedding centroids of test speakers' enrollment set and real utterances from the same speaker, real utterances from a different speaker, and generated utterances adapted to the same speaker. Consistent with the observations from the previous subsection, SEA-ALL performs best. We further consider an adversarial scenario for speaker verification. In contrast to the previous standard speaker verification setup where we now select a verification utterance with either a real utterance from the same speaker or a synthetic sample from a model adapted to the same speaker. Under this setup, the speaker verification system is challenged by synthetic samples and acts as a classifier for real versus generated utterances. The ROC curve of this setup is shown in FIG3 and the models are using the maximum data size setting. Other data size settings can be found in Appendix C. If the generated samples are indistinguishable from real utterances, the ROC curve approaches the diagonal line (that is, the verification system fails to separate real and generated voices). Importantly, SEA-ALL manages to confuse the verification system especially for the VCTK dataset where the ROC curve is almost inline with the diagonal line with an AUC of 0.56. This paper studied three variants of meta-learning for sample efficient adaptive TTS. The adaptation method that fine-tunes the entire model, with the speaker embedding vector first optimized, shows impressive performance even with only 10 seconds of audio from new speakers. When adapted with a few minutes of data, our model matches the state-of-the-art performance in sample naturalness. Moreover, it outperforms other recent works in matching the new speaker's voice. We also demon-: ROC curve for real versus generated utterance detection. The utterances were generated using models with 5 and 10 minutes of training data per speaker from LibriSpeech and VCTK respectively. Lower curve indicate that the verification system is having a harder time distinguishing real from generated samples.strated that the generated samples achieved a similar level of voice similarity to real utterances from the same speaker, when measured by a text independent speaker verification model. Our paper considers the adaptation to new voices with clean, high-quality training data collected in a controlled environment. The few-shot learning of voices with noisy data is beyond the scope of this paper and remains a challenging open research problem. A requirement for less training data to adapt the model, however, increases the potential for both beneficial and harmful applications of text-to-speech technologies such as the creation of synthesized media. While the requirements for this particular model (including the high-quality training data collected in a controlled environment and equally high quality data from the speakers to which we adapt, as described in Section 5.1) present barriers to misuse, more research must be conducted to mitigate and detect instances of misuse of text-to-speech technologies in general. Our encoding network is illustrated as the summation of two sub-network outputs in FIG4. The first subnetwork is a pre-trained speaker verification model (TI-SV) BID8, comprising 3 LSTM layers and a single linear layer. This model maps a waveform sequence of arbitrary length to a fixed 256-dimensional d-vector with a sliding window, and is trained from approximately 36M utterances from 18K speakers extracted from anonymized voice search logs. On top of this we add a shallow MLP to project the output d-vector to the speaker embedding space. The second sub-network comprises 16 1-D convolutional layers. This network reduces the temporal resolution to 256 ms per frame (for 16 kHz audio), then averages across time and projects into the speaker embedding space. The purpose of this network is to extract residual speaker information present in the demonstration waveforms but not captured by the pre-trained TI-SV model. Here we provide the DET curves of speaker verification problem for models with different training data sizes in addition to those shown in Section 5.4.1. Figure 8: Detection error trade-off (DET) curve for speaker verification, using the TI-SV speaker verification model BID8. The utterances were generated using 1 minute or 10 seconds of utterance from LibriSpeech and VCTK. EER is marked with a dot. We provide the ROC curves of the speaker verification problem with adversarial examples from adaptation models with different training data sizes in addition to those shown in Section 5.4.2.: ROC curve for real vs. generated utterance detection. The utterances were generated using 1 minute or 10 seconds of utterance from LibriSpeech and VCTK. Lower curve suggests harder to distinguish real from generated samples.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

Sample efficient algorithms to adapt a text-to-speech model to a new voice style with the state-of-the-art performance.

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.