Question: represent the text to find a very short summary (1-2 sentences max)

Question: State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data supervision can be alleviated with character-level information. While these methods showed encouraging , they are not on par with their supervised counterparts and are limited to pairs of languages sharing a common alphabet. In this work, we show that we can build a bilingual dictionary between two languages without using any parallel corpora, by aligning monolingual word embedding spaces in an unsupervised way. Without using any character information, our model even outperforms existing supervised methods on cross-lingual tasks for some language pairs. Our experiments demonstrate that our method works very well also for distant language pairs, like English-Russian or English-Chinese. We finally describe experiments on the English-Esperanto low-resource language pair, on which there only exists a limited amount of parallel data, to show the potential impact of our method in fully unsupervised machine translation. Our code, embeddings and dictionaries are publicly available. Most successful methods for learning distributed representations of words (e.g. BID32 a); BID34; ) rely on the distributional hypothesis of BID20, which states that words occurring in similar contexts tend to have similar meanings. BID28 show that the skip-gram with negative sampling method of BID32 amounts to factorizing a word-context co-occurrence matrix, whose entries are the pointwise mutual information of the respective word and context pairs. Exploiting word cooccurrence statistics leads to word vectors that reflect the semantic similarities and dissimilarities: similar words are close in the embedding space and conversely. BID31 first noticed that continuous word embedding spaces exhibit similar structures across languages, even when considering distant language pairs like English and Vietnamese. They proposed to exploit this similarity by learning a linear mapping from a source to a target embedding space. They employed a parallel vocabulary of five thousand words as anchor points to learn this mapping and evaluated their approach on a word translation task. Since then, several studies aimed at improving these cross-lingual word embeddings BID12; BID47;; BID0; BID1; BID43 ), but they all rely on bilingual word lexicons. Recent attempts at reducing the need for bilingual supervision BID43 employ identical character strings to form a parallel vocabulary. The iterative method of BID2 gradually aligns embedding spaces, starting from a parallel vocabulary of aligned digits. These methods are however limited to similar languages sharing a common alphabet, such as European languages. Some recent methods explored distribution-based approach BID7 or adversarial training BID50 to obtain cross-lingual word embeddings without any parallel data. While these approaches sound appealing, their performance is significantly below supervised methods. To sum up, current methods have either not reached competitive performance, or they still require parallel data, such as aligned corpora BID18 BID46 or a seed parallel lexicon BID11.In this paper, we introduce a model that either is on par, or outperforms supervised state-of-the-art methods, without employing any cross-lingual annotated data. We only use two large monolingual corpora, one in the source and one in the target language. Our method leverages adversarial training to learn a linear mapping from a source to a target space and operates in two steps. First, in a twoplayer game, a discriminator is trained to distinguish between the mapped source embeddings and the target embeddings, while the mapping (which can be seen as a generator) is jointly trained to fool the discriminator. Second, we extract a synthetic dictionary from the ing shared embedding space and fine-tune the mapping with the closed-form Procrustes solution from BID42. Since the method is unsupervised, cross-lingual data can not be used to select the best model. To overcome this issue, we introduce an unsupervised selection metric that is highly correlated with the mapping quality and that we use both as a stopping criterion and to select the best hyper-parameters. In summary, this paper makes the following main contributions:• We present an unsupervised approach that reaches or outperforms state-of-the-art supervised approaches on several language pairs and on three different evaluation tasks, namely word translation, sentence translation retrieval, and cross-lingual word similarity. On a standard word translation retrieval benchmark, using 200k vocabularies, our method reaches 66.2% accuracy on English-Italian while the best supervised approach is at 63.7%.• We introduce a cross-domain similarity adaptation to mitigate the so-called hubness problem (points tending to be nearest neighbors of many points in high-dimensional spaces). It is inspired by the self-tuning method from BID48, but adapted to our two-domain scenario in which we must consider a bi-partite graph for neighbors. This approach significantly improves the absolute performance, and outperforms the state of the art both in supervised and unsupervised setups on word-translation benchmarks.• We propose an unsupervised criterion that is highly correlated with the quality of the mapping, that can be used both as a stopping criterion and to select the best hyper-parameters.• We release high-quality dictionaries for 12 oriented languages pairs, as well as the corresponding supervised and unsupervised word embeddings.• We demonstrate the effectiveness of our method using an example of a low-resource language pair where parallel corpora are not available (English-Esperanto) for which our method is particularly suited. The paper is organized as follows. Section 2 describes our unsupervised approach with adversarial training and our refinement procedure. We then present our training procedure with unsupervised model selection in Section 3. We report in Section 4 our on several cross-lingual tasks for several language pairs and compare our approach to supervised methods. Finally, we explain how our approach differs from recent related work on learning cross-lingual word embeddings. In this paper, we always assume that we have two sets of embeddings trained independently on monolingual data. Our work focuses on learning a mapping between the two sets such that translations are close in the shared space. BID31 show that they can exploit the similarities of monolingual embedding spaces to learn such a mapping. For this purpose, they use a known dictionary of n = 5000 pairs of words {x i, y i} i∈{1,n}, and learn a linear mapping W between the source and the target space such that DISPLAYFORM0 where d is the dimension of the embeddings, M d (R) is the space of d × d matrices of real numbers, and X and Y are two aligned matrices of size d × n containing the embeddings of the words in the parallel vocabulary. The translation t of any source word s is defined as t = argmax t cos(W x s, y t). There are two distributions of word embeddings, English words in red denoted by X and Italian words in blue denoted by Y, which we want to align/translate. Each dot represents a word in that space. The size of the dot is proportional to the frequency of the words in the training corpus of that language. (B) Using adversarial learning, we learn a rotation matrix W which roughly aligns the two distributions. The green stars are randomly selected words that are fed to the discriminator to determine whether the two word embeddings come from the same distribution. (C) The mapping W is further refined via Procrustes. This method uses frequent words aligned by the previous step as anchor points, and minimizes an energy function that corresponds to a spring system between anchor points. The refined mapping is then used to map all words in the dictionary. (D) Finally, we translate by using the mapping W and a distance metric, dubbed CSLS, that expands the space where there is high density of points (like the area around the word "cat"), so that "hubs" (like the word "cat") become less close to other word vectors than they would otherwise (compare to the same region in panel (A)).In practice, BID31 obtained better on the word translation task using a simple linear mapping, and did not observe any improvement when using more advanced strategies like multilayer neural networks. BID47 showed that these are improved by enforcing an orthogonality constraint on W. In that case, the equation boils down to the Procrustes problem, which advantageously offers a closed form solution obtained from the singular value decomposition (SVD) of Y X T: DISPLAYFORM1 In this paper, we show how to learn this mapping W without cross-lingual supervision; an illustration of the approach is given in FIG0. First, we learn an initial proxy of W by using an adversarial criterion. Then, we use the words that match the best as anchor points for Procrustes. Finally, we improve performance over less frequent words by changing the metric of the space, which leads to spread more of those points in dense regions. Next, we describe the details of each of these steps. In this section, we present our domain-adversarial approach for learning W without cross-lingual supervision. Let X = {x 1, ..., x n} and Y = {y 1, ..., y m} be two sets of n and m word embeddings coming from a source and a target language respectively. A model is trained to discriminate between elements randomly sampled from W X = {W x 1, ..., W x n} and Y. We call this model the discriminator. W is trained to prevent the discriminator from making accurate predictions. As a , this is a two-player game, where the discriminator aims at maximizing its ability to identify the origin of an embedding, and W aims at preventing the discriminator from doing so by making W X and Y as similar as possible. This approach is in line with the work of BID15, who proposed to learn latent representations invariant to the input domain, where in our case, a domain is represented by a language (source or target).Discriminator objective We refer to the discriminator parameters as θ D. We consider the probability P θ D source = 1 z that a vector z is the mapping of a source embedding (as opposed to a target embedding) according to the discriminator. The discriminator loss can be written as: DISPLAYFORM0 Mapping objective In the unsupervised setting, W is now trained so that the discriminator is unable to accurately predict the embedding origins: DISPLAYFORM1 Learning algorithm To train our model, we follow the standard training procedure of deep adversarial networks of BID17. For every input sample, the discriminator and the mapping matrix W are trained successively with stochastic gradient updates to respectively minimize L D and L W. The details of training are given in the next section. The matrix W obtained with adversarial training gives good performance (see Table 1), but the are still not on par with the supervised approach. In fact, the adversarial approach tries to align all words irrespective of their frequencies. However, rare words have embeddings that are less updated and are more likely to appear in different contexts in each corpus, which makes them harder to align. Under the assumption that the mapping is linear, it is then better to infer the global mapping using only the most frequent words as anchors. Besides, the accuracy on the most frequent word pairs is high after adversarial training. To refine our mapping, we build a synthetic parallel vocabulary using the W just learned with adversarial training. Specifically, we consider the most frequent words and retain only mutual nearest neighbors to ensure a high-quality dictionary. Subsequently, we apply the Procrustes solution in on this generated dictionary. Considering the improved solution generated with the Procrustes algorithm, it is possible to generate a more accurate dictionary and apply this method iteratively, similarly to BID2. However, given that the synthetic dictionary obtained using adversarial training is already strong, we only observe small improvements when doing more than one iteration, i.e., the improvements on the word translation task are usually below 1%. In this subsection, our motivation is to produce reliable matching pairs between two languages: we want to improve the comparison metric such that the nearest neighbor of a source word, in the target language, is more likely to have as a nearest neighbor this particular source word. Nearest neighbors are by nature asymmetric: y being a K-NN of x does not imply that x is a K-NN of y. In high-dimensional spaces BID36, this leads to a phenomenon that is detrimental to matching pairs based on a nearest neighbor rule: some vectors, dubbed hubs, are with high probability nearest neighbors of many other points, while others (anti-hubs) are not nearest neighbors of any point. This problem has been observed in different areas, from matching image features in vision BID22 to translating words in text understanding applications. Various solutions have been proposed to mitigate this issue, some being reminiscent of pre-processing already existing in spectral clustering algorithms BID48.However, most studies aiming at mitigating hubness consider a single feature distribution. In our case, we have two domains, one for each language. This particular case is taken into account by, who propose a pairing rule based on reverse ranks, and the inverted soft-max (ISF) by Smith et al. FORMULA0, which we evaluate in our experimental section. These methods are not fully satisfactory because the similarity updates are different for the words of the source and target languages. Additionally, ISF requires to cross-validate a parameter, whose estimation is noisy in an unsupervised setting where we do not have a direct cross-validation criterion. In contrast, we consider a bi-partite neighborhood graph, in which each word of a given dictionary is connected to its K nearest neighbors in the other language. We denote by N T (W x s) the neighborhood, on this bi-partite graph, associated with a mapped source word embedding W x s. All K elements of N T (W x s) are words from the target language. Similarly we denote by N S (y t) the neighborhood associated with a word t of the target language. We consider the mean similarity of a source embedding x s to its target neighborhood as DISPLAYFORM0 where cos(., .) is the cosine similarity. Likewise we denote by r S (y t) the mean similarity of a target word y t to its neighborhood. These quantities are computed for all source and target word vectors with the efficient nearest neighbors implementation by BID23. We use them to define a similarity measure CSLS(., .) between mapped source words and target words, as DISPLAYFORM1 Intuitively, this update increases the similarity associated with isolated word vectors. Conversely it decreases the ones of vectors lying in dense areas. Our experiments show that the CSLS significantly increases the accuracy for word translation retrieval, while not requiring any parameter tuning.3 TRAINING AND ARCHITECTURAL CHOICES We use unsupervised word vectors that were trained using fastText 2. These correspond to monolingual embeddings of dimension 300 trained on Wikipedia corpora; therefore, the mapping W has size 300 × 300. Words are lower-cased, and those that appear less than 5 times are discarded for training. As a post-processing step, we only select the first 200k most frequent words in our experiments. For our discriminator, we use a multilayer perceptron with two hidden layers of size 2048, and Leaky-ReLU activation functions. The input to the discriminator is corrupted with dropout noise with a rate of 0.1. As suggested by BID16, we include a smoothing coefficient s = 0.2 in the discriminator predictions. We use stochastic gradient descent with a batch size of 32, a learning rate of 0.1 and a decay of 0.95 both for the discriminator and W. We divide the learning rate by 2 every time our unsupervised validation criterion decreases. The embedding quality of rare words is generally not as good as the one of frequent words BID29 ), and we observed that feeding the discriminator with rare words had a small, but not negligible negative impact. As a , we only feed the discriminator with the 50,000 most frequent words. At each training step, the word embeddings given to the discriminator are sampled uniformly. Sampling them according to the word frequency did not have any noticeable impact on the . BID43 showed that imposing an orthogonal constraint to the linear operator led to better performance. Using an orthogonal matrix has several advantages. First, it ensures that the monolingual quality of the embeddings is preserved. Indeed, an orthogonal matrix preserves the dot product of vectors, as well as their 2 distances, and is therefore an isometry of the Euclidean space (such as a rotation). Moreover, it made the training procedure more stable in our experiments. In this work, we propose to use a simple update step to ensure that the matrix W stays close to an orthogonal matrix during training BID8 ). Specifically, we alternate the update of our model with the following update rule on the matrix W: where β = 0.01 is usually found to perform well. This method ensures that the matrix stays close to the manifold of orthogonal matrices after each update. In practice, we observe that the eigenvalues of our matrices all have a modulus close to 1, as expected. The refinement step requires to generate a new dictionary at each iteration. In order for the Procrustes solution to work well, it is best to apply it on correct word pairs. As a , we use the CSLS method described in Section 2.3 to select more accurate translation pairs in the dictionary. To increase even more the quality of the dictionary, and ensure that W is learned from correct translation pairs, we only consider mutual nearest neighbors, i.e. pairs of words that are mutually nearest neighbors of each other according to CSLS. This significantly decreases the size of the generated dictionary, but improves its accuracy, as well as the overall performance. Selecting the best model is a challenging, yet important task in the unsupervised setting, as it is not possible to use a validation set (using a validation set would mean that we possess parallel data). To address this issue, we perform model selection using an unsupervised criterion that quantifies the closeness of the source and target embedding spaces. Specifically, we consider the 10k most frequent source words, and use CSLS to generate a translation for each of them. We then compute the average cosine similarity between these deemed translations, and use this average as a validation metric. We found that this simple criterion is better correlated with the performance on the evaluation tasks than optimal transport distances such as the Wasserstein distance BID40 ). FIG1 shows the correlation between the evaluation score and this unsupervised criterion (without stabilization by learning rate shrinkage). We use it as a stopping criterion during training, and also for hyperparameter selection in all our experiments. In this section, we empirically demonstrate the effectiveness of our unsupervised approach on several benchmarks, and compare it with state-of-the-art supervised methods. We first present the cross-lingual evaluation tasks that we consider to evaluate the quality of our cross-lingual word embeddings. Then, we present our baseline model. Last, we compare our unsupervised approach to our baseline and to previous methods. In the appendix, we offer a complementary analysis on the alignment of several sets of English embeddings trained with different methods and corpora. Word translation The task considers the problem of retrieving the translation of given source words. The problem with most available bilingual dictionaries is that they are generated using online tools like Google Translate, and do not take into account the polysemy of words. Failing to capture word polysemy in the vocabulary leads to a wrong evaluation of the quality of the word embedding space. Other dictionaries are generated using phrase tables of machine translation systems, but they are very noisy or trained on relatively small parallel corpora. For this task, we create high-quality en-es es-en en-fr fr-en en-de de-en en-ru ru-en en-zh zh-en en-eo eo-en dictionaries of up to 100k pairs of words using an internal translation tool to alleviate this issue. We make these dictionaries publicly available as part of the MUSE library 3.We report on these bilingual dictionaries, as well on those released by to allow for a direct comparison with previous approaches. For each language pair, we consider 1,500 query source and 200k target words. Following standard practice, we measure how many times one of the correct translations of a source word is retrieved, and report precision@k for k = 1, 5, 10.Cross-lingual semantic word similarity We also evaluate the quality of our cross-lingual word embeddings space using word similarity tasks. This task aims at evaluating how well the cosine similarity between two words of different languages correlates with a human-labeled score. We use the SemEval 2017 competition data (Camacho-Collados et al. FORMULA0) which provides large, highquality and well-balanced datasets composed of nominal pairs that are manually scored according to a well-defined similarity scale. We report Pearson correlation. Sentence translation retrieval Going from the word to the sentence level, we consider bag-ofwords aggregation methods to perform sentence retrieval on the Europarl corpus. We consider 2,000 source sentence queries and 200k target sentences for each language pair and report the precision@k for k = 1, 5, 10, which accounts for the fraction of pairs for which the correct translation of the source words is in the k-th nearest neighbors. We use the idf-weighted average to merge word into sentence embeddings. The idf weights are obtained using other 300k sentences from Europarl. In what follows, we present the on word translation retrieval using our bilingual dictionaries in Table 1 and our comparison to previous work in TAB1 where we significantly outperform previous approaches. We also present on the sentence translation retrieval task in TAB3 and the cross-lingual word similarity task in Table 4. Finally, we present on word-by-word translation for English-Esperanto in Table 5.Baselines In our experiments, we consider a supervised baseline that uses the solution of the Procrustes formula given in, and trained on a dictionary of 5,000 source words. This baseline can be combined with different similarity measures: NN for nearest neighbor similarity, ISF for Inverted SoftMax and the CSLS approach described in Section 2.2.Cross-domain similarity local scaling This approach has a single parameter K defining the size of the neighborhood. The performance is very stable and therefore K does not need cross-validation: the are essentially the same for K = 5, 10 and 50, therefore we set K = 10 in all experiments. In Table 1 provides a strong and robust gain in performance across all language pairs, with up to 7.2% in eneo. We observe that Procrustes-CSLS is almost systematically better than Procrustes-ISF, while being computationally faster and not requiring hyper-parameter tuning. In TAB1, we compare our Procrustes-CSLS approach to previous models presented in BID31;; Smith et al. FORMULA0; BID2 on the English-Italian word translation task, on which state-of-the-art models have been already compared. We show that our Procrustes-CSLS approach obtains an accuracy of 44.9%, outperforming all previous approaches. In TAB3, we also obtain a strong gain in accuracy in the Italian-English sentence retrieval task using CSLS, from 53.5% to 69.5%, outperforming previous approaches by an absolute gain of more than 20%.Impact of the monolingual embeddings For the word translation task, we obtained a significant boost in performance when considering fastText embeddings trained on Wikipedia, as opposed to previously used CBOW embeddings trained on the WaCky datasets BID3 ), as can been seen in TAB1. Among the two factors of variation, we noticed that this boost in performance was mostly due to the change in corpora. The fastText embeddings, which incorporates more syntactic information about the words, obtained only two percent more accuracy compared to CBOW embeddings trained on the same corpus, out of the 18.8% gain. We hypothesize that this gain is due to the similar co-occurrence statistics of Wikipedia corpora. Figure 3 in the appendix shows on the alignment of different monolingual embeddings and concurs with this hypothesis. We also obtained better for monolingual evaluation tasks such as word similarities and word analogies when training our embeddings on the Wikipedia corpora. Adversarial approach Table 1 shows that the adversarial approach provides a strong system for learning cross-lingual embeddings without parallel data. On the es-en and en-fr language pairs, Adv-CSLS obtains a P@1 of 79.7% and 77.8%, which is only 3.2% and 3.3% below the supervised approach. Additionally, we observe that most systems still obtain decent on distant languages that do not share a common alphabet (en-ru and en-zh), for which method exploiting identical character strings are just not applicable BID2 ). This method allows us to build a strong synthetic vocabulary using similarities obtained with CSLS. The gain in absolute accuracy observed with CSLS on the Procrustes method is even more important here, with differences between Adv-NN and Adv-CSLS of up to 8.4% on es-en. As a simple baseline, we tried to match the first two moments of the projected source and target embeddings, which amounts to solving DISPLAYFORM0 and solving the sign ambiguity BID45. This attempt was not successful, which we explain by the fact that this method tries to align only the first two moments, while adversarial training matches all the moments and can learn to focus on specific areas of the distributions instead of considering global statistics. Refinement: closing the gap with supervised approaches The refinement step on the synthetic bilingual vocabulary constructed after adversarial training brings an additional and significant gain in performance, closing the gap between our approach and the supervised baseline. In Table 1, we observe that our unsupervised method even outperforms our strong supervised baseline on en-it and en-es, and is able to retrieve the correct translation of a source word with up to 83% accuracy. The better performance of the unsupervised approach can be explained by the strong similarity of cooccurrence statistics between the languages, and by the limitation in the supervised approach that uses a pre-defined fixed-size vocabulary (of 5,000 unique source words): in our case the refinement step can potentially use more anchor points. In TAB3, we also observe a strong gain in accuracy Table 4: Cross-lingual wordsim task. NASARI (Camacho-Collados et al. FORMULA0) refers to the official SemEval2017 baseline. We report Pearson correlation.en-eo eo-en Dictionary -NN 6.1 11.9 Dictionary -CSLS 11.1 14.3 Table 5: BLEU score on English-Esperanto. Although being a naive approach, word-byword translation is enough to get a rough idea of the input sentence. The quality of the generated dictionary has a significant impact on the BLEU score.(up to 15%) on sentence retrieval using bag-of-words embeddings, which is consistent with the gain observed on the word retrieval task. Application to a low-resource language pair and to machine translation Our method is particularly suited for low-resource languages for which there only exists a very limited amount of parallel data. We apply it to the English-Esperanto language pair. We use the fastText embeddings trained on Wikipedia, and create a dictionary based on an online lexicon. The performance of our unsupervised approach on English-Esperanto is of 28.2%, compared to 29.3% with the supervised method. On Esperanto-English, our unsupervised approach obtains 25.6%, which is 1.3% better than the supervised method. The dictionary we use for that language pair does not take into account the polysemy of words, which explains why the are lower than on other language pairs. People commonly report the P@5 to alleviate this issue. In particular, the P@5 for English-Esperanto and Esperanto-English is of 46.5% and 43.9% respectively. To show the impact of such a dictionary on machine translation, we apply it to the English-Esperanto Tatoeba corpora BID44. We remove all pairs containing sentences with unknown words, ing in about 60k pairs. Then, we translate sentences in both directions by doing word-byword translation. In Table 5, we report the BLEU score with this method, when using a dictionary generated using nearest neighbors, and CSLS. With CSLS, this naive approach obtains 11.1 and 14.3 BLEU on English-Esperanto and Esperanto-English respectively. Table 6 in the appendix shows some examples of sentences in Esperanto translated into English using word-by-word translation. As one can see, the meaning is mostly conveyed in the translated sentences, but the translations contain some simple errors. For instance, the "mi" is translated into "sorry" instead of "i", etc. The translations could easily be improved using a language model. Work on bilingual lexicon induction without parallel corpora has a long tradition, starting with the seminal works by BID37 BID13. Similar to our approach, they exploit the distributional structure, but using discrete word representations such as TF-IDF vectors. Following studies by BID14 BID38 BID41; BID25 BID19 BID21 leverage statistical similarities between two languages to learn small dictionaries of a few hundred words. These methods need to be initialized with a seed bilingual lexicon, using for instance the edit distance between source and target words. This can be seen as prior knowledge, only available for closely related languages. There is also a large amount of studies in statistical decipherment, where the machine translation problem is reduced to a deciphering problem, and the source language is considered as a ciphertext BID39 BID35. Although initially not based on distributional semantics, recent studies show that the use of word embeddings can bring significant improvement in statistical decipherment BID10.The rise of distributed word embeddings has revived some of these approaches, now with the goal of aligning embedding spaces instead of just aligning vocabularies. Cross-lingual word embeddings can be used to extract bilingual lexicons by computing the nearest neighbor of a source word, but also allow other applications such as sentence retrieval or cross-lingual document classification BID24. In general, they are used as building blocks for various cross-lingual language processing systems. More recently, several approaches have been proposed to learn bilingual dictionaries mapping from the source to the target space BID31 BID51 BID12 BID0. In particular, BID47 showed that adding an orthogonality constraint to the mapping can significantly improve performance, and has a closed-form solution. This approach was further referred to as the Procrustes approach in BID43.The hubness problem for cross-lingual word embedding spaces was investigated by. The authors added a correction to the word retrieval algorithm by incorporating a nearest neighbors reciprocity term. More similar to our cross-domain similarity local scaling approach, BID43 introduced the inverted-softmax to down-weight similarities involving oftenretrieved hub words. Intuitively, given a query source word and a candidate target word, they estimate the probability that the candidate translates back to the query, rather than the probability that the query translates to the candidate. Recent work by BID43 leveraged identical character strings in both source and target languages to create a dictionary with low supervision, on which they applied the Procrustes algorithm. Similar to this approach, recent work by BID2 used identical digits and numbers to form an initial seed dictionary, and performed an update similar to our refinement step, but iteratively until convergence. While they showed they could obtain good using as little as twenty parallel words, their method still needs cross-lingual information and is not suitable for languages that do not share a common alphabet. For instance, the method of BID2 on our dataset does not work on the word translation task for any of the language pairs, because the digits were filtered out from the datasets used to train the fastText embeddings. This iterative EMbased algorithm initialized with a seed lexicon has also been explored in other studies BID19 BID26.There has been a few attempts to align monolingual word vector spaces with no supervision. Similar to our work, BID50 employed adversarial training, but their approach is different than ours in multiple ways. First, they rely on sharp drops of the discriminator accuracy for model selection. In our experiments, their model selection criterion does not correlate with the overall model performance, as shown in FIG1. Furthermore, it does not allow for hyper-parameters tuning, since it selects the best model over a single experiment. We argue it is a serious limitation, since the best hyper-parameters vary significantly across language pairs. Despite considering small vocabularies of a few thousand words, their method obtained weak compared to supervised approaches. More recently, BID49 proposed to minimize the earth-mover distance after adversarial training. They compare their only to their supervised baseline trained with a small seed lexicon, which is one to two orders of magnitude smaller than what we report here. In this work, we show for the first time that one can align word embedding spaces without any cross-lingual supervision, i.e., solely based on unaligned datasets of each language, while reaching or outperforming the quality of previous supervised approaches in several cases. Using adversarial training, we are able to initialize a linear mapping between a source and a target space, which we also use to produce a synthetic parallel dictionary. It is then possible to apply the same techniques proposed for supervised techniques, namely a Procrustean optimization. Two key ingredients contribute to the success of our approach: First we propose a simple criterion that is used as an effective unsupervised validation metric. Second we propose the similarity measure CSLS, which mitigates the hubness problem and drastically increases the word translation accuracy. As a , our approach produces high-quality dictionaries between different pairs of languages, with up to 83.3% on the Spanish-English word translation task. This performance is on par with supervised approaches. Our method is also effective on the English-Esperanto pair, thereby showing that it works for lowresource language pairs, and can be used as a first step towards unsupervised machine translation. In order to gain a better understanding of the impact of using similar corpora or similar word embedding methods, we investigated merging two English monolingual embedding spaces using either Wikipedia or the Gigaword corpus BID33 ), and either Skip-Gram, CBOW or fastText methods (see Figure 3). Figure 3: English to English word alignment accuracy. Evolution of word translation retrieval accuracy with regard to word frequency, using either Wikipedia (Wiki) or the Gigaword corpus (Giga), and either skip-gram, continuous bag-of-words (CBOW) or fastText embeddings. The model can learn to perfectly align embeddings trained on the same corpus but with different seeds (a), as well as embeddings learned using different models (overall, when employing CSLS which is more accurate on rare words) (b). However, the model has more trouble aligning embeddings trained on different corpora (Wikipedia and Gigaword) (c). This can be explained by the difference in co-occurrence statistics of the two corpora, particularly on the rarer words. Performance can be further deteriorated by using both different models and different types of corpus (d).Source mi kelkfoje parolas kun mia najbaro tra la barilo. Hypothesis sorry sometimes speaks with my neighbor across the barrier. Reference i sometimes talk to my neighbor across the fence. Source la viro malanta ili ludas la pianon. Hypothesis the man behind they plays the piano. Reference the man behind them is playing the piano. Source bonvole protektu min kontra tiuj malbonaj viroj. Hypothesis gratefully protects hi against those worst men. Reference please defend me from such bad men. Table 6: Esperanto-English. Examples of fully unsupervised word-by-word translations. The translations reflect the meaning of the source sentences, and could potentially be improved using a simple language model.

Aligning languages without the Rosetta Stone: with no parallel data, we construct bilingual dictionaries using adversarial training, cross-domain local scaling, and an accurate proxy criterion for cross-validation.

represent the tldr for finding the corresponding scientific article

Examples:

"New Class of Autoencoders with pseudo invertible architecture" == "We consider the problem of information compression from high dimensional data. Where many studies consider the problem of compression by non-invertible trans- formations, we emphasize the importance of invertible compression. We introduce new class of likelihood-based auto encoders with pseudo bijective architecture, which we call Pseudo Invertible Encoders. We provide the theoretical explanation of their principles. We evaluate Gaussian Pseudo Invertible Encoder on MNIST, where our model outperform WAE and VAE in sharpness of the generated images. We consider the problem of information compression from high dimensional data. Where many studies consider the problem of compression by non-invertible transformations, we emphasize the importance of invertible compression as there are many cases where one cannot or will not decide a priori what part of the information is important and what part is not. Compression of images for person ID in a small company requires less resolution then person ID at an airport. To loose part of the information without harm to the future purpose of viewing the picture requires knowing the purpose upfront. Therefore, the fundamental advantage of invertible information compression is that compression can be undone if a future purpose so requires. Recent advances of classification models have demonstrated that deep learning architectures of proper design do not lead to information loss while still being able to achieve state-of-the-art in classification performance. These i-RevNet models BID5 implement a small but essential modification of the popular RevNet models while achieving invertibility and a performance similar to the standard RevNet BID2. This is of great interest as it contradicts the intuition that information loss is essential to achieve good performance in classification BID13. Despite the requirement of the invertibility, flow-based generating models BID0; BID11; BID6 demonstrate that the combination of bijective mappings allows one to transform the raw distribution of the input data to any desired distribution and perform the manipulation of the data. On the other hand, Auto-Encoders have provided the ideal mechanism to reduce the data to the bare minimum while retaining all essential information for a specific task, the one implemented in the loss function. Variational Auto Encoders (VAE) BID7 and Wasserstein Auto Encoders (WAE) BID14 are performing best. They provide an approach for stable training of autoencoders, which demonstrate good at reconstruction and generation. However, both of these methods involve the optimization of the objective defined on the pixel level. We would emphasise the importance of avoiding the separate decoder part and training the model without relying on the reconstuction quality directly. Combining the best of Invertible mappings and Auto-Encoders, we introduce Pseudo Invertible Encoder. Our model combines bijectives with restriction and extension of the mappings to the dependent sub-manifolds FIG0. The main contributions of this paper are the following:• We introduce new class of likelihood-based Auto-Encoders, which we call Pseudo Invertible Encoders. We provide the theoretical explanation of their principles.• We demonstrate the properties of Gaussian Pseudo Invertible Encoder in manifold learning.• We compare our model with WAE and VAE on MNIST, and report that the sharpness of the images, generated by our models is better. 2 RELATED WORK ResNets BID3 enable Networks to grow even more and thus memory consumption becomes a bottleneck. BID2 propose a Reversible Residual Network (RevNet) where each layer's activations can be reconstructed from the activations of the next layer. By replacing the residual blocks with coupling layers, they mimic the behaviour of residual blocks while being able to retrieve the original input of the layer. RevNet replaces the residual blocks of ResNets, but also accommodates non-invertible components to train more efficiently. By adding a downsampling operator to the coupling layer, i-RevNet circumvents these non-invertible modules BID5. With this they show that losing information is not a necessary condition to learn representations that generalize well on complicated problems. Although i-RevNet circumvents non-invertible modules, data is not compressed and the model is only invertible up to the last layer. All their methods do not allow dimensionality reduction. In current research we build a pseudo invertible model which performs dimensionality reduction. Auto-Encoders were first introduced by BID12 as an unsupervised learning algorithm. They are now widely used as a technique for dimension reduction by compressing input data. By training an encoder and a decoder network, and measuring the distance between original and reconstructed data, data can be represented in a latent space. This latent space can then be used for supervised learning algorithms. Instead of learning a compressed representation of the input data BID7 propose to learn the parameters of a probability distribution that represent the data. tol introduced new class of models -Wasserstein Auto Encoders, which use Optimal Transport to be trained. These methods require the optimization of the objective function which includes the terms defined on pixel level. Our model does not require such optimization. Moreover, it only perform encoding at training time. Here we introduce the approach for obtaining dimensionality reduction invertible mappings. Our method is based on the restriction of the mappings to low-dimensional manifolds, and extension of the inverse mappings with certain constraints (Fig. 2). Given data DISPLAYFORM0 In other words, we are looking for a pair of Figure 2: The schematic representation of the Restriction-Extension approach. The invertible mapping X ↔ Z is preformed by using the dependent sub-manifold R = g(Z) and a pair extended functionsG,G −1.associated functions G and G −1 such that DISPLAYFORM1 We use this residual manifold in order to match the dimensionalities of the hidden and initial spaces. Here we introduce the function g: DISPLAYFORM2 With no loss of generality we can say that R = g(Z). We use the pair of extended functionsG: DISPLAYFORM3 Rather than searching for the invertible dimensionality reduction mapping directly, we seek to find G, the invertible transformation with certain constraints, expressed by R.In search forG, we focus on DISPLAYFORM4 where F is a parametric family of functions invertible on R D. We select the function F θ with parameters θ which satisfy the constraint: DISPLAYFORM5 where DISPLAYFORM6 Taking into account constraint 3, we derive F θ (x) = [z, r], where z ∈ Z and r ∈ R. By combining this with Eq. 2 we have the desired pair of functions: DISPLAYFORM7 The obtained function G is Pseudo Invertible Endocer, or shortly PIE. As we are interested in high dimensional data such as images, the explicit choice of parameters θ is impossible. We choose θ * as a maximizer of the log likelihood of the observed data given the prior p θ (x): DISPLAYFORM0 After a change of variables according to Eq. 4 we obtain Taking into account the constraint 3 we derive the joint distribution for DISPLAYFORM1 DISPLAYFORM2 Dirac's delta function can be viewed as a limit of sequence of Gaussians: DISPLAYFORM3 Let us fix 2 = 2 0 DISPLAYFORM4 Finally, for the log likelihood we have: DISPLAYFORM5 We choose prior distribution p(z) as Standard Gaussian. We search for the parameters by using Gradient Descent. The method relies on the function F θ. This choice is challenging by itself. The currently known classes of real-value bijectives are limited. To overcome this issue, we approximate F θ with a composition of basic bijectives from certain classes: DISPLAYFORM0 where DISPLAYFORM1 Taking into account that a composition of PIE is also PIE, we create a final dimensionality reduction mapping from a sequence of PIEs: such that DISPLAYFORM2 DISPLAYFORM3 where DISPLAYFORM4 Then the log likelihood is represented as DISPLAYFORM5 where J kl is the Jacobian of the k-th function of the l-th PIE. The approximation error here depends only on, according to the Eq. 10. For the simplicity we will now refer to the whole model as PIE. The building blocks of this model are PIE blocks. If we choose the distribution p(z) in Eq. 17 as Standard Gaussian, g l (·) = 0, ∀l and 0 = 1, then the model can be viewed as Normalizing Flow with multi-scale architecture BID1 FIG1. It was demonstrated in BID1 that the model with such architecture achieves semantic compression. This section introduces the basic bijectives for the Pseudo-Invertible Encoder (PIE). We explain what each building bijective consists of and how it fits in the global architecture as shown in FIG2. PIE is composed of a series of convolutional blocks followed by linear blocks, as depicted in FIG2. The convolutional PIE blocks consist of series of coupling layers and 1×1 convolutions. We perform invertible downsampling of the image at the beginning of the convolutional block, by reducing the spatial resolution and increasing the number of channels, keeping the overall number of the variables the same. At the end of the convolutional PIE block, the split of variables is performed. One part of the variables is projected to the residual manifold R while others is feed to the next block. The linear PIE blocks are constructed in the same manner. However, the downsampling is not performed and 1 × 1 convolutions are replaced invertible linear mappings. DISPLAYFORM0 Figure 5: Structure of a coupling block. P partitions the input into two groups of equal length. U unites these group together. In the inverse P −1 and U −1 are the reverse of these operations respectively. In order to enhance the flexibility of the model, we utilize affine coupling layers Fig. 5. We modify the version, introduced in BID1.Given input data x, the output y is obtained by using the mapping: DISPLAYFORM0 Here multiplication and division are performed element-wise. The scalings s 1, s 2 and the biases b 1, b 2 are the functions, parametrized with neural networks. The invertibility is not required for this functions. x 1, x 2 are the non-intersecting partitions of x. For convolutional blocks we partition the tensors by splitting them into halves along the channels. In case of the linear blocks, we just split the features into halves. The log determinant of the Jacobian of coupling layer is given by: DISPLAYFORM1 where log | · | is calculated element-wise. The affine couplings operate on non-intersecting parts of the tensor. In order to capture the various correlations between channels and features, the different mechanism of channel permutations were proposed. BID6 demonstrated that invertible 1×1 convolutions perform better than fixed permutations and reversing of the order of channels BID1.We parametrize Invertible 1 × 1 Convolutions and invertible linear mappings with. Given the vector v, the Householder Matrix is computed as: DISPLAYFORM0 (b) Inverse Figure 6: Structure of the split method. P partitions the input into two sub samples. P −1 unites these sub samples together. The obtained matrix is orthogonal. Therefore, its inverse is just its transpose, which makes the computation of the inverse easier comparing to BID6. The log determinant of the Jacobian of such transformation is equal to 0. We use invertible downsampling to progressively reduce the spatial size of the tensor and increase the number of its channels. The downsampling with the checkerboard patterns Jacobsen et al. FORMULA3; BID1 transforms the tensor of size C ×H ×W into a tensor of size 4C × DISPLAYFORM0 where H, W are the height and the width of the image, and C is the number of the channels. The log determinant of the Jacobian of Downsampling is 0 as it just performs permutation. All the discussed blocks transform the data while preserving its dimensionality. Here we introduce Split block Fig. 6, which is responsible for the projection, restrictions and extension, described in Section 3. It reduces the dimensionality of the data by splitting the variables into two nonintersecting parts z, r of dimensionalities d and D − d, respectively. z is kept and is to be processed by the subsequent blocks. r is constrained to match N (r|g(z), 2 0 I). The mappings is defined as DISPLAYFORM0 For this experiment we trained a Gaussian PIE on the MNIST digits dataset. We build PIE with 2 convolutional blocks, each splitting the data in the last layer to 50% of the input size. Next we add three linear blocks to PIE, reducing the dimensions to 64, 10 and the last block does not reduce the dimensions any further. For each affine transformation we use the three biggest possible Householder reflections. For this experiment we set K l equal to 3. Optimization is done with the Adam optimizer BID8. The model diminishes the number of dimensions from R 784 to R 10.This experiment shows the ability of PIE to learn a manifold with three different constraints; 2 = 0.01, 2 = 0.1 and 2 = 1.0. The are shown in FIG3. As the constraint gets to loose, as shown in the right column, the model is not able to reconstruct anymore FIG3. Lower values for 2 perform better in terms of reconstruction. Too low values, however, sample fuzzy images FIG3. Narrowing down the distribution to sample from increases the models probability to produce accurate images. This is shown in FIG3 where samples are taken from N (0, 0.5). For both 2 = 0.01 and 2 = 0.1 reconstructed images are more accurate. This experiment shows that tightening the constraint by decreasing 2 increases the power of the manifold learned by the model. This is shown again in FIG3 where we diminished the number of dimensions even further from R 10 to R 2 utilizing UMAP BID9. With 2 = 1.0 UMAP created a manifold with a good Gaussian distribution. However, from the manifold created by PIE it was not able to separate distinct digits from each other. Tightening the constraint with a lower 2 moves the manifold created by UMAP further away from a Gaussian distribution, while it is better able to separate classes from each other. It is a well-known problem in VAEs that generated images are smoothened. improves over VAEs by utilizing Wasserstein distance function. To test the sharpness of generated images we convolve the grey-scaled images with the Laplace filter. This filter acts as an edge detector. We compute the variance of the activations and average them over 10000 sampled images. If an image is blurry, it means there are less edges and thus more activations will be close to zero, leading to a smaller variance. In this experiment we compare the sharpness of images generated by PIE with WAE, VAE and the sharpeness of the original images. For VAE and WAE we take the architecture as described in BID10. For PIE we take the architecture as described in section 5.1. Table 1 shows the for this experiment. PIE outperforms both VAE and WAE in terms of sharpeness of generated images. Images generated by PIE are even more sharp then original images from the MNIST dataset. An explanation for this is the use of a checkerboard pattern in the downsampling layer of the PIE convolutional block. With this technique we capture intrinsic properties of the data and are thus able to reconstruct sharper images. In this paper we have proposed the new class of Auto Encoders, which we call Pseudo Invertible Encoder. We provided a theory which bridges the gap between Auto Encoders and Normalizing Flows. The experiments demonstrate that the proposed model learns the manifold structure and generates sharp images."


Question: Questions that require counting a variety of objects in images remain a major challenge in visual question answering (VQA). The most common approaches to VQA involve either classifying answers based on fixed length representations of both the image and question or summing fractional counts estimated from each section of the image. In contrast, we treat counting as a sequential decision process and force our model to make discrete choices of what to count. Specifically, the model sequentially selects from detected objects and learns interactions between objects that influence subsequent selections. A distinction of our approach is its intuitive and interpretable output, as discrete counts are automatically grounded in the image. Furthermore, our method outperforms the state of the art architecture for VQA on multiple metrics that evaluate counting. Visual question answering (VQA) is an important benchmark to test for context-specific reasoning over complex images. While the field has seen substantial progress, counting-based questions have seen the least improvement. Intuitively, counting should involve finding the number of distinct scene elements or objects that meet some criteria, see Fig. 1 for an example. In contrast, the predominant approach to VQA involves representing the visual input with the final feature map of a convolutional neural network (CNN), attending to regions based on an encoding of the question, and classifying the answer from the attention-weighted image features BID32 BID31 b; BID7 BID14. Our intuition about counting seems at odds with the effects of attention, where a weighted average obscures any notion of distinct elements. As such, we are motivated to re-think the typical approach to counting in VQA and propose a method that embraces the discrete nature of the task. Our approach is partly inspired by recent work that represents images as a set of distinct objects, as identified by object detection, and making use of the relationships between these objects BID26. We experiment with counting systems that build off of the vision module used for these two works, which represents each image as a set of detected objects. For training and evaluation, we create a new dataset, HowMany-QA. It is taken from the countingspecific union of VQA 2.0 BID10 and Visual Genome QA .We introduce the Interpretable Reinforcement Learning Counter (IRLC), which treats counting as a sequential decision process. We treat learning to count as learning to enumerate the relevant objects in the scene. As a , IRLC not only returns a count but also the objects supporting its answer. This output is produced through an iterative method. Each step of this sequence has two stages: First, an object is selected to be added to the count. Second, the model adjusts the priority given to unselected objects based on their configuration with the selected objects (Fig. 1). We supervise only the final count and train the decision process using reinforcement learning (RL).Additional experiments highlight the importance of the iterative approach when using this manner of weak supervision. Furthermore, we train the current state of the art model for VQA on HowMany-QA and find that IRLC achieves a higher accuracy and lower count error. Lastly, we compare the Figure 1: IRLC takes as input a counting question and image. Detected objects are added to the returned count through a sequential decision process. The above example illustrates actual model behavior after training.grounded counts of our model to the attentional focus of the state of the art baseline to demonstrate the interpretability gained through our approach. Visual representations for counting. As a standalone problem, counting from images has received some attention but typically within specific problem domains. BID23 explore training a CNN to count directly from synthetic data. Counts can also be estimated by learning to produce density maps for some category of interest (typically people), as in; Oñoro-Rubio & López-; BID34. Density estimation simplifies the more challenging approach of counting by instance-by-instance detection BID19. Methods to detect objects and their bounding boxes have advanced considerably BID8 BID21 BID6 BID35 but tuning redundancy reduction steps in order to count is unreliable. Here, we overcome this limitation by allowing flexible, question specific interactions during counting. Alternative approaches attempt to model subitizing, which describes the human ability to quickly and accurately gauge numerosity when at most a few objects are present. demonstrates that CNNs may be trained towards a similar ability when estimating the number of salient objects in a scene. This approach was extended to counting 80 classes of objects simultaneously in. Their model is trained to estimate counts within each subdivision of the full image, where local counts are typically within the subitizing range. The above studies apply counting to a fixed set of object categories. In contrast, we are interested in counting during visual question answering, where the criteria for counting change from question to question and can be arbitrarily complex. This places our work in a different setting than those that count from the image alone. For example, apply their trained models to a subset of VQA questions, but their analysis was limited to the specific subset of examples where the question and answer labels agreed with the object detection labels they use for training. Here, we overcome this limitation by learning to count directly from question/answer pairs. Visual question answering. The potential of deep learning to fuse visual and linguistic reasoning has been recognized for some time a). Visual question answering poses the challenge of retrieving question-specific information from an associated image, often requiring complex scene understanding and flexible reasoning. In recent years, a number of datasets have been introduced for studying this problem (; BID20 BID37 BID0 BID10). The majority of recent progress has been aimed at the so-named "VQA" datasets BID0 BID10, where counting questions represent roughly 11% of the data. Though our focus is on counting questions specifically, prior work on VQA is highly relevant. An early baseline for VQA represents the question and image at a coarse granularity, respectively using a "bag of words" embedding along with spatially-pooled CNN outputs to classify the answer BID36. In BID20, a similar fixed-length image representation is fused with the question embeddings as input to a recurrent neural network (RNN), from which the answer is classified. Attention. More recent variants have chosen to represent the image at a finer granularity by omitting the spatial pooling of the CNN feature map and instead use attention to focus relevant image regions before producing an answer BID32 BID31 b; BID7 BID14. These works use the spatially-tiled feature vectors output by a CNN to represent the image; others follow the intuition that a more meaningful representation may come from parsing the feature map according to the locations of objects in the scene BID24 BID13. Notably, using object detection was a key design choice for the winning submission for the VQA 2017 challenge. Work directed at VQA with synthetic images (which sidesteps the challenges created by computer vision) has further demonstrated the utility that relationships may provide as an additional form of image annotation BID26.Interpretable VQA. The use of "scene graphs" in real-image VQA would have the desirable property that intermediate model variables would be grounded in concepts explicitly, a step towards making neural reasoning more transparent. A conceptual parallel to this is found in Neural Module Networks BID2 b; BID12, which gain interpretability by grounding the reasoning process itself in defined concepts. The general concept of interpretable VQA has been the subject of recent interest. We address this at the level of counting in VQA. We show that, despite the challenge presented by this particular task, an intuitive approach gains in both performance and interpretability over state of the art. Within the field of VQA, the majority of progress has been aimed at the VQA dataset BID0 and, more recently, VQA 2.0 BID10, which expands the total number of questions in the dataset and attempts to reduce bias by balancing answers to repeated questions. VQA 2.0 consists of 1.1M questions pertaining to the 205K images from COCO . The examples are divided according to the official COCO splits. In addition to VQA 2.0, we incorporate the Visual Genome (VG) dataset . Visual Genome consists of 108K images, roughly half of which are part of COCO. VG includes its own visual question answering dataset. We include examples from that dataset when they pertain to an image in the VQA 2.0 training set. In order to evaluate counting specifically, we define a subset of the QA pairs, which we refer to as HowMany-QA. Our inclusion criteria were designed to filter QA pairs where the question asks for a count, as opposed to simply an answer in the form of a number (Fig 2). For the first condition, we require that the question contains one of the following phrases: "how many", "number of", "amount of", or "count of". We also reject a question if it contains the phrase "number of the", since this phrase frequently refers to a printed number rather than a count (i.e. "what is the number of the bus?"). Lastly, we require that the ground-truth answer is a number between 0 to 20 (inclusive). The original VQA 2.0 train set includes roughly 444K QA pairs, of which 57,606 are labeled as having Figure 2: Examples of question-answer pairs that are excluded from HowMany-QA. This selection exemplifies the common types of "number" questions that do not require counting and therefore distract from our objective: (from left to right) time, general number-based answers, ballparking, and reading numbers from images. Importantly, the standard VQA evaluation metrics do not distinguish these from counting questions; instead, performance is reported for "number" questions as a whole. Due to our filter and focus on counting questions, we cannot make use of the official test data since its annotations are not available. Hence, we divide the validation data into separate development and test sets. More specifically, we apply the above criteria to the official validation data and select 5,000 of the ing QA pairs to serve as the test data. The remaining 17,714 QA pairs are used as the development set. As mentioned above, the HowMany-QA training data is augmented with available QA pairs from Visual Genome, which are selected using the same criteria. A breakdown of the size and composition of HowMany-QA is provided in TAB4. All models compared in this work are trained and evaluated on HowMany-QA. To facilitate future comparison to our work, we have made the training, development, and test question IDs available for download. In this work, we focus specifically on counting in the setting of visual question answering (where the criteria for counting changes on a question-by-question basis). In addition, we are interested in model interpretability. We explore this notion by experimenting with models that are capable of producing question-guided counts which are visually grounded in object proposals. Rather than substantially modifying existing counting approaches -such as -we compare three models whose architectures naturally fit within our experimental scope. These models each produce a count from the outputs of an object detection module and use identical strategies to encode the question and compare it to the detected objects. The models differ only in terms of how these components are used to produce a count FIG0. Our approach is inspired by the strategy of and. Their model, which represents current state of the art in VQA, infers objects as the input to the questionanswering system. This inference is performed using the Faster R-CNN architecture BID21. The Faster R-CNN proposes a set of regions corresponding to objects in the image. It encodes the image as a set of bounding boxes {b 1, ..., b N}, b i ∈ R 4 and complementary set of object encodings {v 1, ..., v N}, v i ∈ R 2048, corresponding to the locations and feature representations of each of the N detected objects, respectively (blue box in FIG0).Rather than train our own vision module from scratch, we make use of the publicly available object proposals learned in. These provide rich, object-centric representations for each image in our dataset. These representations are fixed when learning to count and are shared across each of the QA models we experiment with. Each architecture encodes the question and compares it to each detected object via a scoring function. We define q as the final hidden state of an LSTM BID11 after processing the question and compute a score vector for each object FIG0, green boxes): DISPLAYFORM0 DISPLAYFORM1 Here, x t denotes the word embedding of the question token at position t and s i ∈ R n denotes the score vector encoding the relevance of object i to the question. Following, we implement the scoring function f S: R m → R n as a layer of Gated Tanh Units (GTU) (van den BID28 . [,] denotes vector concatenation. We experiment with jointly training the scoring function to perform caption grounding, which we supervise using region captions from Visual Genome. Region captions provide linguistic descriptions of localized regions within the image, and the goal of caption grounding is to identify which object a given caption describes (details provided in Section B.1 of the Appendix). Caption grounding uses a strategy identical to that for question answering (Eqs. 1 and 2): an LSTM is used to encode each caption and the scoring function f S is used to encode its relevance to each detected object. The weights of the scoring function are tied for counting and caption grounding FIG0. We include from experiments where caption grounding is ignored. Interpretable RL Counter (IRLC). For our proposed model, we aim to learn how to count by learning what to count. We assume that each counting question implicitly refers to a subset of the objects within a scene that meet some variable criteria. In this sense, the goal of our model is to enumerate that subset of objects. To implement this as a sequential decision process, we need to represent the probability of selecting a given action and how each action affects subsequent choices. To that end, we project the object scores s ∈ R N xn to a vector of logits κ ∈ R N, representing how likely each object is to be counted, where N is the number of detected objects: DISPLAYFORM0 And we compute a matrix of interaction terms ρ ∈ R N xN that are used to update the logits κ. The value ρ ij represents how selecting object i will change κ j. We calculate this interaction from a compressed representation of the question (W q), the dot product of the normalized object vectors (v T iv j), the object coordinates (b i and b j), and basic overlap statistics (IoU ij, O ij, and O ji): DISPLAYFORM1 where f ρ: x ∈ R m ⇒ R is a 2-layer MLP with ReLU activations. For each step t of the counting sequence we greedily select the action with the highest value (interpreted as either selecting the next object to count or terminating), and update κ accordingly: DISPLAYFORM2 DISPLAYFORM3 where ζ is a learnable scalar representing the logit value of the terminal action, and κ 0 is the of Equation 3 FIG0. The action a t is expressed as the index of the selected object. ρ(a t, ·) denotes the row of ρ indexed by a t. Each object is only allowed to be counted once. We define the count C as the timestep when the terminal action was selected t: a t = N + 1.This approach bears some similarity to Non-Maximal Suppression (NMS), a staple technique in object detection to suppress redundant proposals. However, our approach is far less rigid and allows the question to determine how similar and/or overlapping objects interact. Training IRLC. Because the process of generating a count requires making discrete decisions, training requires that we use techniques from Reinforcement Learning. Given our formulation, a natural choice is to apply REINFORCE BID29. To do so, we calculate a distribution over action probabilities p t from κ t and generate a count by iteratively sampling actions from the distribution: DISPLAYFORM4 DISPLAYFORM5 We calculate the reward using Self-Critical Sequence training BID22 BID17, a variation of policy gradient. We define E = |C − C GT | to be the count error and define the reward as R = E greedy − E, where E greedy is the baseline count error obtained by greedy action selection (which is also how the count is measured at test time). From this, we define our (unnormalized) counting loss as DISPLAYFORM6 Additionally, we include two auxiliary objectives to aid learning. For each sampled sequence, we measure the total negative policy entropy H across the observed time steps. We also measure the average interaction strength at each time step and collect the total DISPLAYFORM7 where L 1 is the Huber loss from Eq 12. Including the entropy objective is a common strategy when using policy gradient BID30; ) and is used to improve exploration. The interaction penalty is motivated by the a priori expectation that interactions should be sparse. During training, we minimize a weighted sum of the three losses, normalized by the number of decision steps. As before, we provide training and implementation details in the Appendix (Sec. B.2).SoftCount. As a baseline approach, we train a model to count directly from the outputs s of the scoring function. For each object, we project its score vector s i to a scalar value and apply a sigmoid nonlinearity, denoted as σ, to assign the object a count value between 0 and 1. The total count is the sum of these fractional, object-specific count values. We train this model by minimizing the Huber loss associated with the absolute difference e between the predicted count C and the ground truth count C GT: DISPLAYFORM8 DISPLAYFORM9 For evaluation, we round the estimated count C to the nearest integer and limit the output to the maximum ground truth count (in this case, 20).Attention Baseline (UpDown). As a second baseline, we re-implement the QA architecture introduced in, which the authors refer to as UpDown -see also for additional details. We focus on this architecture for three main reasons. First, it represents the current state of the art for VQA 2.0. Second, it was designed to use the visual representations we employ. And, third, it exemplifies the common two-stage approach of deploying question-based attention over image regions (here, detected objects) to get a fixed-length visual representation DISPLAYFORM10 and then classifying the answer based on this average and the question encoding DISPLAYFORM11 DISPLAYFORM12 where s ∈ R N xn denotes the matrix of score vectors for each of the N detected objects and α ∈ R N denotes the attention weights. Here, each function f is implemented as a GTU layer and ⊗ denotes element-wise multiplication. For training, we use a cross entropy loss, with the target given by the ground-truth count. At test time, we use the most probable count given by p. We use two metrics for evaluation. For consistency with past work, we report the standard VQA test metric of accuracy. Since accuracy does not measure the degree of error we also report root-meansquared-error (RMSE), which captures the typical deviation between the estimated and ground-truth count and emphasizes extreme errors. Details are provided in the Appendix (Sec. D).To better understand the performance of the above models, we also report the performance of two non-visual baselines. The first baseline (Guess1) shows the performance when the estimated count is always 1 (the most common answer in the training set). The second baseline (LSTM) learns to predict the count directly from a linear projection of the question embedding q (Eq. 1). IRLC achieves the highest overall accuracy and (with SoftCount) the lowest overall RMSE on the test set TAB6 ). Interestingly, SoftCount clearly lags in accuracy but is competitive in RMSE, arguing that accuracy and RMSE are not redundant. We observe this to from the fact that IRLC is less prone to small errors and very slightly more prone to large errors (which disproportionately impact RMSE). However, whereas UpDown improves in accuracy at the cost of RMSE, IRLC is substantially more accurate without sacrificing overall RMSE.To gain more insight into the performance of these models, we calculate these metrics within the development set after separating the data according to how common the subject of the count is during training 1. We break up the questions into 5 roughly equal-sized bins representing increasingly uncommon subjects. We include a 6th bin for subjects never seen during training. The accuracy and RMSE across the development set are reported for each of these bins in Figure 5.Organizing the data this way reveals two main trends. First, all models perform better when asked to count subjects that were common during training. Second, the performance improvements offered by IRLC over UpDown persist over all groupings of the development data. We introduce a novel analysis to quantify how well counted objects match the subject of the question. To perform this analysis, we form generic questions that refer to the object categories in the COCO object detection dataset. We take the object proposals counted in response to a given question and compare them to the ground truth COCO labels to determine how relevant the counted object proposals are. Our metric takes on a value of 1 when the counted objects perfectly map onto the category to which the question refers. Values around 0 indicate that the counted objects were not relevant to the question. Section D of the Appendix details how the grounding quality metric is calculated. We perform this analysis for each of the 80 COCO categories using the images in the HowMany-QA development set. FIG2 compares the grounding quality of SoftCount and IRLC, where each point represents the average grounding quality for a particular COCO category. As with the previous two metrics, grounding quality is highest for COCO categories that are more common during training. We observe that IRLC consistently grounds its counts in objects that are more relevant to the question than does SoftCount. A paired t-test shows that this trend is statistically significant (p < 10 −15). Examples of failure cases with common and rare subjects ("people" and "ties," respectively). Each example shows the output of IRLC, where boxes correspond to counted objects, and the output of UpDown, where boxes are shaded according to their attention weights. The design of IRLC is inspired by the ideal of interpretable VQA BID4. One hallmark of interpretability is the ability to predict failure modes. We argue that this is made more approachable by requiring IRLC to identify the objects in the scene that it chooses to count. FIG3 illustrates two failure cases that exemplify observed trends in IRLC. In particular, IRLC has little trouble counting people (they are the most common subject) but encounters difficulty with referring phrases (in this case, "sitting on the bench"). When asked to count ties (a rare subject), IRLC includes a sleeve in the output, demonstrating the tendency to misidentify objects with few training examples. These failures are obvious by virtue of the grounded counts, which point out exactly which objects IRLC counted. In comparison, the attention focus of UpDown (representing the closest analogy to a grounded output) does not identify any pattern. From the attention weights, it is unclear which scene elements form the basis of the returned count. Indeed, the two models may share similar deficits. We observe that, in many cases, they produce similar counts. However, we stress that without IRLC and the chance to observe such similarities such deficits of the UpDown model would be difficult to identify. The Appendix includes further visualizations and comparisons of model output, including examples of how IRLC uses the iterative decision process to produce discrete, grounded counts (Sec. A). We present an interpretable approach to counting in visual question answering, based on learning to enumerate objects in a scene. By using RL, we are able to train our model to make binary decisions about whether a detected object contributes to the final count. We experiment with two additional baselines and control for variations due to visual representations and for the mechanism of visuallinguistic comparison. Our approach achieves state of the art for each of the evaluation metrics. In addition, our model identifies the objects that contribute to each count. These groundings provide traction for identifying the aspects of the task that the model has failed to learn and thereby improve not only performance but also interpretability. A EXAMPLES Figure 8: Example outputs produced by each model. For SoftCount, objects are shaded according to the fractional count of each (0=transparent; 1=opaque). For UpDown, we similarly shade the objects but use the attention focus to determine opacity. For IRLC, we plot only the boxes from objects that were selected as part of the count. At each timestep, we illustrate the unchosen boxes in pink, and shade each box according to κ t (corresponding to the probability that the box would be selected at that time step; see main text). We also show the already-selected boxes in blue. For each of the questions, the counting sequence terminates at t = 3, meaning that the returned count C is 3. For each of these questions, that is the correct answer. The example on the far right is a'correct failure,' a case where the correct answer is returned but the counted objects are not related to the question. These kinds of subtle failures are revealed with the grounded counts. We experiment with jointly training counting and caption grounding. The goal of caption grounding is, given a set of objects and a caption, to identify the object that the caption describes. Identical to the first stages of answering the counting question, we use an LSTM to encode the caption and compare it to each of the objects using the scoring function: DISPLAYFORM0 DISPLAYFORM1 where h ∈ R 1024, x t i ∈ R 300 is the embedding for the token at timestep t of the caption, T is the caption length, and f S: R m → R n is the scoring function (Sec. 4.3). The embedding of object i is denoted by v i and the relevance of this object to the caption is encoded by the score vector s i.We project each such score vector to a scalar logit α i and apply a softmax nonlinearity to estimate p ∈ R N, where N is the number of object proposals and p i denotes the probability that the caption describes object proposal i: DISPLAYFORM2 During training, we randomly select four of the images in the batch of examples to use for caption grounding (rather than the full 32 images that make up a batch). To create training data from the region captions in Visual Genome, we assign each caption to one of the detected object proposals. To do so, we compute the intersection over union between the ground truth region that the caption describes and the coordinates of each object proposal. We assign the object proposal with the largest IoU to the caption. If the maximum IoU for the given caption is less than 0.5, we ignore it during training. We compute the grounding probability p for each caption to which we can successfully assign a detection and train using the cross entropy loss averaged over the captions. We weight the loss associated with caption grounding by 0.1 relative to the counting loss. Each of the considered counting models makes use of the same basic architecture for encoding the question and comparing it with each of the detected objects. For each model, we initialized the word embeddings from GloVe BID18 and encoded the question with an LSTM of hidden size 1024. The only differences in the model-specific implementations of the language module was the hidden size of the scoring function f S. We determined these specifics from the optimal settings observed during initial experiments. We use a hidden size of 512 for SoftCount and UpDown and a hidden size of 2048 for IRLC. We observed that the former two models were more prone to overfitting, whereas IRLC benefited from the increased capacity. When training on counting, we optimize using Adam BID15. For SoftCount and UpDown, we use a learning rate of 3x10 −4 and decay the learning rate by 0.8 when the training accuracy plateaus. For IRLC, we use a learning rate of 5x10 −4 and decay the learning rate by 0.99999 every iteration. For all models, we regularize using dropout and apply early stopping based on the development set accuracy (see below).When training IRLC, we apply the sampling procedure 5 times per question and average the losses. We weight the entropy penalty P H and interaction penalty P I (Eq. 10) both by 0.005 relative to the counting loss. These penalty weights yield the best development set accuracy within the hyperparameter search we performed (Fig. 10). IRLC auxiliary loss. We performed a grid search to determine the optimal setting for the weights of the auxiliary losses for training IRLC. From our observations, the entropy penalty is important to balance the exploration of the model during training. In addition, the interaction penalty prevents Figure 11: HowMany-QA test set performance for models trained with the full HowMany-QA training data (blue) and trained without the additional data from Visual Genome (green). degenerate counting strategies. The of the grid search suggest that these auxiliary losses improve performance but become unhelpful if given too much weight (Fig. 10). In any case, IRLC outperforms the baseline models across the range of settings explored. Data augmentation with Visual Genome. Here, we compare performance on the HowMany-QA test set for models trained with and without additional data from Visual Genome QA. In all cases, performance benefits from the additional training data (Fig. 11). On average, excluding Visual Genome from the training data decreases accuracy by 2.7% and increases RMSE by 0.12. Interestingly, the performance of IRLC is most robust to the loss of training data. Ordinality of UpDown output. Whereas the training objectives for SoftCount and IRLC intrinsically reflect the ordinal nature of counts, the same is not true for UpDown. For example, the loss experienced by SoftCount and IRLC reflect the degree of error between the estimated count and the ground truth target; however, UpDown is trained only to place high probability mass on the ground truth value (missing by 1 or by 10 are treated as equally incorrect). We examine the patterns in the output count probabilities from UpDown to ask whether the model learns an ordinal representation despite its non-ordinal training objective. Figure 12 illustrates these trends. When the estimated count is less than 5, the second-most probable count is very frequently adjacent to the most probable count. When the estimated count is larger than 5, the probability distribution is less smooth, such Figure 12: (Left) Average count probability (Eq. 15) from UpDown, grouped according to the estimated count. (Right) Cumulative distribution of the absolute difference between the top two predicted counts, shown for when the most likely count was less than 5 (blue) and when it was greater than or equal to 5 (green). The probability distributions are much less smooth when the estimated count is large.that the second-most probable count is often considerably different than the most probable count. This suggests that UpDown learns ordinality for lower count values (where training data is abundant) but fails to generalize this concept to larger counts (where training data is more sparse). Accuracy. The VQA dataset includes annotations from ten human reviewers per question. The accuracy of a given answer a depends on how many of the provided answers it agrees with. It is scored as correct if at least 3 humans answers agree:Acc (a) = min #humans that said a 3, 1.Each answer's accuracy is averaged over each 10-choose-9 set of human answers. As described in the main text, we only consider examples where the consensus answer was in the range of 0-20. We use all ten labels to calculate accuracy, regardless of whether individual labels deviate from this range. Thee accuracy values we report are taken from the average accuracy over some set of examples. RMSE. This metric simply quantifies the typical deviation between the model count and the groundtruth. Across a set of N, we calculate this metric as DISPLAYFORM0 whereĈ i and C i are the predicted and ground truth counts, respectively, for question i. RMSE is a measurement of error, so lower is better. Grounding Quality. We introduce a new evaluation method for quantifying how relevant the objects counted by a model are to the type of object it was asked to count. This evaluation metric takes advantage of the ground truth labels included in the COCO dataset. These labels annotate each object instance of 80 different categories for each of the images in the development set. We make use of GloVe embeddings to compute semantic similarity. We use GloVe (x) ∈ R 300 to denote the (L2 normalized) GloVe embedding of category x. For each image m, the analysis is carried out in two stages. First, we assign one of the COCO categories (or ) to each of the object proposals used for counting. For each object proposal, we find the object in the COCO labels with the largest IoU. If the IoU is above 0.5, we assign the object proposal to the category of the COCO object, otherwise we assign the object proposal to the . Below, we use k m i to denote the category assigned to object proposal i for image m. Second, for each of the COCO categories present in image m, we use the category q (i.e. q = "car") to build a question (i.e. "How many cars are there?"). For SoftCount and IRLC, the count returned in response to this question is the sum of each object proposal's inferred count value: DISPLAYFORM1 where N m is the number of object proposals in image m and w (m,q) i is the count value given to proposal i. We use the count values to compute a weighted sum of the semantic similarity between the assigned object proposal categories k and the question category q: DISPLAYFORM2 where semantic similarity is estimated from the dot product between the embeddings of the assigned category and the question category. If k m i corresponds to the category, we replace its embedding with a vector of zeros. The final metric is computed for each COCO category by accumulating the over all images that contain a label for that category and normalizing by the net count to get an average: DISPLAYFORM3 The interpretation of this metric is straightforward: on average, how relevant are the counted objects to the subject of the question.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We perform counting for visual question answering; our model produces interpretable outputs by counting directly from detected objects.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We propose a new video understanding benchmark, with tasks that by-design require temporal reasoning to be solved, unlike most existing video datasets.