Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Most of recent work in cross-lingual word embeddings is severely Anglocentric. The vast majority of lexicon induction evaluation dictionaries are between English and another language, and the English embedding space is selected by default as the hub when learning in a multilingual setting. With this work, however, we challenge these practices. First, we show that the choice of hub language can significantly impact downstream lexicon induction performance. Second, we both expand the current evaluation dictionary collection to include all language pairs using triangulation, and also create new dictionaries for under-represented languages. Evaluating established methods over all these language pairs sheds light into their suitability and presents new challenges for the field. Finally, in our analysis we identify general guidelines for strong cross-lingual embeddings baselines, based on more than just Anglocentric experiments. Continuous distributional vectors for representing words (embeddings) have become ubiquitous in modern, neural NLP. Cross-lingual representations additionally represent words from various languages in a shared continuous space, which in turn can be used for Bilingual Lexicon Induction (BLI). BLI is often the first step towards several downstream tasks such as Part-Of-Speech (POS) tagging , parsing , document classification , and machine translation (; b;). Often, such shared representations are learned with a two-step process, whether under bilingual or multilingual settings (hereinafter BWE and MWE, respectively). First, monolingual word embeddings are learned over large swaths of text; such pre-trained word embeddings, in fact, are available for several languages and are widely used, like the fastText Wikipedia vectors . Second, a mapping between the languages is learned, in one of three ways: in a supervised manner if dictionaries or parallel data are available to be used for supervision , under minimal supervision e.g. using only identical strings , or even in a completely unsupervised fashion . Both in bilingual and multilingual settings, it is common that one of the language embedding spaces is the target to which all other languages get aligned to (hereinafter "the hub"). We outline the details in Section 2. Despite all the recent progress in learning cross-lingual embeddings, we identify a major shortcoming to previous work: it is by and large English-centric. Notably, most MWE approaches essentially select English as the hub during training by default, aligning all other language spaces to the English one. We argue and empirically show, however, that English is a poor hub language choice. In BWE settings, on the other hand, it is fairly uncommon to denote which of the two languages is the hub (often this is implied to be the target language). However, we experimentally find that this choice can greatly impact downstream performance, especially when aligning distant languages. This Anglocentricity is even more evident at the evaluation stage. The lexica most commonly used for evaluation are the MUSE lexica which cover 45 languages, but with translations only from and into English. Even still, alternative evaluation dictionaries are also very English-and European-centric: report on on on Spanish-English and Italian-English, and Artetxe et al. (2018a) between English and Italian, German, Finnish, Spanish, and Turkish. We argue that cross-lingual word embedding mapping methods should look beyond English for their evaluation benchmarks because, compared to all others, English is a language with disproportionately large available data and relatively poor inflectional morphology e.g., it lacks case, gender, and complex verbal inflection systems . These two factors allow for an overly easy evaluation setting which does not necessarily generalize to other language pairs. In light of this, equal focus should instead be devoted to evaluation over more diverse language pairs that also include morphologically rich and low-resource languages. With this work, we attempt to address these shortcomings, providing the following contributions: • We show that the choice of the hub when evaluating on diverse language pairs can lead to significantly different performance (e.g., by more than 10 percentage points for BWE over distant languages). We also show that often English is a suboptimal hub for MWE. • We identify some general guidelines for choosing a hub language which could lead to stronger baselines; less isometry between the hub and source and target embedding spaces mildly correlates with performance, as does typological distance (a measure of language similarity based on language family membership trees). For distant languages, multilingual systems should in most cases be preferred over bilingual ones. • We provide resources for training and evaluation on non-Anglocentric language pairs. We outline a simple triangulation method with which we extend the MUSE dictionaries to an additional 2352 lexicons covering 49 languages, and we present on a subset of them. We also create new evaluation lexica for under-resourced languages using Azerbaijani, Belarusian, and Galician as our test cases. We additionally provide recipes for creating such dictionaries for any language pair with available parallel data. In the supervised bilingual setting, as formulated by , given two languages L = {l 1, l 2} and their pre-trained row-aligned embeddings X 1, X 2, respectively, a transformation matrix M is learned such that: The set Ω can potentially impose a constraint over M, such as the very popular constraint of restricting it to be orthogonal . Previous work has empirically found that this simple formulation is competitive with other more complicated alternatives . The orthogonality assumption ensures that there exists a closed-form solution in the form of the Singular Value Decomposition (SVD) of X 1 X T 2. 1 Note that in this case only a single matrix M needs to be learned, because X 1 − M X 2 = M −1 X 1 − X 2, while at the same time a model that minimizes X 1 − M X 2 is as expressive as one minimizing M 1 X 1 − M 2 X 2, and easier to learn. In the minimally supervised or even the unsupervised setting the popular methods follow an iterative refinement approach . Starting with a seed dictionary (e.g. from identical strings or numerals) an initial mapping is learned in the same manner as in the supervised setting. The initial mapping, in turn, is used to expand the seed dictionary with high confidence word translation pairs. The new dictionary is then used to learn a better mapping, and so forth the iterations continue until convergence. We will generally refer to such methods as MUSE-like. Similarly, in a multilingual setting, one could start with N languages L = {l 1, l 2, . . ., l N} and their respective pre-trained embeddings X 1, X 2,..., X N, and then learn N −1 bilingual mappings between a pre-selected target language and all others. Hence, one of the language spaces is treated as a target (the hub) and remains invariant, while all others are mapped into the (now shared) hub language space. Alternatively, those mappings could be jointly learned using the MAT+MPSR methods of -also taking advantage of the inter-dependencies between any two language pairs. Importantly, though, there is no closed form solution for learning the joint mapping, hence a solution needs to be approximated with gradient-based methods. MAT+MPSR generalizes the adversarial approach of to multiple languages, and also follows an iterative refinement 2 In either case, a language is chosen as the hub, and N − 1 mappings for the other languages are learned. Other than MAT+MPSR, the only other unsupervised multilingual approach is that of , who propose to incrementally align multiple languages by adding each new language as a hub. We decided, though, against comparing to this method, because (a) their method requires learning O(N 2) mappings for relatively small improvements and (b) the order in which the languages are added is an additional hyperparameter that would explode the experimental space. Lexicon Induction One of the most common downstream evaluation tasks for the learned crosslingual word mappings is Lexicon Induction (LI), the task of retrieving the most appropriate wordlevel translation for a query word from the mapped embedding spaces. Specialized evaluation (and training) dictionaries have been created for multiple language pairs, with the MUSE dictionaries most often used, providing word translations between English (En) and 48 other high-to mid-resource languages, as well as on all 30 pairs among 6 very similar Romance and Germanic languages (English, French, German, Spanish, Italian, Portuguese). Given the mapped embedding spaces, the translations are retrieved using a distance metric, with Cross-Lingual Similarity Scaling (, CSLS) as the most common and best performing in the literature. Intuitively, CSLS decreases the scores of pairs that lie in dense areas, increasing the scores of rarer words (which are harder to align). The retrieved pairs are compared to the gold standard and evaluated using precision at k (P@k, evaluating how often the correct translation is within the k retrieved nearest neighbours of the query). Throughout this work we report P@1, which is equivalent to accuracy, but we also provide with P@5 and P@10 in the Appendix. As other works have recently noted the typically used evaluation dictionaries cover a narrow breadth of the possible language pairs, with the majority of them focusing in pairs with English (as with the MUSE dictionaries) or among high-resource European languages. In this section, we first outline our method for creating new dictionaries for low resource languages. Then, we describe the simple triangulation process that allows us to create dictionaries among all 49 MUSE languages. Our approach for constructing dictionaries is fairly straightforward, inspired by phrase table extraction techniques from phrase-based MT . Rather than manual inspection, however, which would be impossible for all language pairs, we rely on fairly simple heuristics for controlling the quality of our dictionaries. The first step is collecting publicly available parallel data between English and the low-resource language of interest. We use data from the TED , OpenSubtitles , WikiMatrix , bible , and JW300 (Agić and Vulić, 2019) datasets. 4 This in 354k, 53k, and 623k English-to-X parallel sentences for Azerbaijani (Az), Belarusian (Be), and Galician (Gl) respectively. 5 We align the parallel sentences using fast align , and extract symmetrized alignments using the gdfa heuristic . In order to ensure that we do not extract highly domain-specific word pairs, we only use the TED, OpenSubtitles, and WikiMatrix parts for word-pair extraction. Also, in order to control for quality, we only extract word pairs if they appear in the dataset more than 5 times, and if the alignment probability is higher than 30%. With this process, we end up with about 6k, 7k, and 38k word pairs for Az-En, Be-En, and GlEn respectively. Following standard conventions, we sort the word pairs according to source-side frequency, and use the intermediate-frequency ones for evaluation, typically using the 5000-6500 rank boundaries. The same process can be followed for any language pair with enough volume of parallel data (needed for training a decent word alignment model). In fact, we can produce similar dictionaries for a large number of languages, as the combination of the recently created JW300 and WikiMatrix datasets provide an average of more than 100k parallel sentences in 300 languages. Our second method for creating new dictionaries is inspired from phrase table triangulation ideas from the pre-neural MT community . The concept can be easily explained with an example, visualized in Figure 1. Consider the Portuguese (Pt) word trabalho which, according to the MUSE Pt-En dictionary, has the words job and work as possible En translations. In turn, these two En words can be translated to 4 and 5 Czech (Cs) words respectively. By utilizing the transitive property (which translation should exhibit) we can identify the set of 7 possible Cs translations for the Pt word trabalho. Following this simple triangulation approach, we create 2352 new dictionaries over language pairs among the 49 languages of the MUSE dictionaries. 7 For consistency, we keep the same train and test splits as with MUSE, so that the source-side types are equal across all dictionaries with the same source language. Triangulating through English (which is unavoidable, due to the lack of non-English-centric dictionaries) is suboptimal -English is morphologically poor and lacks gender information. As a , several inflected forms in morphologically-rich languages map to the same English form. Similarly, gendered nouns or adjectives in gendered languages map to English forms that lack gender information. For example, the MUSE Greek-English dictionary lists the word peaceful as the translation for all ειρηνικός, ειρηνική, ειρηνικό, ειρηνικά, which are the male, female, and neutral (singular and plural) inflections of the same adjective. Equivalently, the English-Italian dictionary translates peaceful into either pacifico, pacifici, or pacifica (male singular, male plural, and female singular, respectively; see Table 1). When translating from or into English lacking context, all of those are reasonable translations. When translating between Greek and Italian, though, one should take gender and number into account. Hence, we devise a filtering method for removing blatant mistakes when triangulating morphologically rich languages. We rely on automatic morphological tagging which we can obtain for most of the MUSE languages, using the StanfordNLP toolkit . The morphological tagging uses the Universal Dependencies feature set making the tagging comparable across almost all languages. Our filtering technique iterates through the bridged dictionaries: for a given source word, if we find a translation word with the exact same morphological analysis, we filter out all other translations with the same lemma but different tags. In the case of feature mismatch (for instance, Greek uses 4 cases and 3 genders while Italian has 2 genders and no cases) or if we only find a partial tag match over a feature subset, we filter out translations with disagreeing tags. Coming back to our Greek-Italian example, this means that for the form ειρηνικός we would only keep pacifico as a candidate translation (we show more examples in Table 1). Our filtering technique removes about 17% of the entries in our bridged dictionaries. Naturally, this filtering approach is restricted to languages for which a morphological analyzer is available. Miti- Table 2: Lexicon Induction performance (measured with P@1) over 10 European languages (90 pairs). In each cell, the superscript denotes the hub language that yields the best for that language pair. µ best: average using the best hub language. µ En: average using the En as the hub. The shaded cells are the only language pairs where a bilingual MUSE system outperforms MAT+MSPR. For our main MWE experiments, we train MAT+MPSR systems to align several language subsets varying the hub language. For BWE experiments, we compare MUSE with MAT+MPSR. The differences in LI performance show the importance of the hub language choice with respect to each evaluation pair. As part of our call for moving beyond Anglo-centric evaluation, we also present LI on several new language pairs using our triangulated dictionaries. It is worth noting that we are predominantly interested in comparing the quality of the multilingual alignment when different hub languages are used. Hence, even slightly noisy dictionaries (like our low-resource language ones) are still useful. Even if the skyline performance (from e.g. a perfect system) would not reach 100% accuracy due to noise, the differences between the systems' performance can be revealing. We first focus on 10 European languages of varying morphological complexity and data availability (which affects the quality of the pre-trained word embeddings): Azerbaijani (Az), Belarusian (Be), Czech (Cs), English (En), Galician (Gl), Portuguese (Pt), Russian (Ru), Slovak (Sk), Spanish (Es), and Turkish (Tr). The choice of these languages additionally ensures that for our three low-resource languages (Az, Be, Gl) we include at least one related higher-resource language (Tr, Ru, Pt/Es respectively), allowing for comparative analysis. Table 2 summarizes the best post-hoc performing systems for this experiment. In the second setting, we use a set of 7 more distant languages: English, French (Fr), Hindi (Hi), Korean (Ko), Russian, Swedish (Sv), and Ukrainian (Uk). This language subset has large variance in terms of typology and alphabet. The best performing systems are presented in Table 3. Experimental Setup We train and evaluate all models starting with the pre-trained Wikipedia FastText embeddings for all languages . We focus on the minimally supervised scenario which only uses similar character strings between any languages for supervision in order to mirror the hard, realistic scenario of not having annotated training dictionaries between the languages. We learn MWE with the MAT+MPSR method using the publicly available code. 8 We also use MAT+MPSR for BWE experiments, but we additionally train and compare to MUSE systems 9 . We compare the statistical significance of the difference in performance from two systems using paired bootstrap resampling . Generally, a difference of 0.4-0.5 percentage points evaluated over our lexica is significant with p < 0.05. The hub matters for distant languages When using MUSE, the answer is simple: the closed form solution of the Procrustes problem is provably direction-independent, and we confirm this empirically (we provide complete on MUSE in Table 15 in the Appendix). However, obtaining good performance with such methods requires the orthogonality assumption to hold, which for distant languages is rarely the case . In fact, we find that the gradient-based MAT+MPSR method in a bilingual setting over distant languages exhibits better performance than MUSE. Across Tables 2 and 3, in only a handful of examples (shaded cells) does MUSE outperform MAT+MPSR for BWE. On the other hand, we find that when aligning distant languages with MAT+MPSR, the difference between hub choices can be significant -in Az-En, for instance, using En as the hub leads to more than 7 percentage points difference to using Az. We show some examples in Table 4. On the other hand, when aligning typologically similar languages, the difference is less pronounced. For example, we obtain practically similar performance for Gl-Pt, Az-Tr, or Uk-Ru when using either the source or the target language as the hub. Note, though, that non-negligible differences could still occur, as in the case of Pt-Gl. In most cases, it is the case that the higher-resourced language is a better hub than the lower-resourced one, especially when the number of resources defer significantly (as in the case of Az and Be against any other language). Since BWE settings are not our main focus, we leave an extensive analysis of this observation for future work. MWE: English is rarely the best hub language In multilingual settings, we conclude that the standard practice of choosing English as the hub language is sub-optimal. Out of the 90 evaluation pairs from our European-languages experiment (Table 2) the best hub language is English in only 17 instances (less than 20% of the time). In fact, the average performance (over all evaluation pairs) when using En as the hub (denoted as µ En) is 1.3 percentage points worse than the optimal (µ best). In our distant-languages experiment (Table 3) English is the best choice only for 7 of the 42 evaluation pairs (again, less than 20% of the time). As before, using En as the hub leads to an average drop of one percentage point in performance aggregated over all pairs, compared to the averages of the optimal selection. The rest of the section attempts to provide an explanation for these differences. Expected gain for a hub language choice As vividly outlined by the superscript annotations in Tables 2 and 3, there is not a single hub language that stands out as the best one. Interestingly, all languages, across both experiments, are the best hub language for some evaluation language pair. For example, in our European-languages experiment, Es is the best choice for about 20% of the evaluation pairs, Tr and En are the best for about 17% each, while Gl and Be are the best for only 5 and 3 language pairs respectively. Clearly, not all languages are equally suited to be the hub language for many language pairs. Hence, it would be interesting to quantify how much better one could do by selecting the best hub language compared to a random choice. In order to achieve this, we define the expected gain G l of using language l as follows. Assume that we are interested in mapping N languages into the shared space m l is the accuracy 10 over a specified evaluation pair m when using language l as the hub. The random choice between N languages will have an expected accuracy equal to the average accuracy when using all languages as hub: The gain for that evaluation dataset m when using language l as hub, then, is g. Now, for a collection of M evaluation pairs we simply average their gains, in order to obtain the expected gain for using language l as the hub: The of this computation for both sets of experiments are presented in Figure 2. The bars marked'overall' match our above definition, as they present the expected gain computed over all evaluation language pairs. For good measure, we also present the average gain per language aggregated over the evaluation pairs where that language was indeed the best hub language ('when best' bars). Perhaps unsurprisingly, Az seems to be the worst hub language choice among the 10 European languages of the first experiment, with an expected loss (negative gain) of -0.4. This can be attributed to how distant Az is from all other languages, as well as to the fact that the Az pre-trained embeddings are of lower quality compared to all other languages (as the Az Wikipedia dataset is significantly smaller than the others). Similarly, Hi and Sv show expected loss for our second experiment. Note that English is not a bad hub choice per se -it exhibits a positive expected gain in both sets of experiments. However, there are languages with larger expected gains, like Es and Gl in the European-languages experiment that have a twice-as-large expected gain, while Ru has a 4 times larger expected gain in the distant-languages experiment. Of course, the language subset composition of these experiments could possibly impact those numbers. For example, there are three very related languages (Es, Gl, Pt) in the European languages set, which might boost the expected gain for that subset; however, the trends stand even if we compute the expected gain over a subset of the evaluation pairs, removing all pairs that include Gl or Pt. For example, after removing all Gl , Es has a slightly lower expected gain of 0.32, but is still the language with the largest expected gain. Identifying the best hub language for a given evaluation set The next step is attempting to identify potential characteristics that will allow us make educated decisions with regards to choosing the hub language, given a specific evaluation set. For example, should one choose a language typologically similar to the evaluation source, target, or both? Or should they use the source or the target of the desired evaluation set as the hub? Our first finding is that the best performing hub language will very likely be neither the source nor the target of the evaluation set. In our European-languages experiments, a language different than the source and the target yields the best accuracy for over 93% of the evaluation sets. Similarly, in the distant-languages experiment, there is only a single instance where the best performing hub language is either the source or the target evaluation language (for the Fr-Ru dataset), and for the other 97% of the cases the best option is a third language. We hypothesize that learning mappings for both language spaces of interest (hence rotating both spaces) allows for a more flexible alignment which leads to better downstream performance, compared to when one of the two spaces is fixed. Note that this contradicts the mathematical intuition discussed in Section 2 according to which a model learning a single mapping (keeping another word embedding space fixed) is as expressive as a model that learns two mappings for each of the languages. Our second finding is that the downstream performance correlates with measures of distance between languages and language spaces. The typological distance (d gen) between two languages can be approximated through their genealogical distance over hypothesized language family trees, which we obtain from the URIEL typological database . Also, recently motivated the use of Gromov-Hausdroff (GH) distance as an a priori estimation of how well two language embedding spaces can be aligned under an isometric transformation (which is an assumption most methods rely on). The authors also note that vector space GH distance correlates with typological language distance. We refer the reader to We find that there is a positive correlation between downstream LI performance and the genealogical distances between the source-hub and target-hub languages. The average (over all evaluation pairs) Pearson's correlation coefficient between P@1 and d gen is 0.49 for the distant languages experiment and 0.38 for the European languages one. A similar positive correlation of performance and the sum of the GH distances between the source-hub and target-hub spaces. On our distant languages experiment, the coefficient between P@1 and GH is equal to 0.45, while it is slightly lower (0.34) for our European languages experiment. High correlation examples from each experiment, namely Gl-En and En-Hi, are shown in Figure 3. Bi-, tri-, and multilingual systems The last part of our analysis compares bilingual, trilingual, and multilingual systems, with a focus on the under-represented languages. Through multiple experiments (complete evaluations are listed in the Appendix) we reach two main . On one hand, when evaluating on typologically distant languages, one should use as many languages as possible. In Table 5 we present one such example with on Az-Cs under various settings. On the other hand, when multiple related languages are available, one can achieve higher performance with multilingual systems containing all related languages and one more hub language, rather than learning diverse multilingual mappings using more languages. We confirm the latter observation with experiments on the Slavic (Be, Ru, Uk) and Iberian (Es, Gl, Pt) clusters, and present an example (Ru-Uk) in Table 5. With this work we challenge the standard practices in learning cross-lingual word embeddings. We empirically showed that the choice of the hub language is an important parameter that affects lexicon induction performance in both bilingual (between distant languages) and multilingual settings. More importantly, we hope that by providing new dictionaries and baseline on several language pairs, we will stir the community towards evaluating all methods in challenging scenarios that include under-represented language pairs. Towards this end, our analysis provides insights and general directions for stronger baselines for non-Anglocentric cross-lingual word embeddings. A Does evaluation directionality matter? We also explored whether there are significant differences between the evaluated quality of aligned spaces, when computed on both directions (src-trg and trg-src). We find that the evaluation direction indeed matters a lot, when the languages of the evaluation pair are very distant, in terms of morphological complexity and data availability (which affects the quality of the original embeddings). A prominent example, from our European-languages experiment, are evaluation pairs involving Az or Be. When evaluating on the Az-XX and Be-XX dictionaries, the word translation P@1 is more than 20 percentage points higher than when evaluating on the opposite direction (XX-Az or XX-Be). For example, Es-Az has a mere P@1 of 9.9, while Az-Es achieves a P@1 of 44.9. This observation holds even between very related languages (cf. Ru-Be: 12.8, Be-Ru: 41.1 and Tr-Az: 8.4, Az-Tr: 32.0), which supports our hypothesis that this difference is also due to the quality of the pre-trained embeddings. It is important to note that such directionality differences are not observed when evaluating distant pairs with presumably high-quality pre-trained embeddings e.g. Tr-Sk or Tr-Es; the P@1 for both directions is very close. Here we provide complete evaluation for our multilingual experiments. Tables 6-11 present P@1, P@5, and P@10 respectively, for the experiment on the 10 European languages. Similarly, on the distant languages experiment are shown in Tables 12, 13, and 14. Table 15 presents the P@1 of the bilingual experiments using MUSE. 7.5 7.0 Ru-Be 12.8 9.9 10.7 11.5 11.2 11.0 11.5 12.3 11.0 11.8 11. Tr-Sk 27.5 29.2 27.9 28.5 29.4 27.7 27.9 27.5 25.2 27.9 27.9

Represent the TLDR for finding the corresponding scientific article

The choice of the hub (target) language affects the quality of cross-lingual embeddings, which shouldn't be evaluated only  on English-centric dictionaries.

represent the short summary for finding the corresponding scientific article (long) Fewshot example: "We provide necessary and sufficient analytical forms for the critical points of the square loss functions for various neural networks, and exploit the analytical forms to characterize the landscape properties for the loss functions of these neural networks." == "Due to the success of deep learning to solving a variety of challenging machine learning tasks, there is a rising interest in understanding loss functions for training neural networks from a theoretical aspect. Particularly, the properties of critical points and the landscape around them are of importance to determine the convergence performance of optimization algorithms. In this paper, we provide a necessary and sufficient characterization of the analytical forms for the critical points (as well as global minimizers) of the square loss functions for linear neural networks. We show that the analytical forms of the critical points characterize the values of the corresponding loss functions as well as the necessary and sufficient conditions to achieve global minimum. Furthermore, we exploit the analytical forms of the critical points to characterize the landscape properties for the loss functions of linear neural networks and shallow ReLU networks. One particular is that: While the loss function of linear networks has no spurious local minimum, the loss function of one-hidden-layer nonlinear networks with ReLU activation function does have local minimum that is not global minimum. In the past decade, deep neural networks BID8 have become a popular tool that has successfully solved many challenging tasks in a variety of areas such as machine learning, artificial intelligence, computer vision, and natural language processing, etc. As the understandings of deep neural networks from different aspects are mostly based on empirical studies, there is a rising need and interest to develop understandings of neural networks from theoretical aspects such as generalization error, representation power, and landscape (also referred to as geometry) properties, etc. In particular, the landscape properties of loss functions (that are typically nonconex for neural networks) play a central role to determine the iteration path and convergence performance of optimization algorithms. One major landscape property is the nature of critical points, which can possibly be global minima, local minima, saddle points. There have been intensive efforts in the past into understanding such an issue for various neural networks. For example, it has been shown that every local minimum of the loss function is also a global minimum for shallow linear networks under the autoencoder setting and invertibility assumptions BID1 and for deep linear networks BID11; BID14; respectively under different assumptions. The conditions on the equivalence between local minimum or critical point and global minimum has also been established for various nonlinear neural networks; BID9; BID15; BID17; BID6 under respective assumptions. However, most previous studies did not provide characterization of analytical forms for critical points of loss functions for neural networks with only very few exceptions. In BID1, the authors provided an analytical form for the critical points of the square loss function of shallow linear networks under certain conditions. Such an analytical form further helps to establish the landscape properties around the critical points. Further in BID13, the authors characterized certain sufficient form of critical points for the square loss function of matrix factorization problems and deep linear networks. The focus of this paper is on characterizing the sufficient and necessary forms of critical points for broader scenarios, i.e., shallow and deep linear networks with no assumptions on data matrices and network dimensions, and shallow ReLU networks over certain parameter space. In particular, such analytical forms of critical points capture the corresponding loss function values and the necessary and sufficient conditions to achieve global minimum. This further enables us to establish new landscape properties around these critical points for the loss function of these networks under general settings, and provides alternative (yet simpler and more intuitive) proofs for existing understanding of the landscape properties. OUR CONTRIBUTION 1) For the square loss function of linear networks with one hidden layer, we provide a full (necessary and sufficient) characterization of the analytical forms for its critical points and global minimizers. These generalize the characterization in BID1 to arbitrary network parameter dimensions and any data matrices. Such a generalization further enables us to establish the landscape property, i.e., every local minimum is also a global minimum and all other critical points are saddle points, under no assumptions on parameter dimensions and data matrices. From a technical standpoint, we exploit the analytical forms of critical points to provide a new proof for characterizing the landscape around the critical points under full relaxation of assumptions, where the corresponding approaches in BID1 are not applicable. As a special case of linear networks, the matrix factorization problem satisfies all these landscape properties.2) For the square loss function of deep linear networks, we establish a full (necessary and sufficient) characterization of the analytical forms for its critical points and global minimizers. Such characterizations are new and have not been established in the existing art. Furthermore, such analytical form divides the set of non-global-minimum critical points into different categories. We identify the directions along which the loss function value decreases for two categories of the critical points, for which our directly implies the equivalence between the local minimum and the global minimum. For these cases, our proof generalizes the in BID11 under no assumptions on the network parameter dimensions and data matrices.3) For the square loss function of one-hidden-layer nonlinear neural networks with ReLU activation function, we provide a full characterization of both the existence and the analytical forms of the critical points in certain types of regions in the parameter space. Particularly, in the case where there is one hidden unit, our fully characterize the existence and the analytical forms of the critical points in the entire parameter space. Such characterization were not provided in previous work on nonlinear neural networks. Moreover, we apply our to a concrete example to demonstrate that both local minimum that is not a global minimum and local maximum do exist in such a case. Analytical forms of critical points: Characterizing the analytical form of critical points for loss functions of neural networks dates back to BID1, where the authors provided an analytical form of the critical points for the square loss function of linear networks with one hidden layer. In BID13, the authors provided a sufficient condition of critical points of a generic function, i.e., the fixed point of invariant groups. They then characterized certain sufficient forms of critical points for the square loss function of matrix factorization problems and deep linear networks, whereas our provide sufficient and necessary forms of critical points for deep linear networks via a different approach. Properties of critical points: BID1; BID0 studied the linear autoencoder with one hidden layer and showed the equivalence between the local minimum and the global minimum. Moreover, BID2 generalized these to the complex-valued autoencoder setting. The deep linear networks were studied by some recent work BID11; BID14 , in which the equivalence between the local minimum and the global minimum was established respectively under different assumptions. established a necessary and sufficient condition for a critical point of the deep linear network to be a global minimum. A similar was established in BID7 for deep linear networks under the setting that the widths of intermediate layers are larger than those of the input and output layers. The effect of regularization on the critical points for a two-layer linear network was studied in.For nonlinear neural networks, studied a nonlinear neural network with one hidden layer and sigmoid activation function, and showed that every local minimum is also a global minimum provided that the number of input units equals the number of data samples. BID9 considered a class of multi-layer nonlinear networks with a pyramidal structure, and showed that all critical points of full column rank achieve the zero loss when the sample size is less than the input dimension. These were further generalized to a larger class of nonlinear networks in BID15, in which they also showed that critical points with non-degenerate Hessian are global minimum. BID3 b) connected the loss surface of deep nonlinear networks with the Hamiltonian of the spin-glass model under certain assumptions and characterized the distribution of the local minimum. BID11 further eliminated some of the assumptions in BID3, and established the equivalence between the local minimum and the global minimum by reducing the loss function of the deep nonlinear network to that of the deep linear network. BID17 showed that a two-layer nonlinear network has no bad differentiable local minimum. BID6 studied a one-hidden-layer nonlinear neural network with the parameters restricted in a set of directions of lines, and showed that most local minima are global minima. considered a two-layer ReLU network with Gaussian input data, and showed that critical points in certain region are non-isolated and characterized the critical-point-free regions. Geometric curvature BID10 established the gradient dominance condition of deep linear residual networks, and further established the gradient dominance condition and regularity condition around the global minimizers for deep linear, deep linear residual and shallow nonlinear networks. BID12 studied the property of the Hessian matrix for deep linear residual networks. The local strong convexity property was established in BID16 for overparameterized nonlinear networks with one hidden layer and quadratic activation functions, and was established in for a class of nonlinear networks with one hidden layer and Gaussian input data. further established the local linear convergence of gradient descent method with tensor initialization. BID18 studied a one-hidden-layer nonlinear network with a single output, and showed that the volume of sub-optimal differentiable local minima is exponentially vanishing in comparison with the volume of global minima. BID5 investigated the saddle points in deep neural networks using the from statistical physics and random matrix theory. Notation: The pseudoinverse, column space and null space of a matrix M are denoted by M †, col(M) and ker(M), respectively. For any index sets I, J ⊂ N, M I,J denotes the submatrix of M formed by the entries with the row indices in I and the column indices in J. For positive integers i ≤ j, we define i: j = {i, i + 1, . . ., j − 1, j}. The projection operator onto a linear subspace V is denoted by P V. In this section, we study linear neural networks with one hidden layer. Suppose we have an input data matrix X ∈ R d0×m and a corresponding output data matrix Y ∈ R d2×m, where there are in total m data samples. We are interested in learning a model that maps from X to Y via a linear network with one hidden layer. Specifically, we denote the weight parameters between the output layer and the hidden layer of the network as A 2 ∈ R d2×d1, and denote the weight parameters between the hidden layer and the input layer of the network as A 1 ∈ R d1×d0. We are interested in the square loss function of this linear network, which is given by DISPLAYFORM0 Note that in a special case where X = I, L reduces to a loss function for the matrix factorization problem, to which all our apply. The loss function L has been studied in BID1 under the assumptions that d 2 = d 0 ≥ d 1 and the matrices XX, Y X (XX) −1 XY are invertible. In our study, no assumption is made on either the parameter dimensions or the invertibility of the data matrices. Such full generalization of the in BID1 turns out to be critical for our study of nonlinear shallow neural networks in Section 4.We further define Σ:= Y X † XY and denote its full singular value decomposition as U ΛU. Suppose that Σ has r distinct positive singular values σ 1 > · · · > σ r > 0 with multiplicities m 1,..., m r, respectively, and hasm zero singular values. Recall that DISPLAYFORM1 Our first provides a full characterization of all critical points of L. Theorem 1 (Characterization of critical points). All critical points of L are necessarily and sufficiently characterized by a matrix L 1 ∈ R d1×d0, a block matrix V ∈ R d2×d1 and an invertible matrix C ∈ R d1×d1 via DISPLAYFORM2 DISPLAYFORM3, where both V i ∈ R mi×pi and V ∈ Rm ×p consist of orthonormal columns with the number of columns DISPLAYFORM4 Theorem 1 characterizes the necessary and sufficient forms for all critical points of L. Intuitively, the matrix C captures the invariance of the product A 2 A 1 under an invertible transform, and L 1 captures the degree of freedom of the solution set for linear systems. In general, the set of critical points is uncountable and cannot be fully listed out. However, the analytical forms in eqs. and do allow one to construct some critical points of L by specifying choices of L 1, V, C that fulfill the condition in eq.. For example, choosing L 1 = 0 guarantees eq., in which case eqs. and yield a critical point (C −1 V U Y X †, U V C) for any invertible matrix C and any block matrix V that takes the form specified in Theorem 1. For nonzero L 1, one can fix a proper V and solve the linear equation on C in eq.. If a solution exists, we then obtain the form of a corresponding critical point. We further note that the analytical structures of the critical points are more important, which have direct implications on the global optimality conditions and landscape properties as we show in the remaining part of the section. Remark 1. We note that the block pattern parameters {p i} r i=1 andp denote the number of columns of {V i} r i=1 and V, respectively, and their sum equals the rank of A 2, i.e., DISPLAYFORM5 The parameters p i, i = 1,..., r,p of V contain all useful information of the critical points that determine the function value of L as presented in the following proposition. DISPLAYFORM6 Proposition 1 evaluates the function value L at a critical point using the parameters {p i} r i=1. To explain further, recall that the data matrix Σ has each singular value σ i with multiplicity m i. For each i, the critical point captures p i out of m i singular values σ i. Hence, for a σ i with larger value (i.e., a smaller index i), it is desirable that a critical point captures a larger number p i of them. In this way, the critical point captures more important principle components of the data so that the value of the loss function is further reduced as suggested by Proposition 1. In summary, the parameters {p i} r i=1 characterize how well the learned model fits the data in terms of the value of the loss function. Moreover, the parameters {p i} r i=1 also determine a full characterization of the global minimizers as given below. Proposition 2 (Characterization of global minimizers). A critical point (A 1, A 2) of L is a global minimizer if and only if it falls into the following two cases. DISPLAYFORM7 The analytical form of any global minimizer can be obtained from Theorem 1 with further specification to the above two cases. Proposition 2 establishes the neccessary and sufficient conditions for any critical point to be a global minimizer. If the data matrix Σ has a large number of nonzero singular values, i.e., the first case, one needs to exhaust the representation budget (i.e., rank) of A 2 and capture as many large singular values as the rank allows to achieve the global minimum; Otherwise, A 2 of a global minimizer can be non-full rank and still captures all nonzero singular values. Note that A 2 must be full rank in the case 1, and so is A 1 if we further adopt the assumptions on the network size and data matrices in BID1. Furthermore, the parameters {p i} r i=1 naturally divide all non-global-minimum critical points (A 1, A 2) of L into the following two categories.• (Non-optimal order): The matrix V specified in Theorem 1 satisfies that there exists 1 ≤ i < j ≤ r such that p i < m i and p j > 0.• (Optimal order): rank(A 2) < min{d 2, d 1} and the matrix V specified in Theorem 1 satisfies that DISPLAYFORM8 To understand the above two categories, note that a critical point of L with non-optimal order captures a smaller singular value σ j (since p j > 0) while skipping a larger singular value σ i with a lower index i < j (since p i < m i), and hence cannot be a global minimizer. On the other hand, although a critical point of L with optimal order captures the singular values in the optimal (i.e., decreasing) order, it does not fully utilize the representation budget of A 2 (because A 2 is non-full rank) to further capture nonzero singular values and reduce the function value, and hence cannot be a global minimizer either. Next, we show that these two types of non-global-minimum critical points have different landscape properties around them. Throughout, a matrix M is called the perturbation of M if it lies in an arbitrarily small neighborhood of M.Proposition 3 (Landscape around critical points). The critical points of L have the following landscape properties.1. A non-optimal-order critical point (A 1, A 2) has a perturbation (A 1, A 2) with rank(A 2) = rank(A 2), which achieves a lower function value; 2. An optimal-order critical point (A 1, A 2) has a perturbation (A 1, A 2) with rank(A 2) = rank(A 2) + 1, which achieves a lower function value; 3. Any point in X:= {(A 1, A 2): A 2 A 1 X = 0} has a perturbation (A 1, A 2), which achieves a higher function value;As a consequence, items 1 and 2 imply that any non-global-minimum critical point has a descent direction, and hence cannot be a local minimizer. Thus, any local minimizer must be a global minimizer. Item 3 implies that any point has an ascent direction whenever the output is nonzero. Hence, there does not exist any local/global maximizer in X. Furthermore, item 3 together with items 1 and 2 implies that any non-global-minimum critical point in X has both descent and ascent directions, and hence must be a saddle point. We summarize these facts in the following theorem. Theorem 2 (Landscape of L). The loss function L satisfies: 1) every local minimum is also a global minimum; 2) every non-global-minimum critical point in X is a saddle point. We note that the saddle points in Theorem 2 can be non-strict when the data matrices are singular. As an illustrative example, consider the following loss function of a shallow linear network L(a 2, a 1) = 1 2 (a 2 a 1 x − y) 2, where a 1, a 2, x and y are all scalars. Consider the case y = 0. Then, the Hessian at the saddle point a 1 = 0, a 2 = 1 is [x 2, 0; 0, 0], which does not have any negative eigenvalue. From a technical point of view, the proof of item 1 of Proposition 3 applies that in BID0 and generalizes it to the setting where Σ can have repeated singular values and may not be invertible. To further understand the perturbation scheme from a high level perspective, note that non-optimalorder critical points capture a smaller singular value σ j instead of a larger one σ i with i < j. Thus, one naturally perturbs the singular vector corresponding to σ j along the direction of the singular vector corresponding to σ i. Such a perturbation scheme preserves the rank of A 2 and reduces the value of the loss function. More importantly, the proof of item 2 of Proposition 3 introduces a new technique. As a comparison, BID1 proves a similar as item 2 using the strict convexity of the function, which requires the parameter dimensions to satisfy d 2 = d 0 ≥ d 1 and the data matrices to be invertible. In contrast, our proof completely removes these restrictions by introducing a new perturbation direction and exploiting the analytical forms of critical points in eqs. and and the condition in eq.. The accomplishment of the proof further requires careful choices of perturbation parameters as well as judicious manipulations of matrices. We refer the reader to the supplemental materials for more details. As a high level understanding, since optimal-order critical points capture the singular values in an optimal (i.e., decreasing) order, the previous perturbation scheme for non-optimal-order critical points does not apply. Instead, we increase the rank of A 2 by one in a way that the perturbed matrix captures the next singular value beyond the ones that have already been captured so that the value of the loss function can be further reduced. In this section, we study deep linear networks with ≥ 2 layers. We denote the weight parameters between the layers as A k ∈ R d k ×d k−1 for k = 1,...,, respectively. The input and output data are denoted by X ∈ R d0×m, Y ∈ R d ×m, respectively. We are interested in the square loss function of deep linear networks, which is given by DISPLAYFORM0, respectively, andm(k) zero singular values. Our first provides a full characterization of all critical points of L D, where we denote DISPLAYFORM1 Theorem 3 (Characterization of critical points). All critical points of L D are necessarily and sufficiently characterized by matrices DISPLAYFORM2.., A can be individually expressed out recursively via the following two equations: DISPLAYFORM3 DISPLAYFORM4 Note that the forms of the individual parameters A 1,..., A can be obtained as follows by recursively applying eqs. and. First, eq. with k = 0 yields the form of A (,2). Then, eq. with k = 0 and the form of A (,2) yield the form of A 1. Next, eq. with k = 1 yields the form of A (,3), and then, eq. with k = 1 and the forms of A (,3), A 1 further yield the form of A 2. Inductively, one obtains the expressions of all individual parameter matrices. Furthermore, the first condition in eq. FORMULA13 is a consistency condition that guarantees that the analytical form for the entire product of parameter matrices factorizes into the forms of individual parameter matrices. Similarly to shallow linear networks, while the set of critical points here is also uncountable, Theorem 3 suggests ways to obtain some critical points. For example, if we set L k = 0 for all k (i.e., eq. is satisfied), we can obtain the form of critical points for any invertible C k and proper V k with the structure specified in Theorem 3. For nonzero L k, eq. needs to be verified for given C k and V k to determine a critical point. Similarly to shallow linear networks, the parameters {p i} r i=1,p determine the value of the loss function at the critical points and further specify the analytical form for the global minimizers, as we present in the following two propositions. DISPLAYFORM5 DISPLAYFORM6 In particular, A (,2) can be non-full rank with rank(A (,2) ) = DISPLAYFORM7 The analytical form of any global minimizer can be obtained from Theorem 3 with further specification to the above two cases. In particular for case 1, if we further adopt the invertibility assumptions on data matrices as in BID1 and assume that all parameter matrices are square, then all global minima must correspond to full rank parameter matrices. We next exploit the analytical forms of the critical points to further understand the landscape of the loss function L D. It has been shown in BID11 that every local minimum of L D is also a global minimum, under certain conditions on the parameter dimensions and the invertibility of the data matrices. Here, our characterization of the analytical forms for the critical points allow us to understand such a from an alternative viewpoint. The proofs for certain cases (that we discuss below) are simpler and more intuitive, and no assumption is made on the data matrices and dimensions of the network. Similarly to shallow linear networks, we want to understand the local landscape around the critical points. However, due to the effect of depth, the critical points of L D are more complicated than those of L. Among them, we identify the following subsets of the non-global-minimum critical DISPLAYFORM8 • (Deep-non-optimal order): There exist 0 ≤ k ≤ − 2 such that the matrix V k specified in Theorem 3 satisfies that there exist 1 ≤ i < j ≤ r(k) such that p i (k) < m i (k) and p j (k) > 0.• (Deep-optimal order): (A, A −1) is not a global minimizer of L D with A (−2,1) being fixed, rank(A) < min{d, d −1}, and the matrix V −2 specified in Theorem 3 satisfies that DISPLAYFORM9 The following summarizes the landscape of L D around the above two types of critical points. The loss function L D has the following landscape properties. deep-non-optimal-order critical point (A 1, . . ., A) has a perturbation (A 1, . . ., A k+1, . . ., A) with rank(A) = rank(A), which achieves a lower function value. 2. A deep-optimal-order critical point (A 1, . . ., A) has a perturbation (A 1, . . ., A −1, A) with rank(A) = rank(A) + 1, which achieves a lower function value. 3. Any point in X D:= {(A 1, . . ., A): A (,1) X = 0} has a perturbation (A 1, . . ., A) that achieves a higher function value. Consequently, 1) every local minimum of L D is also a global minimum for the above two types of critical points; and 2) every critical point of these two types in X D is a saddle point. Theorem 4 implies that the landscape of L D for deep linear networks is similar to that of L for shallow linear networks, i.e., the pattern of the parameters {p i (k)} r(k) i=1 implies different descent directions of the function value around the critical points. Our approach does not handle the remaining set of non-global minimizers, i.e., there exists q ≤ −1 such that (A, . . ., A q) is a global minimum point of L D with A (q−1,1) being fixed, and A (,q) is of optimal order. It is unclear how to perturb the intermediate weight parameters using their analytical forms for deep networks, and we leave this as an open problem for the future work. In this section, we study nonlinear neural networks with one hidden layer. In particular, we consider nonlinear networks with ReLU activation function σ: R → R that is defined as σ(x):= max{x, 0}. Our study focuses on the set of differentiable critical points. The weight parameters between the layers are denoted by A 2 ∈ R d2×d1, A 1 ∈ R d1×d0, respectively, and the input and output data are denoted by X ∈ R d0×m, Y ∈ R d2×m, respectively. We are interested in the square loss function which is given by DISPLAYFORM0 where σ acts on A 1 X entrywise. Existing studies on nonlinear networks characterized the sufficient conditions for critical points being global minimum BID9 Since the activation function σ is piecewise linear, the entire parameter space can be partitioned into disjoint cones. In particular, we consider the set of cones K I×J where I ⊂ {1, . . ., d 1}, J ⊂ {1, . . ., m} that satisfy DISPLAYFORM1 where "≥" and "<" represent entrywise comparisons. Within K I×J, the term σ(A 1 X) activates only the entries σ(A 1 X) I:J, and the corresponding loss function L N is equivalent to DISPLAYFORM2 Hence, within K I×J, L N reduces to the loss of a shallow linear network with parameters ((A 2):,I, (A 1) I,: ) and input & output data pair (X :,J, Y :,J). Note that our on shallow linear networks in Section 2 are applicable to all parameter dimensions and data matrices. Thus, Theorem 1 fully characterizes the forms of critical points of L N in K I×J. Moreover, the existence of such critical points can be analytically examined by substituting their forms into eq.. In summary, we obtain the following , where we denote Σ J:= Y:,J X †:,J X:,J Y:,J with the full singular value decomposition U J Λ J U J, and suppose that Σ J has r(J) distinct positive singular values σ 1 (J) > · · · > σ r(J) (J) with multiplicities m 1,..., m r(J), respectively, andm(J) zero singular values. Proposition 6 (Characterization of critical points). All critical points of L N in K I×J for any I ⊂ {1, . . ., d 1}, J ⊂ {1, . . ., m} are necessarily and sufficiently characterized by an L 1 ∈ R |I|×d0, a block matrix V ∈ R d2×|I| and an invertible matrix C ∈ R |I|×|I| such that DISPLAYFORM3 DISPLAYFORM4 ×p consist of orthonormal columns with p i ≤ m i for i = 1,..., r(J),p ≤m such that DISPLAYFORM5 Moreover, a critical point in K I×J exists if and only if there exists such C, V, L 1 that DISPLAYFORM6 Other entries of A 1 X < 0.To further illustrate, we consider a special case where the nonlinear network has one unit in the hidden layer, i.e., d 1 = 1, in which case A 1 and A 2 are row and column vectors, respectively. Then, the entire parameter space can be partitioned into disjoint cones taking the form of K I×J, and I = {1} is the only nontrivial choice. We obtain the following from Proposition 6.Proposition 7 (Characterization of critical points). Consider L N with d 1 = 1 and any J ⊂ {1, . . ., m}. Then, any nonzero critical point of L N within K {1}×J can be necessarily and sufficiently characterized by an 1 ∈ R 1×d0, a block unit vector v ∈ R d2×1 and a scalar c ∈ R such that DISPLAYFORM7 Specifically, v is a unit vector that is supported on the entries corresponding to the same singular value of Σ J. Moreover, a nonzero critical point in K {1}×J exists if and only if there exist such c, v, 1 that satisfy DISPLAYFORM8 DISPLAYFORM9 We note that Proposition 7 characterizes both the existence and the forms of critical points of L N over the entire parameter space for nonlinear networks with a single hidden unit. The condition in eq. FORMULA24 is guaranteed because P ker(v) = 0 for v = 0.To further understand Proposition 7, suppose that there exists a critical point in K {1}×J with v being supported on the entries that correspond to the i-th singular value of Σ J. Then, Proposition 1 implies that DISPLAYFORM10 In particular, the critical point achieves the local minimum DISPLAYFORM11. This is because in this case the critical point is full rank with an optimal order, and hence corresponds to the global minimum of the linear network in eq.. Since the singular values of Σ J may vary with the choice of J, L N may achieve different local minima in different cones. Thus, local minimum that is not global minimum can exist for L N. The following proposition concludes this fact by considering a concrete example. Proposition 8. For one-hidden-layer nonlinear neural networks with ReLU activation function, there exists local minimum that is not global minimum, and there also exists local maximum. FORMULA13 and FORMULA19 hold if c −1 (v) 1,: ≥ 0, 1,: < 0. Similarly to the previous case, choosing c = 1, v =, 1 = (−1, 0) yields a local minimum that achieves the function value L n = 2. Hence, local minimum that is not global minimum does exist. Moreover, in the cone K I×J with I = {1}, J = ∅, the function L N remains to be the constant 5 2, and all points in this cone are local minimum or local maximum. Thus, the landscape of the loss function of nonlinear networks is very different from that of the loss function of linear networks. In this paper, we provide full characterization of the analytical forms of the critical points for the square loss function of three types of neural networks, namely, shallow linear networks, deep linear networks, and shallow ReLU nonlinear networks. We show that such analytical forms of the critical points have direct implications on the values of the corresponding loss functions, achievement of global minimum, and various landscape properties around these critical points. As a consequence, the loss function for linear networks has no spurious local minimum, while such point does exist for nonlinear networks with ReLU activation. In the future, it is interesting to further explore nonlinear neural networks. In particular, we wish to characterize the analytical form of critical points for deep nonlinear networks and over the full parameter space. Such will further facilitate the understanding of the landscape properties around these critical points. Notations: For any matrix M, denote vec(M) as the column vector formed by stacking its columns. Denote the Kronecker product as "⊗". Then, the following useful relationships hold for any dimension compatible matrices M, U, V, W: DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 Recall that a point DISPLAYFORM4 DISPLAYFORM5 We first prove eqs. and. DISPLAYFORM6 Next, we derive the form of A 2. Recall the full singular value decomposition Σ = U ΛU, where Λ is a diagonal matrix with distinct singular values σ 1 >... > σ r > 0 and multiplicities m 1,..., m r, respectively. We also assume that there arem number of zero singular values in Λ. Using the fact that P col(A2) = U P col(U A2) U, the last equality in eq. reduces to DISPLAYFORM7 By the multiplicity pattern of the singular values in Λ, P col(U A2) must be block diagonal. Specifically, we can write P col(U A2) = diag(P 1, . . ., P r, P), where P i ∈ R mi×mi and P ∈ Rm ×m.Also, since P col(U A2) is a projection, P 1,..., P r, P must all be projections. Note that P col(U A2) has rank rank(A 2), and suppose that P 1,..., P r, P have ranks p 1,..., p r,p, respectively. Then, we must have p i ≤ m i for i = 1,..., r,p ≤m and r i=1 p i +p = rank(A 2). Also, note that each projection can be expressed as P i = V i V i with V i ∈ R mi×pi, V ∈ Rm ×p consisting of orthonormal columns. Hence, we can write P col(U A2) = V V where V = diag(V 1, . . ., V r, V). We then conclude that P col(A2) = U P col(U A2) U = U V V U. Thus, A 2 has the same column space as U V, and there must exist an invertible matrix DISPLAYFORM8 Then, plugging A † 2 = C −1 V U into eq. yields the desired form of A 1.We now prove eq.. Note that the above proof is based on the equations DISPLAYFORM9 Hence, the forms of A 1, A 2 in eqs. and need to further satisfy ∇ A2 L = 0. By eq. FORMULA19 and the form of A 2, we obtain that DISPLAYFORM10 This expression, together with the form of A 1 in eq., implies that DISPLAYFORM11 where (i) uses the fact that X † XX = X, (ii) uses the fact that the block pattern of V is compatible with the multiplicity pattern of the singular values in Λ, and hence V V ΛV = ΛV. On the other hand, we also obtain that DISPLAYFORM12 Thus, to satisfy ∇ A2 L = 0 in eq. FORMULA12, we require that DISPLAYFORM13 which is equivalent to DISPLAYFORM14 Lastly, note that (I − U V (U V) ) = P col(U V) ⊥, and (I − V V) = P ker(V), which concludes the proof. By expansion we obtain that L = DISPLAYFORM0. Consider any (A 1, A 2) that satisfies eq. FORMULA4, we have shown that such a point also satisfies eq., which further yields that DISPLAYFORM1 where (i) follows from the fact that Tr(P col(A2) Σ P col(A2) ) = Tr(P col(A2) Σ), and (ii) uses the fact that P col(A2) = U P col(U A2) U. In particular, a critical point (A 1, A 2) satisfies eq.. Moreover, using the form of the critical point A 2 = U V C, eq. FORMULA20 further becomes DISPLAYFORM2 where (i) is due to P col(V C) = P col(V) = V V, and (ii) utilizes the block pattern of V and the multiplicity pattern of Λ that are specified in Theorem 1.: Consider a critical point (A 1, A 2) with the forms given by Theorem 1. By choosing L 1 = 0, the condition in eq. FORMULA4 is guaranteed. Then, we can specify a critical point with any V that satisfies the block pattern specified in Theorem 1, i.e., we can choose any p i, i = 1,..., r,p such that p i ≤ m i for i = 1,..., r,p ≤m and DISPLAYFORM0 m i, the global minimum value is achieved by a full rank A 2 with rank(A 2) = min{d 2, d 1} and DISPLAYFORM1 That is, the singular values are selected in a decreasing order to minimize the function value.: If (A 2, A 1) is a global minimizer and min{d y, d} > r i=1 m i, the global minimum can be achieved by choosing p i = m i for all i = 1,..., r andp ≥ 0. In particular, we do not need a full rank A 2 to achieve the global minimum. For example, we can choose rank(A 2) = r i=1 m i < min{d y, d} with p i = m i for all i = 1,..., r andp = 0. We first prove item 1. Consider a non-optimal-order critical point (A 1, A 2). By Theorem 1, we can write A 2 = U V C where V = [diag(V 1, . . ., V r, V), 0] and V i, i = 1,..., r, V consist of orthonormal columns. Define the orthonormal block diagonal matrix Since (A 1, A 2) is a non-optimal-order critical point, there exists 1 ≤ i < j ≤ r such that p i < m i and p j > 0. Then, consider the following perturbation of U S for some > 0. DISPLAYFORM0 DISPLAYFORM1 with which we further define the perturbation matrix A 2 = M S V C. Also, let the perturbation matrix A 1 be generated by eq. with U ← M and V ← S V. Note that with this construction, (A 1, A 2) satisfies eq., which further implies eq. for (A 1, A 2), i.e., A 2 A 1 X = P col(A2) Y X † X. Thus, eq. holds for the point (A 1, A 2), and we obtain that DISPLAYFORM2 where the last equality uses the fact that S ΛS = Λ, as can be observed from the block pattern of S and the multiplicity pattern of Λ. Also, by the construction of M and the form of S V, a careful calculation shows that only the i, j-th diagonal elements of P col(S U M S V) have changed, i.e., DISPLAYFORM3 As the index i, j correspond to the singular values σ i, σ j, respectively, and σ i > σ j, one obtain that DISPLAYFORM4 Thus, the construction of the point (A 2, A 1) achieves a lower function value for any > 0. Letting → 0 and noticing that M is a perturbation of U S, the point (A 2, A 1) can be in an arbitrary neighborhood of (A 2, A 1). Lastly, note that rank(A 2) = rank(A 2). This completes the proof of item 1.Next, we prove item 2. Consider an optimal-order critical point (A 1, A 2). Then, A 2 must be non-full rank, since otherwise a full rank A 2 with optimal order corresponds to a global minimizer by Proposition 2. Since there exists some k ≤ r such that 0]. Using this expression, eq. yields that DISPLAYFORM5 DISPLAYFORM6 We now specify our perturbation scheme. Recalling the orthonormal matrix S defined in eq.. Then, we consider the following matrices for some 1, 2 > 0 DISPLAYFORM7 For this purpose, we need to utilize the condition of critical points in eq., which can be equivalently expressed as DISPLAYFORM8 (ii) ⇔ (CL 1) (rank(A2)+1):d1,: XY (I − U S :,1:(q−1) (U S :,1:(q−1) ) ) = 0where (i) follows by taking the transpose and then simplifying, and (ii) uses the fact that V = SS V = S:,1:(q−1) in the case of optimal-order critical point. Calculating the function value at (A 1, A 2), we obtain that DISPLAYFORM9. We next simplify the above three trace terms using eq.. For the first trace term, observe that DISPLAYFORM10 2 Tr(S :,q ΛS :,q) where (i) follows from eq. as S:,q is orthogonal to the columns of S:,1:(q−1). For the second trace term, we obtain that DISPLAYFORM11 = 2Tr(2 U S :,q (CL 1) (rank(A2)+1),: XY U V diag (U V diag) ) + 2Tr(1 2 U S :,q S :,q ΛSS V diag (U V diag) ) (i) = 2Tr(2 U S :,q (CL 1) (rank(A2)+1),: XY U V diag (U V diag) ) + 2Tr(1 2 σ k U S :,q e q S V diag (U V diag) )(ii) = 2Tr(2 U S :,q (CL 1) (rank(A2)+1),: XY U V diag (U V diag) ), where (i) follows from S:,q ΛS = σ k e q, and (ii) follows from e q S V diag = 0. For the third trace term, we obtain that 2Tr(P Y) = 2Tr(2 U S :,q (CL 1) (rank(A2)+1),: XY ) + 2Tr(1 2 U S :,q (U S :,q) Σ) = 2Tr(2 U S :,q (CL 1) (rank(A2)+1),: XY ) + 2Tr(1 2 S :,q ΛS :,q).Combining the expressions for the three trace terms above, we conclude that Consider a critical point (A 1, . . ., A) so that eq. FORMULA4 Observe that the product matrix A (,2) is equivalent to the class of matrices B 2 ∈ R min{d,...,d2}×d1.Consider a critical point (B 2, A 1) of the shallow linear network L:= The proof is similar to that for shallow linear networks. Consider a deep-non-optimal-order critical point (A 1, . . ., A), and define the orthonormal block matrix S k using the blocks of V k in a similar way as eq.. Then, A (l,k+2) takes the form A (l,k+2) = U k S k S k V k C k. Since A (l,k+2) is of non-optimal order, there exists i < j < r(k) such that p i (k) < m i (k) and p j (k) > 0. Thus, we perturb the j-th column of U k S k to be, and denote the ing matrix as M k.Then, we perturb A to be A = M k (U k S k) A so that A A (−1,k+2) = M k S k V k C k. Moreover, we generate A k+1 by eq. with U k ← M k, V k ← S k V k. Note that such construction satisfies eq., and hence also satisfies eq., which further yields that DISPLAYFORM0 With the above equation, the function value at this perturbed point is evaluated as DISPLAYFORM1 Then, a careful calculation shows that only the i, j-th diagonal elements of DISPLAYFORM2 have changed, and are Now consider a deep-optimal-order critical point (A 1, . . ., A). Note that with A (−2,1) fixed to be a constant, the deep linear network reduces to a shallow linear network with parameters (A, A −1). Since (A, A −1) is not a non-global minimum critical point of this shallow linear network and A is of optimal-order, we can apply the perturbation scheme in the proof of Proposition 3 to identify a perturbation (A, A −1) with rank(A) = rank(A) + 1 that achieves a lower function value. Consider any point in X D. Since A (,1) X = 0, we can scale the nonzero row, say, the i-th row (A) i,: A (−1,1) X properly in the same way as that in the proof of Proposition 3 to increase the function value. Lastly, item 1 and item 2 imply that every local minimum is a global minimum for these two types of critical points. Moreover, combining items 1,2 and 3, we conclude that every critical point of these two types in X D is a saddle point."

An empirical investigation of GAN-based alignment of word vector spaces, focusing on cases, where linear transformations provably exist, but training is unstable.


Question: Represent the article for finding a single-sentence TLDR

Question: Interpreting generative adversarial network (GAN) training as approximate divergence minimization has been theoretically insightful, has spurred discussion, and has lead to theoretically and practically interesting extensions such as f-GANs and Wasserstein GANs. For both classic GANs and f-GANs, there is an original variant of training and a "non-saturating" variant which uses an alternative form of generator update. The original variant is theoretically easier to study, but the alternative variant frequently performs better and is recommended for use in practice. The alternative generator update is often regarded as a simple modification to deal with optimization issues, and it appears to be a common misconception that the two variants minimize the same divergence. In this short note we derive the divergences approximately minimized by the original and alternative variants of GAN and f-GAN training. This highlights important differences between the two variants. For example, we show that the alternative variant of KL-GAN training actually minimizes the reverse KL divergence, and that the alternative variant of conventional GAN training minimizes a "softened" version of the reverse KL. We hope these may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

Typical GAN training doesn't optimize Jensen-Shannon, but something like a reverse KL divergence.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the article for finding a single-sentence TLDR

Question: REINFORCE can be used to train models in structured prediction settings to directly optimize the test-time objective. However, the common case of sampling one prediction per datapoint (input) is data-inefficient. We show that by drawing multiple samples (predictions) per datapoint, we can learn with significantly less data, as we freely obtain a REINFORCE baseline to reduce variance. Additionally we derive a REINFORCE estimator with baseline, based on sampling without replacement. Combined with a recent technique to sample sequences without replacement using Stochastic Beam Search, this improves the training procedure for a sequence model that predicts the solution to the Travelling Salesman Problem. REINFORCE is a well known policy optimization algorithm that learns directly from experience. Variants of it have been used to train models for a wide range of structured prediction tasks, such as Neural Machine Translation BID12 BID0, Image Captioning (b) and predicting solutions (tours) for the Travelling Salesman Problem (TSP) BID1 BID6. As opposed to maximum likelihood (supervised) learning, the appeal of using REINFORCE for structured prediction is that it directly optimizes the test-time performance. When using REINFORCE, often for each datapoint (e.g. a sentence, image or TSP instance) only a single sample/prediction (e.g. a translation, caption or tour) is used to construct a gradient estimate. From a classic Reinforcement Learning (RL) point of view, this makes sense, as we may not be able to evaluate multiple sampled actions for a state (datapoint). However, from a data point of view, this is inefficient if we can actually evaluate multiple samples, such as in a structured prediction setting. Reinforcement Learning with multiple samples/predictions for a single datapoint has been used before (e.g. BID14 ;), but we use the samples as counterfactual information by constructing a (local, for a single datapoint) REINFORCE baseline. A similar idea was applied for variational inference by BID10.Many structured prediction tasks can be formulated in terms of sequence modelling, which is the focus of this paper. In most sequence modelling tasks, the objective is a deterministic function of the predicted sequence. As a , duplicate sampled sequences are uninformative and therefore do not improve the quality of the gradient estimate. To solve this problem, we propose to use sampling without replacement to construct a better gradient estimate. This is inspired by recent work by BID7, who introduce Stochastic Beam Search as a method to sample sequences without replacement, and use this to construct a (normalized) importance-weighted estimator for (sentence level) BLEU score. We extend this idea to estimate policy gradients using REINFORCE, and we show how to use the same set of samples (without replacement) to construct a baseline. This way we can leverage sampling without replacement to improve training of sequence models. In our experiment, we consider the TSP and show that using REINFORCE with multiple samples is beneficial compared to single sample REINFORCE, both computationally and in terms of data-efficiency. Additionally, for a sample size of 4 − 8 samples per datapoint, sampling without replacement in slightly faster learning. The REINFORCE estimator allows to estimate gradients of the expectation E y∼p θ (y) [f (y)] by the relation: DISPLAYFORM0 If we also have a context, or datapoint, x (such as a source sentence), we may write p θ (y|x) and f (y, x), but in this paper, we leave dependence on x implicit. Extension of the derived estimators to a minibatch of datapoints x is straightforward. Typically, we estimate the expectation using samples y 1,..., y k and we may reduce variance of the estimator by using a baseline B i that is independent of the sample y i (but may depend on the other samples y j, j = i): DISPLAYFORM1 In practice, often a single sample y is used (per datapoint x, as we already have a batch of datapoints) to compute the estimate, e.g. k = 1, but in this paper we consider k > 1. In this paper, we consider a parametric distribution over discrete structures (sequences). Enumerating all n possible sequences as y 1,..., y n, we indicate with y i the i-th possible outcome, which has log-probability φ i = log p θ (y i) defined by the model. We can use the Gumbel-Max trick BID4 BID9 to sample y according to this distribution as follows: let G i ∼ Gumbel (a standard Gumbel distribution) for i = 1,..., n i.i.d., and let y = y i *, where DISPLAYFORM0 For a proof we refer to BID9. In a slight abuse of notation, we write G φi = φ i + G i, and we call G φi the (Gumbel-) perturbed log-probability of y i.The Gumbel-Max trick can be extended to the Gumbel-Top-k trick BID7 to draw an ordered sample without replacement, by taking the top k largest perturbed log-probabilities (instead of just one, the argmax). The is equivalent to sequential sampling without replacement, where after an element y is sampled, it is removed from the domain and the remaining probabilities are renormalized. The Gumbel-Top-k trick is equivalent to Weighted Reservoir Sampling BID3, as was noted by BID15. The ordered sample is also known as a partial ranking according to the Plackett-Luce model BID11 BID8.For a sequence model with exponentially large domain, naive application of the Gumbel-Top-k trick is infeasible, but an equivalent can be obtained using Stochastic Beam Search BID7. This modification of beam search expands the k partial sequences with maximum (Gumbel) perturbed log-probability, effectively replacing the standard top k operation by sampling without replacement. The ing top k completed sequences are a sample without replacement from the sequence model, by the equivalence to the Gumbel-Top-k trick. For details we refer to BID7. For many applications we need to estimate the expectation of a function f (y), where y is the realization of a variable with a discrete probability distribution p θ (y). When using Monte Carlo (MC) sampling (with replacement), we write y i to indicate the i-th sample in a set of samples. In contrast, when sampling without replacement we find it convenient to write y i (with superscript i) to refer to the i-th possible value in the domain, so (like we did in Section 2.2) we can enumerate the domain with n possible values as y 1,..., y n. This notation allows us to write out the expectation of f (y): DISPLAYFORM0 Published at the ICLR 2019 workshop: Deep RL Meets Structured Prediction Using MC sampling with replacement, we estimate equation 3 using k samples y 1,..., y k: DISPLAYFORM1 When sampling without replacement using the Gumbel-Top-k trick (Section 2.2) we write S as the set of k largest indices of G φi (i.e. S = arg top k{G φi : i ∈ {1, ..., n}}), so the sample (of size k) without replacement is {y i : i ∈ S}. We can use the sample S with the estimator derived by BID16, based on priority sampling BID2. This means that, to correct for the effects of sampling without replacement, we include importance weights DISPLAYFORM2 Here κ is the (k + 1)-th largest value of {G φi : i ∈ {1, ..., n}}, i.e. the (k + 1)-th largest Gumbel perturbed log-probability, and q θ,a (y DISPLAYFORM3) is the probability that the perturbed log-probability of y i exceeds a. Then we can use the following estimator: DISPLAYFORM4 This estimator is unbiased, and we include a copy of the proof by BID7 (adapted from the proofs by BID2 and Vieira FORMULA0) in Appendix A, as this introduces notation and is the basis for the proof in Appendix C.Intuition behind this estimator comes from the related threshold sampling scenario, where instead of fixing the sample size k, we fix the threshold a and define a variably sized sample S = {i ∈ {1, ..., n}: G φi > a}. With threshold sampling, each element y i in the domain is sampled independently with probability P (G φi > a) = q θ,a (y i), and DISPLAYFORM5 is a standard importance weight. As it turns out, instead of having a fixed threshold a, we can fix the sample size k and use κ as empirical threshold (as i ∈ S if G φi > κ), and still obtain an unbiased estimator BID2 BID16.As was shown by BID7, in practice it is preferred to normalize the importance weights to reduce variance. This means that we compute the normalization W (S) = i∈S p θ (y i) q θ,κ (y i) and obtain the following (biased) estimator: DISPLAYFORM6 Typically REINFORCE is applied with a single sample y per datapoint x (e.g. one translation per source sentence, or, in our experiment, a single tour per TSP instance). In some cases, it may be preferred to take multiple samples y per datapoint x as this requires less data. Taking multiple samples also gives us counterfactual information which can be used to construct a strong (local) baseline. Additionally, we obtain computational benefits, as for encoder-decoder models we can obtain multiple samples using only a single pass through the encoder. With replacement, we can use the estimator in equation 2, where we can construct a baseline B i for the i-th term based on the other samples j = i: DISPLAYFORM0 The form in equation 8 is convenient for implementation as it allows to compute a fixed'baseline' B = 1 k k j=1 f (y j) once and correct for the bias (as B depends on y i) by normalizing using DISPLAYFORM1 For details and a proof of unbiasedness we refer to Appendix C. The basic REINFORCE without replacement estimator follows from combining equation 1 with equation 5 for an unbiased estimator: DISPLAYFORM0 Similar to equation 6, we can compute a lower variance but biased variant by normalizing the importance weights using the normalization W (S) = i∈S DISPLAYFORM1. When sampling without replacement, the individual samples are dependent, and therefore we cannot simply define a baseline based on the other samples as we did in Section 3.1. However, similar to the'baseline' DISPLAYFORM2 based on the complete sample S (without replacement), using equation 5: DISPLAYFORM3 Using this baseline introduces a bias that we cannot simply correct for by a constant term (as we did in equation 8), as the importance weights depend on y i. Instead, we weight the individual terms by DISPLAYFORM4 This estimator is unbiased and we give the full proof in Appendix C.For the normalized version, we use the normalization W (S) = i∈S p θ (y i) q θ,κ (y i) for the baseline, and DISPLAYFORM5 to normalize the outer terms: DISPLAYFORM6 It seems odd to normalize the terms in the outer sum by We consider the task of predicting the solution for instances of the Travelling Salesman Problem (TSP) BID17 BID1 BID6. The problem is to find the order in which to visit locations (specified by their x, y coordinates) to minimize total travelling distance. A policy is trained using REINFORCE to minimize the expected length of a tour (sequence of locations) predicted by the model. The Attention Model by BID6 is a sequence model that considers each instance as a fully connected graph of nodes which are processed by an encoder. The decoder then produces the tour as a sequence of nodes to visit, one node at a time, where it autoregressively uses as input the node visited in the previous step.. REINFORCE is used with replacement (WR) and without replacement (WOR) using k = 4 (top row) or k = 8 (bottom row) samples per instance, and a local baseline based on the k samples for each instance. We compare against REINFORCE using one sample per instance, either with a baseline that is the average of the batch, or the strong greedy rollout baseline by BID6 that requires an additional rollout of the model. We use the source code by BID6 1 to reproduce their TSP experiment with 20 nodes (as larger instances diminish the benefit of sampling without replacement). We implement REIN-FORCE estimators based on multiple samples, either sampled with replacement (WR) or without replacement (WOR) using Stochastic Beam Search BID7. We compare the following four estimators:• Single sample with a batch baseline. Here we compute the standard REINFORCE estimator (equation 2) with a single sample (k = 1). We use a batch of 512 instances (datapoints) and as baseline we take the average of the tour lengths in the batch, hence each instances uses the same baseline. This is implemented as using the exponential moving average baseline by BID6 with β = 0.• Single sample with a greedy rollout baseline, and batch size 512. As baseline, we use a greedy rollout: for each instance x we take the length of the tour that is obtained by greedily selecting the next location according to an earlier (frozen) version of the model. This baseline, similar to self-critical training BID13, corresponds to the best found by BID6, superior to using an exponential moving average or learned value function. However, the greedy rollout requires an additional forward pass through the model.• Multiple samples with replacement (WR) with a local baseline. Here we compute the estimator in equation 8 based on k = 4, 8 samples. We use a batch size of 512 k, so the total number of samples is the same. The baseline is local as it is different for each datapoint, but it does not require additional model evaluations like the greedy rollout.• Multiple samples without replacement (WOR) with a local baseline. Here we use the (biased) normalized without replacement estimator in equation 11 with k = 4, 8 samples and batch size 512 k. Samples are drawn without replacement using Stochastic Beam Search BID7. For fair comparison, we do not take a (k + 1)-th sample to compute κ, but sacrifice the k-th sample and compute the summation in equation 11 with the remaining k − 1 (3 or 7) samples. Note that a single sample with a local baseline is not possible, which is why we use the batch baseline. The model architecture and training hyperparameters (except batch size) are as in the paper by BID6. We present the in terms of the validation set (not used for additional tuning) optimality gap during training in Figure 1, using k = 4 (top row) and k = 8 (bottom row). We found diminishing returns for larger k. The left column presents the in terms of the number of gradient update steps (minibatches). We see that sampling without replacement performs on par (k = 8) or slightly better than using the strong but computationally expensive greedy rollout baseline or using multiple samples with replacement. The standard batch baseline performs significantly worse. The estimators based on multiple samples do not lose (much) final performance, while using significantly less instances. In the right column, where are presented in terms of the number of instances, this effectiveness is confirmed, and we observe that sampling without replacement is preferred to sampling with replacement. The difference is small, but there is also not much room for improvement as are close to optimal. The benefit of learning with less data may be small if data is easily generated (as in our setting), but there is also a significant computational benefit as we need significantly fewer encoder evaluations. In this paper, we have derived REINFORCE estimators based on drawing multiple samples, with and without replacement, and evaluated the effectiveness of the proposed estimators in a structured prediction setting: the prediction of tours for the TSP. The derived estimators yield comparable to recent using REINFORCE with a strong greedy rollout baseline, at greater data-efficiency and computational efficiency. These estimators are especially well suited for structured prediction settings, where the domain is too large to compute exact gradients, but we are able to take multiple samples for the same datapoint, and the objective is a deterministic function of the sampled prediction. We hope the proposed estimators have potential to be used to improve training efficiency in more structured prediction settings, for example in the context of Neural Machine Translation or Image Captioning, where depending on the entropy of the model, sampling without replacement may yield a beneficial improvement. We include here in full the proof by BID7, as this introduces necessary notation and helps understanding of the proof in Appendix C.A.1 PROOF OF UNBIASEDNESS OF PRIORITY SAMPLING ESTIMATOR BY KOOL ET AL.The following proof is adapted from the proofs by BID2 and BID16. For generality of the proof, we write DISPLAYFORM0, and we consider general keys h i (not necessarily Gumbel perturbations).We assume we have a probability distribution over a finite domain 1,..., n with normalized probabilities p i, e.g. n i=1 p i = 1. For a given function f (i) we want to estimate the expectation DISPLAYFORM1 Each element i has an associated random key h i and we define q i (a) = P (h i > a). This way, if we know the threshold a it holds that q i (a) = P (i ∈ S) is the probability that element i is in the sample S. As was noted by BID16, the actual distribution of the key does not influence the unbiasedness of the estimator but does determine the effective sampling scheme. Using the Gumbel perturbed log-probabilities as keys (e.g. h i = G φi) is equivalent to the PPSWOR scheme described by BID16.We define shorthand notation h 1:n = {h 1, ..., h n}, h −i = {h 1, ..., h i−1, h i+1, ..., h n} = h 1:n \{h i}. For a given sample size k, let κ be the (k + 1)-th largest element of h 1:n, so κ is the empirical threshold. Let κ i be the k-th largest element of h −i (the k-th largest of all other elements).Similar to BID2 we will show that every element i in our sample contributes an unbiased estimate of E[f (i)], so that the total estimator is unbiased. Formally, we will prove that DISPLAYFORM2 from which the follows: DISPLAYFORM3 To prove equation 12, we make use of the observation (slightly rephrased) by BID2 that conditioning on h −i, we know κ i and the event i ∈ S implies that κ = κ i since i will only be in the sample if h i > κ i which means that κ i is the k + 1-th largest value of h −i ∪ {h i} = h 1:n. The reverse is also true (if κ = κ i then h i must be larger than κ i since otherwise the k + 1-th largest value of h 1:n will be smaller than κ i). DISPLAYFORM4 We will now prove that the REINFORCE estimator based on multiple samples with the sample average as baseline FORMULA10 is unbiased. Let y 1:k = {y 1, ..., y k} be the set of independent samples (with replacement) from p θ (y). First we show that using the batch mean as baseline is equivalent to using the mean of the other elements in the batch, up to a constant DISPLAYFORM0 Note that k−1 k goes to 1 as the batch size k increases and we do not need to include it (and we can simply compute the biased mean) as it can be absorbed into the learning rate. Since y j is independent of y i, unbiasedness follows: DISPLAYFORM1 The proof that the REINFORCE estimator based on multiple samples without replacement with baseline (equation 10) is unbiased follows from adapting and combining the proofs in Appendix A and B. Additionally to q i (a) = P (h i > a) we define q ij (a) = P (h i > a ∩ h j > a) = P (h i > a)P (h j > a) = q i (a)q j (a) for i = j and q ii (a) = P (h i > a) = q i (a). For convenience we define shorthand for the conditional q j|i (a) = qij (a) qi(a), so q j|i (a) = q j (a) for j = i and q i|i (a) = 1. Furthermore, we define h −ij = h 1:n \ {h i, h j} and define κ ij (i = j) as the (k − 1)-th (not k-th!) largest element of h −ij, and κ ii = κ i, e.g. the k-th largest element of h −i.We denote with with {i, j ∈ S} = {i ∈ S ∩ j ∈ S} the event that both i and j are in the sample, also for i = j which simply means {i ∈ S}. First we generalize P (i ∈ S|h −i) = q i (κ i) to the pairwise conditional inclusion probability P (i, j ∈ S|h −ij). DISPLAYFORM2 Proof. For i = j: DISPLAYFORM3 For i = j: Assuming w.l.o.g. h i < h j there are the following scenario's:• κ ij < h i < h j. In this case, after adding h i and h j to h −ij, κ ij will be the (k + 1)-th largest element so κ = κ ij and i ∈ S and j ∈ S since h j > h i > κ = κ ij.• h i < κ ij < h j or h i < h j < κ ij. In both cases, there are at least (k − 1) + 1 = k elements higher than h i so i ∈ S.Therefore it follows that {i, j ∈ S|u −ij} = {h i > κ ij ∩ h j > κ ij |u −ij} and additionally this event implies κ = κ ij. Now the follows: DISPLAYFORM4 =P (h i > κ ij |u −ij)P (h j > κ ij |u −ij) =q i (κ ij)q j (κ ij) = q ij (κ ij)Using this Lemma we can prove the following Lemma: Lemma 2. DISPLAYFORM5 Note that the expectation is w.r.t. the keys h 1:n which define the random variables κ and S = {i : h i > κ}.Proof. DISPLAYFORM6 u −ij, i, j ∈ S P (i, j ∈ S|u −ij) + 0 · (1 − P (i, j ∈ S|u −ij)) DISPLAYFORM7 Theorem 1. Let B(S) = j∈S p θ (y j) qj (κ) f (y j). Then the following is an unbiased estimator: DISPLAYFORM8 Proof. First note that, when i ∈ S, we can rewrite: DISPLAYFORM9 Using equation 16, we can rewrite (similar to equation 13) DISPLAYFORM10 Substituting this into equation 15 and normalizing the outer importance weights by W (S) we see that this term cancels to obtain DISPLAYFORM11

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that by drawing multiple samples (predictions) per input (datapoint), we can learn with less data as we freely obtain a REINFORCE baseline.

represent this text to find a much much longer article about it with say hundreds-thousands of words

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.