Question: Represent the article for finding a single-sentence TLDR

Question: Neural architecture search (NAS) has made rapid progress incomputervision,wherebynewstate-of-the-arthave beenachievedinaseriesoftaskswithautomaticallysearched neural network (NN) architectures. In contrast, NAS has not made comparable advances in natural language understanding (NLU). Corresponding to encoder-aggregator meta architecture of typical neural networks models for NLU tasks (Gong et al. 2018), we re-deﬁne the search space, by splittingitinto twoparts:encodersearchspace,andaggregator search space. Encoder search space contains basic operations such as convolutions, RNNs, multi-head attention and its sparse variants, star-transformers. Dynamic routing is included in the aggregator search space, along with max (avg) pooling and self-attention pooling. Our search algorithm is then fulﬁlled via DARTS, a differentiable neural architecture search framework. We progressively reduce the search space every few epochs, which further reduces the search time and resource costs. Experiments on ﬁve benchmark data-sets show that, the new neural networks we generate can achieve performances comparable to the state-of-the-art models that does not involve language model pre-training. Neural architecture search (NAS) has recently attracted intensive attention. On one hand, promising methodological innovation for NAS have been developed, e.g. the seminal gradient-based NAS approach DARTS (Liu, Simonyan, and Yang 2018), followed by improvements such as SNAS (Xie et al. 2018), P-DARTS, PC-DARTS (Xu et al. 2019), etc. On the other hand, NAS has helped to discover better models to for a variety of vision tasks, e.g., image classification (Zoph and Le 2017; Zoph et al. 2017; Cai, Zhu, and Han 2018), semantic segmentation, object detection (Ghiasi, Lin, and Le 2019), superresolution (Ahn, Kang, and Sohn 2018), etc. For natural language processing tasks, NAS is relatively less studied. Except for the general methodology-wise innovations NASNet (Zoph and Le 2016), ENAS (Pham et al. 2018) and DARTS (Liu, Simonyan, and Yang 2018) which pay slight extra effort on searching for new RNN cells on Copyright c 2020, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. language modeling (LM) tasks, there is little studies tailored to the NLU task. One such an example is the evolved transformer (So, Liang, and Le 2019), which uses the evolutionbased NAS algorithm to search for better transformer architecture for machine translation. Although state-of-the-art performance has been achieved on 4 machine translation tasks, the computation cost is exceedingly high since they have to evaluate a large number of models. In fact, NAS has not been fully investigated for a wide variety of fundamental natural language understanding (NLU) tasks, such as classification (e.g. or sentiment analysis), natural language inference (NLI), sequence tagging tasks such as named entity recognition (NER). Especially, there is no existing work on the effectiveness of one-shot architecture search (Bender et al. 2018) methods on NLU tasks, which could also otherwise significantly reduce the search cost as done in vision tasks. A typical neural network architecture for NLU includes an encoder which contextualizes the embedded text inputs and extracts higher-level features, and an aggregator that aggregates the encoded inputs to a fix-length vector to make a prediction (Gong et al. 2018). In terms of encoders, many previous NAS literature restrict the search space to nonlinear maps such as tanh and sigmoid, and the objective to be the discovery of a new recurrent cell to form a new type of recurrent neural network (RNN). However, other than RNNs, there are many other available encoders, for example, convolutional networks (CNN) (Kim 2014), and attentionbased model such as transformer (Vaswani et al. 2017), etc. In addition, recent works e.g. star-transformer (Guo et al. 2019) have proposed more sparse versions of transformer to reduce the computational complexity and improve the generalization when there is no pre-trained language model. In addition, as far as we know, there is no existing work on searching for an aggregator. A collection of aggregators are available (Gong et al. 2018). However, one have to choose manually in a trial-and-error fashion. In this work, we design an encoder search space that contains a rich collection of encoders. The involved operations include: i) the zero map and identity map; ii) the two most commonly used RNNs, LSTM (Hochreiter and Schmidhuber 1997) and GRU (Cho et al. 2014); iii) highway network (Srivastava, Greff, and Schmidhuber 2015); iv) a series of convolutional networks with different kernel sizes; v) multi-head attention from (Vaswani et al. 2017); vi) startransformer (Guo et al. 2019) and its variants, which will be explained later in the next section. The combination of encoder operations is searched in a encoder search cell, which is a directed acyclic graph (DAG) of intermediate nodes collected by the encoder operations from the encoder search space. To further reduce the human designs, we propose to search for a suitable aggregator along with the search of encoder cell via an aggregator search cell which includes max (average) pooling, self-attention pooling and dynamic routing (Gong et al. 2018). The aggregator search cell is a DAG with only one step in which the only node is connected to the inputs by a mixture of aggregators. Our search strategy is mainly based on DARTS (Liu, Simonyan, and Yang 2018). To reduce computation cost, we employ a progressive search space reduction strategy similar to P-DARTS. Experiments are performed on three different kinds of NLU tasks, i.e., text classification, NLI and NER, with 5 benchmark datasets. For fair comparison, we only compare our with former state-of-the-art (SOTA) models without large-scale LM pre-training, or any other outside resources like knowledge bases, or any human designed features. Results have shown that with the help of NAS on our search space, we achieve that are comparable to the SOTA on these 5 tasks, indicating the effectiveness of NAS in the field of NLU research. Our work contributes the field by the following aspects: • We re-define the search space for neural architecture search in NLU tasks, by extending and modifying the encoder search space from the evolved transformer, and define the aggregator search space. • To the best of our knowledge, we are the first to conduct NAS experiments on NLU tasks such as classification, NLI, NER tasks, with one-shot NAS. • Our approach achieves the that are comparable to the state-of-the-art models designed by human experts, on various NLU tasks (classification, NLI, NER), by using neural architecture search over the search space defined above. In addition, we demonstrate the effectiveness of one-shot architecture search for NLU tasks. • We propose a modularized version of star-transformer and its variant, thus including a sparse version of transformer into the search space, which is also novel in the literature. The ing advantage is that the search cost can be reduced notably and the network's generalization capability can also be improved. Related Work Recently, a new research field named neural architecture search (NAS) has been drawing more and more attention. The goal is to find automatic mechanisms for generating new neural architectures to replace conventional handcrafted ones. Recently, it is widely applied to computer vision tasks, such as image classification (Zoph and Le 2017; Zoph et al. 2017; Cai, Zhu, and Han 2018), semantic segmentation, object detection (Ghiasi, Lin, and Le 2019), super-resolution (Ahn, Kang, and Sohn 2018), etc. However, NAS is less well studied in the field of natural language understanding (NLU). Recent works (Zoph and Le 2016; Pham et al. 2018; Liu, Simonyan, and Yang 2018) search new recurrent cells for the language modeling (LM) task on the PennTreebank dataset 1. The recurrent cell discovered by (Liu, Simonyan, and Yang 2018) achieves the test perplexity of 56.1, which is competitive with the stateof-the-art model enhanced by a mixture of softmaxes. The evolved transformer (So, Liang, and Le 2019) applies NAS to discover better versions of the transformer architecture. Eploying an evolution-based search algorithm, and the vanilla transformer as the initial population, it generates a better transformer architecture that consistently outperform the vanilla transformer on 4 benchmark machine translation tasks. Our work contributes by going beyond the RNN structure and re-defining the search space to include a richer connection of operations. Our work is implemented on DARTS (Liu, Simonyan, and Yang 2018) and P-DARTS. DARTS relaxes the search space to be continuous, so that the architecture can be optimized with respect to its validation set performance by gradient descent. Due to its simplicity, DARTS has inspired a series follow-up work to improve the search stability and efficiency. Based on DARTS, P-DARTS ) divides the search process into multiple stages and progressively increase the network depth at the end of each stage. Our work contributes to the gradient-based NAS (and more generally, one-shot NAS) research by investigating its effectiveness in discovering new NN architectures for a series of NLU tasks. Our search space design takes advantages of the recent advances in the NLU field. One of the most import advances in sentence encoding is the application of various self-attention mechanisms, among which the transformer (Vaswani et al. 2017) is the most prominent one, which has become ubiquitous in NLU research. Specifically, the QANet ) modifies the transformer architecture to obtain the first place on the SQuaD leaderboard 2. The transformer is powerful due to its multi-head self-attention mechanism, which can well capture the contextual information. However, the transformer maybe be difficult to train and generalize well on a small or medium sized data-set (Guo et al. 2019). Thus, many other self-attention operations are proposed, e.g., dynamic self-attention (Yoon, Lee, and Lee 2018) and DiSAN (Shen et al. 2018). Recently, (Guo et al. 2019) propose the star-transformer, a sparser version of the multi-head attention model, and achieves competitive on a series of benchmark datasets like SST-1, SNLI, CoNLL2003. On the aggregation side, an important advancement is the application of capsule networks and dynamic routing policy in text classification Gong et al. 2018). Capsule networks can dynamically decide what and how much information need to be transferred from each word to the final encoding of the text sequence, thus achieving better even with simple encoders (Gong et al. 2018). Our work is built upon these work and contributes by: i) include some of the most prominent attention based encoders and aggregators into the search space, and experiment on whether NAS can generate new architectures that have competitive ; ii) we are the first to propose the aggregator search space; iii) we include a modularized version of the star-transformer and its variant into the search space, thus we are the first to combine the dense and sparse multi-head self-attention operations into the same search space. We first analyze the meta architectures for a series of NLU tasks, based on which we will define the search space As pointed out by (Gong et al. 2018), an NLP model with text inputs and discrete labels (possibly a sequence of labels) can be assembled by the following components: an embedding layer, an encoding layer, an aggregation layer and a prediction layer. Embedding layers usually are static pretrained embeddings like word2vec (Mikolov et al. 2013), or contextualized embedding like ELMO (Peters et al. 2018) and BERT (Devlin et al. 2018). We mainly focus on the encoding layer and aggregation layer. A text sequence with words S = w 0, w 1,..., w L−1 is mapped into a d-dimensional embedding vector space as X = x 0, x 1,..., x L−1. The encoding layer integrates the information inside the embedding layer and extract higherlevel features. The encoded inputs are denoted as H = h 0, h 1,..., h L−1. When the task at hand requires predicting a single label, the final prediction layer requires a fix-length vector. Thus an aggregator is needed to aggregate the information inside sequences of various length to a single fixlength vector, namely h *. In this work, we investigate the neural architecture search for three different tasks: classification (CLS), natural language inference (NLI) and named entity recognition (NER). The meta architectures of neural networks are depicted in Figure 1. For classification, the encoder is followed by an aggregator whose output will be passed to the prediction layer. For the NLI task, the encoding and aggregating for the premise and the hypothesis are the same as CLS task, and the aggregated vectors, h * 1, h * 2, will interact via an interaction layer before being passed into the prediction layer. Following (Chen et al. 2016), we define the interaction layer Note that in this work, we will not consider any sort of cross attentions between the two inputs before or after the interaction layer. In addition, due to limited resources for search, we restrict that the encoder and aggregator are shared by both inputs. For the NER task, the aggregator is not required. Note that in this work, we will not consider adding a CRF layer after the enocoder, as done by some other NER models e.g. (Lample et al. 2016). Recall our goal here is to discover and evaluate new model architectures. Based on the above discussion, we propose to divide the search space into two subspace: encoder search space and aggregator search space. In (Liu, Simonyan, and Yang 2018) and (Pham et al. 2018), the objective is to discover new variants of RNNs, so their search space are a collection of linear or non-linear maps, such as, tanh and sigmoid. In this work, we will define the encoder space at a more coarse granularity, allowing us to build a richer search space. As (Liu, Simonyan, and Yang 2018), the encoder search space contains the zero map and the identity map. We then include 1-d convolutional networks (conv1d), and two of RNNs, namely, LSTM and GRU, and the highway network. Highway network is introduced to help train deeper neural network models (see e.g., yu2018qanet). These basic models have been widely used in NLP tasks, such as classification (Kim 2014), Question-answering, NER (Lample et al. 2016), relation extraction (Zeng et al. 2015), to name just a few. Note that we will use the depth-wise separable convolutions (Chollet 2017) instead of the vanilla convolutions, since the former are more parameter efficient. Recent years has witnessed the architecture of Transformers (Vaswani et al. 2017) becoming ubiquitous in research community. At its core, the multi-head self-attention mechanism has shown its superior ability to model long-distance contexts. In this work, similar to (So, Liang, and Le 2019), we include the multi-head attention layer, excluding the residual connection that is usually added in a transformer block, into the search space. The point-wise feed-forward layer contains conv1d and residual connection, so we will not include it as a basic operation. Although transformer has shown its great expressiveness capability, it is difficult to train and easy to over-fit on a small or medium dataset (Guo et al. 2019). One reason is that the Algorithm 2: Reversed star-transformer module Input: multi-head attention let each token pay attention to every token in the sentence, ing in over-parametrization. Thus some more sparse variants are proposed e.g. sparse transformers (Child et al. 2019). We include the star-transformer proposed by (Guo et al. 2019). The original design of star-transformers requires the relay node being initialized with average pooling and then updated iteratively in a few layers that follows. For better modularization, we modify the Algorithm 1 of (Guo et al. 2019) to Algorithm 3 below. Note that without iterative updating, the relay node is initialized by simple averaging, during which the information of the sentence may not be well preserved. Thus, we propose to first enrich the semantic representation of relay node via multi-head attention between the initialized relay node and the inputs, before we update the satellite nodes. This variant is called the reversed star-transformer, and the algorithm is described in Algorithm 3. Due to limited resources, for attention based operations, we will only set the attention head number to be 2. Now we formally define the encoder search space, which consists of the following operations: • Special zero operation, denoted as null; • Skip connection, denoted as identity; • Highway network, denoted as highway; • 1-d separable convolutions, with kernel size k, where k = 1, 3, 5, denoted as sep conv1d 1, sep conv1d 3 and sep conv1d 5; • Two RNNs, which are denoted as lstm and gru; • Attention-based operations, with attention head number set to be 2, including: multi-head attention (multi head attn 2), star-transformer (star trans 2) and reversed star-transformer (reversed star trans 2). There are several different aggregation operations. The most common two are the max pooling and the average pooling. Self-attention technique is also used for aggregation. It assigns each word a weight to indicate the importance of a word depending on the task on hand. A few words that are crucial to the task will be emphasized while the "boring" words are ignored. We also include dynamic routing (Gong et al. 2018) into our aggregator operation space. Unless specified, we use a dynamic routing aggregator with 4 capsules and 3 iteration steps. Now we formally define the aggregator search space, which includes the following modules: • Max pooling, denoted as max-pool; • Average pooling, denoted as avg-pool; • Self-attention pooling, denoted as self-attn-pool; • Dynamic routing, denoted as dynamic-routing. In this work we use Differentiable Architecture Search (DARTS) (Liu, Simonyan, and Yang 2018) as our architecture search framework. The goal is to search for an encoder cell alongside with an aggregator cell. Employing the terminology in (Liu, Simonyan, and Yang 2018), a cell is defined as a directed acyclic graph (DAG) of N nodes, x 0, x 1, · · ·, x N −1, where each node is a network layer, i.e., performing a specific mathematical function. We denote the search space as Φ i,j, in which each element represents the information flow connecting node i to node j, which consists of a set of operations weighted by the architecture parameters α (i,j), and is thus formulated as: where i < j. An intermediate node can be represented as Given multiplier m, a hyperparameter determining the number of nodes whose are included as the cell's output, the output of the cell is The NLU tasks we are dealing with are small or medium sized, so we will not consider stacking multiple encoder cells in this paper. Our search space is large, thus we employ a progressive search space reduction strategy during search, which is similar to P-DARTS ). The procedure is as follows: i) start the search with the whole operation space; ii) let k denote the epoch interval length for a reduction. after every k epochs, we drop the operation in Φ i,j having the lowest score; iii) after each reduction, the search re-starts; iv) repeat step ii) and step iii) till the order of encoder search space drops to 5, and the order of the aggregator search space drops to 1; v) the search procedure continues with the remaining search space till convergence. After obtaining the continuous architecture encoding α, deriving the aggregator is simple, since we only need to select the most likely aggregation operation. For the encoder cell, we consider two approaches to derive the final discrete network architecture: • Following (Liu, Simonyan, and Yang 2018), retain up to k strongest predecessors for each intermediate node, where k = 1, 2. Note this approach ignores the appearance of the null operation if it obtains the highest score. • Directly replace every mixed operation as the most likely operation by taking the arg-max. If the best operation connecting two intermediate nodes is a null operation, this connection is dropped. In this way, we may find a sparser new encoder cell. In our experiments, for each search or evaluation, we assign 2 CPU cores, 20G memory and 1 Tesla P100 GPU card. We conduct experiments on three different kinds of tasks with 5 benchmark datasets, whose statistics are shown in Table 1. Specifically, SST-1 and SST-2 (Socher et al. 2013) are two text classification data-sets. Sci-tail and MedNLI are NLI datasets, and CoNLL2003 (Sang and De Meulder 2003) is a benchmark NER datasets 3. SST-1 Stanford Sentiment Treebank is a movie review dataset which has been parsed and further splitted to train/dev/test set (Socher et al. 2013). SST-2 This dataset is a binary-class version of SST-1, with neutral reviews removed. SciTail This is a textual entailment dataset derived from a science question answering (SciQ) dataset. MedNLI It is a NLI dataset annotated by doctors, grounded in the medical history of patients (Romanov and Shivade 2018). CoNLL2003 This dataset consists of 200k training words which have been annotated as Person, Organization, Location, Miscellaneous, or Other (non-named entity). Our experimental protocols follow (Liu, Simonyan, and Yang 2018). Experiments on each task consist of two stages, architecture search and architecture evaluation. In the search stage, we search for the cells on the train set, and determine the best pair of encoder cell and aggregator cell based on the performance on validation set. In the second stage, we derive discrete architectures from the cells, train them from scratch, and report their performance on the test set. Initially the search space consists of all the operations in Section Search Space Design. For every 3 epochs we carry out search space reduction once, till the search space is halved. The number of intermediate nodes N for the encoder cell ranges from 1 to 3. With N equal to 3, the search cell takes up 9G GPU memory. Note that the number of nodes for the aggregator cell can only be 1. Similar with ENAS (Pham et al. 2018) and DARTS (Liu, Simonyan, and Yang 2018), we enable layer normalization in each node to prevent gradient explosion during architecture search, and disable it during architecture evaluation. For architecture search, both the embedding and hidden size are set to 300. Word embedding is initialized from pretrained Glove (Pennington, Socher, and Manning 2014). We randomly initialize word vectors for words that do not appear in Glove. The batch size is 32 for the classification tasks and 16 for the others. The learning rate is 1e-4 for both network parameters and architecture parameters, and the weight decay is set to 1e-5, and the dropout rate is set to 0.5. Dropout is applied after embedding look-up, after encoding layer, and to the output layer. The learning rate is warmed up for 1000 steps and then it decreases linearly. The max number of epochs is set 60. We use Adam (Kingma and Ba 2015) to optimize both the network parameters and architecture parameters. The search takes around 1.5 GPU day (Tesla P100) for SST-1, and 0.3 for the SciTail tasks. We run each search configurations twice with different random seeds and pick the best cell based on the validation performance, and each search run in at most 3 different dicrete NN architectures. The hyper-parameters are the same with those in the search stage. Results on SST Results on SST-1 and SST-2 datasets are listed in Table 2. On the SST-1, DARTS generate a network architecture (DARTS-SST-1-V0) that performs better than most of the traditional NN models. Not that the encoder cell of DARTS-SST-1-V0 contains only RNN and CNN operations, but the exact details of combination of different level of features are impossible to design manually. The best ar- (Le and Mikolov 2014) 48.7 87.8 MT-LSTM (F2S) 49.1 87.2 Tree-LSTM (Tai, Socher, and Manning 2015) 51.0 88.0 CNN-Tensor (Lei, Barzilay, and Jaakkola 2015) 51.2 -BiLSTM + max pooling (Gong et al. 2018) 48.0 87.0 BiLSTM + average pooling (Gong et al. 2018) 46.2 85.2 BiLSTM + self-att (Gong et al. 2018) 48.2 86.4 BiLSTM + dynamic routing (Gong et al. 2018) 50.5 87.6 Emb + self-att (Shen et al. 2018) 48.9 -DiSAN (Shen et al. 2018) 51.7 -BiLSTM + self-att (Yoon, Lee, and Lee 2018) 50.4 88.2 CNN + self-att (Yoon, Lee, and Lee 2018) 50.6 88.3 Dynamic self-att (Yoon, Lee, and Lee 2018) 50.6 88.5 Transformer (Guo et al. 2019) 50 chitecture (DARTS-SST-2-V0) we obtained on the SST-2 dataset involves a star-transformer operation and an identity map. Note that since (Guo et al. 2019) did not provide on SST-2, we use the code from fastNLP 4 to run the transformer and the original star-transformer on SST-2. The given by us are all the average of 10 different runs. We can see that DARTS-SST-2-V0 can obtain comparable to the SOTA on SST-2. We also experiment on the transferability of the learned architectures. From Table 2, we can see that DARTS-SST-2-V0 performs worse than DARTS-SST-1-V0 on SST-1 with a significant margin, but DARTS-SST-1-V0 also performs competitively on SST-2. Results on NLI tasks Among the architecture candidates derived from the search on SciTail, we find that the one obtained by accepting the null operation when it gets the highest score (DARTS-SciTail-V0) performs best. In addition, this search run gives the average pooling as the aggregator instead of dynamic-routing. The are presented in Table 3: Test accuracy (%) on the SciTail dataset. Model ACC 600D ESIM 70.6 Decomposable Attention 72.3 DGEM 72.3 AdvEntuRe 79.0 HCRN (Tay, Luu, and Hui 2018) 80.0 DeIsTe (Yin, Schütze, and Roth 2018) 82.1 CAFE (Yin, Schütze, and Roth 2018) 83.3 MIMN 84.0 ConSeqNet 85.2 HBMP (Mihaylov et al. 2018) 86.0 star-transformer (Guo et al. 2019) 79 Table 3. DARTS-SciTail-V0 achieves a competitive performance on the test set, outperforming the baseline models such as ESIM and decomposable attention by a large margin. It also outperforms the of the star-transformer and transformer even after extensively parameters tuning. Our model is actually the best one that has no inter-sentence attentions other than the final interaction before the prediction layer, and uses no outside resources, no manually designed features and no extra training mechanism like adversarial training. As we can see from Figure 5 that, on the MedNLI dataset, the search gives out a architecture (DARTS-MedNLI-V0) that quite resembles the original implementation of the multi-head attention inside the transformer block, except the residual connection is replaced by a sep conv with kernel size 3. DARTS-MedNLI-V0 performs worse than the original star-transformer, but it is better than the original transformer, and the baseline ESIM and InferSent. We also look into the transferability between the two task. We find that although the datasets are from different domains, the architecture searched on one performs comparable on the other. For the NER task CoNLL2003, since our goal is to compare the previous models and the models discovered by NAS, we do not include the CRF layer which is proven to be a standard component of the best NER models (Lample et al. 2016). We do not use any outside resources like gazetteers, or manually designed features like suffix features and capitalization features, or charac- ter embedding. To eliminate implementation discrepancies, we re-run all the ourselves for this task. Figure 6 gives the searched architecture (DARTS-CoNLL2003-V0) and Table 5 gives out the . The LSTM sets a strong baseline, and the star-transformer and GRU performs significantly worse. Our DARTS-CoNLL2003-V0 works slightly better than LSTM. This paper addresses NAS for a series of NLU tasks. Corresponding to the encoder-aggregator architecture of typical NN models for NLU (Gong et al. 2018), we redefine the search space, by splitting it into encoder search space and aggregator search space. Our search strategy is based on DARTS (Liu, Simonyan, and Yang 2018) and P-DARTS. Experiments shows that architectures discovered by NAS achieves that are comparable to the previous SOTA models. In the further, we would like to investigate one-shot architecture search on more large-scale NLU tasks.

Represent the TLDR for finding the corresponding scientific article

Neural Architecture Search for a series of Natural Language Understanding tasks. Design the search space for NLU tasks. And Apply differentiable architecture search to discover new models

An empirical study that provides a novel perspective on few-shot learning, in which a fine-tuning method shows comparable accuracy to more complex state-of-the-art methods in several classification tasks.


Question: Represent the scientific article to find a short TLDR.

Question: Network embedding (NE) methods aim to learn low-dimensional representations of network nodes as vectors, typically in Euclidean space. These representations are then used for a variety of downstream prediction tasks. Link prediction is one of the most popular choices for assessing the performance of NE methods. However, the complexity of link prediction requires a carefully designed evaluation pipeline to provide consistent, reproducible and comparable . We argue this has not been considered sufficiently in recent works. The main goal of this paper is to overcome difficulties associated with evaluation pipelines and reproducibility of . We introduce EvalNE, an evaluation framework to transparently assess and compare the performance of NE methods on link prediction. EvalNE provides automation and abstraction for tasks such as hyper-parameter tuning, model validation, edge sampling, computation of edge embeddings and model validation. The framework integrates efficient procedures for edge and non-edge sampling and can be used to easily evaluate any off-the-shelf embedding method. The framework is freely available as a Python toolbox. Finally, demonstrating the usefulness of EvalNE in practice, we conduct an empirical study in which we try to replicate and analyse experimental sections of several influential papers. Link prediction is an important task with applications in a wide range of fields such as computer science, social sciences, biology, and medicine BID6 BID14 BID15 BID22. It amounts to estimating the likelihood for the existence of edges, between pairs of nodes that do not form an edge in the input graph. Many Network Embedding (NE) methods (e.g., BID0 BID2 BID5 BID8 BID10 BID12 BID17 BID18 BID19 have recently been applied to solving link prediction problems, showing promising . These methods map nodes in the network to vectors in IR d . This embedding is then used for a variety of tasks such as visualization, multi-label classification, clustering or link prediction. The challenges of evaluating NE methods for link prediction We argue that the practical performance of most NE methods is poorly understood and that experiments in many papers are difficult to compare due to variation in experimental setup and evaluation procedures. In this paper, we focus on a number of difficulties specific to the evaluation of NE methods for link prediction. Link prediction is a particularly challenging task to evaluate as it involve a number design choices, which can confound the and are prone to errors.1) Train-test splitting of graphs For example, a typical implicit assumption is that the input graph is not complete, and the purpose is to accurately predict the missing edges. To evaluate the performance of an NE method for link prediction, one thus needs an (incomplete) training graph along with a (more) complete version of that graph for testing. Much research has been devoted to determining the best approach to generate these training graphs BID6 BID14 BID22. Strong theoretical and empirical evidence suggest that in order to fairly evaluate link prediction methods, snapshots of the network at different points in time should be used for training and testing. In this way, the link prediction methods are tested on the natural evolutions of the networks. However, the availability of such snapshots is uncommon and raises additional questions, such as how to choose the time intervals for splitting the network. For these reasons, authors typically resort to sampling sets of edges from the input graphs and using the ing sub-graphs for training BID5 BID8 BID10 BID12. The remaining edges are used as positive test examples. The process of sampling edges is not standardized and varies between scientific works. The relative sizes of the train and test sets, for example, is a user-defined parameter which varies significantly. In BID8; BID10 the authors use a 50-50 train-test split, in BID5 ) a 60-40, in Lai et al. (2017 an 80-20 and in BID20 values ranging from 30-70 up to 80-20.A related problem is that, in addition to the 'positive' train and test edges, often also 'negative' edges (or non-edges) are required. Sometimes these are used to derive the embedding, while in other cases they are used only to train the classifier that predicts links. These sets of non-edges can be selected according to different strategies (Kotnis & Nastase) and can be of various sizes.2) From node embeddings to edge predictions Furthermore, most NE methods simply provide node embeddings. From these, edge embeddings need to be derived prior to performing predictions. There are several approaches for deriving edge embeddings which also seem to have a strong impact on the performance of different methods BID8.3) Evaluation measures Also the metrics used to evaluate the accuracy varies, e.g., from AUC-ROC BID10, to precision-recall BID21, to precision@k BID20. Finally, it appears to be common practice in recent literature to use recommended default settings for existing methods, while tuning the hyper-parameters for the method being introduced. When the recommended default settings were informed by experiments on other graphs than those used in the study at hand, this can paint an unduly unfavorable picture. For none of the above problems a gold standard exists which would clear the preferable choice. There are many valid choices and this, together with the fact that not all values of parameters and choices are usually reported have led to a situation where no one can see the forest for the trees. To address this problems, we propose EvalNE, a framework that simplifies the complex and time consuming process of evaluating NE methods for link prediction. EvalNE automates many parts of the evaluation process: hyper-parameter tuning, selection of train and test edges, negative sampling, and more. The framework: Implements guidelines from research in the area of link prediction evaluation and sampling BID6 BID14 BID22. Includes (novel) efficient edge and non-edge sampling algorithms. Provides the most widely used edge embedding methods. Evaluates the scalability and accuracy of methods, through wall clock time and a range of fixed-threshold metrics and threshold curves. Integrates in a single operation the evaluation of any number of NE methods coded in any language on an unbounded array of networks. Finally, EvalNE ensures reproducibility and comparability of the evaluations and . By making it easier to reproduce, compare, and understand evaluation pipelines, we regain the ability to assess the real strengths and weaknesses of existing NE methods, paving the way for more rapid and more reliable progress in the area. The remainder of this paper is organized as follows. In Section 2 we discuss related work. Section 3 presents the proposed evaluation framework including the novel edge samplig strategy. Empirical of attempts to reproduce evaluation setups of several papers and of the proposed edge sampling strategy are reported in Section 4. Finally Section 5 concludes this paper. The open-source toolbox is freely available at [scrapped for anonymity]. The evaluation of link prediction has been studied in several works BID6 BID14 BID16 BID22. These papers focus on the impact of different train set sampling strategies, negative sampling, and fair evaluations criteria. Link prediction as evaluation for the representations learned by a NE algorithm was first introduced in the pioneering work of BID8. The survey by BID23 points out the importance of link prediction as an application of network representation learning. The authors also signal the inconsistencies in the evaluation of different NE approaches and conduct an empirical analysis of many of these methods on a wide range of datasets. Their empirical study, however, only Figure 1: Diagram of the types of methods which can be evaluated using EvalNE. Gray blocks represent modules provided by the library and white blocks are the user-specified methods to be evaluated. The library allows for the evaluation of end-to-end prediction methods (several LP heuristics are included as baselines here), edge embedding methods, and node embedding methods.focuses on vertex classification and clustering. The importance of standard evaluation frameworks as tools to bridge the gap between research and application is discussed in BID9.To the best of out knowledge only two frameworks for the evaluation of NE methods currently exist. OpenNE is a recently proposed toolbox for evaluating NE methods on multi-label classification. The toolbox also includes implementations of several state-of-the-art embedding methods. GEM BID7 ) is a similar framework which also implements a variety of embedding methods and includes basic evaluation on multi-label classification, visualization, and link prediction tasks. These frameworks, however, are focused on the implementations of embedding methods rather than the evaluation pipeline. Furthermore, these libraries are limited to the NE methods provided by the authors or require new implementations which comply with pre-defined interfaces. In this section we discuss the key aspects of EvalNE. The framework has been designed as a pipeline of interconnected and interchangeable building blocks, as illustrated by Figure 1. The modular structure of our framework simplifies code maintenance and the addition of new features, and allows for flexible model evaluation. EvalNE can be used to evaluate methods providing node embeddings, edge embeddings, or similarity scores (we include in this category the link prediction heuristics). Next we describe the frameworks and its components in more detail as well as the software design. The core building blocks of EvalNE are the data split and model evaluation. These bocks constitute the most basic pipeline for assessing the quality of link predictions. However, in order to extend the evaluation process to other types of embedding methods we also provide building blocks for data manipulation and preprocessing, learning edge embeddings from node embeddings, binary classification and a range of LP heuristics which can be used as baselines. Before presenting in detail each of these building blocks we introduce some needed notation. We represent an undirected weighted network as G = (V, E, W) with vertex set V = {1, . . ., N}, edge set E ✓ V ⇥ V and weight matrix W 2 IR N⇥N. Edges are represented as unordered pairs e = (u, v) 2 E with weights w e 2. E train and E test denote the training and testing edge sets. We represent a d-dimensional node embedding as X = (x 1, x 2, . . ., x N) where X 2 IR N⇥d.Preprocessing The toolbox offers a variety of functions to load, store, and manipulate networks. These include methods to prune nodes based on degree, remove self-loops, relabel nodes, obtain sets of specific types of edges, restrict networks to their main connected components and obtain common network statistics. The preprocessing functions of EvalNE build on top of and can be used in combination with those provided by other mainstream software packages (e.g. Networkx).Data split As pointed out in Section 1, in order to perform link prediction on a given input graph G, sets of train and test edges are required. The set of training edges is generally required to span all nodes in G and induce a train graph G train with a single connected component, because embeddings of independent components will be far away from and unrelated to each other. Most studies resort to a naive algorithm for deriving a connected G train. The procedure removes edges from an input graph iteratively until the required number of train edges remain. The removed edges are used as test samples. In each iteration, the connectedness of the graph is checked and an edge is only removed if it does not cause the graph to become disconnected. This requirement is generally satisfied by running a Breadth First Search (BFS) on the graph after each edge removal, which is a costly operation (O(|V | + |E|)).Integrated in our evaluation framework, we include a novel algorithm to perform the train-test splits which, as we will show in Section 4, is orders of magnitude faster yet equally simple. Our algorithm, also accounts for the fact that the training graph G train must span all nodes in G and contain a single connected component. Given as input an undirected graph G = (V, E, W) and a target number of train edges m, the proposed algorithm proceeds as follows:1. Obtain a uniform spanning tree ST of G 2. Initialize the set of training edges E train to all edges in ST 3. Add m |E train | edges to E train selected uniformly at random wihout replacement from DISPLAYFORM0 We select a spanning tree uniformly at random from the set of all possible ones using Broder's algorithm BID1 ):1. Select a random vertex s of G and start a random walk on the graph until every vertex is visited. For each vertex i 2 V \ s collect the edge e = (j, i) that corresponds to the first entrance to vertex i. Let T be this collection of edges. 2. Output the set T.On expectation, the complexity of the uniform spanning tree generation is O(n log n) BID1 and the addition of random edges can be efficiently done in O(|E train |). DISPLAYFORM1 we first construct an equivalent undirected version G by adding reciprocals for every edge in E ⇤. We then run the same algorithm described above on G and include in the training set only those edges present in the initial directed graph G ⇤. This method in a weakly connected train graph spanning the same nodes as the original G ⇤.In addition to train and test edges, sets of train and test non-edges (also referred to as negative samples) are required in order to evaluate link prediction. These are edges between pairs of vertices u, v such that e = (u, v) / 2 E. The proposed toolbox can compute these non-edges according to either the open world or the closed world assumption. The two strategies only differ in the selection of the train non-edges. Under the open world assumption, train non-edges are selected such that they are not in E train. Thus, this strategy allows overlapping between train non-edges and test real edges. Under the closed world assumption, we consider the non-edges to be known a priori and therefore select them so that they are neither in E train nor in E test.The number of train and test edges and non-edges are user-defined parameters. For the train set size, fractions between 50% and 90% of total edges E of G are recommended. For values below 50%, the ing train graph will often not preserve the properties of G BID13. The LP heuristics take as input a train graph G train = (V, E train, W) spanning the same set of vertices as the initial graph G but containing only the edges in the train set and output scores of node similarity which can be directly used for link prediction. EvalNE includes the following heuristics: common neighbours (CN), Jaccard coefficient (JC), Adamic-Adar index (AA), resource allocation index (RA), preferential attachment (PA) and Katz. In addition to these, a random prediction model is provided for reference. This method simply outputs a uniform random value in as the likelihood of a link between any pair of given vertices (u, v).For the case of undirected graphs we use the usual definitions of the heuristics (see Appendix A 1) and for directed graphs we restrict our analysis to either the in-neighbourhood (i (u)) or the outneighbourhood (o (u)) of the nodes u 2 V.Node to edge embedding Unlike for the abovementioned LP heuristics where the output can be directly used for link prediction, for NE methods this is generally not the case. Most authors only 1 https://www.dropbox.com/sh/8whq0di1sb9pz8m/AADl1DIrWRWOjtwuVum_TxnGa?dl=0 release code to compute node embeddings. Thus, an additional step of learning edge embeddings is required in order to perform link predictions via binary classification. The edge representations can be learned in an unsupervised fashion, for any given edge e = (u, v), by applying a binary operator over the embeddings of nodes u and v, i.e. x u and x v respectively: BID8 propose the following alternatives for the operator which we include in our evaluation framework: average, hadamard, weighted L 1 and weighted L 2 (See Appendix B for equations). Additional user-defined operators can be also easily integrated. DISPLAYFORM0 Binary classification Most NE methods (e.g., BID5 BID8 BID10 BID12 rely on a logistic regression classifier to predict the probability of links given the edge embeddings. In EvalNE we implement logistic regression with 10-fold cross validation. The framework, however, is flexible and allows for any other binary classifier to be used. Evaluation The proposed framework can evaluate the scalability, parameter sensitivity and accuracy of embedding methods. We asses the scalability directly by measuring wall clock time. The performance can be easily reported for different values of embedding dimensionality and hyperparameter values. Finally, the link prediction accuracy is reported using two types of metrics: fixedthreshold metrics and threshold curves. Fixed-threshold metrics summarize method performance to single values. EvalNE provides the following: confusion matrix (TP, FN, FP, TN), precision, recall, fallout, miss, accuracy, F-score and AUC-ROC. Threshold curves present the performance of the methods for a range of threshold values between 0 and 1. EvalNE includes precision-recall and receiver operating curves BID4. The framework provides recommendations of the most suitable metrics based on the evaluation setup. The EvalNE framework is provided as a Python toolbox compatible with Python2 and Python3 which can run on Linux, MacOS, and Microsoft Windows. The toolbox depends only on a small number of popular open-source Python packages, and the coding style and documentation comply with strict formats. The documentation provided contains instructions on the installation, examples of high-level and low-level use and integration with existing code. EvalNE can be used both as a command line tool and an API. As a command line tool it exposes a configuration file which determines the complete experimental setup a user wants to test, including methods to evaluate, networks and edge splits. For convenience, several pre-filled configuration files that reproduce experimental sections of influential NE papers are provided. When used as an API, the framework exposes a modular design with blocks providing independent and self contained functionalities. The user interacts with the library through an evaluator object that integrates all the building blocks and orchestrates the method evaluation pipeline. This section aims to demonstrate the usefulness and flexibility of EvalNE. To this end we have selected and replicated the experimental sections of four papers from the NE literature, i.e. Node2vec BID8, CNE BID10, PRUNE BID12 and SDNE BID20. We also report experiments comparing the proposed edge sampling algorithm to the naive approach. Experimental setups The experimental settings we aimed to replicate are as follows:Node2vec This work used the following embedding methods and link prediction heuristics in the evaluation: Node2vec, DeepWalk, LINE, Spectral Clustering, CN, JC, AA, and PA. Experiments are performed on the Facebook, PPI, and AstroPh datasets with 50-50 train-test splits. The same number of edges and non-edges are used throughout the experiment and the edge embedding methods used are average, hadamard, weighted L 1 and weighted L 1. Results are reported in terms of AUC-ROC. sizes for the non-edge sets. The node to edge embedding operator used is not reported by the authors and the are presented in terms of AUC-ROC. In this setting, the network used are directed. CNE The evaluation setup is like the Node2vec paper, with the following differences: CNE and Metapath2vec are also evaluated; and evaluation is done also on the BlogCatalog, Wikipedia, and StudentDB networks. Results are reported for the hadamard operator only. This paper comes from a research group that we have occasional collaborations with. We had access to the full code for their evaluation pipeline, but the EvalNE codebase was implemented completely separately. SDNE The paper reports experiments for the following NE methods: SDNE, LINE, DeepWalk, GraRep, LapEig and CN. The link prediction experiments are performed on the GR-QC dataset with a 0.85 train-test fraction. The are reported only for the hadamard operator and in terms of precision@k for a colection of values between 2 and 10000. The number of train non-edges is the same as that of train edges while all the remaining non-endges in the graph are used for testing. We replicated these setting with the following exceptions for node2vec we did not include spectral clustering and for PRUNE we could not obtain NRCL and lacked the computational resources to evaluate the webspam dataset. In all cases we report the same metrics as the original papers and, except were reported differently, average the over three repetitions. We use the closed-world assumption for the non-edges, logistic regression for binary classification and tuned the hyper-parameters equivalently as reported in each paper. All experiments were run using specific configuration files created for each setting which will be made public. Regarding the implementations, we evaluated the LP heuristics included in EvalNE; original code by the authors for Deepwalk, Node2vec, LINE, PRUNE, CNE, and Metapath2vec, and for the remaining ones, the implementations in the OpenNE 2 library (See Appendix C for references to exact implementations). Table 1 contains the main characteristics of all the networks used. Network sizes are those corresponding to the main connected components. For each of the experimental settings reproduced we present a table containing the original values reported in the paper, and between parentheses the difference between our and this value TAB2. Positive values in parentheses thus indicate that the our are higher than the ones reported in the original papers by that margin, while negative values indicate the opposite. The obtained show that our experiments align fairly well with those reported in the CNE and PRUNE papers. The only exception is the of Metapath2vec on the Facebook dataset, which substantially differs from the CNE paper. For Node2vec and SDNE, the differences are larger and occasionally severe (differences over 0.15 are marked in bold in all tables). Possible explanation for this are the use of different implementations of the NE methods, different not reported default parameters or the use of parallelization. We have studied the effect of each of these possible causes and found 1) important differences in accuracy when comparing different implementations of LINE and Node2vec on identical settings, 2) different initial default parameters for distinct implmentations of deepwalk, and 3) performance degradation for Metapath2vec when using parallelization (See Appendix D for more details).In addition, we have also observed important differences when computing the AUC-ROC directly from class labels or from class probabilities. In order to reproduce the CNE experiments we used class probabilities (which is the appropriate choice), while for Node2vec class labels appear to have been used. These illustrate the need to: (a) create reproducible pipelines for experiments, and (b) report specifics about the parameter settings and precise implementations used in the evaluation. We now present our comparing the proposed edge and non-edge sampling strategy in terms of accuracy and scalability to the naive approach. For the accuracy experiment we selected the Facebook, PPI, and arXiv datasets. We used both the proposed and naive edge split strategies and performed link predictions with the CN, JC, AA, and PA heuristics. We output all the metrics available in EvalNE and compute their absolute difference. Recall is the metric presenting the highest deviation so we collect these in TAB1. For the AUC-ROC, one of the most widely used metrics, the maximum deviation is of 0.01 reached on the arXiv dataset by the PA heuristic. Consistently, across methods and datasets, the obtained with the naive approach were slightly higher than those using our method. Regarding the scalability experiments, in FIG2 we summarize the execution time in seconds of both methods on four different datasets. This experiment shows that the proposed method is several order of magnitude faster than the naive approach and independent on the number of train and test edges required. The execution time of the naive approach further increases as more test edges are required. In FIG3 we select the BlogCatalog dataset and restrict it to sub-graphs of different prec@100 prec@200 prec@300 prec@500 prec@800 prec@1000 prec@10000 SDNE 1 (0.00) 1 (0 number of nodes ranging from 400 up to 10000. The execution times using both sampling strategies are reported in FIG3 .For our proposed edge sampling we also evaluated the variance in method performance for different number of experiment repetitions (or equivalently, the effect of averaging the over different number of edge splits). In this case we used the same datasets, NE methods and LP heuristics as in the Node2vec experiment and 50-50 and 90-10 train-test splits. We compared the obtained in a single run with those averaged over 5 independent runs for both split fractions. For the 50-50 train-test split the average difference observed over all methods, datasets and metrics is 3.4 · 10 3 with a variance of 1.8 · 10 5 and a maximum difference of 0.0293. For the 90-10 split, the average difference is 3.0 · 10 3 the variance of 1.0 · 10 5 and maximum difference 0.0186. These indicate that a single train and test split already gives a sufficiently good estimate of the generalization error of the models evaluated. Thus, experiment repeats are not needed for networks of similar sizes. These observations seem to hold for different train-test split fractions. The recent surge of research in the area of network embeddings has ed in a wide variety of data sets, metrics, and setups for evaluating and comparing the utility of embedding methods. Comparability across studies is lacking and not all evaluations are equally sound. This highlights the need for specific tools and pipelines to ensure the correct evaluation of these methods. Particularly, the use of representation learning for link prediction tasks requires train and test sampling, non-edge sampling, and in many cases selection of edge embedding methods and binary classifiers. The evaluation procedure, thus, becomes an ensemble of tasks which allow for many errors or inconsistencies. In this work we have proposed EvalNE, a novel framework that can be used to evaluate any network embedding method for link prediction. Our pipeline automates the selection of train and test edge sets, simplifies the process of tuning model parameters and reports the accuracy of the methods according to many criteria. Our experiments highlight the importance of the edge sampling strategy and parameter tuning for evaluating NE methods. We have also introduced a scalable procedure to select edge sets from given networks and showed empirically that is orders or magnitude faster than the naive approaches used in recent literature.

Represent the TLDR for finding the corresponding scientific article!

In this paper we introduce EvalNE, a Python toolbox for automating the evaluation of network embedding methods on link prediction and ensuring the reproducibility of results.

Represent the short summary for finding the corresponding scientific article (long)

We point out important problems with the common practice of using the best single model performance for comparing deep learning architectures, and we propose a method that corrects these flaws.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Deep learning models can be efficiently optimized via stochastic gradient descent, but there is little theoretical evidence to support this. A key question in optimization is to understand when the optimization landscape of a neural network is amenable to gradient-based optimization. We focus on a simple neural network two-layer ReLU network with two hidden units, and show that all local minimizers are global. This combined with recent work of; show that gradient descent converges to the global minimizer. For the duration of this paper, we will assume that x is standard normal in R n and all expectations are with respect to the standard normal. The population loss function is: DISPLAYFORM0 Define DISPLAYFORM1 so the loss can be rewritten as (ignoring additive constants, then multiplied by 4): DISPLAYFORM2 g(w i, w j) − 2g(w i, w * j).From BID0 we get DISPLAYFORM3 and DISPLAYFORM4 In this paper, we study the landscape of f over the manifold R = {w 1 = w 2 = 1}. The manifold gradient descent algorithm is: DISPLAYFORM5 where P R is the orthogonal projector onto the manifold R, and ∇ R is the manifold gradient of f. In order to analyze the global convergence of manifold gradient descent, we need a characterization of all critical points. We show that f (W) have no spurious local minimizer on the manifold R. Theorem 4.1. Assume wThe next theorem shows that manifold gradient descent with random initialization converges to the global minimizer Theorem 4.2. With probability one, manifold gradient descent will converge to the global minimizers. Proof. The objective function f is infinitely differentiable on manifold R. Using Corollary 6 of BID16, manifold gradient descent will converge to a local minimizer with probability one. Since the only local minima for function f are w 1 = wThe second observation is we only need to compute the gradient on the manifold and check whether it's zero. Define m(w 1) = sin θ 1 ∂f ∂w11 − cos θ 1 ∂f ∂w12 and m(w 2) = sin θ 2 ∂f ∂w21 − cos θ 2 ∂f ∂w22. Then for w 1 and w 2, the norm of the manifold gradients are |m(w 1)| and |m(w 2)|. Thus, we only need to check whether the value of function m is 0 and get rid of the absolute value sign. Then we apply the polar coordinates onto the manifold gradients, and obtain: m(w 2) = 1 π (π − θ w1,w2) sin(θ 2 − θ 1) + cos θ 2 − sin θ 2 + 1 π θ w2,w * 1 sin θ 2 − θ w2,w * 2 cos θ 2.The last observation we need for this theorem is that we must divide this problem into several cases because each angle in is a piecewise linear function. If we discuss each case independently, the ing functions are linear in the angles. The details are in Appendix B. After the calculation of all cases, we found the positions of all the critical points: WLOG assume θ 1 ≤ θ 2, then there are four critical points in the 2D case: (θ 1, θ 2) = (0,). After finding all the critical points, we compute the manifold Hessian matrix for those points and show that there is a direction of negative curvature. The details can be found in Appendix C. The next step is to reduce it to a three dimensional problem. As stated in the two-dimensional case, the gradient is in span{w 1, w 2, w * 1, w * 2}, which is four-dimensional. However, using the following lemma, we can reduce it to three dimensions and simplify the whole problem. Lemma 4.4. If (w 1, w 2) is a critical point, then there exists a set of standard orthogonal basis (e 1, e 2, e 3) such that e 1 = w Even if we simplify the problem into three dimensional case, it still seems to be impossible to identify all critical points explicitly. Our method to analyze the landscape of the loss surface is to find the properties of critical points and then show all saddle points and local maximizers have a direction of negative curvature. The following two lemmas captures the main geometrical properties of the critical points in three dimensional case. More detailed properties are given is Section 5.2 Lemma 4.5.arccos(−w 11) arccos(−w 21) = arccos(−w 12) arccos(−w 22) = − w 23 w 13.The ratio in Lemma 4.5 captures an important property of all critical points. For simplicity, based on D.5, we define DISPLAYFORM0.Then from the properties of θ 1, θ 2 and upper bound the value of k 0 we get Lemma 4.6. θ 1 = θ 2.That lemma shows that w 1 and w 2 must be on a plane whose projection onto span{w * 1, w * 2} is the bisector of w * 1 and w * 2. Combining this with the computation of Hessian, we conclude that we have found negative curvature for all possible critical points, which leads to the following proposition. Here we provide some detailed proofs which are important for the understanding of the main theorem. In general case, the following lemma shows we only need three dimension. Lemma 5.1. If (w 1, w 2) is a critical point, then there exists a set of standard orthogonal basis (e 1, e 2, e 3) such that e 1 = w * 1, e 2 = w * 2 and w 1, w 2 lies in span{e 1, e 2, e 3}.Proof. If (w 1, w 2) is a critical point, then DISPLAYFORM0 where matrix (I − w 1 w T 1) projects a vector onto the tangent space of w 1. Since DISPLAYFORM1 we get DISPLAYFORM2 which means that DISPLAYFORM3 )w * 2 lies in the direction of w 1. If θ w1,w2 = π, i.e., w 1 = −w 2, then of course the four vectors have rank at most 3, so we can find the proper basis. If θ w1,w2 < π, then we know that there exists a real number r such that DISPLAYFORM4 Since θ w1,w2 < π, we know that the four vectors w 1, w 2, w * 1 and w * 2 are linear dependent. Thus, they have rank at most 3 and we can find the proper basis. Next we will focus on the properties of critical points. Assume (w 1, w 2) is one of the critical points, from lemma D.1 we can find a set of standard orthogonal basis (e 1, e 2, e 3) such that e 1 = w * 1, e 2 = w * 2 and w 1, w 2 lies in span{e 1, e 2, e 3}. Furthermore, assume w 1 = w 11 e 1 + w 12 e 2 + w 13 e 3 and w 2 = w 21 e 1 + w 22 e 2 + w 23 e 3, i.e., w 1 = (w 11, w 12, w 13) and w 2 = (w 21, w 22, w 23). Since we have already found out all the critical points when w 13 = w 23 = 0, in the following we assume w Proof. Adapting from the proof of lemma D.4 and we know that DISPLAYFORM0 Similarly, we have DISPLAYFORM1 Taking the first component of FORMULA0 and FORMULA0 gives us DISPLAYFORM2 DISPLAYFORM3 Thus, DISPLAYFORM4 Similarly, we get DISPLAYFORM5 Since ∀i, j ∈, π − θ wi,w * j = arccos(−θ wij), we know that DISPLAYFORM6 Using this equation, we obtain several properties of critical points. The following two lemmas show the basic properties of critical points in three dimensional case. Completed proofs are given in Appendix B and C. Lemma 5.3. θ w1,w2 < π. Lemma 5.4. w 13 * w 23 < 0.These two lemmas restrict the position of critical points in some specific domains. Then we construct a new function F in order to get more precise analysis. Define DISPLAYFORM7 From the properties of that particular function and upper bound the value of k 0 we get Lemma 5.5. θ 1 = θ 2.That lemma shows that w 1 and w 2 must be on a plane whose projection onto span{w * 1, w * 2} is the bisector of w * 1 and w * 2. Although we cannot identify the critical points explicitly, we will show these geometric properties already capture the direction of negative curvature. In this section, we partially characterize the structure of the critical points when w * 1, w * 2 are nonorthogonal, but form an acute angle. In other words, the angle between w * 1 and w * 2 is α ∈ (0, π 2). Let us first consider the 2D cases, i.e., both w 1 and w 2 are in the span of w * 1 and w * 2. Similar to the original problem, after the technique of changing variables(i.e., using polar coordinates and assume θ 1 and θ 2 are the angles of w 1 and w 2 in polar coordinates), we divide the whole plane into 4 parts, which are the angle in [0, α], [α, π], [π, π + α] and [π + α, 2π). We have the following lemma: Lemma 6.1. Assume w * 1 = w * 2 = 1, w * T 1 w * 2 > 0 and w 1, w 2 ∈ span{w * 1, w * 2}. When w 1 and w 2 are in the same part(one of four parts), the only critical points except the global minima are those when both w 1 and w 2 are on the bisector of w * 1 and w * 2.Proof. The complete proof is given in appendix E, the techniques are nearly the same as things in the original problem and a bit harder, so to be brief, we omit the proof details here. For the three-dimensional cases cases of this new problem, it's interesting that the first few lemmatas are still true. Specifically, Lemma D.1(restated as Lemma 4.4) to Lemma D.5(restated as Lemma 4.5) are still correct. The proof is very similar to the proofs of those lemmas, except we need modification to the coefficients of terms in the expressions of the manifold gradients. We did experiments to verify the theoretical . Since our are restricted to the case of K = 2 hidden units, it is also natural to investigate whether general two-layer ReLU networks also have the property that all local minima are global minima. Unfortunately as we show via numerical simulation, this is not the case. We consider the cases of K from 2 to 11 hidden units and we set the dimension d = K. For each K, the true parameters are orthogonal to each other. For each K, we run projected gradient descent with 300 different random initializations, and count the number of local minimum (critical points where the manifold Hessian is positive definite) with non-zero training error. If we reach a sub-optimal local minimum, we can conclude the loss surface exhibits spurious local minima. The bar plot showing the number of times gradient descent converged to spurious local minima is in FIG2. From the plot, we see there is no spurious local minima from K = 2 to K = 6. However for K ≥ 7, we observe a clear trend that there are more spurious local minima when there are more hidden units. In this paper, we provided recovery guarantee of stochastic gradient descent with random initialization for learning a two-layer neural network with two hidden nodes, unit-norm weights, ReLU activation functions and Gaussian inputs. Experiments are also done to verify our . For future work, here we list some possible directions. This paper focused on a ReLU network with only two hidden units,. And the teaching weights must be orthogonal. Those are many conditions, in which we think there are some conditions that are not quite essential, e.g., the orthogonal assumption. In experiments we have already seen that even if they are not orthogonal, it still has some good properties such as the positions of critical points. Therefore, in the future we can further relax or abandon some of the assumptions of this paper and preserve or improve the we have. The neural network we discussed in this paper is in some sense very simple and far from practice, although it is already the most complex model when we want to analyze the whole loss surface. By experiments we have found that when it comes to seven hidden nodes with orthogonal true parameters, there will be some bad local minima, i.e., there are some local minima that are not global. We believe that research in this paper can capture the characteristics of the whole loss surface and can help analyze the loss surface when there are three or even more hidden units, which may give some bounds on the performance of bad local minima and help us understand the specific non-convexity of loss surfaces. Consider a neural network with 2 hidden nodes and ReLU as the activation function: DISPLAYFORM0 where σ(x) = max(0, x) is the ReLU function. First we study the 2-D case, i.e., the input and all parameters are two dimensional. Assume that the input follows standard normal distribution. The loss function is population loss: DISPLAYFORM1 Define DISPLAYFORM2 then from BID0 we get DISPLAYFORM3 Thus, DISPLAYFORM4 Moreover, from FORMULA1 we get DISPLAYFORM5 Assume w * 1 = w * 2 and w * T 1 w * 2 = 0. WLOG, let e 1 = w * 1 and e 2 = w * 2. Then we know that ∀i, j ∈, g(w * i, w * j) is a constant number. Thus, define the objective function(which equals to 4l(W) up to an additive constant) DISPLAYFORM6 Thus, DISPLAYFORM7 Similarly, for w 2, the gradient is ∂f DISPLAYFORM8 Assume that w 1 = (w 11, w 12) and w 2 = (w 21, w 22), then the gradient can be expressed in this form: DISPLAYFORM9 and DISPLAYFORM10 Because of symmetry, for w 2, the gradient is DISPLAYFORM11 and DISPLAYFORM12 B CRITICAL POINTS IN 2D CASES In 2D cases, we can translate W to polar coordinates and fix w 1 = w 2 = 1, so there are two variables left: θ 1 and θ 2, i.e., w 1 = (cos θ 1, sin θ 1) and w 2 = (cos θ 2, sin θ 2).For manifold gradient, we only need to consider its norm and check whether it's zero. Under review as a conference paper at ICLR 2018To make life easier, it's better to simplify the m functions a bit using w 1 = (cos θ 1, sin θ 1) and w 2 = (cos θ 2, sin θ 2): DISPLAYFORM0 Similarly, DISPLAYFORM1 Then we can divide them into several cases and analyze them one by one to specify the positions and properties of the critical points. DISPLAYFORM2 The norm of the manifold gradient w.r.t. w 1 is DISPLAYFORM3 Similarly, the norm of m(w 2) is DISPLAYFORM4 Define DISPLAYFORM5 If m(w 1) = m(w 2) = 0, then DISPLAYFORM6 and DISPLAYFORM7 Thus, DISPLAYFORM8 Also note that FORMULA3 and we get DISPLAYFORM9 DISPLAYFORM10 and the inequality becomes equality only then DISPLAYFORM11 ≥ 0. Note that is because cos(2θ) − cos θ is always non-positive when 0 ≤ θ ≤ DISPLAYFORM12 In a word, there are two critical points in this case: DISPLAYFORM13 The norm of the manifold gradient w.r.t. w 1 is DISPLAYFORM14 Similarly, DISPLAYFORM15 Define DISPLAYFORM16 DISPLAYFORM17 and the inequality becomes equality only then θ = π 2 or θ = π. DISPLAYFORM18 Note that the inequality becomes equality only when θ cos θ = 0 and DISPLAYFORM19 and DISPLAYFORM20 Thus, DISPLAYFORM21 However, we know that h 2 (θ 1) < 0 and h 2 (θ 1) < 0, which makes a contradiction. In a word, there is no critical point in this case. DISPLAYFORM22 The norm of the manifold gradient w.r.t. w 1 is DISPLAYFORM23 Similarly, the norm of m(w 2) is DISPLAYFORM24 Define DISPLAYFORM25 Let θ = θ + π, then DISPLAYFORM26 DISPLAYFORM27 Moreover, ∀θ ∈ [π, DISPLAYFORM28 DISPLAYFORM29 DISPLAYFORM30 = 0.Also, when θ ∈ [π, DISPLAYFORM31 so h 3 is an increasing function when θ ∈ [π, From Lemma B.1, DISPLAYFORM32 DISPLAYFORM33 DISPLAYFORM34 DISPLAYFORM35 Note that becomes equality only when θ 1 = π or θ 1 = DISPLAYFORM36 Actually, this is symmetric to the B.3, so in this part I would like to specify this kind of symmetry. We have already assumed that θ 1 ≤ θ 2 without loss of generality, and under this assumption, we can find another symmetry: From w 1 and w 2, using line y = x as symmetry axis, we can get two new vectors w 1 and w 2. w 1 is not necessarily the image of w 1 because we need to preserve the assumption that θ 1 ≤ θ 2, but there exists one and only one mapping such that θ 1 ≤ θ 2. In this kind of symmetry, the angles, including θ w1,w2 and θ wi,w * j where i, j ∈, are the same, so the two symmetric cases share the same gradients, thus the symmetric critical points. We use (i, j),where i, j ∈, to represent the case that θ 1 is in the ith quadrant and θ 2 is in the jth one. Using this kind of symmetry, we conclude that is equivalent to and is equivalent to, so there are 4 cases left which are,, and. DISPLAYFORM37 Similar to previous cases, DISPLAYFORM38 and m(w 2) = 1 π (π − θ 2 + θ 1) sin(θ 2 − θ 1) + cos θ 2 − sin θ 2 DISPLAYFORM39 Using previous definitions, we conclude that DISPLAYFORM40 DISPLAYFORM41 If m(w 1) = m(w 2) = 0, then m (w 1) + m(w 2) = 0, i.e., DISPLAYFORM42 From we know that DISPLAYFORM43 Thus, using lemma B.2, DISPLAYFORM44 That means the only case that h 1 (θ 1) + h 2 (θ 2) = 0 is when the inequality becomes equality, which means that cos θ 1 = 1 and h 2 (θ 1 + π 2) = h 2 (θ 2) = − 1 2. Thus, we must have θ 1 = 0, and θ 2 = π 2 or θ 2 = π. Plugging them back in FORMULA0 and FORMULA0, we can verify that the first one is a critical point while the other is not. Since (θ 1, θ 2) = (0, π 2) has been counted in case 1, there are no new critical points in this case. DISPLAYFORM45 Similar to previous cases, DISPLAYFORM46 and m(w 2) = 1 π (π − θ w1,w2) sin(θ 2 − θ 1) + cos θ 2 − sin θ 2 DISPLAYFORM47 Thus, using previous definitions DISPLAYFORM48 and DISPLAYFORM49 If m(w 1) = m(w 2) = 0, then m(w 1) + m(w 2) = 0, i.e., DISPLAYFORM50 DISPLAYFORM51 Then we have the following lemma: Lemma B.3. When 0 ≤ θ ≤ Proof. From, h 3 (θ + π) = h 1 (θ) − cos θ + sin θ. Thus, H(θ) = 2h 1 (θ) − cos θ + sin θ DISPLAYFORM52 When 0 ≤ θ ≤ π 4, since sin θ is a concave function for θ, we know that DISPLAYFORM53 Thus, DISPLAYFORM54 To make H(θ) = 0, we must have DISPLAYFORM55 Thus, DISPLAYFORM56 And to make H(θ) = 0, the only possibility is θ = π 2, which ends the proof. Remember that if m(w 1) = m(w 2) = 0, then we have h 3 (θ 2) = −h 1 (θ 1).If h 1 (θ 1) > 0, i.e., 0 ≤ θ 1 < π 4, then from lemma B.3, H(θ 1) ≤ 0, which means that DISPLAYFORM57 Since h 3 is a strictly increasing function, we know that if h 3 (θ 2) = −h 1 (θ 1), then θ 2 ≥ θ 1 + π, so sin(θ 1 − θ 2) ≥ 0, and that means DISPLAYFORM58 Similarly, if h 1 (θ 1) < 0, i.e., DISPLAYFORM59 Thus, if h 3 (θ 2) = −h 1 (θ 1), then θ 2 ≤ θ 1 + π, so sin(θ 1 − θ 2) ≤ 0, and that means DISPLAYFORM60 The last possibility is h 1 (θ 1) = 0, i.e., θ 1 = π 4. Plugging it into and we know that h 3 (θ 2) = 0, so θ 2 = 5π 4. And that is indeed a critical point. In a word, the only critical point in this case is (θ 1, θ 2) = (DISPLAYFORM61 Like previous cases, DISPLAYFORM62 If m(w 1) = m(w 2) = 0, then m (w 1) + m(w 2) = 0, i.e., DISPLAYFORM63 Let θ = θ 2 − π, then from and FORMULA0, we know that DISPLAYFORM64 Thus, from lemma B.2, DISPLAYFORM65 Therefore, in order to achieve h 2 (θ 1) + h 3 (θ 2) = 0, the only way is let becomes equality, which means that θ 2 = 3π 2 and θ 1 = π 2 or π. Plugging them into FORMULA0 and FORMULA0 we conclude that both of them are not critical points. In a word, there is no critical point in this case. DISPLAYFORM66 Similar to previous cases, DISPLAYFORM67 and m(w 2) = 1 π (π − θ w1,w2) sin(θ 2 − θ 1) + cos θ 2 − sin θ 2 DISPLAYFORM68 2, we must have θ w1,w2 = π 2, so it must be true that DISPLAYFORM69 Therefore, using lemma B.2, DISPLAYFORM70 In a word, there is no critical point in this case. In , based on the assumption that θ 1 ≤ θ 2 there are four critical points in the 2D case: Assume the manifold is R = {(w 1, w 2): w 1 2 = w 2 2 = 1}, then the Hessian on the manifold is DISPLAYFORM0 DISPLAYFORM1 where z = (z 1, z 2) satisfies w DISPLAYFORM2 and DISPLAYFORM3 Then we can get when w 1 = w 2 and w 1 = −w 2, DISPLAYFORM4 So this point is a saddle point. In , we have four critical points: one is global maximal, the other three are saddle points. DISPLAYFORM0 ) is a critical point, then there exists a set of standard orthogonal basis (e 1, e 2, e 3) such that e 1 = w * 1, e 2 = w * 2 and w 1, w 2 lies in span{e 1, e 2, e 3}.Proof. If (w 1, w 2) is a critical point, then DISPLAYFORM1 where matrix (I − w 1 w T 1) projects a vector onto the tangent space of w 1. Since DISPLAYFORM2 we get DISPLAYFORM3 DISPLAYFORM4 )w * 2 lies in the direction of w 1. If θ w1,w2 = π, i.e., w 1 = −w 2, then of course the four vectors have rank at most 3, so we can find the proper basis. If θ w1,w2 < π, then we know that there exists a real number r such that DISPLAYFORM5 Since θ w1,w2 < π, we know that the four vectors w 1, w 2, w * 1 and w * 2 are linear dependent. Thus, they have rank at most 3 and we can find the proper basis. Next we will focus on the properties of critical points. Assume (w 1, w 2) is one of the critical points, from lemma D.1 we can find a set of standard orthogonal basis (e 1, e 2, e 3) such that e 1 = w * 1, e 2 = w * 2 and w 1, w 2 lies in span{e 1, e 2, e 3}. Furthermore, assume w 1 = w 11 e 1 + w 12 e 2 + w 13 e 3 and w 2 = w 21 e 1 + w 22 e 2 + w 23 e 3, i.e., w 1 = (w 11, w 12, w 13) and w 2 = (w 21, w 22, w 23). Since we have already found out all the critical points when w 13 = w 23 = 0, in the following we assume w Proof. If θ w1,w2 = π, then w 1 = −w 2, so w 2 is in the direction of w 1. We have already known from DISPLAYFORM0 )w * 2 lies in span{e 1, e 2}, so w 1 ∈ span{e 1, e 2} and w 2 ∈ span{e 1, e 2}. Thus, w 13 = w 23 = 0 and that contradicts with the assumption. In a word, θ w1,w2 < π. Lemma D.3. w 13 * w 23 = 0.Proof. We have already known from DISPLAYFORM1 lies in the direction of w 1. Writing it in each dimension and we know that there exists a real number r 0 such that DISPLAYFORM2 From lemma D.2 we know that θ w1,w2 < π, so we can define DISPLAYFORM3.Then the equations become DISPLAYFORM4 Similarly, we have DISPLAYFORM5 Since w 2 13 + w 2 23 = 0, at least one of those two variables cannot be 0. WLOG, we assume that w 13 = 0. If w 23 = 0, then from we know that w 13 = 0, which contradicts the assumption. Thus, w 23 = 0, which means that w 13 * w 23 = 0.Lemma D.4. w 13 * w 23 < 0.Proof. Adapting from the proof of lemma D.3, we know that DISPLAYFORM6 DISPLAYFORM7 DISPLAYFORM8 and DISPLAYFORM9 DISPLAYFORM10 DISPLAYFORM11 Furthermore, kk = From lemma D.2 we know that θ w1,w2 < π, and from lemma D.3 we know that both w 1 and w 2 are DISPLAYFORM12 > 0. Therefore, we have DISPLAYFORM13 That means k < 0, so Proof. Adapting from the proof of lemma D.4 and we know that DISPLAYFORM14 Similarly, we have DISPLAYFORM15 Taking the first component of FORMULA0 and FORMULA0 gives us DISPLAYFORM16 DISPLAYFORM17 Thus, DISPLAYFORM18 Similarly, we get DISPLAYFORM19 Since ∀i, j ∈, π − θ wi,w * j = arccos(−θ wij), we know that DISPLAYFORM20 For simplicity, based on D.5, we define k 0 = −k, θ 1 = π − θ w2,w * 1 and θ 2 = π − θ w2,w *. Then DISPLAYFORM0 WLOG, assume k 0 ≥ 1, otherwise we can switch w 1 and w 2.Thus, DISPLAYFORM1 DISPLAYFORM2 Similarly, if we apply the change of variables onto the second component of, we will get DISPLAYFORM3 Thus, DISPLAYFORM4 Proof. Note that when θ ∈ [0, DISPLAYFORM5 is a strict decreasing function w.r.t. θ. Note that G = k 0 + 1 > 0 and DISPLAYFORM6 Then the only thing we need to prove is DISPLAYFORM7 Since the inequality holds only when cos 2, which means k 0 = 3 and k 0 = 1, which makes a contradiction. Thus, DISPLAYFORM8 Therefore, DISPLAYFORM9, which completes the proof. Lemma D.10. F (θ) is either strictly decreasing or first decrease and then increase when θ ∈ (θ 0, DISPLAYFORM10 Proof. DISPLAYFORM11 Define DISPLAYFORM12, then H(θ)·F (θ) < 0(i.e., when H(θ) is positive, F (θ) is decreasing, otherwise F (θ) is increasing), and we know that DISPLAYFORM13 Note that holds because θ > θ 0 ≥ π 2k0. Thus, H(θ) is a strictly decreasing function when θ ∈ (θ 0, π k0]. We can see that DISPLAYFORM14 Thus, if H(DISPLAYFORM15 Otherwise, F (θ) first decrease and then increase when θ ∈ (θ 0, DISPLAYFORM16 Proof. From lemma D.10 we have already known that F (θ) is either strictly decreasing or first decrease and then increase when θ ∈ (θ 0, π k0], so the maximum of the function value on an interval can only be at the endpoints of that interval, which means that we only need to prove F (DISPLAYFORM17 Thus, h(x) is decreasing in [, π]. However, we know that h(DISPLAYFORM18 which means that F ( DISPLAYFORM19 Lemma D.12. θ 1 = θ 2 .Proof. From the proof of lemma D.8 we get DISPLAYFORM20 Thus, DISPLAYFORM21 Using lemma D.9, If z = (tz 1, z 2), ||z 1 || = ||z 2 || = 1 and w E 2D CASES WITH ASSUMPTION RELAXATION Since this section is pretty similar to B, I will try my best to make it brief and point out the most important things in the proof. DISPLAYFORM22 After the changing of variables(i.e., polar coordinates), we know that w 1 = (cos θ 1, sin θ 1) and w 2 = (cos θ 2, sin θ 2). Then when θ is in the first part to the fourth part, the function h will change to four different functions: DISPLAYFORM0 WLOG, we assume θ 1 ≤ θ 2. E.2 0 ≤ θ 1 ≤ θ 2 ≤ α First, it's easy to verify that ∀θ ∈ [0, θ], h 1 (θ) + h 1 (α − θ) = 0.Besides, h 1 (θ) = sin θ + sin(α − θ) − (π − θ) cos θ − (π − α + θ) cos(α − θ)= 2 sin α 2 cos(θ − α 2) − (π − θ) cos θ − (π − α + θ) cos(α − θ)≤ 2 sin α 2 − π 2 (cos θ + cos(α − θ)) DISPLAYFORM1 ≤ 2 sin α 2 − π cos α 2 < 0.When m(w 1) = m(w 2) = 0, we know that h 1 (θ 1)+h 1 (θ 2) = 0, and because of those two properties above, we know that θ 1 + θ 2 = α. Thus, θ 1 ∈ [0, α 2]. And we have the following lemma Lemma E.1. m(w 1) ≤ 0. m(w 1) = sin(α − 2θ 1)(π − α + 2θ 1) − (π − α + θ 1) sin(α − θ 1) + (π − θ 1) sin θ 1 ≥ sin(α − 2θ 1)(π − α + θ 1) − (π − α + θ 1) sin(α − θ 1) + (π − θ 1) sin θ 1≥ sin(α − 2θ 1)(π − α + θ 1) − (π − α + θ 1) sin(α − θ 1) + (π − α 2) sin θ 1= (π − α + θ 1)(sin(α − 2θ 1) − sin(α − θ 1)) + (π − α 2) sin θ 1≥ (π − α 2)(sin(α − 2θ 1) − sin(α − θ 1) + sin θ 1 )= (π − α 2)(sin(α − 2θ 1) − sin θ 1 − sin θ 1 cos(α − 2θ 1) − cos θ 1 sin(α − 2θ 1)) DISPLAYFORM0 Thus, the only possible critical points are m(w 1) = 0, which are 0 and α 2. After verification, we conclude that there are only two critical points in this case: (θ 1, θ 2) = (0, α) or (θ 1, θ 2) = (α 2, α 2).E.3 α ≤ θ 1 ≤ θ 2 ≤ π When m(w 1) = m(w 2) = 0, we know that h 1 (θ 1) + h 1 (θ 2) = 0. However, when θ ∈ [α, π], we know that h 2 (θ) = (π − θ + α) sin(α − θ) − (π − θ) sin θ ≤ 0.The inequality cannot become equal because the possible values of θs such that each term equals zero has no intersection. Thus, h 2 (θ) is always negative, which means that in this case there are no critical points. E.4 π ≤ θ 1 ≤ θ 2 ≤ π + α It's easy to verify that ∀θ ∈ [π, π + α], h 3 (θ) + h 3 (2π + α − θ) = 0. Furthermore, DISPLAYFORM1 > 0.Thus, from m(w 1) = m(w 2) = 0, we know that h 1 (θ 1) + h 1 (θ 2) = 0 we get θ 1 + θ 2 = 2π + α, which means that θ 1 ∈ [π, π + α 2], so we can prove the following lemma: Lemma E.2. m(w 1) ≤ 0.Proof. Let θ = θ 1 − π, then m(w 1) = (π − θ 2 + θ 1) sin(θ 1 − θ 2) + h 3 (θ 1) = (π + θ − α + θ) sin(2θ − α) + h 1 (θ) + π sin θ − π sin(α − θ) ≤ (π + 2θ − α) sin(2θ − α) + sin(α − 2θ)(π + 2θ − α) + π(sin θ − sin(α − θ)) ≤ π(sin θ − cos θ) ≤ 0.The first inequality is from lemma E.1.Thus, the only possible critical points are m(w 1) = 0, which are π and π + α 2. After verification, we conclude that there are only two critical points in this case: (θ 1, θ 2) = (π, π + α) or (θ 1, θ 2) = (π + α 2, π + α 2).

Represent the short summary for finding the corresponding scientific article (long)

Recovery guarantee of stochastic gradient descent with random initialization for learning a two-layer neural network with two hidden nodes, unit-norm weights, ReLU activation functions and Gaussian inputs.

represent the short summary for finding the corresponding scientific article (long).

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.