Question: represent the text to find a very short summary (1-2 sentences max)

Question: This work provides an automatic machine learning (AutoML) modelling architecture called Autostacker. Autostacker improves the prediction accuracy of machine learning baselines by utilizing an innovative hierarchical stacking architecture and an efficient parameter search algorithm. Neither prior domain knowledge about the data nor feature preprocessing is needed. We significantly reduce the time of AutoML with a naturally inspired algorithm - Parallel Hill Climbing (PHC). By parallelizing PHC, Autostacker can provide candidate pipelines with sufficient prediction accuracy within a short amount of time. These pipelines can be used as is or as a starting point for human experts to build on. By focusing on the modelling process, Autostacker breaks the tradition of following fixed order pipelines by exploring not only single model pipeline but also innovative combinations and structures. As we will show in the experiment section, Autostacker achieves significantly better performance both in terms of test accuracy and time cost comparing with human initial trials and recent popular AutoML system. Machine Learning nowadays is the main approach for people to solve prediction problems by utilizing the power of data and algorithms. More and more models have been proposed to solve diverse problems based on the character of these problems. More specifically, different learning targets and collected data correspond to different modelling problems. To solve them, data scientists not only need to know the advantages and disadvantages of various models, they also need to manually tune the hyperparameters within these models. However, understanding thoroughly all of the models and running experiments to tune the hyperparameters involves a lot of effort and cost. Thus, automating the modelling procedure is highly desired both in academic areas and industry. An AutoML system aims at providing an automatically generated baseline with better performance to support data scientists and experts with specific domain knowledge to solve machine learning problems with less effort. The input to AutoML is a cleanly formatted dataset and the output is one or multiple modelling pipelines which enables the data scientists to begin working from a better starting point. There are some pioneering efforts addressing the challenge of finding appropriate configurations of modelling pipelines and providing some mechanisms to automate this process. However, these works often rely on fixed order machine learning pipelines which are obtained by mimicking the traditional working pipelines of human experts. This initial constraint limits the potential of machine to find better pipelines which may or may not be straightforward, and may or may not have been tried by human experts before. In this work, we present an architecture called Autostacker which borrows the stacking BID1 method from ensemble learning, but allows for the discovery of pipelines made up of simply one model or many models combined in an innovative way. All of the automatically generated pipelines from Autostacker will provide a good enough starting point compared with initial trials of human experts. However, there are several challenges to accomplish this:• The quality of the datasets. Even though we are stepping into a big data era, we have to admit that there are still a lot of problems for which it is hard to collect enough data, especially data with little noise, such as historical events, medical research, natural disasters and so on. We tackle this challenge by always using the raw dataset in all of the stacking layers Figure 1: This figure describes the pipeline architecture of Autostacker. Autostacker pipelines consists of one or multiple layers and one or multiple nodes inside each layer. Each node represents a machine learning primitive model, such as SVM, MLP, etc. The number of layers and the number of nodes per layer can be specified beforehand or they can be changeable as part of the hyperparameters. In the first layer, the raw dataset is used as input. Then in the following layers, the prediction from each node will be added to the raw dataset as synthetic features (new colors). The new dataset generated by each layer will be used as input to the next layer.while also adding synthetic features in each stacking layer to fully use the information in the current dataset. More details are provided in the Approach section below.• The generalization ability of the AutoML framework. As mentioned above, existing AutoML frameworks only allow systems to generate an assembly line from data preprocessing and feature engineering to model selection where only a specific single model will be utilized by plugging in a previous model library. In this paper, depending on the computational cost and time cost, we make the number of such primitive models a variable which can be changed dynamically during the pipeline generation process or initialized in the beginning. This means that the simplest pipeline could be a single model, and the most complex pipeline could contain hundreds of primitive models as shown in Figure 1 • The large space of variables. The second challenge mentioned above leads to this problem naturally. Considering the whole AutoML framework, variables include the type of primitive machine learning models, the configuration settings of the framework (for instance, the number of primitive models in each stacking layer) and the hyperparameters in each primitive model. One way to address this issue is to treat this as an optimization problem BID3. Here in this paper, we instead treat this challenge as a search problem. We propose to use a naturally inspired algorithm, Parallel Hill Climbing (PHC), BID10 to effectively search for appropriate candidate pipelines. To make the definition of the problem clear, we will use the terminology listed below throughout this paper:• Primitive and Pipeline: primitive denotes an existed single machine learning model, for example, a DecisionTree. In addition, these also include traditional ensemble learning models, such as Adaboost and Bagging. The pipeline is the form of the output of Autostacker, which is a single primitive or a combination of primitives.• Layer and Node: Figure 1 shows the architecture of Autostacker which is formed by multiple stacking layers and multiple nodes in each layers. Each node represents a machine learning primitive model. Automated Machine Learning has recently gained more attention and there are a variety of related research programs underway and tools coming out. In this section, we first describe recent work in this field and then explain where our work fits in. The current focus in AutoML mainly consists of two parts: machine learning pipeline building and intelligent model hyperparameter search. By extending the fixed pipeline used in works mentioned above, such as Auto-Weka and Autosklearn, one of the most recent and popular framework called TPOT Olson et al. FORMULA0 allows for parallel feature engineering which happens before model prediction. However, all of the works mentioned above follow the same machine learning pipeline. TPOT also uses Evolutionary Algorithms to treat the parameter configuration problem as a search problem. In this work, we use TPOT as one of our baselines. As described above, there is very little work trying to discover innovative pipelines, even with traditional building blocks, such as sklearn. Pushing the ability of machines to be able to discover innovative machine learning building pipelines, such as new combinations or new arrangements, is necessary to cover a larger space of possible architectures. In this work, we encourage Autostacker to fulfill this requirement in two ways: 1. generating models with new combinations and 2. generating models with completely innovative architectures. In terms of the optimization methods, we offer an alternative solution to search for the settings with Parallel Hill Climbing which is very effective, especially when we are faced with a giant possible search space. The success of using this kind of strategy on large scale AutoML is also proved in TPOT. The working process of Autostacker is shown in Figure 2 and the overview of pipeline architecture built by Autostacker is hown in Figure 1. The whole pipeline consists of multiple layers, where each layer contains multiple nodes. These nodes are the primitive machine learning models. The ith layer takes in the dataset X i, and outputs the prediction Y i,j, where Y i,j denotes the prediction of the jth node in the ith layer (i = 0, 1, 2, ..., I, j = 0, 1, 2, ..., J). After each layer's prediction, we add these prediction back to the dataset used as input to the layer as synthetic features, and then use this newly generated dataset as the input of the next layer. With each new layer, the dataset gets more and more synthetic features until the last layer which only consists of a single node. We take the output of the last layer as the final output of this machine learning problem. Again, if we use f k to denote the kth (k = 0, 1, 2, ..., K) feature in the dataset, the final dataset will contain DISPLAYFORM0 features in total and this new dataset will be used in the last layer prediction. N i (0,1,2,...) is the number of nodes in the ith layer. The total number of features in the dataset before the last layer can be specified by users. Unlike the traditional stacking algorithm in ensemble learning, which only feeds the prediction into next layer as inputs, this proposed architecture always keeps the information directly from the raw dataset. Here are the considerations: Figure 2: The overview process of Autostacker and its usage is shown here. First, we start to build the model pipelines by generating initial pipelines with dynamic configurations of architecture and hyperparameters, feeding into PHC algorithm, and looping the process to generate winning pipelines. Then the winning pipelines can be used as better baselines for data scientists, or we can analyze the pattern in the winning pipeline sets for further meta-learning usage.• The number of items in the dataset could be very small. If so, the prediction from each layer could contain very little information about the problem and it is very likely that the primitives bias the outcomes a lot. Accordingly, throwing away the raw dataset could lead to high-biased prediction which is not suitable for generalization, especially for situations where we could have more training data in the future.• Moreover, by combining the new synthetic features with the raw dataset, we implicitly give some features more weight when these features are important for prediction accuracy. Yet we do not delete the raw dataset because we do not fully trust the primitives in individual layers. We can consequently reduce the influences of bias coming from individual primitive and noise coming from the raw dataset. There are multiple hyperparameters within this architecture:• I and J: the maximum number of layers and the maximum number of nodes corresponding to each layer.• H: the hyperparameters in each primitive.• The types of the primitives. Here we provide a dictionary of primitives which only serves as a search space. Note that Autostacker provides two specifications for I and J. The default mode is to let users simply specify the maximum range of I and J. Only two positive integers are needed to enable Autostacker to explore different configurations. There are two advantages here: 1. This mode frees the system of constraints and allows for the discover of further possible innovative pipelines. 2. This speed up the whole process significantly. We will illustrate this point in the Experiment section later. Another choice is to explicitly denote the value of I and J. This allows systems to build pipelines with a specific number of layers and number of nodes per layer based on allowed computational power and time. The search algorithm for finding the appropriate hyperparameters is described in the next section. In this paper, the Parallel Hill Climber (PHC) Algorithm has been chosen as the search algorithm to find the group of hyperparameters which can lead to better baseline model pipelines. PHC is commonly used as baseline algorithm in the development of Evolutionary Algorithm. As we will show later, our system can already achieve significantly better performance with this straightforward baseline algorithm. Algorithm 1 provides the details of this algorithm in our system. First, we generate N completed pipelines by randomly selecting the hyperparameters. Then we run a one step Hill Climber on top of these N pipelines to get another N pipelines. The one-step Hill Climber essentially just randomly changes one of the hyperparameters, for example the number of estimators in a Random Forest Classifier. Now we train these 2N pipelines and evaluate them through cross validation. Then N pipelines with the highest validation accuracies are selected as the seed pipelines for the next generation of Parallel Hill Climber. Once the seed pipelines are ready, another one step Hill Climber will be applied on them and another round of evaluation and selection will be executed afterwards. The same loop continues until the end of all the iterations, where the number of iterations M can be specified by users. eva = EV ALU AT E(eva pip) DISPLAYFORM0 sel pip = SELECT (eva pip, eva , N) iter init = sel pip 10: end for 11: Return sel pip 12: function HILLCLIMBER(list pip) for each integer i in length of list pip do Return sel pip 28: end function This section presents the training and testing procedure. The training process happens in the evaluation step as shown above. Corresponding to our hierarchical framework, the pipeline is trained layer by layer. Inside each layer, each primitive is also trained independently with the same dataset. The next layer will be trained after integrating the previous dataset with with prediction from the previous trained layer. Similarly, the validation process and testing process share the same mechanism but with validation set and test set respectively. After training and validating the pipelines, we pick the first 10 pipelines with the highest validation accuracies as the final output of Autostacker. We believe that these 10 pipelines can provide better baselines for human experts to get started with the problem. Here choosing the top 10 pipelines instead of the first one pipeline directly is based on the consideration that the input might be a small amount of data which is more likely to be unbalanced. Yet unbalanced data cannot guarantee that the performance on the validation process can fully represent that on the testing process. For example, two pipelines with the same validation might behave very differently on the same test dataset. Hence, it is necessary to provide a set of candidates which can be guaranteed to do better on average so that human experts can fine tune the pipelines subsequently. Another significant advantage of our approach is that the system is very flexible to scale up and parallelize. Starting from the initial generation, one-step hill climbing, training, validation to evaluation, each pipeline runs independently, which means that each worker can work on one pipeline alone. There is no frequent communication or sequential decision making among all the workers and each worker can run through the pipeline separately. They only need to share the validation to be ranked by the end of each iteration. Then one shot selection based on the validation accuracy will be applied on the outputs of the parallel workers. More specifically, in terms of the Algorithm 1 we show above, Random, HILLCLIMBER, and EVALUATE function are all very easily parallelized when the system runs. To show the performance of our system, we selected 15 datasets from the benchmark dataset provided in BID9 which collects and cleans datasets from public data resources, such as and UCILichman FORMULA0 etc., as the sample experimental data. According to the published in TPOT, we arbitrarily choose 9 datasets claimed to have better in TPOT comparing with Random Forest Classifier, 4 datasets with worse performance in TPOT and 2 datasets with same performance with Random Forest Classifier in TPOT. We limit the total number of datasets to be 15 to show here to cover all cases of datasets in TPOT. These datasets come from different problem domains and target different machine learning tasks including binary classification and multi-class classification. Autostacker is also compatible with regression problems. We will release on more benchmark datasets as well as the code base. The data is cleaned in terms of filling in the missing values with large negative values or deleting the data points with missing values. Other than that, there is no other data preprocessing or feature preprocessing in Autostacker. It would certainly be possible to use preprocessing on the dataset and features as another building block or hyperparameter in Autostacker, and we also provide this flexibility in our system. Nevertheless, in this paper we focus only on the modelling process to show our contribution to the architecture and automation process. Before each round of the experiment, we shuffle and partition the dataset to 80%/20% as training/testing data. The goal of Autostacker is to provide a better baseline pipeline for data scientists in an automatic way. Thus, the baseline we choose to compare with should be able to represent the prediction ability of pipelines coming from the initial trials of data scientists. The baseline pipeline that we compare with is chosen to be Random Forest Classifier / Regressor with the number of estimators being 500 as ensemble learning models like Random Forest have been shown to work well on average in practice when considering multi-model predictions. We also compare our to those of the TPOT model from BID8 which is one of the most recent and popular AutoML systems. Currently, our primitives are from the scikit-learn library BID11 and XGboost libaray BID2 The full list is in TAB0 in the appendix. In Autostacker, users are allowed to plug in any primitives they like as long as the function signatures are consistent with our current code base. In terms of the basic structure (number of layers and number of nodes per layer) of the candidate pipelines, as we mentioned above, there are two types of settings provided in Autostacker. In this section, we show the performance of the default mode of Autostacker: dynamic configurations. We specify the maximum range of number of layers to be 5 and the maximum range of number of nodes per layer to be 3. In this section, we will show the of the test accuracy and time cost of Autostacker as well as comparisons with the Random Forest and TPOT baselines. The test accuracy is calculated using balanced accuracy BID17. We refer to them as test accuracy in the rest of this paper. We ran 10 rounds of experiments for Random Forest Classifier and 3 to 10 rounds of experiment for TPOT based on the computation and time cost. For Autostacker, 3 rounds of experiments are executed on each dataset and the datasets get shuffled before each round. Thus, the figure contains 30 test in total where each 10 of them come from 1 round experiment. The notches in the box plot represent the 95% confidence intervals of median values. We use one machine with 24 CPUs to parallelize each experiment for each architecture. As shown in FIG5, all the left side columns shows the test accuracy comparisons on the 15 sample datasets. From the comparisons, we can tell several things:• Autostacker achieves 100% better test accuracy compared with Random Forest Baselines, and 13 out of 15 better accuracy compared with TPOT, while the rest Hill Valley with noise and vehicle datasets achieve extremely similar or slight lower accuracy according to median values.• Autostacker is much more robust. Autostakcer can always provide better baselines to speed up further work, while Random Forest fails to give any better on the parity5 dataset, and TPOT fails to provide better baselines than Random Forest Classifier on the breastcancer, pima, ecoli, wine-recognition and cars datasets after spending a couple of hours searching. This kind of guarantee of not being worse on average comes from the characteristic of Autostacker: the innovative stacking architecture which fully utilizes the predictions coming from different primitive models as well as the whole information of the raw dataset. In the meantime, Autostacker does not give up the single model case if it is better than a stacking architecture. Hence, Autostacker essentially contains the Random Forst Baseline we are using here. All the right side columns in FIG5 show the time cost among comparisons. Autostacker largely reduce the time usage up to 6 times comparing with TPOT on all the sample datasets. In of the experiment , the output of Autostacker can improve the baseline pipeline sufficiently enough for human experts to start with better pipelines within a short mount of time, and Autostacker achieves significantly better performance on sample datasets than all baseline comparisons. During the experiments and research process, we noticed that Autostacker still has several limitations. Here we will describe these limitations and possible future solutions:• The ability to automate the machine learning process for large scale datasets is limited. Nowadays, there are more sophisticated models or deep learning approaches which achieve very good on large scale datasets and multi-task problems. Our current primitive library and modelling structure is very limited at solving these problems. One of the future solutions could be to incorporate more advanced primitives and to choose to use them when necessary.• Autostacker can be made more efficient with better search algorithms. There are a lot of modern evolutionary algorithms, and some of them are based on the Parallel Hill Climber that we use in this work. We believe that Autostacker could be made faster by incorporating them. We also believe traditional methods and knowledge from statistics and probability will be very helpful to better understand the output of Autostacker, such as by answering questions like: why do was a particular pipeline chosen as one of the final candidate pipelines? In this work, we contribute to automating the machine learning modelling process by proposing Autostacker, a machine learning system with an innovative architecture for automatic modelling and a well-behaved efficient search algorithm. We show how this system works and what the performance of this system is, comparing with human initial trails and related state of art techniques. We also demonstrate the scaling and parallelization ability of our system. In , we automate the machine learning modelling process by providing an efficient, flexible and well-behaved system which provides the potential to be generalized into complicated problems and is able to be integrated with data and feature processing modules.

Represent the short summary for finding the corresponding scientific article (long)

Automate machine learning system with efficient search algorithm and innovative structure to provide better model baselines.

represent the tldr for finding the corresponding scientific article

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the scientific article to find a short TLDR

Question: Surrogate models can be used to accelerate approximate Bayesian computation (ABC). In one such framework the discrepancy between simulated and observed data is modelled with a Gaussian process. So far principled strategies have been proposed only for sequential selection of the simulation locations. To address this limitation, we develop Bayesian optimal design strategies to parallellise the expensive simulations. We also touch the problem of quantifying the uncertainty of the ABC posterior due to the limited budget of simulations. Approximate Bayesian computation is used for Bayesian inference when the analytic form of the likelihood function of a statistical model of interest is either unavailable or too costly to evaluate, but simulating the model is feasible. Unfortunately, many models e.g. in genomics and epidemiology (; ;) and climate science are costly to simulate making sampling-based ABC inference algorithms infeasible. To increase sample-efficiency of ABC, various methods using surrogate models such as neural networks (; ; ;) and Gaussian processes (; ; ; Järvenpää et al., 2018 Järvenpää et al.,, 2019a have been proposed. In one promising surrogate-based ABC framework the discrepancy between the observed and simulated data is modelled with a Gaussian process (GP) (; Järvenpää et al., 2018 Järvenpää et al.,, 2019a . Sequential Bayesian experimental design (or active learning) methods to select the simulation locations so as to maximise the sample-efficiency in this framework were proposed by Järvenpää et al. (2019a). However, one often has access to multiple computers to run some of the simulations in parallel. In this work, motivated by the related problem of batch Bayesian optimisation (; ; ;) and the parallel GP-based method by Järvenpää et al. (2019b) for inference tasks where noisy and potentially expensive log-likelihood evaluations can be obtained, we resolve this limitation by developing principled batch simulation methods which considerably decrease the wall-time needed for ABC inference. The posterior distribution is often summarised for further decision making using e.g. expectation and variance. When the computational resources for ABC inference are limited, it would be important to assess the accuracy of such summaries, but this has not been explicitly acknowledged in earlier work. We devise an approximate numerical method to propagate the uncertainty of the discrepancy, represented by the GP model, to the ing ABC posterior summaries. We call our ing framework as Bayesian ABC in analogy with the related problems of Bayesian quadrature (; ;) and Bayesian optimisation (BO) . Let π(θ) denote the prior density of the (continuous) parameters θ ∈ Θ ⊂ R p of a statistical model of interest and π(x obs | θ) corresponding intractable likelihood function. Standard ABC algorithms such as the ABC rejection sampler target the approximate posterior batch setting where b simulations are simultaneously selected to be computed in parallel at each iteration of our algorithm. Consider a loss function l: D 2 → R + so that l(π ABC, d) quantifies the penalty of reporting d ∈ D as our ABC posterior when the true one is π ABC ∈ D. Given D t, the one-batch-ahead Bayes-optimal selection of the next batch of b, where In Eq. 3, an expectation over b future discrepancy evaluations ∆ * = (∆ * 1, . . ., ∆ * b) at locations θ * needs to be computed assuming ∆ * follows the current GP model. The expectation is taken of the Bayes risk L(Π f Dt∪D *) ing from the nested decision problem of choosing the estimator d, assuming ∆ * are known and merged with current data. Using a loss function based onπ ) 2 dθ between the unnormalised ABC posteriorπ f ABC and its estimatord, then the optimal estimator is the mean in Eq. 11. The ing expected integrated variance (EIV) acquisition function, denoted as where and T is Owen's T function. We use greedy optimisation as is also common in batch BO (see, e.g., ; and the integral over Θ is computed using importance sampling. We can also show that the corresponding L 1 loss function produces the marginal median in Eq. 13 of the Appendix as the optimal estimator. The ing acquisition function, called expected integrated MAD (EIMAD), in addition to some heuristically-motivated batch methods used as baselines (called MAXV, MAXMAD), are developed in Appendix A.2. Pointwise marginal uncertainty of the unnormalised ABC posteriorπ f ABC was used for selecting the simulation locations. However, knowingπ f ABC and its marginal uncertainty in some individual θ-values is not very helpful for understanding the accuracy of the final estimate of π f ABC. Computing the distribution of e.g. ABC posterior expectation or marginals using π f ABC in Eq. 2 is clearly more intuitive. Unfortunately, such computations are difficult due to the nonlinear dependence on the infinite-dimensional quantity f. We propose a simulation-based approach where we combine drawing of GP sample paths and normalised importance sampling. For full details and an illustration, see Appendix A.3 and Fig. 3. We use two real-world simulation models to compare the performance of the sequential and synchronous batch versions of the acquisition methods. As a simple baseline, we consider random points (RAND) drawn from the prior. ABC-MCMC with extensive simulations is used to compute the ground truth ABC posterior. Median total variation distance (TV) over 50 repeated simulations is used to measure the quality of approximations. See Appendix B for further details and C for additional . Lorenz model. This model describes the dynamics of slow weather variables and their dependence on unobserved fast weather variables. The model is represented by a coupled stochastic differential equation which can only be solved numerically ing in an intractable likelihood. The model has two parameters θ = (θ 1, θ 2) which we estimate from timeseries data. for full details and the experimental set-up that we also use, with the exception that we set θ ∼ U(×[0, 0.5]). The are shown in Fig. 1(a). Furthermore, Fig. 1 (b-c) demonstrates the uncertainty quantification of the expectation of the model-based ABC posterior. The effect of batch size is shown in Fig. 2(c). Bacterial infections model. This model describes transmission dynamics of bacterial infections in day care centers and features intractable likelihood function . We estimate the internal, external and co-infection parameters β ∈, Λ ∈ and θ ∈, respectively, using true data and uniform priors. The discrepancy is formed as in. The with all methods are shown in Fig. 2(a) and Fig. 2(b) shows the effect of batch size for the two best methods. Discussion. We obtain reasonable posterior approximations considering the very limited budget of simulations. EIV and EIMAD tend to produce more stable and accurate ABC posterior estimates than MAXV and MAXMAD. Difference in approximation quality between EIV and EIMAD, both based on the same Bayesian decision theoretic framework but different loss functions, was small. In all cases, our batch strategies produced similar evaluation locations as the corresponding sequential methods. This suggests that substantial improvements in wall-time can be obtained when the simulations are costly. The convergence of the uncertainty in the ABC posterior expectation in Fig. 1(b-c) is approximately towards the true ABC posterior expectation due to a slight GP misspecification. The ABC posterior marginals of the bacterial infection model in Appendix C contain some uncertainty after 600 iterations which our approach allows to rigorously quantify. Developing more effective (analytical) methods for computing these uncertainty estimates is an interesting topic for future work. corresponding vector of test point θ. For further details of GP regression, see e.g.. Formulas for the mean, median and variance ofπ f ABC were derived by Järvenpää et al. (2019a) in the case of a zero mean GP prior. It is easy to see that these formulas hold also for our more general GP model. For example, where med denotes the marginal (i.e. elementwise) median. The EIMAD acquisition function, denoted as L m t (θ *) can be shown to be where, similarly as for EIV in Eq. 4, T is Owen's T function and a t is given by Eq. 12. MAD stands for mean absolute deviation (around median). We do not show a detailed derivation of EIV and EIMAD acquisition functions here but only note that these can be obtained using similar computations as in Järvenpää et al. (2019a,b). We consider also a heuristic acquisition function which evaluates where the pointwise uncertainty ofπ f ABC (θ) is highest. Such intuitive strategy is sometimes called as uncertainty sampling and used, e.g., by , Järvenpää et al. (2019a) and. When variance is used as the measure of uncertainty ofπ f ABC (θ), we call the method as MAXV and when MAD is used, we obtain an alternative strategy called analogously MAXMAD. The ing acquisition functions can be computed analytically. Specifically, the variance is computed using Eq. 14. A similar formula can be derived for MAD. Finally, we propose a heuristic approach from BO to parallellise MAXV and MAXMAD strategies: The first point in the batch is chosen as in the sequential case. The further points are iteratively selected as the locations where the expected variance (or MAD), taken with respect to the discrepancy values of the pending points, that is points that have been already chosen to the current batch, is highest. The ing acquisition functions are immediately obtained as the integrands of Eq. 4 and 15. A.3. Uncertainty quantification of the ABC posterior (1 one MCMC sampling from the instrumental density and scales as O(nt 2) so that n can be large. Total cost is O((n +ñ)t 2 +ñ 2 (t + s) +ñ 3 ). We briefly describe some key details of our algorithm and the experiments. Locations for fitting the initial GP model are sampled from the uniform prior in all cases. We take 10 initial points for 2D and 20 for 3D cases. We use b = 0, B ij = 10 2 1 i=j and include basis functions of the form 1, θ i, θ 2 i. The discrepancy ∆ θ is assumed smooth and we use the squared exponential covariance function are given weakly informative priors and their values are obtained using MAP estimation at each iteration. Owen's T function values are computed using a C-implementation of the algorithm by. For simplicity and to ensure meaningful comparisons to ground-truth, we fix ε to certain small predefined values although, in practice, its value is set adaptively (Järvenpää et al., 2019a) or based on pilot runs. We compute the estimate of the unnormalised ABC posterior using the Eq. 11 for MAXV, EIV, RAND and Eq. 13 for MAXMAD, EIMAD. Adaptive MCMC is used to sample from the ing ABC posterior estimates and from instrumental densities needed for IS approximations. TV denotes the median total variation distance between the estimated ABC posterior and the true one (2D) or the average TV between their marginal TV values (3D) computed numerically over 50 repeated runs. Iteration (i.e. number of batches chosen) serves as a proxy for wall-time. The number of simulations i.e. the maximum value of t is fixed in all experiments and the batch methods thus finish earlier. Mahalanobis distance was used as the discrepancy for Lorenz model. The simulation model was run 500 times to estimate the covariance matrix of the six summary statistics by at the true parameter and the the inverse of the covariance matrix was used in the Mahalanobis distance. Of course, such discrepancy is unavailable in practice because the true parameter is unknown and the computational budget limited. However, as the main goal of this paper is to approximate any given ABC posterior with a limited simulation budget, we chose our target ABC posterior this way. defined a discrepancy for the bacterial infections model by summing four L 1 -distances computed between certain individual summaries. For details, see example 7 in. We used the same discrepancy except that we further took square root of their discrepancy function. We obtained a similar ABC posterior as the original article where ABC-PMC algorithm and slightly different approach for comparing the data sets were used. show the 10 initial points and the black dots 100 additional points selected using each acquisition function (the last two batches in the second row are however highlighted by red plus-signs and crosses). The TV value in the title shows the total variation distance between the true and estimated ABC posteriors for each particular case. Fig. 5 and 6 show typical estimated ABC posterior densities of the Lorenz and bacterial infections models, respectively. These are shown to demonstrate the accuracy obtainable with very limited simulations. These particular were obtained with the sequential EIV method using 600 iterations corresponding to 610 simulations (Lorenz model) or 620 simulations (bacterial infections model). Fig. 7 illustrates the ABC posterior uncertainty quantification for the bacterial infections model. Sequential EIV method was used and one typical case is shown. The suggest that while the ABC posterior is well estimated at the last iteration, there is some uncertainty left about its exact shape.

We propose principled batch Bayesian experimental design strategies and a method for uncertainty quantification of the posterior summaries in a Gaussian process surrogate-based approximate Bayesian computation framework.

Represent the short summary for finding the corresponding scientific article (long)

General analysis of sign-based methods (e.g. signSGD) for non-convex optimization, built on intuitive bounds on success probabilities.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Randomly initialized first-order optimization algorithms are the method of choice for solving many high-dimensional nonconvex problems in machine learning, yet general theoretical guarantees cannot rule out convergence to critical points of poor objective value. For some highly structured nonconvex problems however, the success of gradient descent can be understood by studying the geometry of the objective. We study one such problem -- complete orthogonal dictionary learning, and provide converge guarantees for randomly initialized gradient descent to the neighborhood of a global optimum. The ing rates scale as low order polynomials in the dimension even though the objective possesses an exponential number of saddle points. This efficient convergence can be viewed as a consequence of negative curvature normal to the stable manifolds associated with saddle points, and we provide evidence that this feature is shared by other nonconvex problems of importance as well. Many central problems in machine learning and signal processing are most naturally formulated as optimization problems. These problems are often both nonconvex and highdimensional. High dimensionality makes the evaluation of second-order information prohibitively expensive, and thus randomly initialized first-order methods are usually employed instead. This has prompted great interest in recent years in understanding the behavior of gradient descent on nonconvex objectives (18; 14; 17; 11). General analysis of first-and second-order methods on such problems can provide guarantees for convergence to critical points but these may be highly suboptimal, since nonconvex optimization is in general an NP-hard probem BID3. Outside of a convex setting one must assume additional structure in order to make statements about convergence to optimal or high quality solutions. It is a curious fact that for certain classes of problems such as ones that involve sparsification (25; 6) or matrix/tensor recovery (21; 19; 1) first-order methods can be used effectively. Even for some highly nonconvex problems where there is no ground truth available such as the training of neural networks first-order methods converge to high-quality solutions.Dictionary learning is a problem of inferring a sparse representation of data that was originally developed in the neuroscience literature, and has since seen a number of important applications including image denoising, compressive signal acquisition and signal classification (13; 26). In this work we study a formulation of the dictionary learning problem that can be solved efficiently using randomly initialized gradient descent despite possessing a number of saddle points exponential in the dimension. A feature that appears to enable efficient optimization is the existence of sufficient negative curvature in the directions normal to the stable manifolds of all critical points that are not global minima BID0. This property ensures that the regions of the space that feed into small gradient regions under gradient flow do not dominate the parameter space. FIG0 illustrates the value of this property: negative curvature prevents measure from concentrating about the stable manifold. As a consequence randomly initialized gradient methods avoid the "slow region" of around the saddle point. Negative curvature helps gradient descent. Red: "slow region" of small gradient around a saddle point. Green: stable manifold associated with the saddle point. Black: points that flow to the slow region. Left: global negative curvature normal to the stable manifold. Right: positive curvature normal to the stable manifold -randomly initialized gradient descent is more likely to encounter the slow region. The main of this work is a convergence rate for randomly initialized gradient descent for complete orthogonal dictionary learning to the neighborhood of a global minimum of the objective. Our are probabilistic since they rely on initialization in certain regions of the parameter space, yet they allow one to flexibly trade off between the maximal number of iterations in the bound and the probability of the bound holding. While our focus is on dictionary learning, it has been recently shown that for other important nonconvex problems such as phase retrieval BID7 performance guarantees for randomly initialized gradient descent can be obtained as well. In fact, in Appendix C we show that negative curvature normal to the stable manifolds of saddle points (illustrated in FIG0) is also a feature of the population objective of generalized phase retrieval, and can be used to obtain an efficient convergence rate. Easy nonconvex problems. There are two basic impediments to solving nonconvex problems globally: (i) spurious local minimizers, and (ii) flat saddle points, which can cause methods to stagnate in the vicinity of critical points that are not minimizers. The latter difficulty has motivated the study of strict saddle functions (36; BID13, which have the property that at every point in the domain of optimization, there is a large gradient, a direction of strict negative curvature, or the function is strongly convex. By leveraging this curvature information, it is possible to escape saddle points and obtain a local minimizer in polynomial time.2 Perhaps more surprisingly, many known strict saddle functions also have the property that every local minimizer is global; for these problems, this implies that efficient methods find global solutions. Examples of problems with this property include variants of sparse dictionary learning, phase retrieval, tensor decomposition BID13, community detection and phase synchronization BID4.Minimizing strict saddle functions. Strict saddle functions have the property that at every saddle point there is a direction of strict negative curvature. A natural approach to escape such saddle points is to use second order methods (e.g., trust region BID8 or curvilinear search BID14) that explicitly leverage curvature information. Alternatively, one can attempt to escape saddle points using first order information only. However, some care is needed: canonical first order methods such as gradient descent will not obtain minimizers if initialized at a saddle point (or at a point that flows to one) -at any critical point, gradient descent simply stops. A natural remedy is to randomly perturb the iterate whenever needed. A line of recent works shows that noisy gradient methods of this form efficiently optimize strict saddle functions (24; 12; 20). For example, obtains rates on strict saddle functions that match the optimal rates for smooth convex programs up to a polylogarithmic dependence on dimension. Randomly initialized gradient descent? The aforementioned are broad, and nearly optimal. Nevertheless, important questions about the behavior of first order methods for nonconvex optimization remain unanswered. For example: in every one of the aforemented benign nonconvex optimization problems, randomly initialized gradient descent rapidly obtains a minimizer. This may seem unsurprising: general considerations indicate that the stable manifolds associated with non-minimizing critical points have measure zero, this implies that a variety of small-stepping first order methods converge to minimizers in the large-time limit. However, it is not difficult to construct strict saddle problems that are not amenable to efficient optimization by randomly initialized gradient descent -see BID11 for an example. This contrast between the excellent empirical performance of randomly initialized first order methods and worst case examples suggests that there are important geometric and/or topological properties of "easy nonconvex problems" that are not captured by the strict saddle hypothesis. Hence, the motivation of this paper is twofold: (i) to provide theoretical corroboration (in certain specific situations) for what is arguably the simplest, most natural, and most widely used first order method, and (ii) to contribute to the ongoing effort to identify conditions which make nonconvex problems amenable to efficient optimization. Suppose we are given data matrix Y = y 1,... y p ∈ R n×p. The dictionary learning problem asks us to find a concise representation of the data BID12, of the form Y ≈ AX, where X is a sparse matrix. In the complete, orthogonal dictionary learning problem, we restrict the matrix A to have orthonormal columns (A ∈ O(n)). This variation of dictionary learning is useful for finding concise representations of small datasets (e.g., patches from a single image, in MRI FORMULA2).To analyze the behavior of dictionary learning algorithms theoretically, it useful to posit that Y = A 0 X 0 for some true dictionary A 0 ∈ O(n) and sparse coefficient matrix X 0 ∈ R n×p, and ask whether a given algorithm recovers the pair (A 0, X 0). BID3 In this work, we further assume that the sparse matrix X 0 is random, with entries i.i.d. Bernoulli-Gaussian 5. For simplicity, we will let A 0 = I; our arguments extend directly to general A 0 via the simple change of variables q → A * 0 q. showed that under mild conditions, the complete dictionary recovery problem can be reduced to the geometric problem of finding a sparse vector in a linear subspace. Notice that because A 0 is orthogonal, row(Y) = row(X 0). Because X 0 is a sparse random matrix, the rows of X 0 are sparse vectors. Under mild conditions, they are the sparsest vectors in the row space of Y, and hence can be recovered by solving the conceptual optimization problem min q DISPLAYFORM0 This is not a well-structured optimization problem: the objective is discontinuous, and the constraint set is open. A natural remedy is to replace the 0 norm with a smooth sparsity surrogate, and to break the scale ambiguity by constraining q to the sphere, giving DISPLAYFORM1 Here, we choose h µ (t) = µ log(cosh(t/µ)) as a smooth sparsity surrogate. This objective was analyzed in, which showed that (i) although this optimization problem is nonconvex, when the data are sufficiently large, with high probability every local optimizer is near a signed column of the true dictionary A 0, (ii) every other critical point has a direction of strict negative curvature, and (iii) as a consequence, a second-order Riemannian trust region method efficiently recovers a column of A 0. BID5 The Riemannian trust region method is of mostly theoretical interest: it solves complicated (albeit polynomial time) subproblems that involve the Hessian of f DL. Note the similarity to the dictionary learning objective. Right: The objective for complete orthogonal dictionary learning (discussed in section 6) for n = 3.In practice, simple iterative methods, including randomly initialized gradient descent are also observed to rapidly obtain high-quality solutions. In the sequel, we will give a geometric explanation for this phenomenon, and bound the rate of convergence of randomly initialized gradient descent to the neighborhood of a column of A 0. Our analysis of f DL is probabilistic in nature: it argues that with high probability in the sparse matrix X 0, randomly initialized gradient descent rapidly produces a minimizer. To isolate more clearly the key intuitions behind this analysis, we first analyze the simpler separable objective DISPLAYFORM2 Figure 2 plots both f Sep and f DL as functions over the sphere. Notice that many of the key geometric features in f DL are present in f Sep; indeed, f Sep can be seen as an "ultrasparse" version of f DL in which the columns of the true sparse matrix X 0 are taken to have only one nonzero entry. A virtue of this model function is that its critical points and their stable manifolds have simple closed form expressions (see Lemma 1). Our problems of interest have the form DISPLAYFORM0 where f: R n → R is a smooth function. We let ∇f (q) and ∇ 2 f (q) denote the Euclidean gradient and hessian (over R n), and let grad [f] (q) and Hess [f] (q) denote their Riemannian counterparts (over S n−1). We will obtain for Riemannian gradient descent defined by the update DISPLAYFORM1 for some step size η > 0, where exp q: T q S n−1 → S n−1 is the exponential map. The Riemannian gradient on the sphere is given by grad[f] (q) = (I − qq *)∇f (q).We let A denote the set of critical points of f over S n−1 -these are the pointsq s.t. grad [f] (q) = 0. We letȂ denote the set of local minimizers, and "A its complement. Both f Sep and f DL are Morse functions on S n−1, 7 we can assign an index α to everyq ∈ A, which is the number of negative eigenvalues of Hess [f] (q).Our goal is to understand when gradient descent efficiently converges to a local minimizer. In the small-step limit, gradient descent follows gradient flow lines γ: R → M, which are solution curves of the ordinary differential equatioṅ DISPLAYFORM2 To each critical point α ∈ A of index λ, there is an associated stable manifold of dimension dim(M) − λ, which is roughly speaking, the set of points that flow to α under gradient flow: DISPLAYFORM3 7 Strictly speaking, fDL is Morse with high probability, due to of. Negative curvature and efficient gradient descent. The union of the light blue, orange and yellow sets is the set C. In the light blue region, there is negative curvature normal to ∂C, while in the orange region the gradient norm is large, as illustrated by the arrows. There is a single global minimizer in the yellow region. For the separable objective, the stable manifolds of the saddles and maximizers all lie on ∂C (the black circles denote the critical points, which are either maximizers " ", saddles " ", or minimizers " "). The red dots denote ∂C ζ with ζ = 0.2.Our analysis uses the following convenient coordinate chart DISPLAYFORM0 where w ∈ B 1. We also define two useful sets: DISPLAYFORM1 Since the problems considered here are symmetric with respect to a signed permutation of the coordinates we can consider a certain C and the will hold for the other symmetric sections as well. We will show that at every point in C aside from a neighborhood of a global minimizer for the separable objective (or a solution to the dictionary problem that may only be a local minimizer), there is either a large gradient component in the direction of the minimizer or negative curvature in a direction normal to ∂C. For the case of the separable objective, one can show that the stable manifolds of the saddles lie on this boundary, and hence this curvature is normal to the stable manifolds of the saddles and allows rapid progress away from small gradient regions and towards a global minimizer BID7. These regions are depicted in Figure 3.In the sequel, we will make the above ideas precise for the two specific nonconvex optimization problems discussed in Section 3 and use this to obtain a convergence rate to a neighborhood of a global minimizer. Our analysis are specific to these problems. However, as we will describe in more detail later, they hinge on important geometric characteristics of these problems which make them amenable to efficient optimization, which may obtain in much broader classes of problems. In this section, we study the behavior of randomly initialized gradient descent on the separable function f Sep. We begin by characterizing the critical points:Lemma 1 (Critical points of f Sep). The critical points of the separable problem are DISPLAYFORM0 For every α ∈ A and corresponding a(α), for µ < c √ n log n the stable manifold of α takes the form DISPLAYFORM1 where c > 0 is a numerical constant. Proof. Please see Appendix ABy inspecting the dimension of the stable manifolds, it is easy to verify that that there are 2n global minimizers at the 1-sparse vectors on the sphere ± e i, 2 n maximizers at the least sparse vectors and an exponential number of saddle points of intermediate sparsity. This is because the dimension of W s (α) is simply the dimension of b in 6, and it follows directly from the stable manifold theorem that only minimizers will have a stable manifold of dimension n − 1. The objective thus possesses no spurious local minimizers. When referring to critical points and stable manifolds from now on we refer only to those that are contained in C or on its boundary. It is evident from Lemma 1 that the critical points in "A all lie on ∂C and that α∈ " A W s (α) = ∂C, and there is a minimizer at its center given by q = e n. We now turn to making precise the notion that negative curvature normal to stable manifolds of saddle points enables gradient descent to rapidly exit small gradient regions. We do this by defining vector fields u (i) (q), i ∈ [n − 1] such that each field is normal to a continuous piece of ∂C ζ and points outwards relative to C ζ defined in 4. By showing that the Riemannian gradient projected in this direction is positive and proportional to ζ, we are then able to show that gradient descent acts to increase ζ(q(w)) = qn w ∞ − 1 geometrically. This corresponds to the behavior illustrated in the light blue region in Figure 3. DISPLAYFORM0 DISPLAYFORM1 where c > 0 is a numerical constant. Proof. Please see Appendix A.Since we will use this property of the gradient in C ζ to derive a convergence rate, we will be interested in bounding the probability that gradient descent initialized randomly with respect to a uniform measure on the sphere is initialized in C ζ. This will require bounding the volume of this set, which is done in the following lemma: DISPLAYFORM2 Proof. Please see Appendix D.3. Using the above, one can obtain the following convergence rate:Theorem 1 (Gradient descent convergence rate for separable function). For any 0 < ζ 0 < 1, r > µ log on the separable objective with µ < c2 √ n log n, enters an L ∞ ball of radius r around a global minimizer in T < C η √ n r 2 + log 1 ζ 0 iterations with probability DISPLAYFORM0 Proof. Please see Appendix A.We have thus obtained a convergence rate for gradient descent that relies on the negative curvature around the stable manifolds of the saddles to rapidly move from these regions of the space towards the vicinity of a global minimizer. This is evinced by the logarithmic dependence of the rate on ζ. As was shown for orthogonal dictionary learning in, we also expect a linear convergence rate due to strong convexity in the neighborhood of a minimizer, but do not take this into account in the current analysis. The proofs in this section will be along the same lines as those of Section 5. While we will not describe the positions of the critical points explicitly, the similarity between this objective and the separable function motivates a similar argument. It will be shown that initialization in some C ζ will guarantee that Riemannian gradient descent makes uniform progress in function value until reaching the neighborhood of a global minimizer. We will first consider the population objective which corresponds to the infinite data limit DISPLAYFORM0 and then bounding the finite sample size fluctuations of the relevant quantities. We begin with a lemma analogous to Lemma 2:Lemma 4 (Dictionary learning population gradient). For w ∈ C ζ, r < |w i |, µ < c 1 r DISPLAYFORM1 √ ζ the dictionary learning population objective 8 obeys DISPLAYFORM2 where c θ depends only on θ, c 1 is a positive numerical constant and u (i) is defined in 7.Proof. Please see Appendix BUsing this , we obtain the desired convergence rate for the population objective, presented in Lemma 11 in Appendix B. After accounting for finite sample size fluctuations in the gradient, one obtains a rate of convergence to the neighborhood of a solution (which is some signed basis vector due to our choice A 0 = I) Theorem 2 (Gradient descent convergence rate for dictionary learning). DISPLAYFORM3, Riemannian gradient descent with step size η < c5θs n log np on the dictionary learning objective 1 with µ < c6 √ ζ0 DISPLAYFORM4, enters a ball of radius c 3 s from a target solution in DISPLAYFORM5 iterations with probability DISPLAYFORM6 where y = DISPLAYFORM7, P y is given in Lemma 10 and c i, C i are positive constants. Proof. Please see Appendix BThe two terms in the rate correspond to an initial geometric increase in the distance from the set containing the small gradient regions around saddle points, followed by convergence to the vicinity of a minimizer in a region where the gradient norm is large. The latter is based on on the geometry of this objective provided in. The above analysis suggests that second-order properties -namely negative curvature normal to the stable manifolds of saddle points -play an important role in the success of randomly initialized gradient descent in the solution of complete orthogonal dictionary learning. This was done by furnishing a convergence rate guarantee that holds when the random initialization is not in regions that feed into small gradient regions around saddle points, and bounding the probability of such an initialization. In Appendix C we provide an additional example of a nonconvex problem that for which an efficient rate can be obtained based on an analysis that relies on negative curvature normal to stable manifolds of saddles -generalized phase retrieval. An interesting direction of further work is to more precisely characterize the class of functions that share this feature. The effect of curvature can be seen in the dependence of the maximal number of iterations T on the parameter ζ 0. This parameter controlled the volume of regions where initialization would lead to slow progress and the failure probability of the bound 1 − P was linear in ζ 0, while T depended logarithmically on ζ 0. This logarithmic dependence is due to a geometric increase in the distance from the stable manifolds of the saddles during gradient descent, which is a consequence of negative curvature. Note that the choice of ζ 0 allows one to flexibly trade off between T and 1 − P. By decreasing ζ 0, the bound holds with higher probability, at the price of an increase in T. This is because the volume of acceptable initializations now contains regions of smaller minimal gradient norm. In a sense, the is an extrapolation of works such as that analyze the ζ 0 = 0 case to finite ζ 0.Our analysis uses precise knowledge of the location of the stable manifolds of saddle points. For less symmetric problems, including variants of sparse blind deconvolution and overcomplete tensor decomposition, there is no closed form expression for the stable manifolds. However, it is still possible to coarsely localize them in regions containing negative curvature. Understanding the implications of this geometric structure for randomly initialized first-order methods is an important direction for future work. One may hope that studying simple model problems and identifying structures (here, negative curvature orthogonal to the stable manifold) that enable efficient optimization will inspire approaches to broader classes of problems. One problem of obvious interest is the training of deep neural networks for classification, which shares certain high-level features with the problems discussed in this paper. The objective is also highly nonconvex and is conjectured to contain a proliferation of saddle points BID10, yet these appear to be avoided by first-order methods BID15 for reasons that are still quite poorly understood beyond the two-layer case. Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. DISPLAYFORM0. Thus critical points are ones where either tanh(q µ) = 0 (which cannot happen on S n−1) or tanh(q µ) is in the nullspace of (I − qq *), which implies tanh(q µ) = cq for some constant b. The equation tanh(x µ) = bx has either a single solution at the origin or 3 solutions at {0, ±r(b)} for some r(b). Since this equation must be solves simultaneously for every element of q, we obtain ∀i ∈ [n]: q i ∈ {0, ±r(b)}. To obtain solutions on the sphere, one then uses the freedom we have in choosing b (and thus r(b)) such that q = 1. The ing set of critical points is thus DISPLAYFORM1 To prove the form of the stable manifolds, we first show that for q i such that |q i | = q ∞ and any q j such that |q j | + ∆ = |q i | and sufficiently small ∆ > 0, we have DISPLAYFORM2 For ease of notation we now assume q i, q j > 0 and hence ∆ = q i − q j, otherwise the argument can be repeated exactly with absolute values instead. The above inequality can then be written as DISPLAYFORM3 If we now define DISPLAYFORM4 where the O(∆ 2) term is bounded. Defining a vector r ∈ R n by DISPLAYFORM5 we have r 2 = 1. Since tanh(x) is concave for x > 0, and |r i | ≤ 1, we find DISPLAYFORM6 From DISPLAYFORM7 and thus q j ≥ 1 √ n − ∆. Using this inequality and properties of the hyperbolic secant we obtain DISPLAYFORM8 and plugging in µ = c √ n log n for some c < 1 DISPLAYFORM9 log n + log log n + log 4).We can bound this quantity by a constant, say h 2 ≤ 1 2, by requiring DISPLAYFORM10 ) log n + log log n ≤ − log 8and for and c < 1, using − log n + log log n < 0 we have DISPLAYFORM11 Since ∆ can be taken arbitrarily small, it is clear that c can be chosen in an n-independent manner such that A ≤ − log 8. We then find DISPLAYFORM12 since this inequality is strict, ∆ can be chosen small enough such that O(∆ 2) < ∆(h 1 − h 2) and hence h > 0, proving 9.It follows that under negative gradient flow, a point with |q j | < ||q|| ∞ cannot flow to a point q such that |q j | = ||q || ∞. From the form of the critical points, for every such j, q must thus flow to a point such that q j = 0 (the value of the j coordinate cannot pass through 0 to a point where |q j | = ||q || ∞ since from smoothness of the objective this would require passing some q with q j = 0, at which point grad [f Sep] (q) j = 0).As for the maximal magnitude coordinates, if there is more than one coordinate satisfying |q i1 | = |q i2 | = q ∞, it is clear from symmetry that at any subsequent point q along the gradient flow line q i1 = q i2. These coordinates cannot change sign since from the smoothness of the objective this would require that they pass through a point where they have magnitude smaller than 1/ √ n, at which point some other coordinate must have a larger magnitude (in order not to violate the spherical constraint), contradicting the above for non-maximal elements. It follows that the sign pattern of these elements is preserved during the flow. Thus there is a single critical point to which any q can flow, and this is given by setting all the coordinates with |q j | < q ∞ to 0 and multiplying the remaining coordinates by a positive constant to ensure the ing vector is on S n. Denoting this critical point by α, there is a vector b such that q = P S n−1 [a(α) + b] and supp(a(α)) ∩ supp(b) = ∅, b ∞ < 1 with the form of a(α) given by 5. The collection of all such points defines the stable manifold of α. Proof of Lemma 2: (Separable objective gradient projection). i) We consider the sign(w i) = 1 case; the sign(w i) = −1 case follows directly. Recalling that DISPLAYFORM13 qn, we first prove DISPLAYFORM14 for some c > 0 whose form will be determined later. The inequality clearly holds for w i = q n.To DISPLAYFORM15 verify that it holds for smaller values of w i as well, we now show that ∂ ∂w i tanh w i µ − tanh q n µ w i q n − c(q n − w i) < 0 which will ensure that it holds for all w i. We define s 2 = 1 − ||w|| 2 + w 2 i and denote q n = s 2 − w 2 i to extract the w i dependence, givingWhere in the last inequality we used properties of the sech function and q n ≥ w i. We thus want to show DISPLAYFORM16 and it follows that 10 holds. For µ < 1 BID15 we are guaranteed that c > 0.From examining the RHS of 10 (and plugging in q n = s 2 − w 2 i) we see that any lower bound on the gradient of an element w j applies also to any element |w i | ≤ |w j |. Since for |w j | = ||w|| ∞ we have q n − w j = w j ζ, for every log(1 µ)µ ≤ w i we obtain the bound DISPLAYFORM17 Proof of Theorem 1: (Gradient descent convergence rate for separable function).We obtain a convergence rate by first bounding the number of iterations of Riemannian gradient descent in C ζ0 \C 1, and then considering DISPLAYFORM18. Choosing c 2 so that µ < 1 2, we can apply Lemma 2, and for u defined in 7, we thus have DISPLAYFORM19 Since from Lemma 7 the Riemannian gradient norm is bounded by √ n, we can choose c 1, c 2 such that µ log(DISPLAYFORM20 . This choice of η then satisfies the conditions of Lemma 17 with r = µ log( DISPLAYFORM21, M = √ n, which gives that after a gradient step DISPLAYFORM22 for some suitably chosenc > 0. If we now define by w (t) the t-th iterate of Riemannian gradient descent and DISPLAYFORM23 and the number of iterations required to exit C ζ0 \C 1 is DISPLAYFORM24 To bound the remaining iterations, we use Lemma 2 to obtain that for every w ∈ C ζ0 \B ∞ r, DISPLAYFORM25 where we have used ||u DISPLAYFORM26 We thus have DISPLAYFORM27 Choosing DISPLAYFORM28 where L is the gradient Lipschitz constant of f s, from Lemma 5 we obtain DISPLAYFORM29 According to Lemma B, L = 1/µ and thus the above holds if we demand η < µ 2. Combining 12 and 13 gives DISPLAYFORM30.To obtain the final rate, we use in g(w 0) − g * ≤ √ n andcη < 1 ⇒ 1 log(1+cη) <C cη for somẽ C > 0. Thus one can choose C > 0 such that DISPLAYFORM31 From Lemma 1 the ball B ∞ r contains a global minimizer of the objective, located at the origin. The probability of initializing in Ȃ C ζ0 is simply given from Lemma 3 and by summing over the 2n possible choices of C ζ0, one for each global minimizer (corresponding to a single signed basis vector)., where L is a lipschitz constant for ∇f (q), one has DISPLAYFORM0 Proof. Just as in the euclidean setting, we can obtain a lower bound on progress in function values of iterates of the Riemannian gradient descent algorithm from a lower bound on the Riemannian gradient. Consider f: S n−1 → R, which has L-lipschitz gradient. Let q k denote the current iterate of Riemannian gradient descent, and let t k > 0 denote the step size. Then we can form the Taylor approximation to f • Exp q k (v) at 0 q k: DISPLAYFORM1 where the matrix norm is the operator norm on R n×n. Using the gradient-lipschitz property of f, we readily compute DISPLAYFORM0 since ∇f = 0 and q k ∈ S n−1. We thus have DISPLAYFORM1 If we put v = −t k grad[f](q k) and write q k+1 = Exp q k (−t k grad [f] (q k)), the previous expression becomes DISPLAYFORM2. Thus progress in objective value is guaranteed by lower-bounding the Riemannian gradient. As in the euclidean setting, summing the previous expression over iterations k now yields DISPLAYFORM3 Plugging in a constant step size gives the desired . Lemma 6 (Lipschitz constant of ∇f). For any x 1, x 2 ∈ R n, it holds DISPLAYFORM4 Proof. It will be enough to study a single coordinate function of ∇f. Using a derivative given in section D.1, we have for DISPLAYFORM5 A bound on the magnitude of the derivative of this smooth function implies a lipschitz constant for x → tanh(x/µ). To find the bound, we differentiate again and find the critical points of the function. We have, using the chain rule, d dx DISPLAYFORM6 (e x/µ + e −x/µ) 3. The denominator of this final expression vanishes nowhere. Hence, the only critical point satisfies x/µ = −x/µ, which implies x = 0. Therefore it holds DISPLAYFORM7 which shows that tanh(x/µ) is (1/µ)-lipschitz. Now let x 1 and x 2 be any two points of R n. Then one has DISPLAYFORM8 completing the proof. Proof of Lemma 4:(Dictionary learning population gradient). For simplicity we consider the case sign(w i) = 1. The converse follows by a similar argument. We have DISPLAYFORM0 Following the notation of, we write x j = b j v j where b j ∼ Bern(θ), v j ∼ N and denote the vectors of these variables by J, v respectively. Defining DISPLAYFORM1 and similarly the second term in 15 is, with DISPLAYFORM0 We already have a lower bound in Lemma 20 of that we can use for the second term, so we need an upper bound for the first term. Following from p. 865, we define DISPLAYFORM0, and defining DISPLAYFORM1 Where b k = (−β) k (k + 1). Using B.3 from Lemma 40 in we have DISPLAYFORM2 Where Φ c (x) is the complementary Gaussian CDF (The exchange of summation and expectation is justified since Y > 0 implies Z ∈, see proof of Lemma 18 in for details). Using the following bounds DISPLAYFORM3 2 /2 by applying the upper (lower) bound to the even (odd) terms in the sum, and then adding a non-negative quantity, we obtain DISPLAYFORM4 and using Lemma 17 in ) and taking T → ∞ so that β → 1 we have DISPLAYFORM5 DISPLAYFORM6 giving the upper bound DISPLAYFORM7 while the lower bound (Lemma 20 in) is DISPLAYFORM8 After conditioning on J \{n, i} the variables X + q n v n, X + q i v i are Gaussian. We can thus plug the bounds into 16 to obtain DISPLAYFORM0 the term in the expectation is positive since q n > ||w|| ∞ (1 + ζ) > w i giving DISPLAYFORM1 To extract the ζ dependence we plug in q n > w i (1 + ζ) and develop to first order in ζ (since the ing function of ζ is convex) giving DISPLAYFORM2 Given some ζ and r such that w i > r, if we now choose µ such that µ < Lemma 8 (Point-wise concentration of projected gradient). For u (i) defined in 7, the gradient of the objective 1 obeys DISPLAYFORM3 Proof of Lemma 8: (Point-wise concentration of projected gradient). If we denote by x i a column of the data matrix with entries x i j ∼ BG(θ), we have DISPLAYFORM4. Since tanh(x) is bounded by 1, DISPLAYFORM5 Invoking Lemma 21 from and u 2 = 1 + DISPLAYFORM6 and using Lemma 36 in with R = √ 2, σ = √ 2 we have DISPLAYFORM7 Lemma 9 (Projection Lipschitz Constant). The Lipschitz constant for DISPLAYFORM8 Proof of Lemma 9: (Projection Lipschitz Constant). We have DISPLAYFORM9 where we have defined DISPLAYFORM10 We also use the fact that tanh is bounded by 1 and s(w) is bounded by X ∞. We can then use Lemma 23 in to obtain DISPLAYFORM11 Lemma 10 (Uniformized gradient fluctuations). For all w ∈ C ζ, i ∈ [n], with probability P > P y we have DISPLAYFORM12 where DISPLAYFORM13 Proof of Lemma 10:(Uniformized gradient fluctuations). For X ∈ R n×p with i.i.d. BG(θ) entries, we define the event E ∞ ≡ {1 ≤ X ∞ ≤ 4 log(np)}. We have DISPLAYFORM14 For any ε ∈ we can construct an ε-net N for C ζ \B 2 1/20 DISPLAYFORM15. If we choose ε = y(θ,ζ) DISPLAYFORM16 We then denote by E g the event DISPLAYFORM17 2 in the of Lemma 8 gives that for all w ∈ C ζ, i ∈ [n], DISPLAYFORM18 py(θ, ζ) Proof of Lemma 11: (Gradient descent convergence rate for dictionary learning -population). The rate will be obtained by splitting C ζ0 into three regions. We consider convergence to B 2 s since this set contains a global minimizer. Note that the balls in the proof are defined with respect to w. DISPLAYFORM19 The analysis in this region is completely analogous to that in the first part of the proof of Lemma 1. For every point in this set we have DISPLAYFORM20. From Lemma 16 we know that, since for every point in this region r 3 ζ < 1, we have DISPLAYFORM21 DISPLAYFORM22 r = z(r, ζ) and we thus demand µ < √ ζ0 DISPLAYFORM23 and obtain from Lemma 4 that for |w i | > r DISPLAYFORM24. We now require η <, M = √ θn (since the maximal norm of the Riemannian gradient is √ θn from Lemma 12), obtaining that at every iteration in this region ζ ≥ ζ 1 + √ nc DL 2(8000(n − 1)) 3/2 η and the maximal number of iterations required to obtain ζ > 8 and exit this region is given by DISPLAYFORM25 According to Proposition 7 in, which we can apply since s ≥ DISPLAYFORM26. Defining h(q) = w 2 2, and denoting by q an update of Riemannian gradient descent with step size η, we have (using a Lagrange remainder term) DISPLAYFORM27 where in the last line we used q = cos(gη)q − sin(gη) DISPLAYFORM28 we obtain (using 18) DISPLAYFORM29 and thus choosing η < DISPLAYFORM30 we find DISPLAYFORM31 Under review as a conference paper at ICLR 2019 and in our region of interest w 2 < w 2 −csθη for somec > 0 and thus summing over iterations, we obtain for someC 2 > 0 DISPLAYFORM32 From Lemma 12, M = √ θn and thus with a suitably chosen c 2 > 0, η < c2s n satisfies the above requirement on η as well as the previous requirements, since θ < 1. Combining these gives, we find that when initializing in C ζ0, the maximal number of iterations required for Riemannian gradient descent to enter B 2 s is DISPLAYFORM0 for some suitably chosen C 1, where t 1, t 2 are given in 17,19. The probability of such an initialization is given by the probability of initializing in one of the 2n possible choices of C ζ, which is bounded in Lemma 3.Once w ∈ B 2 s, the distance in R n−1 between w and a solution to the problem (which is a signed basis vector, given by the point w = 0 or an analog on a different symmetric section of the sphere) is no larger than s, which in turn implies that the Riemannian distance between ϕ(w) and a solution is no larger than c 3 s for some c 3 > 0. We note that the conditions on µ can be satisfied by requiring µ < DISPLAYFORM1 where X is the data matrix with i.i.d. BG(θ) entries. Proof. Denoting x ≡ (x, x n) we have DISPLAYFORM2 and using Jensen's inequality, convexity of the L 2 norm and the triangle inequality to obtain DISPLAYFORM3 Similarly, in the finite sample size case one obtains DISPLAYFORM4 Proof of Theorem 2: (Gradient descent convergence rate for dictionary learning). The proof will follow exactly that of Lemma 11, with the finite sample size fluctuations decreasing the guaranteed change in ζ or ||w|| at every iteration (for the initial and final stages respectively) which will adversely affect the bounds. DISPLAYFORM5 To control the fluctuations in the gradient projection, we choose DISPLAYFORM6 which can be satisfied by choosing y(θ, ζ 0) = DISPLAYFORM7 for an appropriate c 7 > 0. According to Lemma 10, with probability greater than P y we then have DISPLAYFORM8 With the same condition on µ as in Lemma 11, combined with the uniformized bound on finite sample fluctuations, we have that at every point in this set DISPLAYFORM9. According to Lemma 12 the Riemannian gradient norm is bounded by M = √ n X ∞. Choosing r, b as in Lemma 11, we require η < for some chosenc > 0. We then obtain DISPLAYFORM10 for a suitably chosen C 2 > 0. The final bound on the rate is obtained by summing over the terms for the three regions as in the population case, and convergence is again to a distance of less than c 3 s from a local minimizer. The probability of achieving this rate is obtained by taking a union bound over the probability of initialization in C ζ0 (given in Lemma 3) and the probabilities of the bounds on the gradient fluctuations holding (from Lemma 10 and FORMULA7). Note that the fluctuation bound events imply by construction the event E ∞ = {1 ≤ X ∞ ≤ 4 log(np)} hence we can replace X ∞ in the conditions on η above by 4 log(np). The conditions on η, µ can be satisfied by requiring η < c5θs n log np, µ < c6 √ ζ0 n 5/4 for suitably chosen c 5, c 6 > 0. The bound on the number of iterations can be simplified to the form in the theorem statement as in the population case. We show below that negative curvature normal to stable manifolds of saddle points in strict saddle functions is a feature that is found not only in dictionary learning, and can be used to obtain efficient convergence rates for other nonconvex problems as well, by presenting an analysis of generalized phase retrieval that is along similar lines to the dictionary learning analysis. We stress that this contribution is not novel since a more thorough analysis was carried out by BID7. The ing rates are also suboptimal, and pertain only to the population objective. Generalized phase retrieval is the problem of recovering a vector x ∈ C n given a set of magnitudes of projections y k = |x * a k | onto a known set of vectors a k ∈ C n. It arises in numerous domains including microscopy, acoustics BID1, and quantum mechanics (see for a review). Clearly x can only be recovered up to a global phase. We consider the setting where the elements of every a k are i. DISPLAYFORM0 We analyze the least squares formulation of the problem given by DISPLAYFORM1 Taking the expectation (large p limit) of the above objective and organizing its derivatives using Wirtinger calculus FORMULA2, we obtain DISPLAYFORM2 For the remainder of this section, we analyze this objective, leaving the consideration of finite sample size effects to future work. In it was shown that aside from the manifold of minimȃ DISPLAYFORM0 the only critical points of E[f] are a maximum at z = 0 and a manifold of saddle points given by DISPLAYFORM1 where W ≡ {z|z * x = 0}. We decompose z as DISPLAYFORM2 where ζ > 0, w ∈ W. This gives z 2 = w 2 + ζ 2. The choice of w, ζ, φ is unique up to factors of 2π in φ, as can be seen by taking an inner product with x. Since the gradient decomposes as follows: DISPLAYFORM3 the directions e iφ x x, w w are unaffected by gradient descent and thus the problem reduces to a two-dimensional one in the space (ζ, w). Note also that the objective for this twodimensional problem is a Morse function, despite the fact that in the original space there was a manifold of saddle points. It is also clear from this decomposition of the gradient that the stable manifolds of the saddles are precisely the set W.It is evident from 24 that the dispersive property does not hold globally in this case. For z / ∈ B ||x|| we see that gradient descent will cause ζ to decrease, implying positive curvature normal to the stable manifolds of the saddles. This is a consequence of the global geometry of the objective. Despite this, in the region of the space that is more "interesting", namely B ||x||, we do observe the dispersive property, and can use it to obtain a convergence rate for gradient descent. We define a set that contains the regions that feeds into small gradient regions around saddle points within B ||x|| by DISPLAYFORM4 We will show that, as in the case of orthogonal dictionary learning, we can both bound the probability of initializing in (a subset of) the complement of Q ζ0 and obtain a rate for convergence of gradient descent in the case of such an initialization. plane. The full red curves are the boundaries between the sets S 1, S 2, S 3, S 4 used in the analysis. The dashed red line is the boundary of the set Q ζ0 that contains small gradient regions around critical points that are not minima. The maximizer and saddle point are shown in dark green, while the minimizer is in pink. These are used to find the change in ζ, w at every iteration in each region: DISPLAYFORM5 We now show that gradient descent initialized in S 1 \Q ζ0 cannot exit ∪ 2 we are guaranteed from Lemma 13 that at every iteration ζ ≥ ζ 0. Thus the region with ζ < ζ 0 can only be entered if gradient descent is initialized in it. It follows that initialization in S 1 \Q ζ0 rules out entering Q ζ0 at any future iteration of gradient descent. Since this guarantees that regions that feed into small gradient regions are avoided, an efficient convergence rate can again be obtained. Theorem 3 (Gradient descent convergence rate for generalized phase retrieval). Gradient descent on 22 with step size η < DISPLAYFORM0 iterations with probability DISPLAYFORM1 ii) Since only a step from S 4 can decrease ζ, we have that for the initial point z 2 > x 2.Combined with DISPLAYFORM2 this gives DISPLAYFORM3 and using the lower bound (1 − 2η x 2 c)ζ ≤ ζ we obtain DISPLAYFORM4 where in the last inequality we used c < DISPLAYFORM5 Proof of Lemma 14. We use the fact that for the next iterate we have DISPLAYFORM6 We will also repeatedly use η < DISPLAYFORM7 which is a shown in Lemma 13. DISPLAYFORM8 We want to show DISPLAYFORM9 (1 + c) x 2.1) We have z ∈ S 3 ⇒ z 2 = (1 − ε) x 2 for some ε ≤ c and using 28 we must show DISPLAYFORM10 Proof of Theorem 3: (Gradient descent convergence rate for generalized phase retrieval). We now bound the number of iterations that gradient descent, after random initialization in S 1, requires to reach a point where one of the convergence criteria detailed in Lemma 15 is fulfilled. From Lemma 14, we know that after initialization in S 1 we need to consider only the set DISPLAYFORM11 S i. The number of iterations in each set will be determined by the bounds on the change in ζ, ||w|| detailed in 27. Assuming we initialize with some ζ = ζ 0. Then the maximal number of iterations in this region is DISPLAYFORM0 since after this many iterations DISPLAYFORM1 The only concern is that after an iteration in S 3 ∪ S 4 the next iteration might be in S 2.To account for this situation, we find the maximal number of iterations required to reach S 3 ∪ S 4 again. This is obtained from the bound on ζ in Lemma 13.Using this , and the fact that for every iteration in S 2 we are guaranteed ζ ≥ (1 + 2η x 2 c)ζ the number of iterations required to reach S 3 ∪ S 4 again is given by DISPLAYFORM2 The final rate to convergence is DISPLAYFORM0 C.9 Probability of the bound holdingThe bound applies to an initialization with ζ ≥ ζ 0, hence in S 1 \Q ζ0. Assuming uniform initialization in S 1, the set Q ζ0 is simply a band of width 2ζ 0 around the equator of the ball B x / √ 2 (in R 2n, using the natural identification of C n with R 2n). This volume can be calculated by integrating over 2n − 1 dimensional balls of varying radius. DISPLAYFORM1 and by V (n) = π n/2 n 2 Γ(n 2) the hypersphere volume, the probability of initializing in S 1 ∩ Q ζ0 (and thus in a region that feeds into small gradient regions around saddle points) is DISPLAYFORM2. For small ζ we again find that P(fail) scales linearly with ζ, as was the case for the previous problems considered. Proof of Lemma 3: (Volume of C ζ). We are interested in the relative volume DISPLAYFORM0 Vol(S n−1) ≡ V ζ. Using the standard solid angle formula, it is given by DISPLAYFORM1 This integral admits no closed form solution but one can construct a linear approximation around small ζ and show that it is convex. Thus the approximation provides a lower bound for V ζ and an upper bound on the failure probability. From symmetry considerations the zero-order term is V 0 = 1 2n. The first-order term is given by DISPLAYFORM2 We now require an upper bound for the second integral since we are interested in a lower bound for V ζ. We can express it in terms of the second moment of the L ∞ norm of a Gaussian vector as follows: where µ(X) is the Gaussian measure on the vector X ∈ R n. We can bound the first term using Combining these bounds, the leading order behavior of the gradient is DISPLAYFORM3 This linear approximation is indeed a lower bound, since using integration by parts twice we have and this is the smallest L ∞ ball containing C ζ.Proof. Given the surface of some L ∞ ball for w, we can ask what is the minimal ζ such that ∂C ζm intersects this surface. This amounts to finding the minimal q n given some w ∞. Yet this is clearly obtained by setting all the coordinates of w to be equal to w ∞ (this is possible since we are guaranteed q n ≥ w ∞ ⇒ w ∞ ≤ where one instead maximizes q n with some fixed w ∞ .Given some surface of an L 2 ball, we can ask what is the minimal C ζ such that C ζ ⊆ B 2 r. This is equivalent to finding the maximal ζ M such that ∂C ζ M intersects the surface of the L 2 ball. Since q n is fixed, maximizing ζ is equivalent to minimizing w ∞. This is done by setting w ∞ = w √ n−1, which gives DISPLAYFORM4 The statement in the lemma follows from combining these .. If we now combine this with the fact that after a Riemannian gradient step cos(gη)q i − sin(gη) ≤ q i ≤ cos(gη)q i + sin(gη), the above condition on η implies the inequality (*), which in turn ensures that |w i | < r ⇒ |w i | < w ∞: |w i | < |w i | + sin(gη) < r + gη < (*)(1 − g 2 η 2)b − gη < cos(gη) w ∞ − sin(gη) ≤ w ∞ Due to the above analysis, it is evident that any w i such that |w i | = w ∞ obeys |w i | > r, from which it follows that we can use 31 to obtain q n w ∞ − 1 = ζ ≥ ζ 1 + √ n 2 ηc(w)

Represent the TLDR for finding the corresponding scientific article

We provide an efficient convergence rate for gradient descent on the complete orthogonal dictionary learning objective based on a geometric analysis.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.