Question: Represent the text to find a very short summary (1-2 sentences max).

Question: We introduce an attention mechanism to improve feature extraction for deep active learning (AL) in the semi-supervised setting. The proposed attention mechanism is based on recent methods to visually explain predictions made by DNNs. We apply the proposed explanation-based attention to MNIST and SVHN classification. The conducted experiments show accuracy improvements for the original and class-imbalanced datasets with the same number of training examples and faster long-tail convergence compared to uncertainty-based methods. Deep active learning (AL) minimizes the number of expensive annotations needed to train DNNs by selecting a subset of relevant data points from a large unlabeled dataset BID7. This subset is annotated and added to the training dataset in a single pool of data points or, more often, in an iterative fashion. The goal is to maximize prediction accuracy while minimizing the product of pool size × number of iterations. A proxy for this goal could be the task of matching feature distributions between the validation and the AL-selected training datasets. In density-based AL approaches, data selection is typically performed using a simple L 2 -distance metric BID10. The image retrieval field BID17 has advanced much further in this area. For example, recent state-of-the-art image retrieval systems are based on DNNbased feature extraction BID0 with attention mechanisms BID8. The latter estimates an attention mask to weight importance of the extracted features and it is trained along with the feature extraction. Inspired by this, we employ image retrieval techniques and propose a novel attention mechanism for deep AL. Unlike supervised self-attention in BID8 BID14, our attention mechanism is not trained with the model. It relies on recent methods to generate visual explanations and to attribute feature importance values BID13. We show the effectiveness of such explanation-based attention (EBA) mechanism for AL when combined with multi-scale feature extraction on a number of image classification datasets. We also conduct experiments for distorted class-imbalanced training data which is a more realistic assumption for unlabeled data. AL is a well-studied approach to decrease annotation costs in a traditional machine learning pipelines BID11. Recently, AL has been applied to data-demanding DNN-based systems in semi-supervised or weakly-supervised settings. Though AL is an attractive direction, existing methods struggle to deal with high-dimensional data e.g. images. We believe this is related to the lack of class and instance-level feature importance information as well as the inability to capture spatially-localized features. To overcome these limitations, we are interested in estimating spatiallymultiscale features and using our EBA mechanism to select only the most discriminative features. BID16 proposed to augment the training dataset by labeling the least confident data points and heuristically pseudo-labeling high confidence predictions. We believe the softmax output is not a reliable proxy for the goals of AL i.e. for selecting images using feature distribution matching between validation and train data. Unlike BID16, we use pseudo labels only to estimate EBA vectors and find similarities between discriminative features. BID2 introduced a measure of uncertainty for approximate Bayesian inference that can be estimated using stochastic forward passes through a DNN with dropout layers. An acquisition function then selects data points with the highest uncertainty which is measured at the output of softmax using several metrics. Recent work BID1 extended this method by using an ensemble of networks for uncertainty estimation and achieved superior accuracy. formulated feature similarity-based selection as a geometric core-set approach which outperforms greedy k-center clustering. Though their method can complement our approach, we are focusing on the novel feature extraction. For instance, they employed a simple L 2 distance similarity measure for the activations of the last fully-connected layer. The most similar work to ours, by BID15, uses the gradients as a measure of importance for dataset subsampling and analysis. However, our approach formulates the problem as a multi-scale EBA for AL application and goes beyond a less robust single-step gradient attention. Other related works are online importance sampling methods BID9 and the influence functions approach in BID4. Online importance sampling upweights samples within the mini-batch during supervised training using gradient similarity while influence functions analyze data point importance using computationally challenging second-order gradient information. Pool-based AL. Let (X, y) be an input-label pair. There is a validation dataset {(X v i, y v i)} i∈M of size M and a collection of training pairs {(X i, y i)} i∈N of size N for which, initially, only a small random subset or pool of labels indexed by N 1 is known. The validation dataset approximates the distribution of test data. At every bth iteration the AL algorithm selects a pool of P new labels to be annotated and added to existing training pairs which creates a training dataset indexed by N b.A DNN Φ(X, θ) is optimized by minimizing a loss function (DISPLAYFORM0 to model parameters θ. However, the actual task is to minimize validation loss expressed by M DISPLAYFORM1 Therefore, an oracle AL algorithm achieves minimum of validation loss using the smallest b × P product. In this work, we are interested not in finding an oracle acquisition function, but in a method to extract relevant features for such function. We use a low-complexity greedy k-center algorithm to select the data points in the unlabeled training collection which are most similar to the misclassified entries in the validation dataset. Feature descriptors. Let F j i ∈ R C×H×W, where C, H, and W are the number of channels, the height, and the width, respectively be the output of the jth layer of DNN for input image X i . Then, a feature vector or descriptor of length L can be defined as DISPLAYFORM2, where function φ(·) is a conventional average pooling operation from BID0. In a multi-scale case, descriptor is a concatenation of multiple feature vectors DISPLAYFORM3 A descriptor matrix for the validation dataset V d ∈ R L×M and training dataset S d ∈ R L×N can be calculated using forward passes. Practically, descriptors can be compressed for storage efficiency reasons using PCA, quantization, etc. Then, a match kernel BID6, e.g. cosine similarity, can be used to match features in both datasets. Assuming that vectors d i are L 2 -normalized, the cosine-similarity matrix is simply DISPLAYFORM4 Explanation-based attention. Feature maps F i extracted by Φ(X, θ) and pooled by φ(·) contain features that: a) are not class and instance-level discriminative (in other words, not disentangled), b) spatially represent features for a plurality of objects in the input. We would like to upweight discriminative features that satisfy a) and b) using an attention mechanism. One approach would be to use self-attention BID14 at the cost of modifying network architecture and intervening into the training process. Instead, we propose to use EBA that is generated only for feature selection. The EBA mechanism attributes feature importance values w.r.t. to the output predictions. Unlike a visual explanation task, which estimates importance heatmaps in the input (image) space, we propose to estimate feature importance tensors A i of the internal DNN representations F i. Attention tensors A i can be efficiently calculated using a series of backpropagation passes. Using one of backpropagation-based methods called integrated gradients (IG) from BID13, A j i can be estimated as DISPLAYFORM5 where K is the number of steps to approximate the continuous integral by a linear path. Other forms of are possible: from the simplest saliency method for which K = 1 BID12 to more advanced methods with randomly sampled input features BID3.Due to lack of labels y i in, we use common pseudo-labeling strategy: y i = 1 arg maxŷi. It is schematically shown in FIG0. Unlike BID16, pseudo-labels are used only to calculate similarity without additional hyperparameters rather than to perform a threshold-selected greedy augmentation. The EBA A i can be converted to multi-scale attention vector using the same processing a i = φ(A i) ∈ R L×1, which, by analogy, forms validation V a ∈ R L×M and train attention matrices S a ∈ R L×N. The latter processing is implemented in most modern frameworks and, therefore, the complexity to generate A i is only K forward-backward passes. Summary for the proposed method. A random subset of N 1 training data points is annotated and a DNN Φ(X, θ) optimized for this subset. Then, the AL algorithm iteratively (b = 2, 3 . . .) performs following steps: 1) generates descriptor-attention matrix pairs DISPLAYFORM6 is element-wise product, 3) selects P relevant data points from the remaining subset using acquisition function arg max i∈N\N b−1 (R(X i), Φ) and 4) retrains Φ(X, θ) using augmented subset N b. Our method as well as uncertainty-based methods from BID2 are applied to the MNIST and SVHN classification. We evaluate AL with the original and distorted training data because unlabeled collection of data points cannot be a-priori perfectly selected. Hence, we introduce a class imbalance which is defined as the ratio of {0 . . . 4} to {5 . . . 9} digits. The following methods have been employed: random sampling, uncertainty-based (uncert), greedy selection using similarity matching without (top-P:none) and with EBA. The latter is estimated by saliency (top-P:grad) or IG (top-P:ig). We rerun experiments 10 times for MNIST and 5 times for SVHN with all-randomized initial parameters. Mean accuracy and standard deviation are reported. DNN parameters are trained from scratch initially and after each AL iteration. Mini-batch size is chosen by cross-validation. MNIST. The dataset train/val/test split is 50K/10K/10K. The LeNet is used with the following hyperparameters: epochs=50, batch-size=25, lr=0.05, lr-decay=0.1 every 15 epochs, uncert methods and IG EBA use K = 128 passes and L is 20 for single-scale (before fc1 layer) and 90 for multiscale descriptors (all layers are concatenated). Figure 2(a) shows that feature-only matching (top-P:none L20) outperforms random selection by ≈ 1% while EBA (top-P:ig L90) adds another 1% of accuracy when there is no class imbalance. High class imbalance (Figure 2(c) ) increases that gap: up to 20% for feature-only matching and 25% with EBA. The highest accuracy is achieved by multi- 90 91 92 93 94 95 96 97 98 99 Top-1 Accuracy, % full random uncert:varMC uncert:entMC top-P:none_L20 top-P:grad_L20 top-P:ig_L20 top-P:ig_L90 top-P:igAbl_L20 top-P:igAbl_L90 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 Fraction of full training dataset, % (b) 75 77 79 81 83 85 87 89 91 93 95 97 Top-1 Accuracy, % full random uncert:varMC uncert:entMC top-P:none_L20 top-P:grad_L20 top-P:ig_L20 top-P:ig_L90 top-P:igAbl_L20 top-P:igAbl_L90 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 Fraction of full training dataset, % (c) 58 61 64 67 70 73 76 79 82 85 88 91 Top-1 Accuracy, % full random uncert:varMC uncert:entMC top-P:none_L20 top-P:grad_L20 top-P:ig_L20 top-P:ig_L90 top-P:igAbl_L20 top-P:igAbl_L90Figure 2: MNIST test dataset accuracy for 3 class imbalance ratios: a) 1 (no imbalance), b) 10 and c) 100. Total 9 AL iterations (b = 10) are performed each with P = 250 pool size. Top-1 Accuracy, % full random uncert:entMC uncert:baldMC top-P:none_L256 top-P:grad_L256 top-P:ig_L256 top-P:ig_L384Figure 3: SVHN test dataset accuracy for 3 class imbalance ratios: a) 1 (no imbalance), b) 10 and c) 100. Total 9 AL iterations (b = 10) are performed each with P = 2, 500 pool size.scale EBA estimated by IG. EBA-based methods outperform the best uncertainty-based variation ratio (uncert:varMC) approach for all class imbalance settings except the last one where its accuracy is higher by less than 1% when b = 4. This might be related to small-scale MNIST and pseudo-label noise for EBA. To study the effects of pseudo-labeling, we plot true-label configurations (marked by "Abl") as well. The accuracy gap between EBA using true-and pseudo-labels is small with no class imbalance, but much larger (up to 25%) when class imbalance ratio is 100 during first AL iterations. The dataset train/validation/test split is 500K/104K/26K. A typical 8-layer CNN is used with the following hyperparameters: epochs=35, batch-size=25, lr=0.1, lr-decay=0.1 every 15 epochs, uncert methods and IG EBA use K = 128 and L is 256 for single-scale (before fc1 layer) and 384 for two-scale descriptors (+ layer before conv7). Figure 3 shows that the gap between random selection and the best EBA-based AL method grows from 2% to more than 12% when the unlabeled training collection has more class imbalance. The gap between full training dataset accuracy increases for larger-scale SVHN as well. This in even faster convergence for the proposed AL relative to random selection. Accuracies of the uncert methods are closer to each other than for MNIST, which may signal their declining effectiveness for large-scale data. The proposed EBA-based methods outperform all uncertainty-based methods for SVHN in the first AL iterations (up to +2.5%) and later arrive at approximately equal . We applied recent image retrieval feature-extraction techniques to deep AL and introduced a novel EBA mechanism to improve feature-similarity matching. First feasibility experiments on MNIST and SVHN datasets showed advantages of EBA to improve density-based AL. Rather than performing AL for the well-picked training datasets, we also considered more realistic and challenging scenarios with class-imbalanced training collections where the proposed method emphasized the importance of additional feature supervision. In future research, EBA could be evaluated with other types of data distortions and biases: within-class bias, adversarial examples, etc. Furthermore, such applications as object detection and image segmentation may benefit more from EBA because multiscale attention can focus on spatially-important features.

We introduce an attention mechanism to improve feature extraction for deep active learning (AL) in the semi-supervised setting.

We propose Janossy pooling, a method for learning deep permutation invariant functions designed to exploit relationships within the input sequence and tractable inference strategies such as a stochastic optimization procedure we call piSGD


Question: Represent the text to find a very short summary (1-2 sentences max):

Question: We apply canonical forms of gradient complexes (barcodes) to explore neural networks loss surfaces. We present an algorithm for calculations of the objective function's barcodes of minima. Our experiments confirm two principal observations: the barcodes of minima are located in a small lower part of the range of values of objective function and increase of the neural network's depth brings down the minima's barcodes. This has natural implications for the neural network learning and the ability to generalize. The learning via finding minima of objective functions is the principal strategy underlying majority of learning algorithms. For example, in Neural Network training, the objective function's input is model parameters (weights) and the objective function's output is the loss on training dataset. The graph of the loss function, often called loss surface, typically has complex structure (e.g. see loss surface visualisations by): non-convexity, many local minima, flat regions, steep slopes. These obstacles harm exploration of the loss surface and complicate searching for optimal network weights. The optimization of modern neural networks is based on the gradient descent algorithm. The global topological characteristics of the gradient vector field trajectories are captured by the Morse complex via decomposing the parameter space into cells of uniform flow, see; and references therein. The invariants of Morse complex called "canonical forms"(or barcodes) constitute the fundamental summary of the topology of the gradient vector field flow. The "canonical forms", or barcodes, in this context are decompositions of the change of topology of the sublevel sets of objective function into simple "birth-death" phenomena of topological feautures of different dimensions. The calculation of the barcodes for different functions constitutes the essence of the topological data analysis. The currently available software packages for the calculation of barcodes of functions, also called "sublevel persistence", are GUDHI, Dionysus, PHAT, and TDA package which incorporates all three previous packages B.T.. They are based on the algorithm, described in , see also appendix and e.g. and references therein. This algorithm which has complexity of O(n 3). These packages can currently handle calculations of barcodes for functions defined on a grid of up to 10 6 points, and in dimensions two and three. Thus all current packages have the scalability issues. We describe a new algorithm for computations of the barcodes of functions in lowest degree. Our algorithm works with functions defined on randomly sampled or specifically chosen point clouds. Point cloud based methods are known to work better than grid based methods in optimization related problems . We also use the fact that the definition of the barcode of lowest degree can be reformulated in geometrical terms (see definition 1 in section 2). The previously known algorithms were based on the more algebraic approach as in definition 3. Our algorithm has complexity of O(n log(n)). It was tested in dimensions up to 16 and with number of points of up to 10 8. In this work, we develop a methodology to describe the properties of the loss surface of the neural network via topological features of local minima. We emphasize that the value of the objective function at the minimum can be viewed as only a part of its topological characteristic from the "canonical form" (barcode). The second half can be described as the value of objective function at the index-one saddle, which can be naturally associated with each local minimum. The difference between the values of objective function at the associated index-one saddle and at the local minimum is a topological invariant of the minimum. For optimization algorithms this quantity measures, in particular, the obligatory penalty for moving from the given local minimum to a lower minimum. The main contributions of the paper are as follows: Applying the one-to-one correspondence between local minima and 1-saddles to exploration of loss surfaces. For each local minimum p there is canonically defined 1-saddle q (see Section 2). The 1-saddle associated with p can be described as follows. The 1-saddle q is precisely the point where the connected component of the sublevel set Θ f ≤c = {θ ∈ Θ | f (θ) ≤ c} containing the minimum p merges with another connected component of the sublevel set whose minimum is lower. This correspondence between the local minima and the 1-saddles, killing a connected component of Θ f ≤c, is one-to-one. The segment [f (p), f (q)] is then the "canonical form" invariant attached to the minimum p. The set of all such segments is the barcode ("canonical form") of minima invariant of f. It is a robust topological invariant of objective function. It is invariant in particular under the action of homeomorphisms of Θ. Full "canonical form" invariants give a concise summary of the topology of objective function and of the global structure of its gradient flow. Algorithm for calculations of the barcodes (canonical invariants) of minima. We describe an algorithm for calculation of the canonical invariants of minima. The algorithm works with function's values on a a randomly sampled or specifically chosen set of points. The local minima give birth to clusters of points in sublevel sets. The algorithm works by looking at neighbors of each point with lower value of the function and deciding if this point belongs to the existing clusters, gives birth to a new cluster (minimum), or merges two or more clusters (index one saddle). A variant of the algorithm has complexity of O(n log(n)), where n is the cardinality of the set of points. Calculations confirming observations on behaviour of neural networks loss functions barcodes. We calculate the canonical invariants (barcodes) of minima for small fully-connected neural networks of up to three hidden layers and verify that all segments of minima's barcode belong to a small lower part of the total range of loss function's values and that with the increase in the neural network depth the minima's barcodes descend lower. The usefulness of our approach and algorithms is clearly not limited to the optimization problems. Our algorithm permits really fast computation of the canonical form invariants (persistence barcodes) of many functions which were not accessible until now. These sublevel persistence barcodes have been successfully applied in different discipline, to mention just a few: cognitive science (M. K.), cosmology , see e.g. and references therein. Our viewpoint should also have applications in chemistry and material science where 1-saddle points on potential energy landscapes correspond to transition states and minima are stable states corresponding to different materials or protein foldings (see e.g. ,). The article is structured as follows. First we describe three definitions of barcodes of minima. After that our algorithm for their calculation is described. In the last part we give examples of calculations, including the loss functions of simple neural nets. The "canonical form" invariants (barcodes) give a concise summary of topological features of functions (see , and references therein). These invariants describe a decomposition of the change of topology of the function into the finite sum of "birth"-"death" of elementary features. We propose to apply these invariants as a tool for exploring topology of loss surfaces. In this work we concentrate on the part of these canonical form invariants, describing the "birth"-"death" phenomena of connected components of sublevel sets of the function. However it should be stressed that this approach works similarly also for "almost minima", i.e. for the critical points (manifolds) of small indexes, which are often the terminal points of the optimization algorithms in very high dimensions. We give three definitions of the "canonical form" invariants of minima. The values of parameter c at which the topology of sublevel set Let p be one of minima of f. When c increases from f (p)− to f (p)+, a new connected component of the set Θ f ≤c is born (see fig 1a, the connected components S 1, S 2, S 3 of sublevel set are born at the blue, green and red minima correspondingly. If p is a minimum, which is not global, then, when c is increased, the connected component of Θ f ≤c born at p merges with a connected component born at a lower minimum. Let q is the merging point where this happens. The intersection of the set Θ f <f (q) with any small neighborhood of q has two connected components. This is the index-one saddle q associated with p. (a) "Death" of the connected component S3. The connected component S3 of sublevel set merges with connected component S2 at red saddle, red saddle is associated with the red minimum. (b) "Death" of the connected component S4. The connected component S4 of sublevel set merges with connected component S1 at violet saddle, violet saddle is associated with the violet minimum (c) "Death" of the connected component S2. The connected component S2 of sublevel set merges with connected component S1 at green saddle, green saddle is associated with the green minimum. Figure 1: Merging of connected components of sublevel sets at saddles. Note that the green saddle is associated with the green minimum which is separated by another minimum from the green saddle. Also these two subsets of small neighborhood of q belong to two different connected components of the whole set Θ f <f (q). The 1-saddles of this type are called "+" ("plus") or "death" type. The described correspondence between local minima and 1-saddles of this type is one-to-one. In a similar way, the 1-saddle q associated with p can be described also as follows. Proposition 2.1. Consider various paths γ starting from the local minimum p and going to a lower minimum. Let m γ ∈ Θ is the maximum of the restriction of f to such path γ. Then 1-saddle q which is paired with the local minimum p is the minimum over the set of all such paths γ of the maxima m γ: The correspondence in the opposite direction can be described analogously. Let q is a 1-saddle point of such type that the two branches of the set Θ f ≤f (q)− near q belong to two different connected components of Θ f ≤f (q)−. A new connected component of the set Θ f ≤c is formed when c decreases from f (q) + to f (q) −. The restriction of f to each of the two connected components has its global minimum. Proposition 2.2. Given a 1-saddle q, the minimum p which is paired with q is the new minimum of f on the connected component of the set Θ f ≤c which is formed when c decreases from f (q) + to f (q) −. The two branches of the set Θ f ≤f (q)− near q can also belong to the same connected components of this set. Then such saddle is of "birth" type and it is naturally paired with index-two saddle of "death" type (see theorem 2.3). Chain complex is the algebraic counterpart of intuitive idea representing complicated geometric objects as a decomposition into simple pieces. It converts such a decomposition into a collection of vector spaces and linear maps. A chain complex (C *, ∂ *) is a sequence of finite-dimensional k-vector spaces and linear operators The j−th homology of the chain complex (C *, ∂ *) is the quotient A chain complex C * is called R−filtered if C * is equipped with an increasing sequence of sub- by a finite set of real numbers s 1 < s 2 <... < s max. Theorem 2.3. Any R−filtered chain complex C * can be brought by a linear transformation preserving the filtration to "canonical form", a canonically defined direct sum of R−filtered complexes of two types: one-dimensional complexes with trivial differential ∂ j (e i) = 0 and two-dimensional complexes with trivial homology ∂ j (e i2) = e i1. The ing canonical form is uniquely determined. The full barcode is a visualization of the decomposition of an R−filtered complexes according to the theorem 2.3. Each filtered 2-dimensional complex with trivial homology ∂ j (e i2) = e i1, e i1 = F ≤s1, e i1, e i2 = F ≤s2 describes a topological feature in dimension j which is "born" at s 1 and which "dies" at s 2. It is represented by segment [s 1, s 2] in the degree-j barcode. And each filtered 1-dimensional complex with trivial differential, ∂ j e i = 0, e i = F ≤r describes a topological feature in dimension j which is "born" at r and never "dies". It is represented by the half-line [r, +∞[ in the degree-j barcode. The proof of the theorem is given in Appendix. Essentially, one can bring an R−filtered complex to the required canonical form by induction, starting from the lowest basis elements of degree one, in such a way that the manipulation of degree j basis elements does not destroy the canonical form in degree j − 1 and in lower filtration pieces in degree j. Let f : Θ → R is smooth, or more generally, piece-wise smooth continuous function such that the sublevel sets Θ f ≤c = {θ ∈ Θ | f (θ) ≤ c} are compact. One filtered complex naturally associated with function f and such that the subcomplexes F s C * compute the homology of sublevel sets Θ f ≤s is the gradient (Morse) complex, see e.g.; and references therein. Without loss of generality the function f can be assumed smooth here, otherwise one can always replace f by its smoothing. By adding a small perturbation such as a regularization term we can also assume that critical points of f are non-degenerate. The generators of the gradient (Morse) complex correspond to the critical points of f. The differential is defined by counting gradient trajectories between critical points when their number is finite. The canonical form of the gradient (Morse) complex describes a decomposition of the gradient flow associated with f into standard simple pieces. Let p be a minimum, which is not a global minimum. Then the generator corresponding to p represents trivial homology class in the canonical form, since the homology class of its connected component is already represented by the global minimum. Then p is the lower generator of a two-dimensional complex with trivial homology in the canonical form. I.e. p is paired with an index-one saddle q in the canonical form. The segment [f (p), f (q)] is then the canonical invariant (barcode) corresponding to the minimum p. The full canonical form of the gradient (Morse) complex of all indexes is a summary of global structure of the objective function's gradient flow. The total number of different topological features in sublevel sets Θ f ≤c of the objective function can be read immediately from the barcode. Namely the number of intersections of horizontal line at level c with segments in the index j barcode gives the number of independent topological features of dimension j in Θ f ≤c. The description of the barcode of minima on manifold Θ with nonempty boundary ∂Θ is modified in the following way. A connected component can be also born at a local minimum of restriction of f to the boundary f | ∂Θ, if gradf is pointed inside manifold Θ. The merging of two connected components can also happen at an index-one critical point of f | ∂Θ, if gradf is pointed inside Θ. In this section we describe the developed algorithm for calculation of the canonical form invariants of local minima. The computation exploits the first definition of barcodes (see Section 2), which is based on the evolution on the connected components of the sublevel sets. To analyse the surface of the given function f: Θ → R, we first build its approximation by finite graph-based construction. To do this, we consider a random subset of points {θ 1, . . ., θ N} ∈ Θ and build a graph with these points as vertices. The edges connect close points. Thus, for every vertex θ n, by comparing f (θ n) with f (θ n) for neighbors θ n of θ n, we are able to understand the local topology near the point θ n. At the same time, connected componenets of sublevel sets Θ f ≤c = {θ ∈ Θ | f (θ) ≤ c} will naturally correspond to connected components of the subgraph on point θ n, such that f (θ n) ≤ c. Two technical details here are the choice of points θ n and the definition of closeness, i.e. when to connect points by an edge. In our experiments, we sample points uniformly from some rectangular box of interest. To add edges, we compute the oriented k-Nearest Neighbor Graph on the given points and then drop the orientation of edges. Thus, every node in the obtained graph has a degree in [k, 2k]. In all our experiments we use k = 2D, where D is the dimension of f's input. Next we describe our algorithm, which computes barcodes of a function from its graph-based approximation described above. The key idea is to monitor the evolution of the connected components of the sublevel sets of the graph, i.e. sets Θ c = {θ n | f (θ n) ≤ c} for increasing c. For simplicity we assume that points θ are ordered w.r.t. the value of function f, i.e. for n < n we have f (θ n) < f (θ n). In this case we are interested in the evolution of connected components throughout the process sequential adding of vertices θ 1, θ 2,..., θ N to graph, starting from an empty graph. We denote the subgraph on vertices θ 1,..., θ n by Θ n. When we add new vertex θ n+1 to θ n, there are three possibilities for connected componenets to evolve: 1. Vertex θ n+1 has zero degree in Θ n+1. This means that θ n+1 is a local minimum of f and it forms a new connected component in the sublevel set. 2. All the neighbors of θ n+1 in Θ n+1 belong to one connected component in Θ n. 3. All the neighbors of θ n+1 in Θ n+1 belong to ≥ 2 connected components s 1, s 2,..., s K ⊂ Θ n. Thus, all these components will form a single connected component in Θ n+1. Algorithm 1: Barcodes of minima computation for function on a graph. Input: Connected undirected graph G = (V, E); function f on graph vertices. Output: Barcodes: a list of "birth"-"death" pairs. In the third case, according to definition 1 of Section 2 the point θ n+1 is a 1-saddle point. Thus, one of the components s k swallows the rest. This is the component which has the lowest minimal value. For other components, 2 this gives their barcodes: for s k the birth-death pair is min We summarize the procedure in the following algorithm 1. Note that we assume that the input graph is connected (otherwise the algorithm can be run on separate connected components). In the practical implementation of the algorithm, we precompute the values of function f at all the vertices of G. Besides that, we use the disjoint set data structure to store and merge connected components during the process. We also keep and update the global minima in each component. We did not include these tricks into the algorithm's pseuso-code in order to keep it simple. The ing complexity of the algorithm is O(N log N) in the number of points. Here it is important to note that the procedure of graph creation may be itself time-consuming. In our case, the most time consuming operation is nearest neighbor search. In our code, we used efficient HNSW Algorithm for aproximate NN search by. In this section we apply our algorithm to describing the surfaces of functions. In Subsection 4.1 we apply the algorithm to toy visual examples. In Subsection 4.2 we apply the algorithm to analyse the loss surfaces of small neural networks. In this subsection we demonstrate the application of the algorithm to simple toy functions f: R D → R. For D ∈ {1, 2} we consider three following functions: 1. Polynomial of a single variable of degree 4 with 2 local minima (see Fig. 2a): 2. Camel function with 3 humps, i.e. 3 local minima (see Fig. 2b): 3. Camel function with 6 humps, i.e. 6 local minima (see Fig. 2c): Function plots with their corresponding barcodes of minima are given in Figure 2. The barcode of the global minimum is represented by the dashed half-line which goes to infinity. In this section we compute and analyse barcodes of small fully connected neural networks with up to three hidden layers. For several architectures of the neural networks many on the loss surface and its local minima are known (see e.g. and references therein). Different geometrical and topological properties of loss surfaces were studied in;;;. There is no ground truth on how should the best loss surface of a neural network looks like. Nevertheless, there exists many common opinions on this topic. First of all, from practical optimization point of view, the desired local (or global) minima should be easily reached via basic training methods such as Stochastic Gradient Descent, see. Usually this requires more-or-less stable slopes of the surface to prevent instabilities such as gradient explosions or vanishing gradients. Secondly, the value of obtained minimum is typically desired to be close to global, i.e. attain smallest training error. Thirdly, from the generalization point of view, such minima are required to provide small loss on the testing set. Although in general it is assumed that the good local optimum is the one that is flat, some recent development provide completely contrary arguments and examples, e.g. sharp minima that generalize well. Besides the optimization of the weights for a given architecture, neural network training implies also a choice of the architecture of the network, as well as the loss function to be used for training. In fact, it is the choice of the architecture and the loss function that determines the shape of the loss surface. Thus, proper selection of the network's architecture may simplify the loss surface and lead to potential improvements in the weight optimization procedure. We have analyzed very tiny neural networks. However our method permits full exploration of the loss surface as opposed to stochastical exploration of higher-dimensional loss surfaces. Let us emphasize that even from practical point of view it is important to understand first the behavior of barcodes in simplest examples where all hyper-parameters optimization schemes can be easily turned off. For every analysed neural network the objective function is its mean squared error for predicting (randomly selected) function g: [−π, π] → R given by plus l 2 −regularization. The error is computed for prediction on uniformly distributed inputs x ∈ {−π + 2π 100 k | k = 0, 1, . . ., 100}. The neural networks considered were fully connected one-hidden layer with 2, 3 and 4 neurons, two-hidden layers with 2x2, 3x2 and 3x3 neurons, and three hidden layers with 2x2x2 and 3x2x2 neurons. We have calculated the barcodes of the loss functions on the hyper-cubical sets Θ which were chosen based on the typical range of parameters of minima. The are as shown in Figure 3. We summarize our findings into two main observations: In this work we have introduced a methodology for analysing the plots of functions, in particular, loss surfaces of neural networks. The methodology is based on computing topological invariants called canonical forms or barcodes. To compute barcodes we used a graph-based construction which approximates the function plot. Then we apply the algorithm we developed to compute the barcodes of minima on the graph. Our experimental of computing barcodes for small neural networks lead to two principal observations. First all barcodes sit in a tiny lower part of the total function's range. Secondly, with increase of the depth of neural network the barcodes descend lower. From the practical point of view, this means that gradient descent optimization cannot stuck in high local minima, and it is also not difficult to get from one local minimum to another (with smaller value) during learning. The method we developed has several further research directions. Although we tested the method on small neural networks, it is possible to apply it to large-scale modern neural networks such as convolutional networks (i.e. ResNet, VGG, AlexNet, U-Net, see) for imageprocessing based tasks. However, in this case the graph-based approximation we use requires wise choice of representative graph vertices, which is a hardcore in high-dimensional spaces (dense filling of area by points is computationally intractable). Another direction is to study the connections between the barcode of local minima and the generalization properties of given minimum and of neural network. There are clearly also connections, deserving further investigation, between the barcodes of minima and concerning the rate of convergency during learning of neural networks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We apply canonical forms of gradient complexes (barcodes) to explore neural networks loss surfaces.

represent the short summary for finding the corresponding scientific article (long)

The loss surface is *very* degenerate, and there are no barriers between large batch and small batch solutions.