Question: represent the article for finding a single-sentence tldr

Question: We present SOSELETO (SOurce SELEction for Target Optimization), a new method for exploiting a source dataset to solve a classification problem on a target dataset. SOSELETO is based on the following simple intuition: some source examples are more informative than others for the target problem. To capture this intuition, source samples are each given weights; these weights are solved for jointly with the source and target classification problems via a bilevel optimization scheme. The target therefore gets to choose the source samples which are most informative for its own classification task. Furthermore, the bilevel nature of the optimization acts as a kind of regularization on the target, mitigating overfitting. SOSELETO may be applied to both classic transfer learning, as well as the problem of training on datasets with noisy labels; we show state of the art on both of these problems. Deep learning has demonstrated remarkable successes in tasks where large training sets are available. Yet, its usefulness is still limited in many important problems that lack such data. A natural question is then how one may apply the techniques of deep learning within these relatively data-poor regimes. A standard approach that seems to work relatively well is transfer learning. Despite its success, we claim that this approach misses an essential insight: some source examples are more informative than others for the target classification problem. Unfortunately, we don't know a priori which source examples will be important. Thus, we propose to learn this source filtering as part of an end-to-end training process. The ing algorithm is SOSELETO: SOurce SELEction for Target Optimization. Each training sample in the source dataset is given a weight, representing its importance. A shared source/target representation is then optimized by means of a bilevel optimization. In the interior level, the source minimizes its classification loss with respect to the representation and classification layer parameters, for fixed values of the sample weights. In the exterior level, the target minimizes its classification loss with respect to both the source sample weights and its own classification layer. The sample weights implicitly control the representation through the interior level. The target therefore gets to choose the source samples which are most informative for its own classification task. Furthermore, the bilevel nature of the optimization acts as a kind of regularization on the target, mitigating overfitting, as the target does not directly control the representation parameters. The entire processtraining of the shared representation, source and target classifiers, and source weights -happens simultaneously. Related Work The most common techniques for transfer learning are feature extraction e.g. and fine-tuning, e.g. BID8. An older survey of transfer learning techniques may be found in BID20. Domain adaptation BID23 involves knowledge transfer when the source and target classes are the same. Earlier techniques aligned the source and target via matching of feature space statistics BID3 BID15; subsequent work used adversarial methods to improve the domain adaptation performance BID6;. In this paper, we are more interested in transfer learning where the source and target classes are different. BID16; BID21 BID1 b) address domain adaptation that is closer to our setting. BID2 examines "partial transfer learning" in which there is partial overlap between source and target classes (often the target classes are a subset of the source). This setting is also dealt with in BID0. Like SOSELETO, BID7 propose selecting a portion of the source dataset, however, the selection is done prior to training and is not end-to-end. In, an adversarial loss aligns the source and target representations in a few-shot setting. Instance reweighting is a well studied technique for domain adaptation, demonstrated e.g. in Covariate Shift methods BID24 BID25 BID26. While in these works, the source and target label spaces are the same, we allow them to be different -even entirely non-overlapping. Crucially, we do not make assumptions on the similarity of the distributions nor do we explicitly optimize for it. The same distinction applies for the recent work of BID9, and for the partial overlap assumption of. In addition, these two works propose an unsupervised approach, whereas our proposed method is completely supervised. Classification with noisy labels is a longstanding problem in the machine learning literature, see the review paper BID5. Within the realm of deep learning, it has been observed that with sufficiently large data, learning with label noise -without modification to the learning algorithms -actually leads to reasonably high accuracy BID10 BID28 BID22 BID4. We consider the setting where the large noisy dataset is accompanied by a small clean dataset. BID27 introduced a noise layer into the CNN that adapts the output to align with the noisy label distribution. proposed to predict simultaneously the clean label and the type of noise; consider the same setting, but with additional information in the form of a knowledge graph on labels. BID18 conditioned the gradient propagation on the agreement of two separate networks. BID14 BID7 combine ideas of learning with label noise with instance reweighting. We consider the problem of classifying a data-poor target set, by utilizing a data-rich source set. The sets and their corresponding labels are denoted by {( DISPLAYFORM0 respectively, where, n t n s . The key insight is that not all source examples contribute equally useful information in regards to the target problem. For example, suppose that the source set consists of a broad collection of natural images; whereas the target set consists exclusively of various breeds of dog. We would assume that images of dogs, as well as wolves, cats and even objects with similar textures like rugs in the source set may help in the target classification task; On the flip side, it is probably less likely that images of airplanes and beaches will be relevant (though not impossible). However, the idea is not to come with any preconceived notions (semantic or otherwise) as to which source images will help; rather, the goal is to let the algorithm choose the relevant source images, in an end-to-end fashion. We assume that the source and target networks share the same architecture, except for the last classification layers. In particular, the architecture is given by F (x; θ, φ), where φ denotes the last classification layer(s), and θ constitutes all of the "representation" layers. The source and target networks are thus F (x; θ, φ s), and F (x; θ, φ t), respectively. By assigning a weight α j ∈ to each source example, we define the weighted source loss as: DISPLAYFORM1 is a per example classification loss (e.g. cross-entropy). The weights α j will allow us to decide which source images are most relevant for the target classification task. The target loss is standard: DISPLAYFORM2 As noted in Section 1, this formulation allows us to address both the transfer learning problem as well as learning with label noise. In the former case, the source and target may have non-overlapping label spaces; high weights will indicate which source examples have relevant knowledge for the target classification task. In the latter case, the source is the noisy dataset, the target is the clean dataset, and they share a classifier (i.e. φ t = φ s) as well as a label space; high weights will indicate which source examples are likely to be reliable. In either case, the target is much smaller than the source. The question now becomes: how can we combine the source and target losses into a single optimization problem? A simple idea is to create a weighted sum of source and target losses. Unfortunately, issues are likely to arise regardless of the weight chosen. If the target is weighted equally to the source, then overfitting may likely given the small size of the target. On the other hand, if the weights are proportional to the size of the two sets, then the source will simply drown out the target. We propose instead to use bilevel optimization. Specifically, in the interior level we find the optimal features and source classifier as a function of the weights α, by minimizing the source loss: DISPLAYFORM3 In the exterior level, we minimize the target loss, but only through access to the source weights; that is, we solve: min DISPLAYFORM4 Why might we expect this bilevel formulation to succeed? The key is that the target only has access to the features in an indirect manner, by controlling which source examples will dominate the source classification problem. This form of regularization, mitigates overfitting, which is the main threat when dealing with a small set such as the target. We adopt a stochastic approach for optimizing the bilevel problem. Specifically, at each iteration, we take a gradient step in the interior level problem: DISPLAYFORM5 and Q(θ, φ s) is a matrix whose j th column is given by DISPLAYFORM6 ). An identical descent equation exists for the classifier φ s, which we omit for clarity. Given this iterative version of the interior level of the bilevel optimization, we may now turn to the exterior level. Plugging Equation into Equation FORMULA4 gives: min α,φ t L t (θ m − λ p Qα, φ t). We can then take a gradient descent step of this equation, yielding: DISPLAYFORM7 where we have made use of the fact that λ p is small. Of course, there will also be a descent equation for the classifier φ t. The ing update scheme is quite intuitive: source example weights are update according to how well they align with the target aggregated gradient. Finally, we clip the weight values to be in the range (a detailed explanation to this step is provided in Appendix B). For completeness we have included a summary of SOSELETO algorithm in Appendix A. Convergence properties are discussed in Appendix C. Noisy Labels: Synthetic Experiment We illustrate how SOSELETO works on the problem of learning with noisy labels, on a synthetic example, see FIG0. (a) The source set consists of 500 points in R 2, separated into two classes by the y-axis. However, 20% of the points are are assigned a wrong label. The target set consists of 50 "clean" points (For clarity, the target set is not illustrated.) SOSELETO is run for 100 epochs. FIG0 and 1(c), we choose a threshold of 0.1 on the weights α, and colour the points accordingly. In particular, in FIG0 (b) the clean (i.e. correctly labelled) instances which are above the threshold are labelled in green, while those below the threshold are labelled in red; as can be seen, all of the clean points lie above the threshold for this choice of threshold, meaning that SOSELETO has correctly identified all of the clean points. In FIG0 (c), the noisy (i.e. incorrectly labelled) instances which are below the threshold are labelled in green; and those above the threshold are labelled in red. In this case, SOSELETO correctly identifies most of these noisy labels by assigning them small weights, while the few mislabeled points (shown in red), are all near the separator. In FIG0, a plot is shown of mean weight vs. training epoch for clean instances and noisy instances. As expected, the algorithm nicely separates the two sets. FIG0 (e) demonstrates that the good behaviour is fairly robust to the threshold chosen. Noisy Labels: CIFAR-10 We now turn to a real-world setting of the problem of learning with label noise using a noisy version of CIFAR-10 (FORMULA3 set aside 10K clean examples for pre-training, a necessary step in both of these algorithms. In contrast, we use half that size. We also compare with a baseline of training only on noisy labels. In all cases, "Quick" cif architecture has been used. We set: λ p = 10 −4, target and source batch-sizes: 32, and 256. Performance on a 10K test set are shown in Table 1 for three noise levels. Our method surpasses previous methods on all three noise levels. We provide further analysis in Appendix D.Transfer Learning: SVHN 0-4 to MNIST 5-9 We now examine the performance of SOSELETO on a transfer learning task, using a challenging setting where: (a) source and target sets have disjoint label sets, and (b) the target set is very small. The source dataset is chosen as digits 0-4 of SVHN BID19. The target dataset is a very small subset of digits 5-9 from MNIST LeCun et al. We compare our with the following techniques: target only, which indicates training on just the target set; standard fine-tuning; and a fine-tuned version thereof; and two variants of the Label Efficient Learning technique described in. Except for the target only and fine-tuning approaches, all other approaches rely on a huge number of unlabelled target data. By contrast, we do not make use of any of this data. For each of the above methods, the simple LeNet architecture BID12 was used. We set: λ p = 10 −2, source and the target batch-sizes: 32, 10. The performance of the various methods on MNIST's test set is shown in TAB2, split into two parts: techniques which use the 30K examples of unlabelled data, and those which do not. SOSELETO has superior performance to all techniques except the Label Efficient. In Appendix E we further analyze which SVHN instances are considered more useful than others by SOSELETO in a partial class overlap setting, namely transferring SVHN 0-9 to MNIST 0-4.Although not initially designed to use unlabelled data, one may do so using the learned SOSELETO classifier to classify the unlabelled data, generating noisy labels. SOSELETO can now be run in a label-noise mode. In the case of n t = 25, this procedure elevates the accuracy to above 92%. We have presented SOSELETO, a technique for exploiting a source dataset to learn a target classification task. This exploitation takes the form of joint training through bilevel optimization, in which the source loss is weighted by sample, and is optimized with respect to the network parameters; while the target loss is optimized with respect to these weights and its own classifier. We have empirically shown the effectiveness of the algorithm on both learning with label noise, as well as transfer learning problems. An interesting direction for future research involves incorporating an additional domain alignment term. We note that SOSELETO is architecture-agnostic, and may be extended beyond classification tasks. DISPLAYFORM0 end while SOSELETO consists of alternating the interior and exterior descent operations, along with the descent equations for the source and target classifiers φ s and φ t. As usual, the whole operation is done on a mini-batch basis, rather than using the entire set; note that if processing is done in parallel, then source mini-batches are taken to be non-overlapping, so as to avoid conflicts in the weight updates. A summary of SOSELETO algorithm appears in 1. Note that the target derivatives ∂L t /∂θ and ∂L t /∂φ t are evaluated over a target mini-batch; we suppress this for clarity. In terms of time-complexity, we note that each iteration requires both a source batch and a target batch; assuming identical batch sizes, this means that SOSELETO requires about twice the time as the ordinary source classification problem. Regarding space-complexity, in addition to the ordinary network parameters we need to store the source weights α. Thus, the additional relative spacecomplexity required is the ratio of the source dataset size to the number of network parameters. This is obviously problem and architecture dependent; a typical number might be given by taking the source dataset to be Imagenet ILSVRC-2012 (size 1.2M) and the architecture to be ResNeXt-101 (size 44.3M parameters), yielding a relative space increase of about 3%. Recall that our goal is to explicitly require that α j ∈. We may achieve this by requiring DISPLAYFORM0 where the new variable β j ∈ R, and σ(·) is a kind of piecewise linear sigmoid function. Now we will wish to replace the Update Equation, the update for α, with a corresponding update equation for β. This is straightforward. Define the Jacobian ∂α/∂β by DISPLAYFORM1 Then we modify Equation to read DISPLAYFORM2 The Jacobian is easy to compute analytically: DISPLAYFORM3 Due to this very simple form, it is easy to see that β m will never lie outside of; and thus that α m = β m for all time. Hence, we can simply replace this equation with DISPLAYFORM4 where CLIP clips the values below 0 to be 0; and above 1 to be 1. SOWETO is only an approximation to the solution of a bilevel optimization problem. As a , it is not entirely clear whether it will even converge. In this section, we demonstrate a set of sufficient conditions for SOWETO to converge to a local minimum of the target loss L t.To this end, let us examine the change in the target loss from iteration m to m + 1: DISPLAYFORM0 Now, we can use the evolution of the weights α. Specifically, we substitute Equation into the above, to get DISPLAYFORM1 where ∆L indicates the change in the target loss, to first order. Note that the latter two terms in ∆L F O t are both negative, and will therefore cause the first order approximation of the target loss to decrease, as desired. As regards the first term, matters are unclear. However, it is clear that if we set the learning rate λ α sufficiently large, the second term will eventually dominate the first term, and the target loss will be decreased. Indeed, we can do a slightly finer analysis. Ignoring the final term (which is always negative), and setting v = Q T ∂Lt ∂θ, we have that DISPLAYFORM0 where in the second line we have used the fact that all elements of α are in; and in the third line, we have used a standard bound on the L 1 norm of a vector. Thus, a sufficient condition for the first order approximation of the target loss to decrease is if DISPLAYFORM1 If this is true at all iterations, then the target loss will continually decrease and converge to a local minimum (given that the loss is bounded from below by 0). We perform further analysis by examining the α-values that SOSELETO chooses on convergence, see FIG3. To visualize the , we imagine thresholding the training samples in the source set on the basis of their α-values; we only keep those samples with α greater than a given threshold. By increasing the threshold, we both reduce the total number of samples available, as well as change the effective noise level, which is the fraction of remaining samples which have incorrect labels. We may therefore plot these two quantities against each other, as shown in FIG3; we show three plots, one for each noise level. Looking at these plots, we see for example that for the 30% noise level, if we take the half of the training samples with the highest α-values, we are left with only about 4% which have incorrect labels. We can therefore see that SOSELETO has effectively filtered out the incorrect labels in this instance. For the 40% and 50% noise levels, the corresponding numbers are about 10% and 20% incorrect labels; while not as effective in the 30% noise level, SOSELETO is still operating as designed. Further evidence for this is provided by the large slopes of all three curves on the righthand side of the graph. APPENDIX E ANALYZING SVHN 0-9 TO MNIST 5-9 SOSELETO is capable of automatically pruning unhelpful instances at train time. The experiment presented in Section 3 demonstrates how SOSELETO can improve classification of MNIST 5-9 by making use of different digits from a different dataset (SVHN 0-4). To further reason about which instances are chosen as useful, we have conducted another experiment: SVHN 0-9 to MNIST 5-9. There is now a partial overlap in classes between source and target. Our findings are summarized in what follows. An immediate effect of increasing the source set, was a dramatic improvement in accuracy to 90.3%.Measuring the percentage of "good" instances (i.e. instances with weight above a certain threshold) didn't reveal a strong correlation with the labels. In Figure 3 we show this for a threshold of 0.8. As can be seen, labels 7-9 are slightly higher than the rest but there is no strong evidence of labels 5-9 being more useful than 0-4, as one might hope for. That said, a more careful examination of low-and high-weighted instances, revealed that the usefulness of an instance, as determined by SOSELETO, is more tied to its appearance: namely, whether the digit is centered, at a similar size as MNIST, the amount of blur, and rotation. In Figure 4 we show a random sample of some "good" and "bad" (i.e. high and low weights, respectively). A close look reveals that "good" instances often tend to be complete, centered, axis aligned, and at a good size (wrt MNIST sizes). Especially interesting was that, among the "bad" instances, we found about 3 − 5% wrongly labeled instances! In FIG5 we display several especially problematic instances of the SVHN, all of which are labeled as "0" in the dataset. As can be seen, some examples are very clear errors. The highly weighted instances, on the other hand, had almost no such errors.

Learning with limited training data by exploiting "helpful" instances from a rich data source.

represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR:

Question: Derivative-free optimization (DFO) using trust region methods is frequently used for machine learning applications, such as (hyper-)parameter optimization without the derivatives of objective functions known. Inspired by the recent work in continuous-time minimizers, our work models the common trust region methods with the exploration-exploitation using a dynamical system coupling a pair of dynamical processes. While the first exploration process searches the minimum of the blackbox function through minimizing a time-evolving surrogation function, another exploitation process updates the surrogation function time-to-time using the points traversed by the exploration process. The efficiency of derivative-free optimization thus depends on ways the two processes couple. In this paper, we propose a novel dynamical system, namely \ThePrev---\underline{S}tochastic \underline{H}amiltonian \underline{E}xploration and \underline{E}xploitation, that surrogates the subregions of blackbox function using a time-evolving quadratic function, then explores and tracks the minimum of the quadratic functions using a fast-converging Hamiltonian system. The \ThePrev\ algorithm is later provided as a discrete-time numerical approximation to the system. To further accelerate optimization, we present \TheName\ that parallelizes multiple \ThePrev\ threads for concurrent exploration and exploitation. Experiment based on a wide range of machine learning applications show that \TheName\ outperform a boarder range of derivative-free optimization algorithms with faster convergence speed under the same settings. Derivative-free optimization (DFO) techniques BID31, such as Bayesian optimization algorithms BID43 BID24, non-differentiable coordinate descent BID4, natural gradient method BID12 BID14, and natural evolution strategies BID41, have been widely used for black-box function optimization. DFO techniques have been viewed as one of promising solutions, when the first-order/higher-order derivatives of the objective functions are not available. For example, to train large-scale machine learning models, parameter tuning is sometimes required. The problem to find the best parameters from the high-dimensional parameter space is frequently formalized as a black-box optimization problem, as the function that maps the specific parameter settings to the performance of models is not known BID11 BID9 BID48 BID21. The evaluation of the black-box function is often computationally expensive, and there thus needs DFO algorithms to converge fast with global/local minimum guarantee. Backgrounds. To ensure the performance of DFO algorithms, a series of pioneering work has been done BID5 BID36 BID16 BID2 BID11. Especially, Powell et al. (; BID33 proposed Trust-Region methods that intends to "surrogate" the DFO solutions through exploring the minimum in the trust regions of the blackbox objective functions, where the trust regions are tightly approximated using model functions (e.g., quadratic functions or Gaussian process) via interpolation. Such two processes for exploration and exploitation are usually alternatively iterated, so as to pursue the global/local minimum BID2. With exploration and exploitation BID7, a wide range of algorithms have been proposed using trust region for DFO surrogation BID34 BID38 BID43 BID45 BID42 BID46 BID39 BID0 BID30 BID23 BID1.Technical Challenges. Though trust region methods have been successfully used for derivative-free optimization for decades, the drawbacks of these methods are still significant:• The computational and storage complexity for (convex) surrogates is extremely high. To approximate the trust regions of blackbox functions, quadratic functions BID34 BID39 and Gaussian process BID43 BID46 BID23 are frequently used as (convex) surrogates. However, fitting the quadratic functions and Gaussian process through interpolation is quite time-consuming with high sample complexity. For example, using quadratic functions as surrogates (i.e., approximation to the second-order Taylor's expansion) needs to estimate the gradient and inverse Hessian matrix BID34 BID39, where a large number of samples are required to avoid ill-conditioned inverse Hessian approximation; while the surrogate function in GP is nonconvex, which is even more sophisticated to optimize.• The convergence of trust region methods cannot be guaranteed for high-dimensional nonconvex DFO. Compared to the derivative-based algorithms such as stochastic gradient descent and accelerated gradient methods BID3 BID44, the convergence of DFO algorithms usually are not theoretically guaranteed. Jamieson et al. BID16 provided the lower bound for algorithms based on boolean-based comparison of function evaluation. It shows that DFO algorithms can converge at Ω(1/ √ T) rate in the best case (T refers to the total number of iterations), without assumptions on convexity and smoothness, even when the evaluation of black-box function is noisy. Our Intuitions. To tackle the technical challenges, we are motivated to study novel trust region methods with following properties 1. Low-complexity Quadratic Surrogates with Limited Memory. To lower the computational complexity, we propose to use quadratic functions with identity Hessian matrices as surrogates. Rather than incorporating all evaluated samples in quadratic form approximation, our algorithm only works with the most-recently evaluated sample points. In this way, the memory consumption required can be further reduced. However, the use of identity Hessian matrices for quadratic form loses the information about the distribution (e.g., Fisher information or covariance BID13) of evaluated sample points. 2. Fast Quadratic Exploration with Stochastic Hamiltonian Dynamical Systems. Though it is difficult to improve the convergence rate of the DFO algorithms in general nonconvex settings with less oracle calls (i.e., times of function evaluation), one can make the exploration over the quadratic trust region even faster. Note that exploration requires to cover a trust region rather than running on the fastest path (e.g., the gradient flow BID15) towards the minimum of trust region. In this case, there needs an exploration mechanism traversing the whole quadratic trust region in a fast manner and (asymptotically) approaching to the minimum. FIG0 illustrates the examples of exploration processes over the quadratic region via its gradient flows (i.e., gradient descent) or using Hamiltonian dynamics with gradients BID25 as well as their stochastic variants with explicit perturbation, all in the same length of time. It shows that the stochastic Hamiltonian dynamics (shown in FIG0 (d)) can well balance the needs of fast-approaching the minimum while sampling the quadratic region with its trajectories. Compared to the (stochastic) gradient flow, which leads to the convergence to the minimum in the fast manner, the stochastic Hamiltonian system are expected to well explore the quadratic trust region with the convergence kept. Inspired by theoretical convergence consequences of Hamiltonian dynamics with Quadratic form BID44 BID25, we propose to use stochastic Hamiltonian dynamical system for exploring the quadratic surrogates. 3. Multiple Quadratic Trust Regions with Parallel Exploration-Exploitation. Instead of using one quadratic cone as the surrogate, our method constructs the trust regions using multiple quadratic surrogates, where every surrogate is centered by one sample point. In this way, the information of multiple sample points can be still preserved. Further, to enjoy the speedup of parallel computation, the proposed method can be accelerated through exploring the minimum from multiple trust regions (using multiple Hamiltonian dynamical sys- Our work is inspired by the recent progress in the continuous-time convex minimizers BID44 BID15 BID47 on convex functions, where the optimization algorithms are considered as the discrete-time numerical approximation to some (stochastic) ordinary differential equations (ODEs) or dynamics, such as Itô processes for SGD algorithms BID15 or Hamiltonian systems for Nesterov's accelerated SGD BID44. We intend to first study the new ODE and dynamical system as a continuous-time DFO minimizer that addresses above three research issues. With the new ODE, we aim at proposing the discrete-time approximation as the algorithms for black-box optimization. Our Contributions. Specifically, we make following contributions. To address the three technical challenges, a continuous-time minimizer for derivative-free optimization based on a Hamiltonian system coupling two processes for exploration and exploitation respectively. Based on the proposed dynamical system, an algorithm, namely SHE 2 -Stochastic Hamiltonian Exploration and Exploitation, as a discrete-time version of the proposed dynamical system, as well as P-SHE 2 that parallelizes SHE 2 for acceleration. With the proposed algorithms, a series of experiments to evaluate SHE 2 and P-SHE 2 using real-world applications. The two algorithms outperform a wide range of DFO algorithms with better convergence. To the best of our knowledge, this work is the first to use a Hamiltonian system with coupled process for DFO algorithm design and analysis. In this section, we first review the most relevant work of trust region methods for DFO problem, then present the preliminaries of this work. The trust region algorithms can be categorized by the model functions used for surrogates. Generally, there are two types of algorithms adopted: Gaussian Process (GP) BID43 BID45 BID46 BID23 or Quadratic functions BID34 BID39 BID30 for surrogation. Blessed by the power of Bayesian nonparameteric statistics, Gaussian process can well fit the trust regions, with confidence bounds measured, using samples evaluated by the blackbox function. However, the GP-based surrogation cannot work in high dimension and cannot scale-up with large number of samples. To solved this problem, GP-based surrogation algorithms using the kernel gradients BID46 and mini-batch BID23 have been recently studied. On the other hand, the quadratic surrogation BID34 indeed approximates the trust region through interpolating the second-order Taylor expansion of the blackbox objective. With incoming points evaluated, there frequently needs to numerically estimate and adapt the inverse Hessian matrix and gradient vector, which is extremely time-consuming and sample-inefficiency (with sample BID34). Following such settings, BID39 proposed to a second-order algorithm for blackbox variational inference based on quadratic surrogation, while BID30 ) leveraged a Gaussian Mixture Model (multiple quadratic surrogations) to fit the policy search space over blackbox probabilistic distribution for policy optimization. A novel convex model generalizing the quadratic surrogation has been recently proposed to characterize the loss for structured prediction BID1. DISPLAYFORM0 In addition, some evolutionary strategies, such as Covariance Matrix Adaptation Evolution Strategy (CMA-ES) BID13 BID0, indeed behave as a sort of quadratic surrogate as well. Compared to the common quadratic surrogate, CMA-ES models the energy of blackbox function using a multivariate Gaussian distribution. For every iteration, CMA-ES draws a batch of multiple samples from the distribution, then statistically updates parameters of the distribution using the samples with blackbox evaluation. CMA-ES can be further accelerated with parallel blackbox function evaluation and has been used for hyperparameter optimization of deep learning BID22. Here, we review the Nesterov's accelerated method for quadratic function minimization. We particularly are interested in the ODE of Nesterov's accelerated method and interpret behavior of the ODE as a Hamiltonian dynamical system. Corollary 1 (ODE of Nesterov's Accelerated Method). According to BID44, the discretetime numerical format of the Nesterov's accelerated method BID27 can be viewed as an ODE as follow. Z DISPLAYFORM0 where f (X) is defined as the objective function for minimization and ∇ X f (Z(t)) refers to the gradient of the function on the point Z(t). Above ODE can converge with strongly theoretical consequences if the function f (X) is convex with some smoothness assumptions BID44. Corollary 2 (Convergence of Eq 1 over Quadratic Loss). Let's set f (X) = DISPLAYFORM1 According to the ODE analysis of Nestereov's accelerated method BID44, the ODE listed in Eq 1 converges with increasing time t at the following rate: DISPLAYFORM2 where X refers to the initial status of the ODE. The proof has been given in BID44. In this section, we present the proposed Hamiltonian system for Black-Box minimization via exploration and exploitation. Then, we introduce the algorithms and analyze its approximation to the dynamical systems. Given a black-box objective function f (X) and X ∈ R d, we propose to search the minimum of f (X) in the d-dimensional vector space R d, using a novel Hamiltonian system, derived from the ODE of Nesterov's accelerated method and Eq 1, yet without the derivative of f (X) needed. Definition 1 (Quadratic Loss Function). Given two d-dimensional vectors X and Y, we characterizes the Euclid distance between the two vectors using the function as follow. DISPLAYFORM0 where the partial derivative of Q on X should be DISPLAYFORM1 Definition 2 (Stochastic Hamiltonian Exploration and Exploitation). As was shown in Eq. 4, a Hamiltonian system is designed with following two coupled processes: exploration process X(t) and exploitation process Y (t), where t refers to the searching time. These two processes are coupled withn each other. Specifically, the exploration process X(t) in Eq 4 uses a second order ODE to track the dynamic process Y (t), while the exploiting process Y (t) always memorizes the minimum point (i.e., X(τ)) that have been reached by X(t) from time 0 to t. DISPLAYFORM2 where DISPLAYFORM3 indicates the fastest direction to track Y (t) from X(t); and the perturbation term ζ(t) referring to an unbiased random noise with controllable bound β t ← −(α t + 3/t) and γ t ← 3/t; 5: DISPLAYFORM4 6: DISPLAYFORM5 else 12: DISPLAYFORM6 end if 14: end for 15: return Y T; ζ(t) 2 ≤ would help the system escape from an unstable stationary point in even shorter time. In the above dynamical system, we treat Y (t) as the minimizer of the black-box function f (X). 2 approximates the black-box function f (X) using a simple yet effective quadratic function, then leverages the ODE listed in Eq 1 to approximate the minimum with the quadratic function. With the new trajectories traversed by Eq 1, the quadratic function would be updated. Through repeating such surrogation-approximation-updating procedures, the ODE continuously tracks the time-dependent evolution of quadratic loss functions and finally stops at a stationary point when the quadratic loss functions is no longer updated (even with new trajectories traversed). Remark 1. We can use the analytical BID44 to interpret the dynamical system (in Eq 4) as an adaptive perturbated dynamical system that intends to minimize the Euclid distance between X(t) and Y (t) at each time t. The memory complexity of this continuous-time minimizer is O, where a Markov process Y (t) is to used to memorize the status quo of local minimum during exploration and exploitation. Theorem 1 (Convergence of SHE 2 Dynamics). Let's denote x * as a possible local minimum point of the landscape function f (x). We have as t → ∞, with high probability, that X(t) → x *, where X(t) is the solution to.Please refer to the Lemma 1 and Lemma 2 in the Appendix for the proof of above theorems. We will discuss the rate of convergence, when introducing SHE 2 algorithm as a discrete-time approximation to SHE 2.3.2 SHE 2: ALGORITHM DESIGN AND ANALYSIS Given a black-box function f (x) and a sequence of non-negative step-size α t (t=0, 1, 2, . . ., T), which is small enough, as well as the scale of perturbation ε, we propose to implement SHE 2 as Algorithm 1. The output of algorithm Y T refers to the value of Y t in the last iteration (i.e., the t th iteration). The whole algorithm only uses the evaluation of function f (x) for comparisons, without computing its derivatives. In each iteration, only the variable Y t is dedicated to memorize the local minimum in the sequence of X 1, X 2,..., X t. Thus the memory complexity of SHE 2 is O.In terms of convergence, Jamieson et al BID16 provided an universal lower bound on the convergence rate of DFO based on the "boolean-valued" comparison of (noisy) function evaluation. SHE 2 should enjoy the same convergence rate Ω(1/ √ T) without addressing any further assumptions. Here, we would demonstrate that the proposed algorithm behaves as a discrete-time approximation to the dynamical systems of X(t) and Y (t) addressed in Eq 4, while as α t → 0 the sequence of X t and Y t (for 1 ≤ t ≤ T) would converge to the behavior of continuous-time minimizer -coupled processes X(t) and Y (t).Given an appropriate constant step-size α t → 0 for t = 1, 2..., T, we can rewrite the the sequences X t described in lines 4-7 of Algorithm 1 as the following Stochastic Differential Equation (SDE) of X ω (t) with the random noise ω(t): DISPLAYFORM0 where ζ(t) refers to the continuous-time dynamics of sequence ζ 1, ζ 2,..., ζ T and |ζ(t)| 2 = ε for every time t. Through combining above two ODEs and Lemma 1, we can obtain the SDE of X(t) based on the perturbation ζ(t) as: DISPLAYFORM1 The sequence Y t (t=0, 1, 2, · · · T) always exploits the minimum point that has been already found by X t at time t. Thus, we can consider Y t is the discrete-time of Y (t) that exploits the minimum traversed by X(t). In this way, we can consider the coupled sequences of X t and Y t (for 1 ≤ t ≤ T) as the discrete-time form of the proposed dynamical system with X(t) and Y (t). To enjoy the speedup of parallel computation, we propose a new Hamiltonian dynamical system with a set of ODEs that leverage multiple pairs of coupled processes for exploration and exploitation in parallel. Then, we present the algorithm design as a discrete-time approximation to the ODEs. 2 DYNAMICAL SYSTEM Given a black-box objective function f (X) and X ∈ R d, we propose to search the minimum of f (X) in the d-dimensional vector space R d, using following systems. Definition 3 (Parallel Stochastic Hamiltonian Exploration and Exploitation). As was shown in Eq. 7, a Hamiltonian system is designed with N pairs of coupled exploration-exploitation processes: X i (t) and Y i (t) for 1 ≤ i ≤ N that explores and exploits the minimum in-parallel from N (random/unique) starting points, and an overall exploitation process Y (t) memorizing the local minimum traversed by the all N pairs of coupled processes. Specifically, for each pair of coupled processes, a new surrogation model Q δ (X i (t), Y i (t), Y (t)) has been proposed to measure the joint distance from X i (t) to Y i (t) and Y (t) respectively, where δ > 0 refers to a trade-off factor weighted-averaging the two distances. DISPLAYFORM0 where DISPLAYFORM1 indicates the fastest direction to track Y i (t) and Y (t), jointly, from X(t). In the above dynamical system, we treat Y (t) as the minimizer of the black-box function f (X). Remark 2. We understand the dynamical system listed in Eq 7 as a perturbated dynamical system with multiple state variables, where all variables are coupled to search the minimum of f (X) through X i (t) (for 1 ≤ i ≤ N). The memory complexity of this continuous-time minimizer is O(N), where every Markov process Y i (t) is to used to memorize the status quo of local minimum traversed by the corresponding processes. 6: for t = 1, 2..., T do 7:for j = 1, 2, 3,..., N in Parallel do 8:/* X(t) update for the j th SHE 2 thread*/ 9:β t ← −(α t + 3/t) and γ t ← 3/t;10: DISPLAYFORM0 11: DISPLAYFORM1 12: DISPLAYFORM2 13: DISPLAYFORM3 DISPLAYFORM4 where ζ j (t) refers to the continuous-time dynamics of sequence ζ j 1, ζ j 2,..., ζ j T and |ζ j (t)| 2 = ε for every time t. Through combining above three ODEs and Eq. 8, we can obtain the ODE of X(t) as: DISPLAYFORM5 Using same the settings, we can conclude that X j t would have similar behavior as X i (t) (for 1 ≤ i ≤ N in Eq 7). Thus, Algorithm 2 can be viewed as a discrete-time approximation of dynamical systems in Eq 7. Since the sequence Y t always exploits the minimum point that has been found by all N threads at every time t, we can use the algorithm output Y T as the minimizer of f (x). The proposed P-SHE 2 algorithm can be viewed as a particle swarm optimizer Kennedy FORMULA1 with inverse-scale step-size settings. Compared to Particle Swarm, which usually adopts constant stepsize settings (i.e., α t, β t and γ t are fixed as a constant value), P-SHE 2 proposes to use a small α t, while setting β t = −(α t + 3/t) and γ t = 3/t for each (the t th) iteration. Such settings help the optimizer approximates to the Nesterov's scheme, so as to enjoy faster convergence speed, under certain assumption. In terms of contribution, our research made as yet an rigorous analysis for Particle Swarm through linking it to to Nesterov's scheme; BID44. We provide three sets of experiments to validate our algorithms. In the first set of experiments, we demonstrate the performance of SHE 2 and P-SHE 2 to minimize two non-convex functions through the comparisons to a set of DFO optimizers, including Gaussian Process optimization algorithms (GP-UCB) , Powell's BOBYQA methods BID35, Limited Memory-BFGS-B (L-BFGS) BID50, Covariance Matrix Adaptation Evolution Strategy (CMA-ES) BID13, and Particle Swarm optimizer (PSO) BID18. For the second set of experiments, we use the same set of algorithms to train logistic regression BID20 and support vector machine BID6 classifiers, on top of benchmark datasets, for supervised learning tasks. In the third set, we use P-SHE 2 to optimize the hyper-parameters of ResNet-50 for the performance tuning on Flower 102 and MIT Indoor 67 benchmark datasets under transfer learning settings. Figure 2 presents the performance comparison between P-SHE 2, SHE 2 and the baseline algorithms using two 2D benchmark nonconvex functions-Franke's function and Peaks function. Figure 2.a and c present the landscape of these two functions, while Figure 2.b and d present the performance evaluation of P-SHE 2, SHE 2 and baseline algorithms on these two functions. All these algorithms are tuned with best parameters and evaluated for 20 times, while averaged performance is presented. Specifically, we illustrate how these algorithms would converge with increasing number of iterations. Obviously, on Franke's function, only P-SHE 2, i.e., the P-SHE 2 algorithm with N = 10 search threads, CMA-ES, GP-UCB algorithms and BOBYQA converge to the global minimum, while the rest of algorithms, including SHE 2 and PSO, converge to the local minimum. Though P-SHE 2 needs more iterations to converge to the global minimum, its per iteration time consumption is significantly lower than the other three convergeable algorithms (shown in Figure 2.e). The same comparison can be also observed from the comparison using Peaks function (in Figures 2.b and 2 .c, only CMA-ES and P-SHE 2 converge to global minimum in the given number of iterations). Compared P-SHE 2 to PSO, they both use 10 search threads with the same computational/memory complexity, while P-SHE 2 converges much faster than PSO. The same phenomena can be also observed from the comparison between SHE 2 and PSO, both of which search with single thread. We can suggest that the adaptive step-size settings inherent from Nesterove's scheme accelerate the convergence speed. We use above algorithms to train logistic regression (LR) and SVM classifiers using Iris (4 features, 3 classes and 150 instances), Breast (32 features, 2 classes and 569 instances) and Wine (13 features, 3 classes and 178 instances) datasets. We treat the loss functions of logistic regression and SVM as black-box functions and parameters (e.g., projection vector β for logistic regression) as optimization outcomes. Note that the number of parameters for multi-class (#class ≥ 3) classification is #class × #f eatures, e.g., 39 for wine data. We don't include GP-UCB in the comparison, as it is extremely time-consuming to scale-up in high-dimensional settings. FIG2 demonstrates how loss function could be minimized by above algorithms with iterations by iterations. For both classifiers on all three datasets, P-SHE 2-the P-SHE 2 algorithms with N = 100 search threads, outperforms all rest algorithms with the most significant loss reduction and the best convergence performance. We also test the accuracy of trained classifiers using the testing datasets. Table. 1 shows that both classifiers trained by P-SHE 2 enjoys the best accuracy among all above DFO algorithms and the accuracy is comparable to those trained using gradientbased optimizers. All above experiments are carried out under 10-folder cross-validation. Note that the accuracy of the classifiers trained by P-SHE 2 is closed to/or even better than some fine-tuned gradient-based solutions BID8 BID40 ). To test the performance of P-SHE 2 for derivative-free optimization with noisy black-box function evaluation, We use P-SHE 2 to optimize the hyper-parameter of ResNet-50 networks for Flower BID29 and MIT Indoor 67 classification BID37 ) tasks. The two networks are pre-trained using ImageNet BID19 and Place365 datasets BID49, respectively. Specifically, we design a black-box function to package the training procedure of the ResNet-50, where 12 continuous parameters, including the learning rate of procedure, type of optimizers (after simple discretization), the probabilistic distribution of image pre-processing operations for randomized data augmentation and so on, are interfaced as the input of the function while the validation loss of the network is returned as the output. We aim at searching the optimal parameters with the lowest validation loss. The experiments are all based on a Xeon E5 cluster with many available TitanX, M40x8, and 1080Ti GPUs. Our experiments compare P-SHE 2 with a wide range of solvers and hyper-parameter tuning tools, including PSO, CMA-ES BID22, GP-UCB BID17 and BOBYQA under the same pre-training/computing settings. Specifically, we adopt the vanilla implementation of GP-UCB and BOBYQA (with single search thread), while P-SHE 2, PSO and CMA-ES are all with 10 search threads for parallel optimization. The experimental show that all these algorithms can well optimize the hyer-parameters of ResNet for the better performance under the same settings, while P-SHE 2 has ever searched the hyperparameters with the lowest validation loss in our experiments (shown in FIG4). Due to the restriction of PyBOBYQA API, we can only provide the function evaluation of the final solution obtained by BOBYQA as a flatline in FIG4. In fact, P-SHE 2, PSO and CMA-ES may spend more GPU hours than GP-UCB and BOBYQA due to the parallel search. For the fair comparison, we also evaluate GP-UCB and BOBYQA with more than 100 iterations til the convergence, where GP-UCB can achieve 0.099854 validation error (which is comparable to the three parallel solvers) for Flower 102 task. Note that we only claim that P-SHE 2 can be used for hyerparameter optimization with decent performance. We don't intend to state that P-SHE 2 is the best for hyerparameter tuning, as the performance of the three parallel solvers are sometimes randon and indeed close to each other. In this paper, we present SHE 2 and P-SHE 2 -two derivative-free optimization algorithms that leverage a Hamiltonian exploration and exploitation dynamical systems for black-box function optimization. Under mild condition SHE 2 algorithm behaves as a discrete-time approximation to a Nestereov's scheme ODE BID44 over the quadratic trust region of the blackbox function. Moreover, we propose P-SHE 2 to further accelerate the minimum search through parallelizing multiple SHE 2 -alike search threads with simple synchronization. Compared to the existing trust region methods, P-SHE 2 uses multiple quadratic trust regions with multiple (coupled) stochastic Hamiltonian dynamics to accelerate the exploration-exploitation processes, while avoiding the needs of Hessian matrix estimation for quadratic function approximation. Instead of interpolating sampled points in one quadratic function, P-SHE 2 defacto constructs one quadratic surrogate (with identity Hessian) for each sampled point and leverages parallel search threads with parallel black-box function evaluation to boost the performance. Experiment show that P-SHE 2 can compete a wide range of DFO algorithms to minimize nonconvex benchmark functions, train supervised learning models via parameter optimization, and fine-tune deep neural networks via hyperparameter optimization. 2 DYNAMICAL SYSTEM Our goal is to show that in the system we have X(t) → x * as t → ∞, where x * is a local minimum point of the landscape function f (x). Definition 4. We say the point x * is a local minimum point of the function f (x) if and only if f (x *) ≤ f (x) for any x ∈ U (x *), where U (x *) which is any open neighborhood around the point x *.Let us first remove the noise ζ(t) in our system. Thus we obtain the following deterministic dynamical system (X 0 (t), Y 0 (t)): DISPLAYFORM0 In the equations (a) and (b) of, the pair of processes (X 0 (t), Y 0 (t)) is a pair of coupled processes. In the next Lemma, we show that X 0 (t) converges to the minimum point of f (x) along the trajectory of X 0 (t) as t → ∞.Lemma 1. For the deterministic system, we have that DISPLAYFORM1 Proof. SetẊ 0 (t) = P (t), we can write equation (a) in in Hamiltonian form DISPLAYFORM2 Set H(X, Y, P) = P 2 2 + Q(X, Y). Then we have DISPLAYFORM3 As we have DISPLAYFORM4, we see from that we have DISPLAYFORM5 Notice that by our construction part (b) of the coupled process, we have f (Y 0 (t)) ≤ f (X 0 (s)) for 0 ≤ s ≤ t. If X 0 (t) − Y 0 (t) = 0, then (X 0 (t) − Y 0 (t)) ·Ẏ 0 (t) = 0. If X 0 (t) − Y 0 (t) = 0, then we see that Y 0 (t) = X 0 (s 0) for some 0 ≤ s 0 < t, and f (X 0 (s 0)) < f (X 0 (t)). By continuity of the trajectory of X 0 (t) as well as the function f (x), we see that in this caseẎ 0 (t) = 0, so that (X 0 (t) − Y 0 (t)) ·Ẏ 0 (t) = 0. Thus we see that actually gives d dt H(X 0 (t), Y 0 (t), P (t)) = − 3 t P (t) 2 2 ≤ 0.From here, we know that H(X 0 (t), Y 0 (t), P (t)) keeps decaying until P (t) → 0 and X 0 (t) − Y 0 (t) 2 → 0, as desired. Since f (Y 0 (t)) ≤ min 0≤s≤t f (X 0 (s)), Lemma 1 tells us that as t → ∞, the deterministic process X 0 (t) in approaches the minimum of f along the trajectory traversed by itself. Let us now add the noise ζ(t) to part (a) of, so that we come back to our original system. We would like to argue that with the noise ζ(t), we actually have lim t→∞ min 0≤s≤t f (X(s)) = f (x *), and thus X(t) → x * when t → ∞ as desired. Lemma 2. For the process X(t) in part (a) of the system, we have X(t) → x * as t → ∞, where x * is a local minimum of the landscape function f (x).Proof. We first notice that we have ζ(t) = ε ξ t ξ t 2, where ξ t ∼ N (0, I) is a sequence of i.i.d.normal, so that ζ(t) 2 = ε, Eζ(t) = 0. Viewing as a small random perturbation (see BID10, Chapter 2, Section 1)) of the system we know that for any δ > 0 fixed, we have DISPLAYFORM6 as ε → 0. From here we know that the process X(t) behaves close to X 0 (t) with high probability, so that by Lemma 1 we know that with high probability we have lim t→∞ f (X(t)) − min 0≤s≤t f (X(s)) = 0.Our next step is to improve the above asymptotic to X(t) → x * as t → ∞. Comparing with, we see that it suffices to show DISPLAYFORM7 To demonstrate, we note that when t is large, we can ignore in the damping term 3 tẊ (t) and obtain a friction-less dynamics f (X(τ)).Combining Lemma 1, and ∂ ∂X Q(X, Y) = X − Y we further see that the term ∂ ∂X Q(X, Y) also contribute little in. Thus part (a) of reduces to a very simple equation DISPLAYFORM8 Equation FORMULA1 enables the process X(t) to explore locally in an ergodic way its neighborhood points, so that if FORMULA1 is not valid, then X(t + dt) will explore a nearby point at which f (X(t + dt)) is less that min 0≤s≤t f (X(t)), and thus will move to that point. This leads to a further decay in the value of f (X(t)), which demonstrates that in he limit t → ∞ we must have, and the Lemma concludes. Summarizing, we have the Theorem 1. Here we provide a short discussion on the convergence rate of the algorithm SHE2. In the previous appendix we have demonstrated that the system converges via two steps. Step 1 in Lemma 1 shows that the differential equation modeling Nesterov's accelerated gradient descent (see BID44) helps the process X(t) to "catch up" with the minimum point Y (t) on its path. Step 2 in Lemma 2 shows that when t → ∞ the noise term ζ(t) helps the process X(t) to reach local

Represent the short summary for finding the corresponding scientific article (long)

a new derivative-free optimization algorithms derived from Nesterov's accelerated gradient methods and Hamiltonian dynamics

Represent the TLDR for finding the corresponding scientific article

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the scientific article to find a short TLDR

Question: Binarized Neural networks (BNNs) have been shown to be effective in improving network efficiency during the inference phase, after the network has been trained. However, BNNs only binarize the model parameters and activations during propagations. Therefore, BNNs do not offer significant efficiency improvements during training, since the gradients are still propagated and used with high precision. We show there is no inherent difficulty in training BNNs using "Binarized BackPropagation" (BBP), in which we also binarize the gradients. To avoid significant degradation in test accuracy, we simply increase the number of filter maps in a each convolution layer. Using BBP on dedicated hardware can potentially significantly improve the execution efficiency (\emph{e.g.}, reduce dynamic memory footprint, memory bandwidth and computational energy) and speed up the training process with an appropriate hardware support, even after such an increase in network size. Moreover, our method is ideal for distributed learning as it reduces the communication costs significantly (e.g., by ~32). Using this method, we demonstrate a minimal loss in classification accuracy on several datasets and topologies.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Binarized Back-Propagation all you need for completely binarized training is to is to inflate the size of the network

Inference in large Transformers is expensive due to the self-attention in multiple layers. We show a simple decomposition technique can yield a faster, low memory-footprint model that is just as accurate of the original models.


Question: Represent the scientific article to find a short TLDR

Question: The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propagation. Understanding the theoretical properties of untrained random networks is key to identifying which deep networks may be trained successfully as recently demonstrated by who showed that for deep feedforward neural networks only a specific choice of hyperparameters known as the `edge of chaos' can lead to good performance. We complete this analysis by providing quantitative showing that, for a class of ReLU-like activation functions, the information propagates indeed deeper for an initialization at the edge of chaos. By further extending this analysis, we identify a class of activation functions that improve the information propagation over ReLU-like functions. This class includes the Swish activation, $\phi_{swish}(x) = x \cdot \text{sigmoid}(x)$, used in , and. This provides a theoretical grounding for the excellent empirical performance of $\phi_{swish}$ observed in these contributions. We complement those previous by illustrating the benefit of using a random initialization on the edge of chaos in this context. Deep neural networks have become extremely popular as they achieve state-of-the-art performance on a variety of important applications including language processing and computer vision; see, e.g., BID8. The success of these models has motivated the use of increasingly deep networks and stimulated a large body of work to understand their theoretical properties. It is impossible to provide here a comprehensive summary of the large number of contributions within this field. To cite a few relevant to our contributions, BID11 have shown that neural networks have exponential expressive power with respect to the depth while BID14 obtained similar using a topological measure of expressiveness. We follow here the approach of BID14 and BID16 by investigating the behaviour of random networks in the infinite-width and finite-variance i.i.d. weights context where they can be approximated by a Gaussian process as established by BID10 and BID9.In this paper, our contribution is two-fold. Firstly, we provide an analysis complementing the of BID14 and BID16 and show that initializing a network with a specific choice of hyperparameters known as the'edge of chaos' is linked to a deeper propagation of the information through the network. In particular, we establish that for a class of ReLU-like activation functions, the exponential depth scale introduced in BID16 is replaced by a polynomial depth scale. This implies that the information can propagate deeper when the network is initialized on the edge of chaos. Secondly, we outline the limitations of ReLU-like activation functions by showing that, even on the edge of chaos, the limiting Gaussian Process admits a degenerate kernel as the number of layers goes to infinity. Our main gives sufficient conditions for activation functions to allow a good'information flow' through the network (Proposition 4) (in addition to being non-polynomial and not suffering from the exploding/vanishing gradient problem). These conditions are satisfied by the Swish activation φ swish (x) = x · sigmoid(x) used in BID4, BID2 and BID15. In recent work, BID15 used automated search techniques to identify new activation functions and found experimentally that functions of the form φ(x) = x · sigmoid(βx) appear to perform indeed better than many alternative functions, including ReLU. Our paper provides a theoretical grounding for these . We also complement previous empirical by illustrating the benefits of an initialization on the edge of chaos in this context. All proofs are given in the Supplementary Material. We use similar notations to those of BID14 and BID9. Consider a fully connected random neural network of depth L, widths (N l) 1≤l≤L, weights W 2 ) denotes the normal distribution of mean µ and variance σ 2. For some input a ∈ R d, the propagation of this input through the network is given for an activation function φ: R → R byThroughout the paper we assume that for all l the processes y l i are independent (across i) centred Gaussian processes with covariance kernels κ l and write accordingly y DISPLAYFORM0. This is an idealized version of the true processes corresponding to choosing N l−1 = +∞ (which implies, using Central Limit Theorem, that y l i (a) is a Gaussian variable for any input a). The approximation of y l i by a Gaussian process was first proposed by BID12 in the single layer case and has been recently extended to the multiple layer case by BID9 and BID10. We recall here the expressions of the limiting Gaussian process kernels. For any input DISPLAYFORM1 where F φ is a function that depends only on φ. This gives a recursion to calculate the kernel κ l; see, e.g., BID9 for more details. We can also express the kernel κ l in terms of the correlation c l ab in the l th layer used in the rest of this paper DISPLAYFORM2 where q l−1 a DISPLAYFORM3 b, is the variance, resp. correlation, in the (l − 1) th layer and Z 1, Z 2 are independent standard Gaussian random variables. when it propagates through the network. q l a is updated through the layers by the recursive formula q l a = F (q l−1 a), where F is the'variance function' given by DISPLAYFORM4 Throughout the paper, Z, Z 1, Z 2 will always denote independent standard Gaussian variables. We analyze here the limiting behaviour of q L a and c L a,b as the network depth L goes to infinity under the assumption that φ has a second derivative at least in the distribution sense 1. From now onwards, we will also assume without loss of generality that c 1 ab ≥ 0 (similar can be obtained straightforwardly when c 1 ab ≤ 0). We first need to define the Domains of Convergence associated with an activation function φ. Remark: Typically, q in Definition 1 is a fixed point of the variance function defined in equation 2. Therefore, it is easy to see that for any (σ b, σ w) such that F is increasing and admits at least one fixed point, we have K φ,corr (σ b, σ w) ≥ q where q is the minimal fixed point; i.e. q:= min{x : F (x) = x}. Thus, if we re-scale the input data to have q 1 a ≤ q, the variance q l a converges to q. We can also re-scale the variance σ w of the first layer (only) to assume that q 1 a ≤ q for all inputs a. The next gives sufficient conditions on (σ b, σ w) to be in the domains of convergence of φ. DISPLAYFORM0 DISPLAYFORM1 The proof of Proposition 1 is straightforward. We prove that sup F (x) = σ 2 w M φ and then apply the Banach fixed point theorem; similar ideas are used for C φ,δ.Example: For ReLU activation function, we have M ReLU = 2 and C ReLU,δ ≤ 1 for any δ > 0. almost surely and the outputs of the network are constant functions. FIG3 illustrates this behaviour for d = 2 for ReLU and Tanh using a network of depth L = 10 with N l = 100 neurons per layer. The draws of outputs of these networks are indeed almost constant. To refine this convergence analysis, BID16 established the existence of q and c such that |q l a −q| ∼ e −l/ q and |c l ab −1| ∼ e −l/ c when fixed points exist. The quantities q and c are called'depth scales' since they represent the depth to which the variance and correlation can propagate without being exponentially close to their limits. More precisely, if we write DISPLAYFORM0 ] then the depth scales are given by r = − log(α) −1 and c = − log(χ 1) −1. The equation χ 1 = 1 corresponds to an infinite depth scale of the correlation. It is called the edge of chaos as it separates two phases: an ordered phase where the correlation converges to 1 if χ 1 < 1 and a chaotic phase where χ 1 > 1 and the correlations do not converge to 1. In this chaotic regime, it has been observed in BID16 that the correlations converge to some random value c < 1 when φ(x) = Tanh(x) and that c is independent of the correlation between the inputs. This means that very close inputs (in terms of correlation) lead to very different outputs. Therefore, in the chaotic phase, the output function of the neural network is non-continuous everywhere. Definition 2. For (σ b, σ w) ∈ D φ,var, let q be the limiting variance 2. The Edge of Chaos, hereafter EOC, is the set of values of (σ b, σ w) satisfying DISPLAYFORM1 To further study the EOC regime, the next lemma introduces a function f called the'correlation function' simplifying the analysis of the correlations. It states that the correlations have the same asymptotic behaviour as the time-homogeneous dynamical system c DISPLAYFORM2 The condition on φ in Lemma 1 is violated only by activation functions with exponential growth (which are not used in practice), so from now onwards, we use this approximation in our analysis. Note that being on the EOC is equivalent to (σ b, σ w) satisfying f = 1. In the next section, we analyze this phase transition carefully for a large class of activation functions. (as we will see later EOC = {(0, √ 2)} for ReLU). Unlike the output in FIG3, this output displays much more variability. However, we will prove here that the correlations still converges to 1 even in the EOC regime, albeit at a slower rate. We consider activation functions φ of the form: φ(x) = λx if x > 0 and φ(x) = βx if x ≤ 0. ReLU corresponds to λ = 1 and β = 0. For this class of activation functions, we see (Proposition 2) that the variance is unchanged (q l a = q 1 a) on the EOC, so that q does not formally exist in the sense that the limit of q l a depends on a. However, this does not impact the analysis of the correlations. Proposition 2. Let φ be a ReLU-like function with λ and β defined above. Then for any σ w < 2 λ 2 +β 2 and DISPLAYFORM0 )} and, on the EOC, F (x) = x for any x ≥ 0.This class of activation functions has the interesting property of preserving the variance across layers when the network is initialized on the EOC. However, we show in Proposition 3 below that, even in the EOC regime, the correlations converge to 1 but at a slower rate. We only present the for ReLU but the generalization to the whole class is straightforward. Example: ReLU: The EOC is reduced to the singleton (σ FORMULA4 also performed a similar analysis by using the "Scaled Exponential Linear Unit" activation (SELU) that makes it possible to center the mean and normalize the variance of the post-activation φ(y). The propagation of the correlation was not discussed therein either. In the next , we present the correlation function corresponding to ReLU networks. This was first obtained in BID0.We present an alternative derivation of this and further show that the correlations converge to 1 at a polynomial rate of 1/l 2 instead of an exponential rate. Proposition 3 (ReLU kernel). Consider a ReLU network with parameters (σ DISPLAYFORM1 We now introduce a set of sufficient conditions for activation functions which ensures that it is then possible to tune (σ b, σ w) to slow the convergence of the correlations to 1. This is achieved by making the correlation function f sufficiently close to the identity function. Proposition 4 (Main Result). Let φ be an activation function. Assume that (i) φ = 0, and φ has right and left derivatives in zero and φ (0 +) = 0 or φ (0 −) = 0, and there DISPLAYFORM0, the function F with parameters (σ b, σ w) ∈ EOC is non-decreasing and lim σ b →0 q = 0 where q is the minimal fixed point of F, q:= inf{x : DISPLAYFORM1 Note that ReLU does not satisfy the condition (ii) since the EOC in this case is the singleton (σ 2 b, σ 2 w) =. The of Proposition 4 states that we can make f (x) close to x by considering σ b → 0. However, this is under condition (iii) which states that lim σ b →0 q = 0. Therefore, practically, we cannot take σ b too small. One might wonder whether condition (iii) is necessary for this to hold. The next lemma shows that removing this condition in a useless class of activation functions. The next proposition gives sufficient conditions for bounded activation functions to satisfy all the conditions of Proposition 4. DISPLAYFORM2, xφ(x) > 0 and xφ (x) < 0 for x = 0, and φ satisfies (ii) in Proposition 4. Then, φ satisfies all the conditions of Proposition 4.The conditions in Proposition 5 are easy to verify and are, for example, satisfied by Tanh and Arctan. We can also replace the assumption "φ satisfies (ii) in Proposition 4" by a sufficient condition (see Proposition 7 in the Supplementary Material). Tanh-like activation functions provide better information flow in deep networks compared to ReLU-like functions. However, these functions suffer from the vanishing gradient problem during back-propagation; see, e.g., BID13 and BID7. Thus, an activation function that satisfies the conditions of Proposition 4 (in order to have a good 'information flow') and does not suffer from the vanishing gradient issue is expected to perform better than ReLU. Swish is a good candidate. DISPLAYFORM3 1+e −x satisfies all the conditions of Proposition 4.It is clear that Swish does not suffer from the vanishing gradient problem as it has a gradient close to 1 for large inputs like ReLU. FIG15 (a) displays f for Swish for different values of σ b. We see that f is indeed approaching the identity function when σ b is small, preventing the correlations from converging to 1. FIG15 (b) displays a draw of the output of a neural network of depth 30 and width 100 with Swish activation, and σ b = 0.2. The outputs displays much more variability than the ones of the ReLU network with the same architecture. We present in TAB0 some values of (σ b, σ w) on the EOC as well as the corresponding limiting variance for Swish. As condition (iii) of Proposition 4 is satisfied, the limiting variance q decreases with σ b. Other activation functions that have been shown to outperform empirically ReLU such as ELU BID1 ), SELU BID6 ) and Softplus also satisfy the conditions of Proposition 4 (see Supplementary Material for ELU). The comparison of activation functions satisfying the conditions of Proposition 4 remains an open question. We demonstrate empirically our on the MNIST dataset. In all the figures below, we compare the learning speed (test accuracy with respect to the number of epochs/iterations) for different activation functions and initialization parameters. We use the Adam optimizer with learning rate lr = 0.001. The Python code to reproduce all the experiments will be made available on-line.. In Figure 5, we compare the learning speed of a Swish network for different choices of random initialization. Any initialization other than on the edge of chaos in the optimization algorithm being stuck eventually at a very poor test accuracy of ∼ 0.1 as the depth L increases (equivalent to selecting the output uniformly at random). To understand what is happening in this case, let us recall how the optimization algorithm works. Let is the output of the network, and is the categorical cross-entropy loss. In the ordered phase, we know that the output converges exponentially to a fixed value (same value for all X i), thus a small change in w and b will not change significantly the value of the loss function, therefore the gradient is approximately zero and the gradient descent algorithm will be stuck around the initial value. from the vanishing gradient problem. Consequently, we expect Tanh to perform better than ReLU for shallow networks as opposed to deep networks, where the problem of the vanishing gradient is not encountered. Numerical confirm this fact. FIG18 shows curves of validation accuracy with confidence interval 90% (30 simulations). For depth 5, the learning algorithm converges faster for Tanh compared to ReLu. However, for deeper networks (L ≥ 40), Tanh is stuck at a very low test accuracy, this is due to the fact that a lot of parameters remain essentially unchanged because the gradient is very small. We have complemented here the analysis of BID16 which shows that initializing networks on the EOC provides a better propagation of information across layers. In the ReLU case, such an initialization corresponds to the popular approach proposed in BID3. However, even on the EOC, the correlations still converge to 1 at a polynomial rate for ReLU networks. We have obtained a set of sufficient conditions for activation functions which further improve information propagation when the parameters (σ b, σ w) are on the EOC. The Tanh activation satisfied those conditions but, more interestingly, other functions which do not suffer from the vanishing/exploding gradient problems also verify them. This includes the Swish function used in BID4, BID2 and promoted in BID15 but also.Our have also interesting implications for Bayesian neural networks which have received renewed attention lately; see, e.g., Hernandez-Lobato & Adams FORMULA4 and BID9. They show that if one assigns i.i.d. Gaussian prior distributions to the weights and biases, the ing prior distribution will be concentrated on close to constant functions even on the EOC for ReLU-like activation functions. To obtain much richer priors, our indicate that we need to select not only parameters (σ b, σ w) on the EOC but also an activation function satisfying Proposition 4. We provide in the supplementary material the proofs of the propositions presented in the main document, and we give additive theoretical and experimental . For the sake of clarity we recall the propositions before giving their proofs. A.1 CONVERGENCE TO THE FIXED POINT: PROPOSITION 1 DISPLAYFORM0 Proof. To abbreviate the notation, we use q l:= q l a for some fixed input a. Convergence of the variances: We first consider the asymptotic behaviour of q l = q l a. Recall that q l = F (q l−1) where, DISPLAYFORM1 The first derivative of this function is given by: DISPLAYFORM2 where Using the condition on φ, we see that for σ DISPLAYFORM0, the function F is a contraction mapping, and the Banach fixed-point theorem guarantees the existence of a unique fixed point q of F, with lim l→+∞ q l = q. Note that this fixed point depends only on F, therefore, this is true for any input a, and K φ,var (σ b, σ w) = ∞.Convergence of the covariances: Since M φ < ∞, then for all a, b ∈ R d there exists l 0 such that, for all l > l 0, | q l a − q l b | < δ. Let l > l 0, using Gaussian integration by parts, we have DISPLAYFORM1 We cannot use the Banach fixed point theorem directly because the integrated function here depends on l through q l. For ease of notation, we write c l:= c l ab, we have DISPLAYFORM2 l is a Cauchy sequence and it converges to a limit c ∈.At the limit DISPLAYFORM3 The derivative of this function is given by DISPLAYFORM4 By assumption on φ and the choice of σ w, we have sup x |f (x)| < 1, so that f is a contraction, and has a unique fixed point. Since f = 1, c = 1. The above is true for any a, b, therefore, DISPLAYFORM5 As an illustration we plot in FIG3 the variance for three different inputs with (σ b, σ w) =, as a function of the layer l. In this example, the convergence for Tanh is faster than that of ReLU. DISPLAYFORM6 where u 2 (x):= xZ 1 + √ 1 − x 2 Z 2. The first term goes to zero uniformly in x using the condition on φ and Cauchy-Schwartz inequality. As for the second term, it can be written as DISPLAYFORM7 again, using Cauchy-Schwartz and the condition on φ, both terms can be controlled uniformly in x by an integrable upper bound. We conclude using the Dominated convergence. Proposition 2. Let φ be a ReLU-like function with λ and β defined above. Then for any σ w < 2 λ 2 +β 2 and DISPLAYFORM0 )} and, on the EOC, F (x) = x for any x ≥ 0.Proof. We write q l = q l a throughout the proof. Note first that the variance satisfies the recursion: DISPLAYFORM1 For all σ w < 2 λ 2 +β 2, q = σ Proof. In this case the correlation function f is given by DISPLAYFORM2 • Let x ∈, note that f is differentiable and satisfies, DISPLAYFORM3 which is also differentiable. Simple algebra leads to DISPLAYFORM4 Since arcsin (x) = 1 √ 1−x 2 and f = 1/2, DISPLAYFORM5 We conclude using the fact that arcsin = x arcsin + √ 1 − x 2 and f = 1.• We first derive a Taylor expansion of f near 1. Consider the change of variable x = 1 − t 2 with t close to 0, then DISPLAYFORM6 we obtain that DISPLAYFORM7 DISPLAYFORM8 l < c l+1 then by taking the image by f (which is increasing because f ≥ 0) we have that c l+1 < c l+2, and we know that c 1 = f (c 0) ≥ c 0, so by induction the sequence c l is increasing, and therefore it converges (because it is bounded) to the fixed point of f which is 1. Now let γ l:= 1 − c l ab for a, b fixed. We note s = 2 √ 2 3π, from the series expansion we have that γ l+1 = γ l − sγ DISPLAYFORM9 DISPLAYFORM10 Assume that (i) φ = 0, and φ has right and left derivatives in zero and at least one of them is different from zero (φ (0 +) = 0 or φ (0 −) = 0), and there exists K > 0 such that DISPLAYFORM11, the function F with parameters (σ b, σ w,EOC) is non-decreasing and lim σ b →0 q = 0 where q is the minimal fixed point of F, q:= inf{x : DISPLAYFORM12 Proof. We first prove that K φ,var (σ b, σ w) ≥ q. We assume that σ b > 0, the case σ b = 0 is trivial since in this case q = 0 (the output of the network is zero in this case).Since F is continuous and DISPLAYFORM13 Using the fact that F is non-decreasing for any input a such that q 1 a ≤ q, we have q l is increasing and converges to the fixed point q. Therefore K φ,var (σ b, σ w) ≥ q. Now we prove that on the edge of chaos, we have DISPLAYFORM14 The EOC equation is given by σ 2 w E[φ ( √ qZ) 2 ] = 1. By taking the limit σ b → 0 on the edge of chaos, and using the fact that lim σ b →0 q = 0, we have σ DISPLAYFORM15 so that by taking the limit σ b → 0, and using the dominated convergence theorem, we have that DISPLAYFORM16 q + 1 and equation 6 holds. Finally since f is strictly convex, for all DISPLAYFORM17 q, we conclude using the fact that lim σ b →0 DISPLAYFORM18 Note however that for all σ b > 0, if (σ b, σ w) ∈ EOC, for any inputs a, b, we have lim l→∞ c l a,b = 1. Indeed, since f is usually strictly convex (otherwise, f would be equal to identity on at least a segment of) and f = 1, we have that f is a contraction (because f ≥ 0), therefore the correlation converges to the unique fixed point of f which is 1. Therefore, in most of the cases, the of Proposition 4 should be seen as a way of slowing down the convergence of the correlation to 1. Proof. Using the convexity of f and the of Proposition 4, we have in the limit σ b → 0, DISPLAYFORM19 2 which implies that var(φ ( √ qZ)) = 0. Therefore there exists a constant a 1 such that φ (√ qZ) = a 1 almost surely. This implies φ = a 1 almost everywhere. Proposition 5. Let φ be a bounded function such that φ = 0, φ > 0, φ (x) ≥ 0, φ(−x) = −φ(x), xφ(x) > 0 and xφ (x) < 0 for x = 0, and φ satisfies (ii) in Proposition 4. Then, φ satisfies all the conditions of Proposition 4.Proof. Let φ be an activation function that satisfies the conditions of Proposition 5.(i) we have φ = 0 and φ > 0. Since φ is bounded and 0 < φ < ∞, then there exists K such that DISPLAYFORM20 (ii) The condition (ii) is satisfied by assumption.(iii) Let σ b > 0 and σ w > 0. Using equation 3 together with φ > 0, we have F (x) ≥ 0 so that F is non-decreasing. Moreover, we have DISPLAYFORM21 DISPLAYFORM22. Now let's prove that the function DISPLAYFORM23 is increasing near 0 which means it is an injection near 0, this is sufficient to conclude (because we take q to be the minimal fixed point). After some calculus, we have DISPLAYFORM24 Using Taylor expansion near 0, after a detailed but unenlightening calculation the numerator is equal to −2φ DISPLAYFORM25, therefore the function e is increasing near 0.(iv) Finally, using the notations U 1:= √ qZ 1 and U 2 (x) = √ q(xZ 1 + √ 1 − x 2 Z 2), the first and second derivatives of the correlation function are given by DISPLAYFORM26 where we used Gaussian integration by parts. Let x > 0, we have that DISPLAYFORM27 where we used the fact that (Z 1, Z 2) = (−Z 1, −Z 2) (in distribution) and φ (−y) = −φ (y) for any y. Using xφ (x) ≤ 0, we have 1 {u1≥0} φ (u 1) ≤ 0. We also have for all y > 0, E[φ (U 2 (x))|U 1 = y] < 0, this is a consequence of the fact that φ is an odd function and that for x > 0 and y > 0, the mapping z 2 → xy + √ 1 − x 2 z 2 moves the center of the Gaussian distribution to a strictly positive number, we conclude that f (x) > 0 almost everywhere and assumption (iii) of Proposition 4 is verified. Proof. To abbreviate notation, we note φ:= φ Swish = xe x /(1 + e x) and h:= e x /(1 + e x) is the Sigmoid function. This proof should be seen as a sketch of the ideas and not a rigourous proof.• we have φ = 0 and φ = • As illustrated in TAB0 in the main text, it is easy to see numerically that (ii) is satisfied. Moreover, we observe that lim σ b →0 q = 0, which proves the second part of the (iii).• Now we prove that F > 0, we note g(x):= xφ (x)φ(x). We have DISPLAYFORM28 Define G by DISPLAYFORM29 which holds true for any positive number x. We thus have g(x) > G(x) for all real numbers x. Therefore E[g( √ xZ)] > 0 almost everywhere and F > 0. The second part of (iii) was already proven above.• Let σ b > 0 and σ w > 0 such that q exists. Recall that DISPLAYFORM30 In FIG3, we show the graph of E[φ (U 1)φ (U 2 (x))] for different values of q (from 0.1 to 10, the darkest line being for q = 10). A rigorous proof can be done but is omitted here. We observe that f has very small values when q is large, this is a of the fact that φ is concentrated around 0.Remark: On the edge of chaos, we have σ DISPLAYFORM31 this yields DISPLAYFORM32 The term E[φ ( √ qZ)φ(√ qZ)] is very small compared to 1 (∼ 0.01), therefore F (q) ≈ 1.Notice also that the theoretical corresponds to the equivalent Gaussian process, which is just an approximation of the neural network. Thus, using a value of (σ b, σ w) close to the EOC should not essentially change the quality of the . We can replace the conditions "φ satisfies (ii)" in Proposition 5 by a sufficient condition. However, this condition is not satisfied by Tanh. Proposition 7. Let φ be a bounded function such that DISPLAYFORM0, xφ(x) > 0 and xφ (x) < 0 for x = 0, and |Eφ (xZ) 2 | |x| −2β for large x and some β ∈. Then, φ satisfies all the conditions of Proposition 4.Proof. Let φ be an activation function that satisfies the conditions of Proposition 7. The proof is similar to the one of 5, we only need to show that having |Eφ (xZ) 2 | |x| −2β for large x and some β ∈ implies that (ii) of 4 is verified. We have that σ −β, so that we can make the term σ 2 w |Eφ (√ qZ) 2 | take any value between 0 and ∞. Therefore, there exists σ w such that (σ b, σ w) ∈ EOC, and assumption (ii) of Proposition 4 holds. In the proof of Proposition 5, we used the condition on φ (odd function) to prove that f > 0, however, in some cases when we can explicitly calculate f, we do not need φ to be defined. This is the case for Hard-Tanh, which is a piecewise-linear version of Tanh. We give an explicit calculation of f for the Hard-Tanh activation function which we note HT in what follows. We compare the performance of HT and Tanh based on a metric which we will define later. HT is given by DISPLAYFORM0 Recall the propagation of the variance q DISPLAYFORM1 where HT is the Hard-Tanh function. We have DISPLAYFORM2 This yields DISPLAYFORM3 where DISPLAYFORM4 -EDGE OF CHAOS:To study the correlation behaviour, we will assume that the variance converges to q. We have E(HT ( √ qZ) 2 ) = E(1 − FIG3 shows the EOC curve (condition (ii) is satisfied). FIG3 shows that is non-decreasing and FIG3 illustrates the fact that lim σ b →0 q = 0. Finally, FIG3 shows that function f is convex. Although the figures of F and f are shown just for one value of (σ b, σ w), the are true for any value of (σ b, σ w) on the EOC. TAB2 presents a comparative analysis of the validation accuracy of ReLU and Swish when the depth is larger than the width, in which case the approximation by a Gaussian process is not accurate (notice that in the approximation of a neural network by a Gaussian process, we first let N l → ∞, then we consider the limit of large L). ReLU tends to outperforms Swish when the width is smaller than the depth and both are small, however, we still observe a clear advantage of Swish for deeper architectures.

How to effectively choose Initialization and Activation function for deep neural networks

Represent the short summary for finding the corresponding scientific article (long)

We prove that for a large class of functions f there exists an interval certified robust network approximating f up to arbitrary precision.