Question: Represent the scientific article to find a short TLDR

Question: We study the problem of fitting task-specific learning rate schedules from the perspective of hyperparameter optimization. This allows us to explicitly search for schedules that achieve good generalization. We describe the structure of the gradient of a validation error w.r.t. the learning rates, the hypergradient, and based on this we introduce a novel online algorithm. Our method adaptively interpolates between two recently proposed techniques , featuring increased stability and faster convergence. We show empirically that the proposed technique compares favorably with baselines and related methodsin terms of final test accuracy. Learning rate (LR) adaptation for first-order optimization methods is one of the most widely studied aspects in optimization for learning methods -in particular neural networks -with early work dating back to the origins of connectionism . More recent work focused on developing complex schedules that depend on a small number of hyperparameters (; Orabona & Pál, 2016). Other papers in this area have focused on the optimization of the (regularized) training loss (; ;). While quick optimization is desirable, the true goal of supervised learning is to minimize the generalization error, which is commonly estimated by holding out part of the available data for validation. Hyperparameter optimization (HPO), a related but distinct branch of the literature, specifically focuses on this aspect, with less emphasis on the goal of rapid convergence on a single task. Research in this direction is vast (see for an overview) and includes model-based , model-free , and gradientbased approaches. Additionally, works in the area of learning to optimize have focused on the problem of tuning parameterized optimizers on whole classes of learning problems but require prior expensive optimization and are not designed to speed up training on a single specific task. The goal of this paper is to automatically compute in an online fashion a learning rate schedule for stochastic optimization methods (such as SGD) only on the basis of the given learning task, aiming at producing models with associated small validation error. We study the problem of finding a LR schedule under the framework of gradient-based hyperparameter optimization : we consider as an optimal schedule η * = (η * 0, . . ., η * T −1) ∈ R T + a solution to the following constrained optimization problem min{f T (η) = E(w T (η)): η ∈ R T + } s.t. w 0 =w, w t+1 (η) = Φ t (w t (η), η t ) for t = {0, . . ., T − 1} = [T], where E: R d → R + is an objective function, Φ t: is a (possibly stochastic) weight update dynamics,w ∈ R d represents the initial model weights (parameters) and finally w t are the weights after t iterations. We can think of E as either the training or the validation loss of the model, while the dynamics Φ describe the update rule (such as SGD, SGD-Momentum, Adam etc.). For example in the case of SGD, Φ t (w t, η t) = w t − η t ∇L t (w t), with L t (w t) the (possibly regularized) training loss on the t-th minibatch. The horizon T should be large enough so that the training error can be effectively minimized, in order to avoid underfitting. Note that a too large value of T does not necessarily harm since η k = 0 for k >T is still a feasible solution, implementing early stopping in this setting. Under review as a conference paper at ICLR 2020 Problem can be in principle solved by any HPO technique. However, most HPO techniques, including those based on hypergradients or on a bilevel programming formulation would not be suitable for the present purpose since they require multiple evaluations of f (which, in turn, require executions of the weight optimization routine), thus defeating one of the main goals of determining LR schedules, i.e. speed. In fact, several other researchers (; ; ; ; ;) have investigated related solutions for deriving greedy update rules for the learning rate. A common characteristic of methods in this family is that the LR update rule does not take into account information from the future. At a high level, we argue that any method should attempt to produce updates that approximate the true and computationally unaffordable hypergradient of the final objective with respect to the current learning rate (in relation to this, discusses the bias deriving from greedy or short-horizon optimization). In practice, different methods resort to different approximations or explicitly consider greedily minimizing the performance after a single parameter update (; ;). The type of approximation and the type of objective (i.e. the training or the validation loss) are in principle separate issues although comparative experiments with both objectives and the same approximation are never reported in the literature and validation loss is only used in the experiments reported in. One additional aspect needs to be taken into account: even when the (true or approximate) hypergradient is available, one still needs to introduce additional hyper-hyperparameters in the design of the online learning rate adaptation algorithm. For example in , hyper-hyperparameters include initial value of the learning rate η 0 and the hypergradient learning rate β. We find in our experiments that can be quite sensitive to the choice of these constants. In this work, we make a step forward in understanding the behavior of online gradient-based hyperparameter optimization techniques by (i) analyzing in Section 2 the structure of the true hypergradient that could be used to solve Problem if wall-clock time was not a concern, and (ii) by studying in Section 3 some failure modes of previously proposed methods along with a detailed discussion of the type of approximations that these methods exploit. In Section 4, based on these considerations, we develop a new hypergradient-based algorithm which reliably produces competitive learning rate schedules aimed at lowering the final validation error. The algorithm, which we call MARTHE (Moving Average Real-Time Hyperparameter Estimation), has a moderate computational cost and can be interpreted as a generalization of the algorithms described in and. Unlike previous proposals, MARTHE is almost parameter-free in that it incorporates heuristics to automatically tune its configuration parameters (i.e. hyper-hyperparameters). In Section 5, we empirically compare the quality of different hypergradient approximations in a small scale task where true hypergradient can be exactly computed. In Section 6, we present a set of real world experiments showing the validity of our approach. We finally discuss potential future applications and research directions in Section 7. We study the optimization problem under the perspective of gradient-based hyperparameter optimization, where the learning rate schedule η = (η 0, . . ., η T −1) is treated as a vector of hyperparameters and T is a fixed horizon. Please refer to Appendix A for a summary of the main notation used throughout the paper. Since the learning rates are positive real-valued quantities, assuming both E and Φ are smooth functions, we can compute the gradient of f ∈ R T, which is given by where " " means transpose. The total derivativeẇ T can be computed iteratively with forward-mode algorithmic differentiation aṡ The Jacobian matrices A t and B t depend on w t and η t, but we will leave these dependencies implicit to ease our notation. In the case of SGD 1, A t = I − η t H t (w t) and [B t] j = −δ tj ∇L t (w t), where subscripts denote columns (starting from 0), δ tj = 1 if t = j and 0 otherwise and H t is the Hessian of the training error on the t−th mini-batch 2. We also note that, given the high dimensionality of η, reverse-mode differentiation would in a more efficient (running-time) implementation. We use here forward-mode both because it is easier to interpret and visualize and also because it is closely related to the computational scheme behind MARTHE, as we will show in Section 4. Finally, we note that stochastic approximations of Eq. may be obtained with randomized telescoping sums or hyper-networks based stochastic approximations . Eq. 3 describes the so-called tangent system which is a discrete affine time-variant dynamical system that measures how the parameters of the model would change for infinitesimal variations of the learning rate schedule, after t iterations of the optimization dynamics. Notice that the "translation matrices" B t are very sparse, having, at any iteration, only one non-zero column. This means that [ẇ t] j remains 0 for all j ≥ t: η t affects only the future parameters trajectory. Finally, for a learning rate η t, the derivative (a scalar) is where the last equality holds true for SGD. Eq. can be read as the scalar product between the gradients of the training error at the t-th step and the objective E at the final iterate, transformed by the accumulated (transposed) Jacobians of the optimization dynamics, shorthanded by P T −1 t+1. As it is apparent from Eq., given w t, the hypergradient of η t is affected only by the future trajectory and does not depend explicitly on η t. In its original form, where each learning rate is left free to take any permitted value, Eq. represents a highly nonlinear setup. Although, in principle, it could be solved by projected gradient descent, in practice it is unfeasible even for small problems: evaluating the gradient with forward-mode is inefficient in time, since it requires maintaining a (large) matrix tangent system. Evaluating it with reverse-mode is inefficient in memory, since the entire weight trajectory (w i) T i=0 should be stored. Furthermore, it can be expected that several updates of η are necessary to reach convergence where each update requires computation of f T and the entire parameter trajectory in the weight space. Since this approach is computationally very expensive, we turn out attention to online updates where η t is required to be updated online based only on trajectory information up to time t. Before developing and motivating our proposed technique, we discuss two previous methods to compute the learning rate schedule online. The real-time hyperparameter optimization (RTHO) algorithm suggested in , reminiscent of stochastic meta-descent , is based on forward-mode differentiation and uses information from the entire weight trajectory by accumulating partial hypergradients. Hypergradient descent (HD), proposed in and closely related to the earlier work of , aims at minimizing the loss w.r.t. the learning rate after one step of optimization dynamics. It uses information only from the past and current iterate. Both methods implement update rules of the type where ∆η t is an online estimate of the hypergradient, β > 0 is a step-size or hyper-learning rate and the max ensures positivity 3. To ease the discussion, we omit the stochastic (mini-batch) evaluation of the training error L and possibly of the objective E. The update rules 4 are given by for RTHO and HD respectively, where P t−1 t:= I. As it can be seen ): the correction term r can be interpreted as an "on-trajectory approximations" of longer horizon objectives as we will discuss in Section 4. Although successful in some learning scenarios, we argue that both these update rules suffer from (different) pathological behaviors, as HD may be "shortsighted", being prone to underestimate the learning rate (as noted by), while RTHO may be too slow to adapt to sudden changes of the loss surface or, worse, may be unstable, with updates growing exponentially in magnitude. We exemplify these behaviors in Fig. 1, using two bidimensional test functions 5 from the optimization literature, where we set E = L and we perform 500 steps of gradient descent, from a fixed initial point. The Beale function, on the left, presents sharp peaks and large plateaus. RTHO consistently outperforms HD for all probed values of β that do not lead to divergence (Fig. 3 upper center). This can be easily explained by the fact that in flat regions gradients are small in magnitude, leading to ∆ HD η t to be small as well. RTHO, on the other hand, by accumulating all available partial hypergradients and exploiting second order information, is capable of making faster progress. We use a simplified and smoothed version of the Bukin function N. 6 to show the opposite scenario (Fig. 3 lower center and right). Once the optimization trajectory closes the valley of minimizers y = 0.01x, RTHO fails to discount outdated information, bringing the learning rate first to grow exponentially, and then to suddenly vanish to 0, as the gradient changes direction. HD, on the other hand, correctly damps η and is able to maintain the trajectory close to the valley. These considerations suggest that neither ∆ RTHO nor ∆ HD provide globally useful update directions, as large plateaus and sudden changes on the loss surface are common features of the optimization landscape of neural networks . Our proposed algorithm smoothly and adaptively interpolates between these two methods, as we will present next. In this section, we develop and motivate MARTHE, an algorithm for computing LR schedule online during a single training run. This method maintains an adaptive moving-average over approximations of Eq. of increasingly longer horizon, using the past trajectory and gradients to retain a low computational overhead. Further, we show that RTHO and HD outlined above, can be interpreted as special cases of MARTHE, shedding further light on their behaviour and shortcomings. 4 In the authors the hyperparameter is updated every K iterations. Here we focus on the case K = 1 which better allows for a unifying treatment. HD is developed using as objective the training loss L rather than the validation loss E. We consider here without loss of generality the case of optimizing E. 5 We use the Beale function defined as L(x, y) = (and a simplified smoothed version of Buking N. 6: L(x, y) = ((y − 0.01x) 2 + ε) 1/2 + ε, with ε > 0. Shorter horizon auxiliary objectives. The g K s define a class of shorter horizon objective functions, indexed by K, which correspond to the evaluation of E after K steps of optimization, starting from u ∈ R d and using ξ as the LR schedule 6. Now, the derivative of g K w.r.t. ξ 0, denoted g K, is given by where the last equality holds for SGD dynamics. Once computed on subsets of the original optimization dynamics (w i) T i=0, the derivative reduces for K = 1 to g 1 (w t, η t) = −∇E(w t+1)∇L(w t) (for SGD dynamics), and for Intermediate values of K yield cheaper, shorter horizon approximations of. Approximating the future trajectory with the past. Explicitly using any of the approximations given by g K (w t, η) as ∆η t is, however, still largely impractical, especially for K 1. Indeed, it would be necessary to iterate the map Φ for K steps in the future, with the ing (w t+i) iterations discarded after a single update of the learning rate. For K ∈ [t], we may then consider evaluating g K exactly K steps in the past, that is evaluating HD, which is computationally inexpensive. However, when past iterates are close to future ones (such as in the case of large plateaus), using larger K's would allow in principle to capture longer horizon dependencies present in the hypergradient structure of Eq. 4. Unfortunately the computational efficiency of K = 1 does not generalize to K > 1, since setting ∆η t = g K would require maintaining K different tangent systems. Discounted accumulation of g k s. The definition of the g K s, however, allows one to highlight the recursive nature of the accumulation of g K. Indeed, by maintaining the vector tangent system requires only updating and recomputing the gradient of E for a total cost of O(c(Φ)) per step both in time and memory using fast Jacobians vector products where c(Φ) is the cost of computing the optimization dynamics (typically c(Φ) = O(d)). The parameter µ ∈ allows to control how quickly past history is forgotten. One can notice that ∆ RTHO η t = S t,1 (w 0, (η j) t−1 i=0 ), while µ = 0 recovers ∆ HD η t. Values of µ < 1 help discounting outdated information, while as µ increases so does the horizon of the hypergradient approximations. The computational scheme of Eq. 9 is quite similar to that of forward-mode algorithmic differentiation for computingẇ (see Section 2 and Eq. 3); we note, however, that the "tangent system" in Eq. 9, exploiting the sparsity of the matrices B t, only keeps track of the variations w.r.t the first component ξ 0, drastically reducing the running time. Adapting µ and β online. We may set ∆η t = S t,µ (w 0, (η j) t−1 i=0 ). This still would require choosing a fixed value of µ, which should be validated on a separate set of held-out data. This may add an undesirable overhead on the optimization procedure. Furthermore, as discussed in Section 3, different regions of the loss surface may benefit from different effective approximation horizons. To address these issues, we propose to compute µ online. Ideally, we would like to verify that ∆η t [∇f (η)] t > 0, i.e. whether the proposed update is a descent direction w.r.t. the true hypergradient. While this is unfeasible (since ∇f (η) is unavailable), we can cheaply compute, after the update w t+1 = Φ(w t, η t), the quantity which, ex post, relates µ t to the one-step descent condition for g 1. We set µ t+1 = h µ (q(µ t)) where h µ is a monotone scalar function with range in (note that if µ t = 0 then Eq. 10 is non-negative). For space limitations, we defer the discussion of the choice of h µ and the effect of adapting online the approximation horizons to the Appendix. We can finally define the update rule for MARTHE as We further propose to adapt β online, implementing with this work a suggestion from. We regard the LR schedule as a function of β and apply the same reasoning done for η, keeping µ = 0, to avoid maintaining an additional tangent system which would involve third order derivatives of the training loss L. We then set β t+1 = β t − β∆η t+1 · ∆η t, where, with a little notation override, β becomes a fixed step-size for adapting the hyper-learning rate. This may seem a useless trade-off at first; yet, as we observed experimentally, one major advantage of lifting the adaptive dynamics by one more level is that it injects additional stability in the learning system, in the sense that good values of this last parameter of MARTHE lays in a much broader range range than those of good hyper-learning rates. In fact, we observe that when β is too high, the dynamics diverges within the first few optimization steps; whereas, when it does not, the final performances are rather stable. Algorithm 1 presents the pseudocode of MARTHE. The runtime and memory requirements of the algorithm are dominated by the computation of the variables Z. Being these structurally identical to the tangent propagation of forward mode algorithmic differentiation, we conclude that the runtime complexity is up to four times that of the underlying optimization dynamics Φ and the memory requirement up to two times (see , Sec. 4). We suggest the default values of β 0 = 0 and η 0 = 0 when no prior knowledge of the task at hand is available. Finally, we suggest to wrap Algorithm 1 in a selection procedure for β, where one may start with a high enough β and aggressively diminish it (e.g. decimating it) until the learning system does not diverge. Algorithm 1 MARTHE; requires β, η0, β0 = 0 Initialization of w, η and {Compute µ t, see Eq. 10} β t+1 ← β t − β∆η t+1 · ∆η t {Update hyper-LR} end for In this section, we empirically compare the optimized LR schedules found by approximately solving Problem 1 by gradient descent (denoted LRS-OPT), where the hypergradient is given by Eq. 4, against HD, RTHO & MARTHE schedules where for MARTHE we consider both the adaptation schemes for µ and β presented in the previous section as well as fixed hyper-learning rate and discount factor µ. We are interested in understanding and visualizing the qualitative similarities among the schedules, as well as the effect of µ and β and the adaptation strategies on the final performance measure. To this end, we trained feedforward neural networks, with three layers of 500 hidden units each, on a subset of 7000 MNIST images. We used the cross-entropy loss and SGD as the optimization dynamics Φ, with a mini-batch size of 100. We further sampled 700 images to form the validation set and defined E to be the validation loss after T = 512 optimization steps (about 7 epochs). For LRS-OPT, we randomly generated different mini-batches at each iteration to prevent the schedule from unnaturally adapting to a specific sample progression 7. We initialized η = 0.01 · 1 512 for LRS-OPT and set η 0 = 0.01 for all adaptive methods, and repeated the experiments for 20 random seeds (except LRS-OPT, repeated only for 4 seeds). Results are visualized in Figure 2. Figure 2 (left) shows the LRS-OPT schedules found after 5000 iterations of gradient descent: the plot reveals a strong initialization (random seed) specific behavior of η * for approximately the first 100 steps. The LR schedule then stabilizes or slowly decreases up until around 50 iterations before the final time, at which point it quickly decreases (recall that with LRS-OPT all η i, including η 0, are optimized "independently" and may take any permitted value). Figure 2 (center) present a qualitative comparison between the offline LRS-OPT schedule and the online ones. HD generates schedules that quickly decay to very small values, while RTHO schedule linger or fail to decrease, possibly causing instability and divergence in certain cases. Fixing µ = 0.99 seems to produce schedules that Center: qualitative comparison between offline and online schedules for one random seed. For MARTHE with fixed µ, we report the best performing one. For each method, we report the schedule generated with the value of β that achieves the best average final validation accuracy. Plots for the remaining random seeds can be found in the appendix. Right: Average validation accuracy of over 20 random seeds, for various values of β. When no point is reported it means that the achieved average accuracy for that configuration falls below 88% (or diverged). For reference, the average validation accuracy of the network trained with η = 0.01 · 1512 is 87.5%, while LRS-OPT attains 96.2%. remarkably mimic the the optimized one; yet, unfortunately, this happens only for a small range of values of µ which we expect to be task dependent. Using both the adaptation schemes for µ and β (curve named MARTHE in the plots), allows to reliably find highly non-trivial schedules that capture the general behavior of the optimized one (additional plots in the Appendix). Figure 2 (right) shows the average validation accuracy over 20 runs (rather than loss, for easier interpretation) of the online methods, varying β and discarding values below 88% of validation accuracy. In particular, fixing µ > 0 seems to have a beneficial impact for all tried values of the hyper-learning rate β. Using only the heuristic for adapting µ online (blue line, named h µ in the plot) further helps, but is somewhat sensitive to the choice of β. Using both the adaptive mechanisms, beside improving the final validation accuracy, seems to drastically lower the sensitivity on the choice of this parameter, provided that the learning system does not diverge. Finally, we note that even with this very simple setup, a single run of LRS-OPT (which comprises 5000 optimization steps) takes more than 2 hours on an M-40 NVIDIA GPU. In contrast, all adaptive methods requires less than a minute to conclude (HD being even faster). We run experiments with an extensive set of learning rate scheduling techniques. Specifically, we compare MARTHE against the following fixed LR scheduling strategies: (i) exponential decay (ED) -where the LR schedule is defined by η t = η 1 γ t (ii) staircase decay and (iii) stochastic gradient descent with restarts (SGDR) by. Moreover, we compare against online strategies such as HD and RTHO. For all the experiments, we used a single Volta V100 GPU (AWS P3.2XL). We fix the batch-size at 128 samples for all the methods, and terminate the training procedure after a fixed number of epochs. We set L as the cross entropy loss with weight-decay regularization (with factor of 5 · 10 −4) and set E as the unregularized cross entropy loss on validation data. All the experiments with SGDM have an initial learning rate (η 0) of 0.1 and for Adam, we set it to 3 · 10 −4. For staircase, we decay the learning rate by 90% after every 60 epochs. For exponential decay, we fix a decay factor of 0.99 per epoch, and for SGDR we use T 0 = 10 and T mult = 2. For HD and RTHO, we set the β as 10 −6 and 10 −8 respectively. Momentum for SGDM is kept constant to 0.9. For Adam we used the standard values for the remaining configuration parameters. We run all the experiments with 5 different seeds reporting average and standard deviation, recording accuracy, loss value and generated learning rate schedules. We trained image classification models on two different datasets commonly used to benchmark optimization algorithms for deep neural networks: CIFAR-10 where we trained a VGG-11 network with BatchNorm using SGDM as the inner optimizer, and CIFAR-100 where we trained a ResNet-18 using Adam (. The source code in PyTorch and TensorFlow to reproduce the experiments will be made publicly available. In Figures 3 and 4, we report the of our experiments for CIFAR-10 with VGG-11 and CIFAR-100 with ResNet-18 respectively. For both figures, we report from left to right: accuracy in percentage, validation loss value, and an example of a generated learning rate schedule. MARTHE produces LR schedules that lead to trained models with very competitive final validation accuracy in both the experimental settings, virtually requiring no tuning. For setting the hyper-learning rate step-size of MARTHE we followed the simple procedure outlined at the end of Section 4, while for the other methods we performed a grid search to select the best value of the respective algorithmic parameters. On CIFAR-10, MARTHE obtains a best average accuracy of 92.79% statistically on par with SGDR (92.54%), while clearly outperforming the other two adaptive algorithms. On CIFAR-100, MARTHE leads to faster convergence during then whole training compared to all the other methods, reaching an accuracy of 76.68%, comparable to staircase schedule with 76.40%. We were not able to achieve competitive with SGDR in this setting, despite trying several values of the two main configuration parameters within the suggested range. Further, MARTHE produces aggressive schedules (see Figure 3 and 4 right, for an example) that increase the LR at the beginning of the training, sharply decreasing it after a few epochs. We observe empirically that this leads to improved convergence speed and competitive final accuracy. Additional experimental validation is reported in Appendix D and includes for MARTHE with an initial learning rate η 0 = 0. Finding a good learning rate schedule is an old but crucially important issue in machine learning. This paper makes a step forward, proposing an automatic method to obtain performing LR schedules that uses an adaptive moving average over increasingly long hypergradient approximations. MARTHE interpolates between HD and RTHO taking the best of the two worlds. The implementation of our algorithm is fairly simple within modern automatic differentiation and deep learning environments, adding only a moderate computational overhead over the underlying optimizer complexity. In this work, we studied the case of optimizing the learning rate schedules for image classification tasks; we note, however, that MARTHE is a general technique for finding online hyperparameter schedules (albeit it scales linearly with the number of hyperparameters), possibly implementing a competitive alternative in other application scenarios, such as tuning regularization parameters . We plan to further validate the method both in other learning domains for adapting the LR and also to automatically tune other crucial hyperparameters. We believe that another interesting future research direction could be to learn the adaptive rules for µ and β in a meta learning fashion. Please refer to Table 1 for a summary and description of the notation used throughout the paper. Validation error Total derivative of w w.r.t. η, variable of the tangent system Jacobian of the dynamics w.r.t. the weights SGD: Jacobian of the dynamics w.r.t. the LRS SGD: Product of Jacobians from iteration s to r ∆η t Update for the learning rate at iteration t β ∈ R Variables of the vector tangent system Cf. tangent matrixẇ Heuristics for dampening factor µ See Appendix B B CHOICE OF HEURISTIC FOR ADAPTING µ We introduced in Section 4 a method to compute online the dampening factor µ t based on the quantity We recall that if q(µ t) is positive then the update ∆η t+1 is a descent direction for the one step approximation of the objective f T. We describe here the precise heuristic rule that we use in our experiments. Callq(µ t) = max(min(q(µ t) g 1 (w t, η t) −2, 1), 0) ∈ the normalized, thresholded q(µ t). We propose to set where c t acts as a multiplicative counter, measuring the effective approximation horizon. The ing heuristics is independent on the initialization of µ since Z 0 = 0. We note that whenever µ t is set to 0, the previous hypergradient history is forgotten. Applying this heuristic to the optimization of the two test functions of Sec. 3 reveals to be successful: for the Beale function, h µ selects µ t = 1 for all t, while for the smoothed Bukin, it selects µ t = 0 for around 40% of the iterations, bringing down the minimum optimality gap at 10 −6 for β = 0.0005. We conducted exploratory experiments with variants of h µ which include thresholding between -1 and 1 and "penalizing" updates larger than g 1 without observing statistically significant differences. We also verified that randomly setting µ t to 0 does not implement a successful heuristics, while introducing another undesirable configuration parameter. We believe, however that there is further space of improvement for h µ (and possibly to adapt the hyper-learning rate), since g 1 does not necessarily capture the long term dependencies of Problem 1. Meta-learning these update rules could be an interesting direction that we leave to future investigation. We show in Figure 5 the LR schedules for the experiments described in Section 5 for the remaining random seeds. The random seed controls the different initial points w 0, which is the same for all online methods and for LRS-OPT, and determines the mini-batch progression for the online methods (while for LRS-OPT the mini-batch progression is randomized at each outer iteration). We report in this section additional experimental to complement the analysis of Section 6. Figure 6 shows accuracy, validation loss and learning rates schedules for CIFAR-100 dataset with a ResNet-18 model and SGDM as (inner) optimization methods. In accordance with previous experimental validation we set the initial learning rate η 0 to 0.1 for all methods. We include also relevant statistics for MARTHE 0, that is MARTHE obtained by letting the schedule start at 0 (cyan lines in the plots). Setting η 0 = 0 is clearly not an optimal choice, nevertheless MARTHE is able to obtain competitive also in this disadvantaged scenario, producing schedules that quickly reach high values and then sharply decrease within the first 40 epochs. Figure 6 (left) reports a sample of generated schedule. It is important to highlight how the heuristic methods, such as exponential decay, are not able to handle η 0 = 0. In fact, for non-adaptive methods, η 0 is indeed another configuration parameter that must be tuned. MARTHE 0, on the other hand, constitute a virtually parameterless method (β can be quickly found with the strategy outlined at the end of Section 4) that can be employed in situations where we have no prior knowledge of the task at hand. Conversely as noted by and , too high (initial) learning rates are not well suited for gradient-based adaptive strategies: instability of the inner optimization dynamics indeed propagates to the hypergradient computation, possibly leading to "exploding" hypergradients. Finally, we tried another configuration of parameters for SGDR, in order to make the last restart completing the training exactly after 200 epochs. We selected T 0 = 10 and T mul = 2.264 (i.e. restarts at 10, 33, 84, 200). Figure 7 and Figure 8 report the same experiments of Section 6 with this slightly different baseline. We note that the performance of SGDR remains similar. E SENSITIVITY ANALYSIS OF INADAPTIVE MARTHE WITH RESPECT TO η 0, µ AND β In this section, we study the impact of η 0, µ and β for MARTHE, when our proposed online adaptive methodologies for µ and β are not applied. We think that the sensitivity of the methods is very important for the HPO algorithms to work well in practice, especially when they depend on the choice of some (new) hyperparameters such as µ and β. We show the sensitivity of inadaptive MARTHE with respect to η 0 and µ, fixing β. We used VGG-11 on CIFAR-10 with SGDM as optimizer, but similar can be obtained in the other cases. Figure 9 shows the obtained test accuracy fixing β to 10 −7 (Left) and 10 −8 (Right). The plots show a certain degree of sensitivity, especially with respect to the choice of a (fixed) µ. This suggest that implementing adaptive strategies to compute online the dampening factor µ and the hyper-learning rate β constitute an essential factor to achieve the competitive reported in Section 6 and Appendix D. Figure 9: Sensitivity analysis of inadaptive MARTHE with respect to η 0 and µ fixing the value of β to 10 −7 (Left) and 10 −8 (Right). We used VGG-11 on CIFAR-10 with SGDM as optimizer. Darker colors mean lower final accuracy.

Represent the short summary for finding the corresponding scientific article (long)

MARTHE: a new method to fit task-specific learning rate schedules from the perspective of hyperparameter optimization

represent the tldr for finding the corresponding scientific article!

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the input

Question: Recent years have witnessed some exciting developments in the domain of generating images from scene-based text descriptions. These approaches have primarily focused on generating images from a static text description and are limited to generating images in a single pass. They are unable to generate an image interactively based on an incrementally additive text description (something that is more intuitive and similar to the way we describe an image). We propose a method to generate an image incrementally based on a sequence of graphs of scene descriptions (scene-graphs). We propose a recurrent network architecture that preserves the image content generated in previous steps and modifies the cumulative image as per the newly provided scene information. Our model utilizes Graph Convolutional Networks (GCN) to cater to variable-sized scene graphs along with Generative Adversarial image translation networks to generate realistic multi-object images without needing any intermediate supervision during training. We experiment with Coco-Stuff dataset which has multi-object images along with annotations describing the visual scene and show that our model significantly outperforms other approaches on the same dataset in generating visually consistent images for incrementally growing scene graphs. To truly understand the visual world, our models should be able to not only recognize images but also generate them. Generative Adversarial Networks, proposed by BID3 have proven immensely useful in generating real world images. GANs are composed of a generator and a discriminator that are trained with competing goals. The generator is trained to generate samples towards the true data distribution to fool the discriminator, while the discriminator is optimized to distinguish between real samples from the true data distribution and fake samples produced by the generator. The next step in this area is to generate customized images and videos in response to the individual tastes of a user. A grounding of language semantics in the context of visual modality has wide-reaching impacts in the fields of Robotics, AI, Design and image retrieval. To this end, there has been exciting recent progress on generating images from natural language descriptions. Conditioned on given text descriptions, conditional-GANs BID11 are able to generate images that are highly related to the text meanings. Samples generated by existing text-to-image approaches can roughly reflect the meaning of the given descriptions, but they fail to contain necessary details and vivid object parts. Leading methods for generating images from sentences struggle with complex sentences containing many objects. A recent development in this field has been to represent the information conveyed by a complex sentence more explicitly as a scene graph of objects and their relationships BID7. Scene graphs are a powerful structured representation for both images and language; they have been used for semantic image retrieval BID6 and for evaluating BID0 and improving BID9 image captioning. In our work, we propose to leverage these scene graphs by incrementally expanding them into more complex structures and generating corresponding images. Most of the current approaches lack the ability to generate images incrementally in multiple steps while preserving the contents of the image generated so far. We overcome this shortcoming by conditioning the image generation process over the cumulative image generated over the previous steps and over the unseen parts of the scene graph. This allows our approach to generate high quality complex real-world scenes with several objects by distributing the image generation over multiple steps without losing the context. Recently, BID2 proposed an approach for incremental image generation but their method is limited to synthetic images due to the need of supervision in the intermediate step. Our approach circumvents the need for intermediate supervision by enforcing perceptual regularization and is therefore compatible with training for even real world images (as we show later).A visualization of our framework's outputs with a progressively growing scene graph can be seen in Figure 1. We can see how at each step new objects get inserted into the image generated so far without losing the context. To summarize, we make the following contributions,• We present a framework to generate images from structured scene graphs that allows the images to be interactively modified, while preserving the context and contents of the image generated over previous steps.• Our method does not need any kind of intermediate supervision and hence, is not limited to synthetic images (where you can manually generate ground truth intermediate images). It is therefore useful for generating real-world images (such as for MS-COCO) which, to the best of our knowledge, is the first attempt of its kind. Generating images from text descriptions is of great interest, both from a computer vision perspective, and a broader artificial intelligence perspective. Since the advent of Generative Adversarial Networks (GANs), there have been many efforts in this direction BID5. BID10 proposed a framework based on an LSTM and a conditional GAN to incrementally generate an image using a sentence. The words in the sentence were encoded using word2vec, and passed through an LSTM. A skip-though vector representing the semantic meaning of an entire sentence is used as the conditioning for the GAN. However, all of these works mostly focus on generating images with single objects (such as faces or flowers or birds). Even within these objects, the avenues of variance is quite limited. Generating more complex scenes with multiple objects and specific relationships between those objects is an even harder research problem. BID15 proposed an architecture based on multiple GANs stacked together, generating images in a coarse-to-fine manner. They later also proposed arranging the generators in a tree-like structure for improved BID16. More recently, BID4 proposed an end-to-end pipeline for inferring scene structure and generating images based on text descriptions. A similar approach was taken by BID13, where they used attention-based object and attribute decoders to infer bounding box locations of objects in the scene. However, for images with several objects such as in COCO-Stuff, the captions are often not descriptive enough to capture all the objects. Furthermore, the captions don't describe the relations between the objects in the image effectively. AttnGANs also begin with a low-resolution image, and then improve it over multiple steps to come up with a final image. However, there's no mechanism to capture consistency during incremental image generation. A more detailed failure case analysis is done in Section 4.Most recently, BID7 proposed to use scene-graphs as a convenient intermediate for image synthesis. Scene-graphs provide an efficient and interpretable representation of the objects in an image and their relationships. The input scene graph is processed with a graph convolution network which passes information along edges to compute embedding vectors for all objects. These vectors are used to predict bounding boxes and segmentation masks for all objects, which are combined to form a coarse scene layout. The layout is passed to a cascaded refinement network which generates an output image at increasing spatial scales. The model is trained adversarially against a pair of discriminator networks which ensure that output images look realistic. However, this model does not account for object saliency or temporal consistency in the generated images. It also fails for highly complex scene where there are several objects due to a single-pass image generation. A more detailed failure case analysis is done in Section 4.Another recent work by introduces a conditional text-to-image based generation approach to generate images iteratively, keeping track of ongoing context and history. Their method uses GRU to process text instructions into embeddings which go through conditioning augmentation and are fed into a GAN to generate contents on a canvas. They use a convex combination of feature maps to ensure consistency. Due to the availability of structured information in the form of scene graphs, we can instead filter out the latent embeddings of the already generated objects, thus allowing for better textural consistency. Moreover, their approach suffers from the drawback of needing supervision at every stage of generation for training and is therefore limited to only synthetic scenarios (such as CoDraw and i-CLEVR where intermediate images can be easily rendered). On the contrary, our approach employs perceptual similarity based regularization and effective use of graph-based embeddings to circumvent the need of ground truth for intermediate steps, making it compatible with even real-world images. As in most modern conditional image-generation formulations, we follow a generative-adversarial approach to image generation. Here the adversarial network penalizes the network based on how realistic the generated images are, as well as whether the required objects are present in them. Furthermore, a key part of the task is to preserve the relations between objects as specified in the scene graph in the generated image as well. For our baseline approach for generating images from scene graphs, we adopt the architecture proposed by BID7. The architecture consists of 3 main modules, a Graph Convolution network (GCN), Layout Prediction Network (LN) and a Cascade Refinement Network (CRN), which we describe in more detail below. For brevity, we omit an exhaustive description of these modules and request the reader to refer to the original paper for further details. Figure 2 provides an overview of the full model architecture. Graph Convolution Network. The Graph Convolution Network (GCN) is composed of several graph convolution layers, and can operate natively on graphs. GCN takes an input graph and computes new vectors for each node and edge. Each graph convolution layer propagates information along edges of the graph. The same function is applied to all graph edges, which ensures that a single convolution layer can work with arbitrary shaped graphs. Layout Prediction Network. The GCN outputs an embedding vector for each object. These object embedding vectors are used by the layout prediction network to compute a scene layout by predicting a segmentation mask and bounding box for each object. Mask regression network and a box regression network are used to predict a soft binary mask and a bounding box, respectively. The layout prediction network hence acts as an intermediary between the graph and image domains. Cascade Refinement Network. Given a scene layout, the Cascade Refinement Network (CRN) is responsible for generating an image which respects the object positions in the scene layout. The CRN consists of a series of convolutional refinement modules. The spatial resolution doubles between Figure 2: Model architecture for incremental image generation using scene graphs. The Graph Convolution Network takes as input the scene graph and produces object embeddings which are then fed to the Scene Layout Network. The SLN generates a layout by predicting the bounding boxes and segmentation masks for the objects. These are finally sent to the Cascaded Refinement Network which generates the image corresponding to the graph. This process is called iteratively to add objects in the image.modules; which ensures that image generation is happening in a coarse-to-fine manner. The scene layout is first downsampled to the module input resolution and then fed to the module along with the previous module's output. Both these inputs are concatenated channel-wise and then passed to a pair of 3 × 3 convolution layers. This output is upsampled using nearest-neighbor interpolation and then passed to the next module. The output from the last module is finally passed to 2 convolution layers to produce the output image. Our method allows for preserving context across the sequentially generated images by conditioning subsequent steps of image generation over certain information from previous steps.• We extend BID7 with a recurrent architecture that generates images using incrementally growing scene graphs using the components discussed in previous section as shown in Figure 2.• To ensure that the image generated in the current step preserves the visual context from the previous steps, we replace three channels of the noise passed to CRN with the RGB channels of the image generated in the previous step. This encourages the CRN to generate the new image as similar as possible to the previously generated image.• Moreover, we want the SLN to generate a layout corresponding to only the newly added objects in the scene graph. To this end, we remove the representations generated by GCN corresponding to the objects generated in previous steps.• We do not have any ground truth for the intermediate generated images so we use perceptual loss for images generated in the intermediate steps to enforce the images to be perceptually similar to the ground truth final image. We do have L1 loss between the final image generated and the ground truth. We use an image level and an object level discriminator to ensure realism of the images and presence of the objects. These are trained as in the regular GAN formulation: DISPLAYFORM0 2. Box loss: Penalizes L1 distance between ground truth boxes from MS COCO vs the predicted labels as L box = n i ||b − b || 3. Mask loss: Penalizes difference between the masks predicted vs the ground truth masks, using cross entropy loss.4. L1 pixel loss: Penalizes the difference between the ground truth image from MS COCO and the final generated image at the end of the incremental generation. L1 pixel losses are also used to penalize the difference between the previous and current generated image. DISPLAYFORM1 5. Perceptual Similarity Loss: Serves as a regularization to ensure that the images generated at different steps are contextually similar to each other. Since we do not have ground truth for intermediate steps, we introduce an additional perceptual similarity loss using between the final ground truth image and the different images generated in the intermediate steps. This allows the model to'hallucinate' for the intermediate steps in a way that the contents of the image are similar to the ground truth. DISPLAYFORM2 where P φ is the function computing a latent embedding that captures the perceptual characteristics of an image. Thus we can provide additional supervision on the coordinates of the bounding boxes predicted by the layout network, to explicitly ensure the relations are preserved. We perform experiments on the 2017 COCO-Stuff dataset BID1 which augments a subset of the COCO dataset BID8 with additional stuff categories. The dataset annotates 40K train and 5K validation images with bounding boxes and segmentation masks for 80'thing' categories (people, cars, etc.) and 91'stuff' categories (sky, grass, etc.). We follow the procedure described in BID7 to construct synthetic scene graphs from these annotations based on the 2D image coordinates of the objects, using six mutually exclusive geometric relationships: left of, right of, above, below, inside, and surrounding. We create three splits for each image based on the number of objects in it. We randomly select 50% of the objects for the first split and incrementally add 25% objects for the next two splits. We then synthetically create separate scene graphs for each split. We train incremental generation for three steps, but this can be easily extended to more number of steps. Note that previous works like BID2 have relied on entirely synthetic datasets. For synthetic datasets, intermediate ground truth for all interim stages of the input text can easily be generated. However, for interactive image generation to be truly useful, it needs to generate realistic looking images not constrained to a synthetic dataset's subspace. The challenge with using real datasets like COCO is that we do not have "ground-truth" images corresponding to interim scene graphs (i.e., where all objects of the training image are not present). Only at the final step, with the full scene graph, do we get supervision from the training image. This makes our loss formulation described in the previous section particularly important. To enable comparison against BID7, we follow their dataset preprocessing steps. They ignore objects covering less than 2% of the image, and use images with 3 to 8 objects. They divide the COCO-Stuff 2017 validation set into their own validation and test sets, which contain 24,972 train, 1024 validation, and 2048 test images. For fair comparison, we do the same. captures the scene graph's semantic context, it lacks consistency over multiple passes and the generated images are of poor quality. AttnGAN on the other hand generates higher resolution images but doesn't capture semantic context from the scene graph. Our model addresses both of these shortcomings by incrementally adding objects in accordance with the scene graph and generating higher quality images. We compare the performance of our model against two baselines BID7 and BID14 as can be seen in FIG0. The scene graph in the first step contains two objects and the relationships between them. An additional object and its relationship with other objects is added to the scene graph at each of the next two steps. Note that the two baselines do not generate incrementally but for fair comparison, we generate outputs from them by feeding different amounts of information (scene graph or text) in three independent forward passes. As can be seen, the first baseline BID7 is able to capture the semantic context provided by the graph. However (i) it fails to preserve consistency over multiple passes and generates a completely new image for each scene graph, agnostic of what it had generated at the previous step and Figure 4: Sample outputs of our pipeline, visualized over 3 steps of generation. 3 splits for each image are created, based on the number of objects in that image. Synthetic scene graphs are then generated for each split and used for incremental image generation. While three steps are used for training, this method is quite easily extensible for more number of steps as well.(ii) the images generate are of poor quality. The second baseline BID14 produces visually pleasing and high resolution images but completely fails to capture any semantic context provided from the graph. Our model, on the other hand, is capable of incrementally adding new objects to the image created in the previous step in accordance with the relationships defined in the scene graph. Additionally, the quality of generated images is significantly improved, since at each step the model has to generate only a few objects rather than generating cluttered scenes with multiple objects, hence enabling it to better generate scene semantics. We present comprehensive generated by our model in Figure 4. It can be seen in Figures 4(c,f,h) that the model starts by generating objects as described by the scene graph and outputs noise in the when it does not have enough information provided as input. As the information is added in the scene graph, objects like grass and sky begin to materialize. However it can also be seen that sometimes due to inherent biases in the dataset, the model begins to hallucinate the even when no information is provided explicitly. In Figure 4 (a), even though there is no mention of rock or water in the initial graph, the model is hallucinating objects of similar shapes at similar locations. We use Inception Score BID12 for evaluating the quality of the images generated from our models. Inception Score uses an ImageNet based classifier to provide a quantitative evaluation of how realistic generated images appear. Inception Scores were originally proposed with the two main goal. Firstly, the images generated should contain clear objects (i.e. the images are sharp rather than blurry), (or, for image x and label y, p(y|x) should be low entropy). Secondly, the generative algorithm should output a high diversity of images from all the different classes in ImageNet, or p(y) should be high entropy. Ground Truth BID7 Step 1 (Ours) Step 2 (Ours) Step 3 (Ours) 6.13 3.05 3.68 5.02 4.14 Table 1: Inception Scores for Ground Truth Images, images generated from Sg2im and the three steps from our modelThe inception scores are reported in Table 1. We compare the inception score of the images generated from the baseline model sg2im with the full scene graph of the ground truth images. For our sequential generation model, we report the scores over three steps of generation, where at the third step the scene graph is the full scene graph corresponding to the ground truth image. We observe that due to our modified loss formulation and incremental generation, our model performs better in inception scores in all three steps. Next, we evaluate how well our model retains the generated image from the previous step when adding in information to the image in the subsequent step (i.e., how "visually consistent" the generated images are). We use the perceptual similarity loss proposed by, and report the mean losses in Table 2. As seen both in the table and qualitatively in FIG0, our model is much more successful at preserving context and previously generated content over subsequent steps. Step 1 to Step 2 Step 2 to Step 3 Table 2: Mean Perceptual similarity loss, evaluated between the images generated images of sg2im and our proposed model. Lower is better. In this paper, we proposed an approach to sequentially generate images using incrementally growing scene graphs with context preservation. Through extensive evaluation and qualitative , we demonstrate that our approach is indeed able to generate an image sequence that is consistent over time and preserves the context in terms of objects generated in previous steps. In future, we plan to explore generating end-to-end with text description by augmenting our methodology with module to generate scene graphs from language input. While scene-graphs provide a very convenient modality to capture image semantics, we would like to explore ways to take natural sentences as inputs to modify the underlying scene graph. The current baseline method does single shot generation by passing the entire layout map through the Cascade Refinement Net for the final image generation. We plan to investigate whether the quality of generation can be improved by instead using attention on the GCN embeddings during generation. This could also potentially make the task of only modifying certain regions in the image easier. Further, we plan to explore better architectures for image generation through layouts for higher resolution image generation.

Represent the TLDR for finding the corresponding scientific article

Interactively generating image from incrementally growing scene graphs in multiple steps using GANs while preserving the contents of image generated in previous steps

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Conditional VAE on top of latent spaces of pre-trained generative models that enables  transfer between drastically different domains while preserving locality and semantic alignment.


Question: represent the article for finding a single-sentence tldr

Question: In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images. However, most Convolutional Neural Networks (CNNs) for image classification were developed using biased datasets that contain large objects, in mostly central image positions. To assess whether classical CNN architectures work well for tiny object classification we build a comprehensive testbed containing two datasets: one derived from MNIST digits and one from histopathology images. This testbed allows controlled experiments to stress-test CNN architectures with a broad spectrum of signal-to-noise ratios. Our observations indicate that: There exists a limit to signal-to-noise below which CNNs fail to generalize and that this limit is affected by dataset size - more data leading to better performances; however, the amount of training data required for the model to generalize scales rapidly with the inverse of the object-to-image ratio in general, higher capacity models exhibit better generalization; when knowing the approximate object sizes, adapting receptive field is beneficial; and for very small signal-to-noise ratio the choice of global pooling operation affects optimization, whereas for relatively large signal-to-noise values, all tested global pooling operations exhibit similar performance. Convolutional Neural Networks (CNNs) are the current state-of-the-art approach for image classification (; ; ;). The goal of image classification is to assign an image-level label to an image. Typically, it is assumed that an object (or concept) that correlates with the label is clearly visible and occupies a significant portion of the image; ). Yet, in a variety of real-life applications, such as medical image or hyperspectral image analysis, only a small portion of the input correlates with the label, ing in low signal-to-noise ratio. We define this input image signal-to-noise ratio as Object to Image (O2I) ratio. The O2I ratio range for three real-life datasets is depicted in Figure 1. As can be seen, there exists a distribution shift between standard classification benchmarks and domain specific datasets. For instance, in the ImageNet dataset objects fill at least 1% of the entire image, while in histopathology slices cancer cells can occupy as little as 10 −6 % of the whole image. Recent works have studied CNNs under different noise scenarios, either by performing random input-to-label experiments (; or by directly working with noisy annotations (; ;). While, it has been shown that large amounts of label-corruption noise hinders the CNNs generalization (;, it has been further demonstrated that CNNs can mitigate this label-corruption noise by increasing the size of training data , tuning the optimizer hyperparameters (Jastrzębski et al., 2017) or weighting input training samples . However, all these works focus on input-to-label corruption and do not consider the case of noiseless input-to-label assignments with low and very low O2I ratios. In this paper, we build a novel testbed allowing us to specifically study the performance of CNNs when applied to tiny object classification and to investigate the interplay between input signal-to-noise ratio and model generalization. We create two synthetic datasets inspired by the children's puzzle book Where's Wally? . The first dataset is derived from MNIST digits and allows us for two medical imaging datasets (CAME-LYON17 and MiniMIAS ) as well as one standard computer vision classification dataset (ImageNet ). The ratio is defined as O2I = Aimage, where A object and A image denote the area of the object and the image, respectively. Together with O2I range, we display examples of images jointly with the object area A object (in red). to produce a relatively large number of datapoints with explicit control of the O2I ratio. The second dataset is extracted from histopathology imaging where we crop images around lesions and obtain small number of datapoints with an approximate control of the O2I ratio. To the best of our knowledge these datasets are the first ones designed to explicitly stress-test the behaviour of the CNNs in the low input image signal-to-noise ratio. We develop a classification framework, based on CNNs, and analyze the effects of different factors affecting the model optimization and generalization. Throughout an empirical evaluation, we make the following observations: -Models can be trained in low O2I regime without using any pixel-level annotations and generalize if we leverage enough training data. However, the amount of training data required for the model to generalize scales rapidly with the inverse of the O2I ratio. When considering datasets with fixed size, we observe an O2I ratio limit in which all tested scenarios fail to exceed random performance. -We empirically observe that higher capacity models show better generalization. We hypothesize that high capacity models learn the input noise structure and, as , achieve satisfactory generalization. -We confirm the importance of model inductive bias -in particular, the model's receptive field size. Our suggest that different pooling operations exhibit similar performance, for larger O2I ratios; however, for very small O2I ratios, the type of pooling operation affects the optimization ease, with max-pooling leading to fastest convergence. The code of our testbed will be publicly available at: https://anonymous.url allowing to reproduce all data and ; we hope this work can serve as a valuable resource facilitating further research into the understudied problem of low signal-to-noise classification scenarios. To study the optimization and generalization properties of CNNs, we build two datasets: one derived from the MNIST dataset and another one produced by cropping large resolution images from the CAMELYON dataset . Each dataset allows to evaluate the behaviour of a CNN-based binary classifier when altering different data-related factors of variation such as dataset size, object size, image resolution and class balance. In this subsection, we describe the data generation process. Inspired by the cluttered MNIST dataset , we introduce a scaled up, large resolution cluttered MNIST dataset, suitable for binary image classification. In this dataset, images are obtained by randomly placing a varying number of MNIST digits on a large resolution image canvas. We keep the original 28 × 28 pixels digit resolution and control the O2I ratio by increasing the resolution of the canvas 1. As , we obtain the following O2I ratios {19.1, 4.8, 1.2, 0.3, and 0.075}% that correspond to the following canvas resolutions 64 × 64, 128 × 128, 256 × 256, 512 × 512, and 1024 × 1024 pixels, respectively. As object of interest, we select digit 3. All positive images contain exactly one instance of the digit 3 randomly placed within the image canvas, while negative instances do not contain any instance. We also include distractors (clutter digits): any MNIST digit image sampled with replacement from a set of labels {0, 1, 2, 4, 5, 6, 7, 8, 9}. We maintain approximately constant clutter density over different O2I ratios. Thus, the following O2I ratios {19.1, 4.8, 1.2, 0.3, and 0.075}% correspond to 2, 5, 25, 100, and 400 clutter objects, respectively. For each value of O2I ratio, we obtain 11276, 1972, 4040 of training, validation and test images 2. Fig. 2 depicts example images for different O2I ratios. We refer the interested reader to the supplementary material for details on image generation process as well as additional dataset visualizations. Histopathology: needle CAMELYON (nCAMELYON). The CAMELYON dataset contains gigapixel hystopathology images with pixel-level lesion annotations from 5 different acquisition sites. We use the pixel-wise annotations to extract crops with controlled O2I ratios. Namely, we generate datasets for O2I ratios in the range of (100−50)%, (50−10)%, (10−1)%, and (1 − 0.1)%, and we crop different image resolutions with the size of 128 × 128, 256 × 256, and 512 × 512 pixels. This in training sets of about 20 − 235 unique lesions per dataset configuration (see supplementary for a detailed list of dataset sizes). More precisely, positive examples are created by taking 50 random crops from every contiguous lesion annotation and rejecting the crop if the O2I ratio does not fall within the desired range. Negative images are taken by randomly cropping healthy images and filtering image crops that mostly contain . We ensure the class balance by sampling an equal amount of positive and negative crops. Once the crops were extracted, no pixel-wise information is used during training. Figure 2 shows examples of extracted images used in the nCAMELYON dataset experiments. We refer to the supplementary for more detail about the data extraction process, the ing dataset sizes and more visualizations. Our classification pipelines follow BagNets backbone, which allows us to explicitly control for the network receptive field size. Figure 3 shows a schematic of our approach. As can be seen, the pipelines are built of three components: topological embedding extractor in Test set accuracy vs. O2I ratio for best models, see text for details. which we can control for embedding receptive field, global pooling operation that converts the topological embedding into a global embedding, and a binary classifier that receives the global embedding and outputs binary classification probabilities. By varying the embedding extractor and the pooling operation, we test a set of 48 different architectures. Topological embedding extractor. The extractor takes as input an image I of size [w img × h img × c img] and outputs a topological embedding E t of shape [w enc × h enc × c enc], where w., h., and c. represent width, height and number of channels. Due to the relatively large image sizes, we train the pipeline with small batch sizes and, thus, we replace BagNet-used BatchNorm operation with Instance Normalization (In the paper, we experiment with four different pooling operations, namely: max, logsumexp, average, and soft attention. In our experiments, we follow the soft attention formulation of . The details about global pooling operations can be found in the supplementary material. In this section, we experimentally test how CNNs' optimization and generalization scale with low and very low O2I ratios. First, we provide details about our experimental setup and then we design experiments to provide empirical evidence to the following questions: Image-level annotations: Is it possible to train classification systems that generalize well in low and very low O2I scenarios? O2I limit vs. dataset size: Is there an O2I ratio limit below which the CNNs will experience generalization difficulties? Does this O2I limit depend on the dataset size? O2I limit vs. model capacity: Do higher capacity models generalize better? Inductive bias -receptive field: Is adjusting receptive field size to match (or exceed) the expected object size beneficial? Global pooling operations: Does the choice of global pooling operation affect model generalization? Finally, we inquire about the optimization ease of the models trained on data with very low O2I ratios. In all our experiments, we used RMSProp with a learning rate of η = 5 · 10 −5 and decayed the learning rate multiplying it by 0.1 at 80, 120 and 160 epochs 3. All models were trained with cross entropy loss for a maximum of 200 epochs. We used an effective batch size of 32. If the batch did not fit into memory we used smaller batches with gradient accumulation. To ensure robustness of our , we run every experiment with six different random seeds and report the mean and standard deviation. Throughout the training we monitored validation accuracy, and reported test set for the model that achieved best validation set performance. (a) mean validation set accuracy heatmap for max pooling operation, and (b) minimum required training set size to achieve the noted validation accuracy. We test training set sizes ∈ {1400, 2819, 5638, 7500, 11276, 22552} and report the minimum amount of training examples that achieve a specific validation performance pooling over different network capacities. In this subsection, we present and discuss the main of our analysis. Unless stated otherwise, the capacity of the ResNet-50 network is about 2.3 · 10 7 parameters. Additional and analysis are presented in the supplementary material. Image-level annotations: For this experiment, we vary the O2I ratio on nMNIST and nCAMELYON to test its influence on the generalization of the network. Figure 4 depicts the for the best configuration according to the validation performance: we use max-pooling and receptive field sizes of 33 × 33 and 9 × 9 pixels for the nMNIST and nCAMELYON datasets, respectively. For the nMNIST dataset, the plot represents the mean over 6 random seeds together with the standard deviation; while for the nCAMELYON dataset we report an average over both the 6 seeds and the crop sizes. We find that the tested CNNs achieve reasonable test set accuracies for the O2I ratios larger than 0.3% for the nMNIST datset and the O2I ratios above 1% for the histopathology dataset. For both datasets, smaller O2I ratios lead to poor or even random test set accuracies. We test the influence of the training set size on model generalization for the nMNIST data, to understand the CNNs' generalization problems for very small O2I ratios. We tested six different dataset sizes 4. Fig. 5a depicts the for max-pooling and a receptive field of 33 × 33 pixels. We observe that larger datasets yield better generalization and this increment is more pronounced for small O2I ratios. For further insights, we plot a heatmap representing the mean validation set 5 for all considered 02Is and training set sizes (Fig. 6a) as well as the minimum number of training examples to achieve a validation accuracy We report only runs that fit the training data. Otherwise we report random accuracy and depict it with a texture on the bars. for (a) the nMNIST dataset and (b) the nCAME-LYON datset. We report only runs that fit the training data. Otherwise we report random accuracy and depict it with a texture on the bars.: nMNIST optimization: (a) number of training epochs needed to fit the 11k training data and (b) the number of successful runs. The textured bars indicate that the model did not fit the training data for all random seeds. of 70% and 85% (Fig. 6b). We observe that in order to achieve good classification generalization the required training set size rapidly increases with the decrease of the O2I ratio. O2I limit vs. capacity: In this experiment, we train networks with different capacities -by uniformly scaling the initial number of filters in convolutional kernels by [6 . We show the CNNs test set performances as a function of the O2I ratio and the network capacity in Figures 5b and 5c for the nMNIST (with 11k training points) and nCAMELYON data, respectively. On nMNIST, we observe a clear trend, where the model test set performance increases with capacity and this boost is larger for smaller O2Is. We hypothesize, that this generalization improvement is due to the model ability to learn-to-ignore the input data noise; with smaller O2I there is more noise to ignore and, thus, higher network capacity is required to solve the task. However, for the nCAMELYON dataset, this trend is not so pronounced and we attribute this to the limited dataset size (more precisely to the small number of unique lesions). These suggest that collecting a very large histopathology dataset might enable training of CNN models using only image level annotations. Inductive bias -receptive field: We report the test accuracy as a function of the O2I ratio and the receptive field size for nMINIST in Figure 7a and for nCAMELYON in Figure 7b. Both plots depict for the global max pooling operation. For nMNIST, we observe that a receptive field that is bigger than the area occupied by one single digit leads to best performances; for example, receptive fields of 33 × 33 and 177 × 177 pixels clearly outperform the smallest tested receptive field of 9 × 9 pixels. However, for the nCAMELYON dataset we observe that the smallest receptive field actually performs best. This suggests that most of the class-relevant information is contained in the texture and that higher receptive fields pick up more spurious correlations, because the capacity of the networks is constant. In this experiment, we compare the performance of four different pooling approaches. We present the relation between test accuracy and pooling function for different O2I ratios with a receptive field of 33 × 33 pixels for nMNIST in Figure 8a and 9 × 9 pixels for nCAMELYON in Figure 8b. On the one hand, for the nMNIST dataset, we observe that for the relatively large O2I ratios, all pooling operations reach similar performance; however, for smaller O2Is we see that max-pooling is the best choice. We hypothesize that the global max pooling operation is best suited to remove nMNIST-type of structured input noise. On the other hand, when using the histopathology dataset, for the smallest O2I mean and soft attention poolings reach best performances; however, these outcomes might be affected by the relatively small nCAMELYON dataset used for training. Optimization: In our large scale nMNIST experiments (when using ≈ 11k datapoints), we observed that some configurations have problems fitting the training data 7. In some runs, after significant efforts put into CNNs hyperparamenter selection, the training accuracy was close to random. To investigate this issue further, we followed the setup of randomized experiments from (; and we substituted the nMNIST datapoints with samples from an isotropic Gaussian distribution. On the one hand, we observed that all the tested setups of our pipeline were able to memorize the Gaussian samples, while, on the other hand, most setups were failing to memorize the same-size, nMNIST datataset for small and very small O2I ratios. We argue that the nMNIST structured noise and its compositionality may be a "harder" type of noise for the CNNs than Gaussian isotropic noise. To provide further experimental evidence, we depict average time-to-fit the training data (in epochs) in Fig. 9a as well as number of successful optimizations in Fig. 9b for different O2I ratios and pooling methods 8. We observe that the optimization gets progressively harder with decreasing O2I ratio (with max pooling being the most robust). Moreover, we note that the are consistent across different random seeds, where all runs either succeed or fail to converge. Reasoning about tiny objects is of high interest in many computer vision areas, such as medical imaging (; ; ; ;) and remote sensing . To overcome the low signal-to-noise ratio, most approaches rely on manual dataset "curation" and collect additional pixel-level annotations such as landmark positions , bounding boxes or segmentation maps . This additional annotation allows to transform the original needle-in-a-haystack problem into a less noisy but imbalanced classification problem (; ; Bándi et al., 2019). However, collecting pixel level annotations has a significant cost and might require expert knowledge, and as such, is a bottleneck in the data collection process. Other approaches leverage the fact that task-relevant information is often not uniformly distributed across input data, e.g. by using attention mechanisms to process very high-dimensional inputs (; ; ;). However, those approaches are mainly motivated from a computational perspective trying to reduce the computational footprint at inference time. Some recent research has also studied attention based approaches both in the context of multi-instance learning and histopathology image classification . However, neither of the works report the exact O2I ratio used in the experiments. In this subsection, we briefly highlight the dimensions of optimization and generalization of CNN that are handy in low O2I classification scenarios. Model capacity. For fixed training accuracy, over-parametrized CNNs tend to generalize better . In addition, when properly regularized and given a fixed size dataset, higher capacity models tend to provide better performance . However, finding proper regularization is not trivial . Dataset size. CNN performance improves logarithmically with dataset size . Moreover, in order to fully exploit the data benefit, the model capacity should scale jointly with the dataset size . Model inductive biases. Inductive biases limit the space of possible solutions that a neural network can learn . Incorporating these biases is an effective way to include data (or domain) specific knowledge in the model. Perhaps the most successful inductive bias is the use of convolutions in CNNs. Different CNN architectures (e. g. altering network connectivity) also lead to improved model performance . Additionally, it has been shown on the ImageNet dataset that CNN accuracy scales logarithmically with the size of the receptive field . Although low input image signal-to-noise scenarios have been extensively studied in signal processing field (e.g. in tasks such as image reconstruction), less attention has been devoted to low signal-tonoise classification scenarios. Thus, in this paper we identified an unexplored machine learning problem, namely image classification in low and very low signal-to-noise ratios. In order to study such scenarios, we built two datasets that allowed us to perform controlled experiments by manipulating the input image signal-to-noise ratio and highlighted that CNNs struggle to show good generalization for low and very low signal-to-noise ratios even for a relatively elementary MNIST-based dataset. Finally, we ran a series of controlled experiments 9 that explore both a variety of CNNs' architectural choices and the importance of training data scale for the low and very low signal-to-noise classification. One of our main observation was that properly designed CNNs can be trained in low O2I regime without using any pixel-level annotations and generalize if we leverage enough training data; however, the amount of training data required for the model to generalize scales rapidly with the inverse of the O2I ratio. Thus, with our paper (and the code release) we invite the community to work on data-efficient solutions to low and very low signal-to-noise classification. Our experimental study exhibits limitations: First, due to the lack of large scale datasets that allow for explicit control of the input signal-to-noise ratios, we were forced to use the synthetically built nMNIST dataset for most of our analysis. As a real life dataset, we used crops from the histopathology CAMELYON dataset; however, due to relatively a small number of unique lesions we were unable to scale the histopathology experiments to the extent as the nMNIST experiments, and, as , some might be affected by the limited dataset size. Other large scale computer vision datasets like MS COCO exhibit correlations of the object of interest with the image . For MS COCO, the smallest O2I ratios are for the object category "sports ball" which on average occupies between 0.3% and 0.4% of an image and its presence tends to be correlated with the image (e. g. presence of sports fields and players). However, future research could examine a setup in which negative images contain objects of the categories "person" and "baseball bat" and positive images also contain "sports ball". Second, all the tested models improve the generalization with larger dataset sizes; however, scaling datasets such as CAMELYON to tens of thousands of samples might be prohibitively expensive. Instead, further research should be devoted to developing computationally-scalable, data-efficient inductive biases that can handle very low signal-to-noise ratios with limited dataset sizes. Future work, could explore the knowledge of the low O2I ratio and therefore sparse signal as an inductive bias. Finally, we studied low signal-to-noise scenarios only for binary classification scenarios 10; further investigation should be devoted to multiclass problems. We hope that this study will stimulate the research in image classification for low signal-to-noise input scenarios. In this section, we provide additional details about the datasets used in our experiments. The needle MNIST (nMNIST) dataset is designed as a binary classification problem: Is there a 3 in this image?'. To generate nMINST, we use the original training, validation and testing splits of the MNIST dataset and generate different nMINST subsets by varying the object-to-image (O2I) ratio, ing in O2I ratios of 19.1%, 4.8%, 1.2%, 0.3%, and 0.075%. We define positive images as the ones containing exactly one digit 3 and negative images as images without any instance of it. We keep the original MNIST digit size and place digits randomly onto a clear canvas to generate a sample of the nMNIST dataset. More precisely, we adapt the O2I ratio by changing the the canvas size, ing in nMNIST image resolution being in 64 × 64, 128 × 128, 256 × 256, 512 × 512, and 1024 × 1024 pixels. To assign MNIST digits to canvas, we split the MNIST digits into two subsets: digit-3 versus clutter (any digit from a set of {0, 1, 2, 4, 5, 6, 7, 8, 9}). For the positive nMNIST images, we sample one digit 3 (without replacement) and n digits (with replacement) from the digit-3 and clutter subsets, respectively. For the negative nMNIST images, we sample n + 1 instances from the clutter subset. We adapt n to keep approximately constant object density for all canvas and choose n to be 2, 5, 25, 100, and 400 for canvas resolutions 64 × 64, 128 × 128, 256 × 256, 512 × 512, and 1024 × 1024, respectively. As , for each value of O2I ratio, we obtain 11276, 1972, 4040 of training, validation and testing images, out of which 50% are negative and 50% are positive images. We present both positive and negative samples for different O2I ratios in Figure 10. The needle CAMELYON (nCAMELYON) is designed as a binary classification task: Are there breast cancer metastases in the image or not?. We rely on the pixel-level annotations within CAMELYON to extract samples for nCAMELYON. We use downsampling level 3 from the original whole slide image using the MultiResolution Image interface released with the original CAMELYON dataset. For positive examples, we identify contiguous regions within the annotations, and take 50 random crops around each contiguous region ensuring that the full contiguous region is inside the crop, and total number of lesion pixels inside the crop are in the desired O2I ratio. The negative crops are taken from healthy images randomly filtering for images that are mostly using a heuristic that the average green pixel value in the crop is below 200. Since the CAMELYON dataset contains images acquired by 5 different centers, we split training, validation and test sets center-wise to avoid any contamination of data across the three sets. All crops coming from center 3 are part of the validation set, and all crops coming from center 4 are part of the test set. All images are generated for resolutions 128 × 128, 256 × 256, 512 × 512, and 1024 × 1024 and are split into 4 different O2I ratios: (100 − 50)%, (50 − 10)%, (10 − 1)%, and (1 − 0.1)%. Figure 11 shows examples of images from nCAMELYON dataset, Table 1 presents number of unique lesions in each dataset, and Table 2 depicts number of dataset images stratified for image resolution and O2I ratios. Because center 3 does not contain lesions of suitable size for crops of with resolution 128 × 128 and O2I ratio (50 − 100)%, we do not include those training runs in our analysis. In this section, we provide additional details about the pipeline used in the experiments. More precisely, we formally define global pooling operations and provide detailed description of the different architectures. In our experiments, we are testing four different global pooling functions: max-pooling, meanpooling, logsumexp and soft attention. The max pooling operation simply returns the maximum value per each channel in the topological embedding. This operation can be formally defined as: Note, that we use subscript notation to denote dimensions of the embedding. The max pooling operation has a spacing effect on gradient backpropagation, during the backward pass through the model all information will be propagated through the embedding position that corresponds to the maximal value. In order to improve gradient backpropagation, one could apply logsumexp pooling, a soft approximation to max pooling. This pooling operation is defined as: Alternatively, one could use an average pooling operation that computes mean value for each channel in the topological embedding. This pooling operation can be formally defined as follows: Finally, attention based pooling include additional weighting tensor a of dimension (w enc × h enc × c enc) that rescales each topological embedding before averaging them. This operation can be formally defined as: s.t. In our experiments, following , we parametrize the soft-attention mechanisms as a [w,h] = sof tmax(f (E spat)) [w,h], where f (·) is modelled by two fully connected layers with tanh-activation and 128 hidden units. We adapt the BagNet architecture proposed in . An overview of the architectures for the tested three receptive field sizes is shown in Table 3. We depict the layers of Figure 12: Impact of the training set balance on model accuracy for different pooling operations and receptive field sizes. residual blocks in brackets and perform downsampling using convolutions with stride 2 within the first residual block. Note that the architectures for different receptive fields differ in the number of 3 × 3 convolutions. The rightmost column shows a regular ResNet-50 model. The receptive field is decreased by replacing 3 × 3 convolutions with 1 × 1 convolutions. We increase the number of convolution filters by a factor of 2.5 if the receptive field is reduced to account for the loss of the trainable parameters. Moreover, when testing different network capacities we evenly scale the number of convolutional filters by multiplying with a constant factor of s ∈ {1/4, 1/2, 1, 2}. In this section, we provide additional experimental as well as additional visualizations of the experiments presented in the main body of the paper. In many medical imaging datasets, it is common to be faced with class-imbalanced datasets. Therefore, in this experiment, we use our nMNIST dataset and test CNNs generalization under moderate and severe class imbalanced scenario. We alter the training set class balance by altering the proportion of positive images in the training dataset and use the following balance values 0.01, 0.1, 0.25, 0.5, 0.75, 0.9 and 0.99, where a value of 0.01 means almost no positive examples and 0.99 indicates very low number of negative images available at training time. Moreover, we ensure that the dataset size is constant (≈ 11k) and only the class-balance is modified. We run the experiments using the O2I ratio of 1.2%, three receptive field sizes (9 × 9, 33 × 33 and 177 × 177 pixels) and four pooling operations (mean, max, logsumexp and soft attention). For each balance value, we train 6 models using 6 random seeds and we oversample the underrepresented class. The are depicted in Figure 12. We observe that the model performance drops as the the training data becomes more unbalanced and that max pooling and logsumexp seem to be the most robust to the class imbalance. We also tested the effect of model capacity increase while having access only to a small dataset (3k class-balanced images) and contrast it with a larger dataset of ≈ 11k training images. We run this experiment on the nMNIST dataset using a network with 2.3 · 10 7 parameters using global max pooling operation and there different receptive field sizes: 9 × 9, 33 × 33 and 177 × 177 pixels. The are depicted in Figure 13. It can be seen that the model's capacity increase does not lead to better generalization, for small size datasets of ≈ 3k. In this section, we report additional for all tested global pooling operations on O2I limit vs. dataset size. We plot a heatmaps representing the validation set for all considered 02I and training set sizes (Figure 14) as well as the minimum number of training examples required to achieve a validation accuracy of 70% and 85% (Figure 15) Figure 13: Impact of the network capacity on the generalization performance dependent on the training set size for nMNIST at O2I ratio = 1.2%. The improvement based on the increased network capacity shrinks with smaller training set. Figure 14: Testing the O2I limit. Validation set accuracy heatmap for max, logsumexp, mean and soft attention poolings. We test training set sizes ∈ {1400, 2819, 5638, 7500, 11276, 22552} and report the average validation accuracy. We test the object localization capabilities of the trained classification models by examining their saliency maps. Figure 16 shows examples of the nMNIST dataset with the object bounding box in Figure 15: Testing the O2I limit. Minimum required training set size to achieve the noted validation accuracy. We test training set sizes ∈ {1400, 2819, 5638, 7500, 11276, 22552} and report the minimum amount of training examples that achieve a specific validation performance pooling over different network capacities. Figure 16: Example images from the nMNIST validation set and their corresponding saliency maps in red. We generate the saliency maps by calculating the absolute of the gradients with respect to the input image using max-pooling, a receptive field of 33, and ResNet-50 capacity. From top to bottom, we show random examples for O2I ratios of {19.14, 4.79, 1.20, 0.30}%. We annotate the object of interest with a blue outline. The captions show the true label y and the predictionŷ. Figure 17: Average precision for detecting the object of interest using the saliency maps for nMNIST. We adapt and use the localize an object by the maximum magnitude of the saliency. We use the magnitude of the saliency as the confidence of the detection. We count wrongly localised objects both as false positive and false negative. For images without object of interest, the we increase the false positive count only. We plot for max-pooling, a receptive field of 33, a training set with 11276 examples and ResNet-50 capacity. (a) shows the dependence of the AP on the pooling method using RF = 33 × 33, (b) shows the dependence on the receptive field using max-pooling. blue and the magnitude of the saliency in red. We rescale the saliency to for better contrast. However, this prevents the comparison of absolute saliency values across different images. In samples containing an object of interest, the models correctly assign high saliency to the regions surrounding the relevant object. On negative examples, the network assigns homogenous importance to all objects. We localise an object of interest as the location with maximum saliency. We follow to quantitatively examine the object detection performance using the saliency maps of the models. We plot the corresponding average precision in Figure 17. We find that the detection performance deteriorates for smaller O2I ratios regardless of the method. This is aligned with the classification accuracy. For small O2I ratios, max-pooling achieves the best detection scores. On larger O2I ratios, logsumexp achieves the best scores.

Represent the TLDR for finding the corresponding scientific article:

We study low- and very-low-signal-to-noise classification scenarios, where objects that correlate with class label occupy tiny proportion of the entire image (e.g. medical or hyperspectral imaging).

Deep neural networks trained with data augmentation do not require any other explicit regularization (such as weight decay and dropout) and exhibit greater adaptaibility to changes in the architecture and the amount of training data.