Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We present a novel black-box adversarial attack algorithm with state-of-the-art model evasion rates for query efficiency under $\ell_\infty$ and $\ell_2$ metrics. It exploits a \textit{sign-based}, rather than magnitude-based, gradient estimation approach that shifts the gradient estimation from continuous to binary black-box optimization. It adaptively constructs queries to estimate the gradient, one query relying upon the previous, rather than re-estimating the gradient each step with random query construction. Its reliance on sign bits yields a smaller memory footprint and it requires neither hyperparameter tuning or dimensionality reduction. Further, its theoretical performance is guaranteed and it can characterize adversarial subspaces better than white-box gradient-aligned subspaces. On two public black-box attack challenges and a model robustly trained against transfer attacks, the algorithm's evasion rates surpass all submitted attacks. For a suite of published models, the algorithm is $3.8\times$ less failure-prone while spending $2.5\times$ fewer queries versus the best combination of state of art algorithms. For example, it evades a standard MNIST model using just $12$ queries on average. Similar performance is observed on a standard IMAGENET model with an average of $579$ queries. Problem. Deep Neural Networks (DNNs) are vulnerable to adversarial examples, which are malicious inputs designed to fool the model's prediction-see for a comprehensive, recent overview of adversarial examples. Research on generating these malicious inputs started in the white-box setting, where access to the gradients of the models is assumed. Since the gradient points to the direction of steepest ascent, an input can be perturbed along the gradient's direction to maximize the network's loss, thereby potentially causing misclassification under class prediction, e.g. with images, or evasion under detection, e.g. with malware. The assumption of access to the underlying gradient does not however reflect real world scenarios. Attack algorithms under a more realistic, restrictive black-box threat model, which assumes access to predictions in lieu of gradients, are therefore studied. Central to their approaches is estimating the gradient. To estimate the magnitudes and signs of the gradient, the community at large has formulated a continuous optimization problem of O(n) complexity where n is the input dimensionality. Most recently work has sought to reduce this complexity by means of data-/time-dependent priors. In this paper, we take a different tact and reduce the central problem to just estimating the signs of the gradients. Our intuition arises from observing that estimating the sign of the top 30% gradient coordinates by magnitude is enough to achieve a rough misclassification rate of 70%. Figure 1 reproducing illustrates this observation for the MNIST dataset-see Appendix A for other datasets. Therefore our goal is to recover the sign of the gradient with high query efficiency so we can use it to generate adversarial examples as effective as those generated by full gradient estimation approaches. Related Work. We organize the related work in two themes, namely Adversarial Example Generation and Sign-Based Optimization. The literature of the first theme primarily divides into white-box and black-box settings. The white-box setting, while not the focus of this work, follows from the works of and who introduced the Fast Gradient Sign Method (FGSM), including several methods to produce adversarial examples for various learning tasks and threat perturbation constraints (; ; ; ; ;). Turning to the blackbox setting and iterative optimization schemes, , without using any gradient information, use a naive policy of perturbing random segments of an image to generate adversarial examples. reduce the dimensions of the feature space using Principal Component Analysis (PCA) and random feature grouping, before estimating gradients. introduce a principled approach by using gradient based optimization. They employ finite differences, a zeroth-order optimization means, to estimate the gradient and then use it to design a gradient-based attack. While this approach successfully generates adversarial examples, it is expensive in how many times the model is queried. substitute traditional finite differences methods with Natural Evolutionary Strategies (NES) to obtain an estimate of the gradient. provide an adaptive random gradient estimation algorithm that balances query counts and distortion, and introduces a trained auto-encoder to achieve attack acceleration. extend this line of work by proposing the idea of gradient priors and bandits: Bandits T D. Our work contrasts with the general approach of these works in two ways: a) We focus on estimating the sign of the gradient and investigate whether this estimation suffices to efficiently generate adversarial examples. b) The above methods employ random sampling in constructing queries to the model while our construction is adaptive. 1 Another approach involves learning adversarial examples for one model (with access to its gradient information) to transfer them against another . use a Generative Adversarial Network (GAN) to generate adversarial examples which are based on small norm-bounded perturbations. These methods involve learning on a different model, which is expensive, and not amenable to comparison with setups-including ours-that directly query the model of interest. Figure 1: Misclassification rate of an MNIST model on the noisy FGSM's adversarial examples as a function of correctly estimated coordinates of sign(∇ x f (x, y)) on 1000 random MNIST images. Estimating the sign of the top 30% gradient coordinates (in terms of their magnitudes) is enough to achieve a rough misclassification rate of 70%. More details can be found in Appendix A. Sign-Based Optimization. In the context of generalpurpose continuous optimization methods, signbased stochastic gradient descent was studied in both zeroth-and first-order setups. In the latter, analyzed signSGD, a sign-based Stochastic Gradient Descent, and showed that it enjoys a faster empirical convergence than SGD in addition to the cost reduction of communicating gradients across multiple workers. extended signSGD to zeroth-order setup with the ZO-SignSGD algorithm. ZO-SignSGD was shown to outperform NES against a blackbox model on MNIST. These approaches use the sign of the gradient (or its zero-order estimate) to achieve better convergence, whereas our approach both estimates and uses the sign of the gradient. Contributions. We present the following contributions at the intersection of adversarial machine learning and black-box (zeroth-order) optimization: 1) We exploit the separability property of the directional derivative of the loss function of the model under attack in the direction of {±1} n vectors, to propose a divide-and-conquer, adaptive, memory-efficient algorithm, we name SignHunter, to estimate the gradient sign bits. 2) We provide a worst-case theoretical guarantee on the number of queries required by SignHunter to perform at least as well as FGSM , which has access to the model's gradient. To our knowledge, no black-box attack from the literature offers a similar performance guarantee. 3) We evaluate our approach on a rigorous set of experiments on both, standard and adversarially hardened models. All other previous works on this topic have published their on a subset of the datasets and threat models we experimentally validate in this work. Through these experiments, we demonstrate that SignHunter's adaptive search for the gradient sign allows it to craft adversarial examples within a mere fraction of the theoretical number of queries thus outperforming FGSM and state-of-the-art black-box attacks. 4) We release a software framework to systematically benchmark adversarial black-box attacks, including SignHunter's, on MNIST, CIFAR10, and IMAGENET models in terms of success rate, query count, and other metrics. 5) We demonstrate how SignHunter can be used to characterize adversarial cones in a black-box setup and in doing so, highlight the gradient masking effect. Notation. Let n denote the dimension of datapoint x. Denote a hidden n-dimensional binary code by q *. That is, q * ∈ H ≡ {−1, +1} n. Further, denote the directional derivative of some function f at a point x in the direction of a vector v by D v f (x) ≡ v T ∇ x f (x) which often can be approximated by the finite difference method. That is, for δ > 0, we have Let Π S (·) be the projection operator onto the set S, B p (x,) be the p ball of radius around x. At the heart of black-box adversarial attacks is generating a perturbation vector to slightly modify the original input x so as to fool the network prediction of its true label y. Put differently, an adversarial example x maximizes the network's loss L(x, y) but still remains -close to the original input x. Although the loss function L can be non-concave, gradient-based techniques are often very successful in crafting an adversarial example. That is, setting the perturbation vector as a step in the direction of ∇ x L(x, y). Consequently, the bulk of black-box attack methods try to estimate the gradient by querying an oracle that returns, for a given input/label pair (x, y), the value of the network's loss L(x, y), consulting prediction or classification accuracy. Using only such value queries, the basic approach relies on the finite difference method to approximate the directional derivative (Eq. 1) of the function L at the input/label pair (x, y) in the direction of a vector v, which, one can construct a linear system of equations to recover the full gradient. Clearly, this approach's query complexity is O(n), which can be prohibitively expensive for large n (e.g., n = 268, 203 for the IMAGENET dataset). Recent works try to mitigate this issue by exploiting data-and/or time-dependent priors (; ; 2019). However, the queries are not adaptive, they are constructed based on i.i.d. random vectors {v i}. They fail to make use of the past queries' responses to construct the new query and recover the full gradient more efficiently. As stated in the introduction, we solve the smaller problem of gradient sign estimation with adaptive queries based on the observation that simply leveraging (noisy) sign bits of the gradient yields successful attacks-see Figure 1. Definition 1. (Gradient Sign Estimation Problem) For an input/label pair (x, y) and a loss function L, let g * = ∇ x L(x, y) be the gradient of L at (x, y) and q * = sign(g *) ∈ H be the sign bit vector of g *. 2 Then the goal of the gradient sign estimation problem is to find a binary vector q ∈ H maximizing the directional derivative from a limited number of (possibly adaptive) function value queries L(x, y). Our goal is to estimate the gradient sign bits of the loss function L of the model under attack at an input/label pair (x, y) from a limited number of loss value adaptive queries L(x, y). To this end, we examine the basic concept of directional derivatives that has been employed in recent black-box 2 Without loss of generality, we encode the sign bit vector in H ≡ {−1, +1} n rather than {0, 1} n. This is a common representation in sign-related literature. Note that the standard sign function has the range of {−1, 0, +1}. Here, we use the non-standard definition whose range is {−1, +1}. This follows from the observation that DNNs' gradients with respect to their inputs are not sparse (, Appendix B.1). 3 The maximization follows from DqL(x, y) = q T g *, which is maximized when q = q * = sign(g *). adversarial attacks. Based on the definition of the directional derivative (Eq. 1), the following can be stated. of the loss function L at an input/label pair (x, y) in the direction of a binary code q is separable. That is, Algorithm 1 SignHunter g: H → R: the black-box function to be maximized over the binary hypercube return done This reformulates the gradient sign estimation problem from single n-dimensional to n 1-dimensional binary black-box optimization problems, reducing the search space of sign bits from 2 n to 2n. Subsequently, one could recover the gradient sign bits with n + 2 queries as follows: i. Start with an arbitrary sign vector q and compute the directional derivative D q L(x, y). Using Eq. 1, this requires two queries: L(x+δq, y) and L(x, y). ii. For the remaining n queries, flip q's bits (coordinates) one by one and compute the corresponding directional derivativeone query each L(x + δq, y). iii. Retain bit flips that maximize the directional derivative D q L(x, y) and revert those otherwise. This, however, still suffers from the O(n) complexity of full gradient estimation methods. Further, each query recovers at most one sign bit and the natural question to ask is: can we recover more sign bits per query? Consider the case where all the gradient coordinates have the same magnitude, i.e., |{|g * i |} 1≤i≤n |= 1, and let the initial guess q 1 have r correct bits and n − r wrong ones. Instead of flipping its bits sequentially, we can flip them all at once to get, then we retain q 2 as our best guess with n − r correct bits, otherwise q 1 remains. In either cases, with three queries, we will recover max(r, n − r) sign bits. One can think of this flip/revert procedure as one of majority voting by the guess's coordinates on whether they agree with their gradient sign's counterparts. To see this, let |g * i |= 1 for all i, then the condition D q2 L(x, y) ≥ D q1 L(x, y) can be written as n − r − r ≥ r − n + r =⇒ n ≥ 2r. If the agree votes r are less than half of the total votes n, then q 2 is retained. Besides flipping all the coordinates, one can employ the same procedure iteratively on a subset (chunk) of the coordinates [q j, . . ., q j+ni] of the guess vector q, recovering max(r i, n i − r i) sign bits, where n i and r i is the length of the ith chunk and the number of its correct signs, respectively. While the magnitudes of gradient coordinates may not have the same value as assumed in the previous example; through empirical evaluation (see Appendix F), we found them to be concentrated. Consequently and with high probability, their votes on retaining or reverting chunks of sign flips are weighted (by their corresponding gradient magnitude) similarly. That said, if we are at a chunk where the distribution of the gradient coordinate magnitudes is uniform, then the flip/revert procedure could favor recovering few sign coordinates with large magnitude counterparts over many sign coordinates with small magnitude counterparts. From our experiments on the noisy FGSM, this still suffices to generate adversarial examples: an attack with 30% correct sign bits (that correspond to the top gradient coordinates magnitudes) is more effective than an attack with 50% correct arbitrary sign bits as shown in Figure 1. Put differently, we would like to recover as many sign bits as possible with as few queries as possible. However, if we can only recover few, they should be those that correspond to coordinates with large gradient magnitude. This notion is in line with the flip/revert procedure. We employ the above observation in a divide-and-conquer search which we refer to as SignHunter. As outlined in Algorithm 1, the technique starts with an initial guess of the sign vector q 1 (s in Algorithm 1). It then proceeds to flip the sign of all the coordinates to get a new sign vector q 2, and revert the flips if the loss oracle returned a value L(x + δq 2, y) (or equivalently the directional derivative) less than the best obtained so far L(x + δq 1, y). SignHunter applies the same rule to the first half of the coordinates, the second half, the first quadrant, the second quadrant, and so on. For a search space of dimension n, SignHunter needs 2 log(n)+1 − 1 sign flips to complete its search. If the query budget is not exhausted by then, one can update x with the recovered signs and restart the procedure at the updated point with a new starting code s. If we start with a sign vector whose Hamming distance to the optimal sign vector q * is n/2: agreeing with q * in the first half of coordinates. In this case, SignHunter needs just four queries to recover the entire sign vector independent of n, whereas the sequential bit flipping still require n + 2 queries. In the next theorem, we show that SignHunter is guaranteed to perform at least as well as FGSM with O(n) oracle queries. Up to our knowledge, no such guarantees exist for any black-box attack from the literature. Theorem 1. (Optimality of SignHunter) Given 2 log(n)+1 queries and that the directional derivative is well approximated by the finite-difference (Eq. 1), SignHunter is at least as effective as FGSM in crafting adversarial examples. The proof can be found in Appendix B. Theorem 1 provides an upper bound on the number of queries required for SignHunter to recover the gradient sign bits, and perform as well as FGSM. In practice (as will be shown in our experiments), SignHunter crafts adversarial examples with a small fraction of this upper bound. The rationale here is that we do not need to recover the sign bits exactly; we rather need a fast convergence to an adversarially helpful sign vector s. In our setup, we use the best sign estimation obtained s so far in a similar fashion to FGSM, whereas full-gradient estimation approaches often employ an iterative scheme of T steps within the perturbation ball B p (x,), calling the gradient estimation routine in every step leading to a search complexity of nT. Instead, our gradient sign estimation routine runs at the top level of our adversarial example generation procedure. Further, SignHunter is amenable to parallel hardware architecture and has a smaller memory footprint (just sign bits) and thus can carry out attacks in batches more efficiently. Crafting black-box adversarial attacks with SignHunter is outlined in Algorithm 2. Algorithm 2 Black-Box Adversarial Example Generation with SignHunter x init: input to be perturbed, yinit: xinit's true label, Bp(.,): p perturbation ball of radius L: loss function of the model under attack 1: δ ← // set finite-difference probe to perturbation bound 2: x o ← x init 3: Define the function g as SignHunter.step s ← SignHunter.get_current_sign_estimate 9: if SignHunter.is_done then 11: Define the function g as in Line 3 (with x o update) SignHunter.init(g) 14: return x We evaluate SignHunter and compare it with established algorithms from the literature: , , and Bandits T D in terms of effectiveness in crafting (without loss of generality) untargeted black-box adversarial examples. To highlight SignHunter's adaptive query construction, we introduce a variant of Algorithm 2, named Rand. At every iteration, Rand's sign vector is sampled uniformly from H. 4. Both ∞ and 2 threat models are considered on the MNIST, CIFAR10, and IMAGENET datasets. Code and data for the experiments can be found at https://bit.ly/3acIHoQ. Experiments Setup. Our experiment setup is similar to . Each attacker is given a budget of 10, 000 oracle queries per attack attempt and is evaluated on 1000 images from the test sets of MNIST, CIFAR10, and the validation set of IMAGENET. We did not find a standard practice for setting the perturbation bound. Figure 2: Performance of black-box attacks in the ∞ and 2 perturbation constraint. The plots show the average number of queries used per successful image for each attack when reaching a specified success rate. 2019), which are smaller than the one used in . We use the observed bound in for CIFAR10. We show based on standard models-i.e., models that are not adversarially hardened. For MNIST and CIFAR10, the naturally trained models from (Hyperparameters Setup. While SignHunter does not have any hyperparameters, to fairly compare it with the other algorithms, we tuned their hyperparameters starting with the default values reported by the corresponding authors. The finite difference probe δ for SignHunter is set to the perturbation bound as it is used for both computing the finite difference and crafting the adversarial examplessee Line 1 in Algorithm 2. This tuning-free aspect of SignHunter offers a robustness advantage over algorithms which require expert hypertuning. Details on the hyperparameter setup are available in Appendix C. Results. Figure 2 shows the trade-off between the success (evasion) rate and the mean number of queries (of the successful attacks, per convention) needed to generate an adversarial example for the MNIST, CIFAR10, and IMAGENET classifiers under the ∞ and 2 perturbation constraints. These plots indicate the average number of queries required for a desired success rate. Table 1 represents a tabulated summary of plots (b) and (e) of Figure 2. 5 We observe the following: For any given success rate, SignHunter dominates the previous state of the art approaches in all settings except the IMAGENET 2 setup, where Bandits T D shows a better query efficiency when the desired success rate is roughly greater than 0.35. This is all the more remarkable because Bandits T D exploits tiles, a data-dependent prior, searching over 50 × 50 × 3 dimensions for IMAGENET, while SignHunter searches over the explicit data 299 × 299 × 3 dimensions: 36× more dimensions. ∞ vs. 2 Perturbation Threat. In view of Bandits T D's advantage, SignHunter is remarkably efficient in the ∞ setup, achieving a 100% evasion using-on average-just 12 queries per image against the MNIST classifier! In the 2 setup, SignHunter's performance degrades-yet it still outperforms the other algorithms. This is expected, since SignHunter perturbs all the coordinates with the same magnitude and the 2 perturbation bound 2 for all the datasets in our experiments is set such that 2 / √ n is significantly less than the ∞ perturbation bound ∞. Take the case of MNIST (n = 28 × 28), where ∞ = 0.3 and 2 = 3. For SignHunter, the 2 setup is equivalent to an ∞ perturbation bound of 3/28 ≈ 0.1. The employed 2 perturbation bounds give the state of the art-continuous optimization based-approaches more perturbation options. For instance, it is possible for NES to perturb just one pixel in an MNIST image by a magnitude of 3; two pixels by a magnitude of 3/ √ 2 ≈ 2.1 each; ten pixels by a magnitude of 3/ √ 10 ≈ 0.9 each, etc. On the other hand, the binary optimization view of SignHunter limits it to always perturb all 28 × 28 pixels by a magnitude of 3/28 ≈ 0.1. Despite its fewer degrees of freedom, SignHunter maintains its effectiveness in the 2 setup. The plots can also be interpreted as a sensitivity assessment of SignHunter as gets smaller going from ∞ to the 2 perturbation threat. SignHunter vs FGSM. The performance of SignHunter is in line with Theorem 1 when compared with the performance of FGSM (the noisy FGSM at k = 100% in Figures 1 and 2 of Appendix A) in both ∞ and 2 setups across all datasets. For instance, FGSM has a failure rate of 0.32 for CIFAR10 2 (Appendix A, Figure 2 (b)), while SignHunter achieves a failure rate of 0.21 with 692.39 < 2n = 2 × 3 × 32 × 32 = 6144 queries (Appendix D, Table 8). Note that for IMAGENET, SignHunter outperforms FGSM with a query budget of 10, 000 queries, a fraction of the theoretical number of queries required 2n = 536, 406 to perform at least as well. Incorporating SignHunter in an iterative framework of perturbing the data point x till the query budget is exhausted (Lines 10 to 14 in Algorithm 2) supports the observation in white-box settings that iterative FGSM-or Projected Gradient Descent (PGD)-is stronger than FGSM . This is evident by the upticks in SignHunter's performance on the MNIST 2 case (Appendix D, Figure 4), which happens after every iteration (after every other 2 × 28 × 28 queries). Gradient Estimation. Plots of the Hamming similarity capture the number of recovered sign bits, while plots of the average cosine similarity capture the value of Eq. 2. Both SignHunter and Bandits T D consistently optimize both metrics. In general, SignHunter (Bandits T D) converges faster especially on the Hamming(cosine) metric as it is estimating the signs(signs and magnitudes) compared to Bandits T D's full gradient (SignHunter's gradient sign) estimation. This is most obvious in the IMAGENET 2 setup (Appendix D, Figure 6). Note that once an attack is successful, the estimated gradient sign at that point is used for the rest of the plot. This explains why, in the ∞ settings, SignHunter's plot does not improve compared to its 2 counterpart, as most of the attacks are successful in the very first few queries made to the loss oracle and no further refined estimation is required. Another possible reason is that the gradient direction can be very local and does not capture the global loss landscape compared to SignHunter's estimation. More on this is discussed in Section 6. SignHunter vs. Rand. Given these , one could argue that SignHunter is effective, because it maximally perturbs datapoints to the vertices of their perturbation balls. 6 However, Rand's poor performance does not support this argument and highlights the effectiveness of SignHunter's adaptive query construction. Except for MNIST and CIFAR10 ∞ settings, Rand performs worse than the full-gradient estimation approaches, although it perturbs datapoints similar to SignHunter. Overall, SignHunter is 3.8× less failure-prone than the state-of-the-art approaches combined, and spends over all the images (successful and unsuccessful attacks) 2.5× less queries. To complement Section 4, we evaluate SignHunter against adversarial training, a way to improve the robustness of DNNs . Specifically, we attacked the secret models used 6 We define perturbation vertices as extreme points of the ball Bp(x,). That is, x ± ∞, where ∞ = when p = ∞ and ∞ = / √ n when p = 2. 7 The number of queries spent is computed based on Tables 7-9 of Appendix D as (1 -fail_rate) * avg_#_queries + fail_rate * 10,000. in public challenges for MNIST and CIFAR10. For IMAGENET, we used ensemble adversarial training, a method that argues security against black-box attacks based on transferability Tramèr et al. (2017a). Appendix E reports the same metrics used in Section 4 as well as a tabulated summary for the discussed below. Public MNIST Black-Box Attack Challenge. In line with the challenge setup, 10, 000 test images were used with an ∞ perturbation bound of = 0.3. Although the secret model is released, we treated it as a black box similar to our experiments in Section 4. No maximum query budget was specified, so we set it to 5, 000 queries. This is equal to the number of iterations given to a PGD attack in the white-box setup of the challenge: 100-steps with 50 random restarts. SignHunter's attacks ed in the lowest model accuracy of 91.47%, outperforming all the submitted attacks to the challenge, with an average number of queries of 233 per successful attack. Note that the attacks submitted to the challenge are based on transferability and do not query the model of interest. On the other hand, the most powerful white-box attack by. The Gradient-Aligned Adversarial Subspace (GAAS) method Tramèr et al. (2017b) provides an approximation of the adversarial cone dimensionality by finding a set of orthogonal perturbations of norm that are all adversarial with respect to the model. By linearizing the model's loss function, this is reduced to finding orthogonal vectors that are maximally aligned with its gradient g * -or its gradient sign q * in the ∞ setup Tramèr et al. (2017a). In Figure 3, we reproduce (Tramèr et al., 2017a, Fig. 2) and show that aligning the orthogonal vectors with SignHunter's estimation (we refer to this approach as SAAS) instead of aligning them with the gradient (GAAS) in a better approximation of the adversarial cone for the two IMAGENET models considered earlier, even when the number of queries given to SignHunter is just a fraction of the dimensionality n. Through its query-efficient finite-difference sign estimation, SignHunter is able to quickly capture the larger-scale variation of the loss landscape in the point's neighborhood, rather than the infinitesimal point-wise variation that the gradient provides, which can be very local. This is important in adversarial settings, where the loss landscape is analyzed in the vicinity of the point; Tramèr et al. (2017a). One interesting observation at k = 1 (note here, r 1 = q *) across all is that GAAS finds adversarial directions for fewer points against the v3 adv-ens4 model than the naturally trained model v3, whereas SAAS reports similar probability of adversarial directions for both. This contrast suggests that ensemble adversarial training Tramèr et al. (2017a) still exhibits the gradient masking effect, where the gradient poorly approximates the global loss landscape. Assuming a black-box threat model, we studied the problem of generating adversarial examples for neural nets and proposed the gradient sign estimation problem as the core challenge in crafting (Tramèr et al., 2017a, Figure 2), for 500 correctly classified points x and ∈ {4, 10, 16}, we plot the probability that we find at least k orthogonal vectors r i -computed based on (Tramèr et al., 2017a, Lemma 7)-such that ||r i || ∞ = and x + r i is misclassified. For both models and for the same points x, SAAS finds more orthogonal adversarial vectors r i than GAAS, thereby providing a better characterization of the space of adversarial examples in the vicinity of a point, albeit without a white-box access to the models. these examples. We formulate the problem as a binary black-box optimization one: maximizing the directional derivative in the direction of {±1} n vectors, approximated by the finite difference of the queries' loss values. The separability property of the directional derivative helped us devise SignHunter, a query-efficient, tuning-free divide-and-conquer algorithm with a small memory footprint that is guaranteed to perform at least as well as FGSM after O(n) queries. No similar guarantee is found in the literature. In practice, SignHunter needs a mere fraction of this number of queries to craft adversarial examples. The algorithm is one of its kind to construct adaptive queries instead of queries that are based on i.i.d. random vectors. Robust to gradient masking, SignHunter can also be used to estimate the dimensionality of adversarial cones. Moreover, SignHunter achieves the highest evasion rate on two public black-box attack challenges and breaks a model that argues robustness against substitute-model attacks. This section shows the performance of the noisy FGSM on standard models (described in Section 1 of the main paper) on the MNIST, CIFAR10 and IMAGENET datasets. In Figure 4, we consider the ∞ threat perturbation constraint. Figure 5 reports the performance for the 2 setup. Similar to , for each k in the experiment, the top k percent of the signs of the coordinates-chosen either randomly (random-k) or by the corresponding magnitude |∂L(x, y)/∂xi| (top-k)-are set correctly, and the rest are set to −1 or +1 at random. The misclassification rate shown considers only images that were correctly classified (with no adversarial perturbation). In accordance with the models' accuracy, there were 987, 962, and 792 such images for MNIST, CIFAR10, and IMAGENET out of the sampled 1000 images, respectively. These figures also serve as a validation for Theorem 1 of the main paper when compared to SignHunter's performance shown in Appendix D., y) ) on random 1000 images from the corresponding evaluation dataset, with the maximum allowed 2 perturbation being set to 3, 127, and 5, respectively. Compared to Figure 4, the performance on MNIST and CIFAR10 drops significantly. In this section, we present a proof of Theorem 1 of Section 3. Note that the theorem makes the assumption that the finite difference is a good approximation of the directional derivative. This assumption has been the core concept behind most of the black-box adversarial attack algorithms and we state it here for completeness. Theorem 1. (Optimality of SignHunter) Given 2 log(n)+1 queries and that the directional derivative is well approximated by the finite-difference (Eq. 1 in the main paper), SignHunter is at least as effective as FGSM in crafting adversarial examples. Proof. Based on the separability property of the directional derivative, the ith coordinate of the gradient sign vector can be recovered as follows: construct two binary codes u and v such that only their ith bit is different. Therefore, we have From the definition of SignHunter, this is carried out for all the n coordinates after 2 log(n)+1 queries. Put it differently, after 2 log(n)+1 queries, SignHunter has flipped every coordinate alone recovering its sign exactly as shown in Eq. 4 above. Therefore, the gradient sign vector is fully recovered, and one can employ the FGSM attack to craft an adversarial example. Note that this is under the assumption that our finite difference approximation of the directional derivative (Eq. 1 in the main paper) is good enough (or at least rank-preserving). This section outlines the experiments setup. To ensure a fair comparison among the considered algorithms, we did our best in tuning their hyperparameters. Initially, the hyperparameters were set to the values reported by the corresponding authors, for which we observed suboptimal performance. We made use of a synthetic concave loss function to efficiently tune the algorithms for each dataset × perturbation constraint combination. The performance curves on the synthetic loss function using the tuned values of the hyperparameters did show consistency with the reported from the literature. For instance, we noted that ZO-SignSGD converges faster than NES, and that BanditsT D outperformed the rest of the algorithms towards the end of query budget. Further, in our adversarial examples generation experiments, we observed failure rate and query efficiency in line with the algorithms' corresponding papers-e.g., compare the performance of BanditsT D and NES in Table 9 of Appendix D with (, Table 1). That said, we invite the community to provide their best tuned attacks. Note that SignHunter does not have any hyperparameters to tune. The finite difference probe δ for SignHunter is set to the perturbation bound as it is used for for both computing the finite difference and crafting the adversarial examples-see Line 1 in Algorithm 2 of the main paper. This tuning-free setup of SignHunter offers a robust edge over the state-of-the-art black-box attacks, which often require expert knowledge to carefully tune their parameters. Table 3 describes the general setup for the experiments. Table 2 lists the sources of the models we attacked in this work, while Tables 4, 5, 6, and 7 outline the algorithms' hyperparameters. Figure 6 shows the performance of the considered algorithms on a synthetic concave loss function after tuning their hyperparameters. All experiments were run on a CUDA-enabled NVIDIA Tesla V100 16GB. A possible explanation of SignHunter's superb performance is that the synthetic loss function is well-behaved in terms of its gradient given an image. That is, most of gradient coordinates share the same sign, since pixels tend to have the same values and the optimal value for all the pixels is the same. Thus, SignHunter will recover the true gradient sign with as few queries as possible (recall the example in Section 3 of the main paper). Moreover, given the structure of the synthetic loss function, the optimal loss value is always at the boundary of the perturbation region; the boundary is where SignHunter samples its perturbations., where x * = xmin+xmax 2 using a query limit of 1000 queries for each image. Note that in all, Bandits T D outperforms both NES and ZO-SignSGD. Also, we observe the same behavior reported by on the fast convergence of ZO-SignSGD compared to NES. We did not tune SignHunter; it does not have any tunable parameters. This section shows of our experiments in crafting adversarial black-box examples on standard models in the form of tables and performance traces, namely Figures 7, 8, and 9; and Tables 8, 9, and 10. Table 8: Summary of attacks effectiveness on MNIST under ∞ and 2 perturbation constraints, and with a query limit of 10, 000 queries. The Failure Rate ∈ column lists the fraction of failed attacks over 1000 images. The Avg. # Queries column reports the average number of queries made to the loss oracle only over successful attacks. Table 10: Summary of attacks effectiveness on IMAGENET under ∞ and 2 perturbation constraints, and with a query limit of 10, 000 queries. The Failure Rate ∈ column lists the fraction of failed attacks over 1000 images. The Avg. # Queries column reports the average number of queries made to the loss oracle only over successful attacks. Hamming Similarity row shows the Hamming similarity of the sign of the attack's estimated gradientĝ with true gradient's sign q *, computed as 1 − ||sign(ĝ) − q * || H /n and averaged over all images. Likewise, plots of the Avg. Cosine Similarity row show the normalized dot product ofĝ and g * averaged over all images. The Success Rate row reports the attacks' cumulative distribution functions for the number of queries required to carry out a successful attack up to the query limit of 10, 000 queries. The Avg. # Queries row reports the average number of queries used per successful image for each attack when reaching a specified success rate: the more effective the attack, the closer its curve is to the bottom right of the plot. Hamming Similarity row shows the Hamming similarity of the sign of the attack's estimated gradientĝ with true gradient's sign q *, computed as 1 − ||sign(ĝ) − q * || H /n and averaged over all images. Likewise, plots of the Avg. Cosine Similarity row show the normalized dot product ofĝ and g * averaged over all images. The Success Rate row reports the attacks' cumulative distribution functions for the number of queries required to carry out a successful attack up to the query limit of 10, 000 queries. The Avg. # Queries row reports the average number of queries used per successful image for each attack when reaching a specified success rate: the more effective the attack, the closer its curve is to the bottom right of the plot. Figure 9: Performance curves of attacks on IMAGENET for ∞ (first column) and 2 (second column) perturbation constraints. Plots of Avg. Loss row reports the loss as a function of the number of queries averaged over all images. The Avg. Hamming Similarity row shows the Hamming similarity of the sign of the attack's estimated gradientĝ with true gradient's sign q *, computed as 1 − ||sign(ĝ) − q * || H /n and averaged over all images. Likewise, plots of the Avg. Cosine Similarity row show the normalized dot product ofĝ and g * averaged over all images. The Success Rate row reports the attacks' cumulative distribution functions for the number of queries required to carry out a successful attack up to the query limit of 10, 000 queries. The Avg. # Queries row reports the average number of queries used per successful image for each attack when reaching a specified success rate: the more effective the attack, the closer its curve is to the bottom right of the plot. This section shows of our experiments in crafting adversarial black-box examples on adversarially trained models in the form of tables and performance traces, namely Tables 11, 12, Model Accuracy SignHunter (Algorithm 2 in the main paper) 91.47% 92.76% PGD against 3 independently & adversarially trained copies of the network FGSM on the CW loss for model B from (Tramèr et al., 2017a) 94.36% FGSM on the CW loss for the naturally trained public network 96.08% PGD on the cross-entropy loss for the naturally trained public network 96.81% Attack using Gaussian Filter for selected pixels on the adversarially trained public network FGSM on the cross-entropy loss for the adversarially trained public network PGD on the cross-entropy loss for the adversarially trained public network SignHunter (Algorithm 2 in the main paper) 47.16% PGD on the cross-entropy loss for the adversarially trained public network PGD on the CW loss for the adversarially trained public network 64.38% FGSM on the CW loss for the adversarially trained public network 67.25% FGSM on the CW loss for the naturally trained public network 85.23% Table 13: Top 1 Error Percentage. The numbers between brackets are computed on 10,000 images from the validation set. The rest are from (Tramèr et al., 2017a, Figure 10 : Performance curves of attacks on the public black-box challenges for MNIST (first column), CIFAR10 (second column) and IMAGENET (third column). Plots of Avg. Loss row reports the loss as a function of the number of queries averaged over all images. The Avg. Hamming Similarity row shows the Hamming similarity of the sign of the attack's estimated gradientĝ with true gradient's sign q *, computed as 1 − ||sign(ĝ) − q * || H /n and averaged over all images. Likewise, plots of the Avg. Cosine Similarity row show the normalized dot product ofĝ and g * averaged over all images. The Success Rate row reports the attacks' cumulative distribution functions for the number of queries required to carry out a successful attack up to the query limit of 5, 000 queries for MNIST and CIFAR10 (1, 000 queries for IMAGENET). The Avg. # Queries row reports the average number of queries used per successful image for each attack when reaching a specified success rate: the more effective the attack, the closer its curve is to the bottom right of the plot. This section illustrates our experiment on the distribution of the magnitudes of gradient coordinates as summarized in Figure 11. How to read the plots: Consider the first histogram in Plot (a) from below; it corresponds to the 1000 th image from the sampled MNIST evaluation set, plotting the histogram of the values {|∂L(x, y)/∂xi|} 1≤i≤n, where the MNIST dataset has dimensionality n = 784. These values are in the range [0, 0.002]. Overall, the values are fairly concentrated-with exceptions, in Plot (e) for instance, the magnitudes of the ∼ 400 th image's gradient coordinates are spread from 0 to ∼ 0.055. Original Images In this section, we show the performance of different sign flip schemes in comparison to SignHunter. Results are summarized in Figure 12. SignHunter's adaptive flips shows a clear advantage over other schemes despite having a worse upper-bound on the query complexity-e.g., Naive can retrieve the signs in n + 2 queries, as discussed in Section 3. In this section, we discuss recent work related to our proposition. Parsimonious Black-Box Adversarial Attacks . Our experiment on the public CIFAR10 black-box attack challenge corresponds to [1, Table 1]. The authors report a 48% success rate (52% model accuracy) with an average number of queries of 1261. On the other hand, our proposed algorithm achieves a 52.84% success rate (47.16% model accuracy) with an average number of queries of 569. Further, (, Table 2) corresponds to our in Appendix D, Table 9; the paper reports a 98.5% success rate with an average number of queries of 722. Our proposed algorithm achieves a 98% success rate with 578.56 average number of queries. Based on these numbers, SignHunter demonstrates better performance than 's attack. Simple Black-Box Attack (SIMBA) . The main distinction is that SIMBA performs a ternary flip over {−δ, 0, +δ} for one random single coordinate at an iteration with δ ≤. On the other hand, SignHunter performs a binary flip {−,} for a group of coordinates at an iteration. Most of's experiments were performed for the 2 perturbation constraint and against models different from those considered in this paper-except for the IMAGENET v3 model, which the authors find much more difficult to attack. The v3 curves at 10, 000 queries in (, Figure 4) for SIMBA (and its variant SIMBA-DCT) look comparable to SignHunter's of Figure 9. For completeness, we implemented SIMBA and evaluated it against the CIFAR10 model in Section 4. The are shown in Figure 13. In line with , SIMBA is a strong baseline in the 2 setup. However, its performance drops significantly in the ∞ setup. Table 4. Harmonica . Both SignHunter and Harmonica seek to optimize a black-box function over the binary hypercube {±1} n, albeit with different assumptions on the objective function. Harmonica assumes that the objective function can be approximated by a sparse and low degree polynomial in the Fourier basis. Our assumption with SignHunter is that the objective function is separable (Property 1, Section 3), this lets us optimize the black-box function with O(n) queries given an initial guess instead of searching over the 2 n vertices. If this assumption is not met, we can restart SignHunter with another guess with a search complexity of O(mn) where m is the number of restarts. With this difference in assumptions of the two algorithms, we conducted an empirical comparison using the two sample problems provided along with Harmonica's authors implementation. As shown in Table 14, the show that SignHunter optimizes the two problems with 8× less number of queries than Harmonica, not to mention the significant computational advantage. In other words, SignHunter in − 2 perturbation setup behaves exactly the same when used in / √ n − ∞ perturbation setup. This is illustrated in Figure 14 To highlight the additional perturbation space that other algorithms have over SignHunter in the 2 setup, we ran NES and BanditsT D as representative examples of standard and dimensionality-reduction-based algorithms against the CIFAR10 model used in Section 4 with an ∞ perturbation setup of = 127/ √ n. In this and and the 2 setup used in Section 4, SignHunter behaves the same, while the performance of NES and BanditsT D drops significantly from their 2 performance due to the reduction in the perturbation space. A possible fix to allow SignHunter to access the additional search space introduced in the 2 setup is to extend the notion of binary sign flips over {+1, −1} to ternary sign flips over {+1, 0, −1} and we intend to explore this thoroughly in a future work. Figure 15: Performance of black-box attacks in the ∞ and 2 perturbation constraints. The plots show the average number of queries used per successful image for each attack when reaching a specified success rate. Note that (b) is similar to the 2 setup examined in Section 4.

Represent the short summary for finding the corresponding scientific article (long)

We present a sign-based, rather than magnitude-based, gradient estimation approach that shifts gradient estimation from continuous to binary black-box optimization.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the following document

Question: Recurrent Neural Networks (RNNs) are widely used models for sequence data. Just like for feedforward networks, it has become common to build "deep" RNNs, i.e., stack multiple recurrent layers to obtain higher-level abstractions of the data. However, this works only for a handful of layers. Unlike feedforward networks, stacking more than a few recurrent units (e.g., LSTM cells) usually hurts model performance, the reason being vanishing or exploding gradients during training. We investigate the training of multi-layer RNNs and examine the magnitude of the gradients as they propagate through the network. We show that, depending on the structure of the basic recurrent unit, the gradients are systematically attenuated or amplified, so that with an increasing depth they tend to vanish, respectively explode. Based on our analysis we design a new type of gated cell that better preserves gradient magnitude, and therefore makes it possible to train deeper RNNs. We experimentally validate our design with five different sequence modelling tasks on three different datasets. The proposed stackable recurrent (STAR) cell allows for substantially deeper recurrent architectures, with improved performance. Recurrent Neural Networks (RNN) have established themselves as a powerful tool for modelling sequential data. They have significantly advanced a number of applications, notably language processing and speech recognition (; ;). The basic building block of an RNN is a computational unit (or cell) that combines two inputs: the data of the current time step in the sequence and the unit's own output from the previous time step. While RNNs are an effective approach that can in principle handle sequences of arbitrary and varying length, they are (in their basic form) challenged by long-term dependencies, since learning those would require the propagation of gradients over many time steps. To alleviate this limitation, gated architectures have been proposed, most prominently Long Short-Term Memory (LSTM) cells and Gated Recurrent Units . They use a gating mechanism to store and propagate information over longer time intervals, thus mitigating the vanishing gradient problem. Although such networks can, in principle, capture long-term dependencies, it is known that more abstract and longer-term features are often represented better by deeper architectures . To that end, multiple recurrent cells are stacked on top of each other in a feedforward manner, i.e., the output (or the hidden state) of the lower cell is connected to the input gate of the next-higher cell. Many works have used such deep recurrent architectures, e.g., , and have shown their ability to extract more complex features from the input and make better predictions. The need for multi-layer RNNs is particularly apparent for image-like input data, where multiple convolutional layers are required to extract a good representation, while the recurrence captures the evolution of each layer over time. Since recurrent architectures are trained by propagating gradients across time, it is convenient to "unwrap" them into a lattice with two axes for depth (abstraction level) and time, see Fig. 1. This view makes it apparent that gradients flow in two directions, namely backwards in time and downwards from deeper to shallower layers. In this paper we ask the question how the basic recurrent unit must be designed to ensure the "vertical" gradient flow across layers is stable and not impaired by vanishing or exploding gradients. We show that stacking several layers of common RNN cells, by their construction, leads to instabilities (e.g., for deep LSTMs the gradients tend to vanish; for deep vanilla RNNs they tend to explode). Our study makes three contributions: (i) We analyse how the magnitude of the gradient changes as it propagates through a cell of the two-dimensional deep RNN lattice. We show that, depending on the inner architecture of the employed RNN cell, gradients tend to be either amplified or attenuated. As the depth increases, the repeated amplification (resp., attenuation) increases the risk of exploding (resp., vanishing) gradients. (ii) We then leverage our analysis to design a new form of gated cell, termed the STAR (stackable recurrent) unit, which better preserves the gradient magnitude inside the RNN lattice. It can therefore be stacked to much greater depth and still remains trainable. (iii) Finally, we compare deep recurrent architectures built from different basic cells in an extensive set of experiments with three popular datasets. The confirm our analysis: training deep recurrent nets fail with most conventional units, whereas the proposed STAR unit allows for significantly deeper architectures. In several cases, the ability to go deeper also leads to improved performance. Vanishing or exploding gradients during training are a long-standing problem of recurrent (and other) neural networks . Perhaps the most effective measure to address them so far has been to introduce gating mechanisms in the RNN structure, as first proposed by in the form of the LSTM (long short-term memory), and later by other architectures such as gated recurrent units . Importantly, RNN training needs proper initialisation. and have shown that initializing the weight matrices with identity and orthogonal matrices can be useful to stabilise the training. and further develop this idea and impose orthogonality throughout the entire training to keep the amplification factor of the weight matrices close to unity, leading to a more stable gradient flow. Unfortunately, it has been shown that such hard orthogonality constraints hurt the representation power of the model and in some cases even destabilise the optimisation. Another line of work has studied ways to mitigate the vanishing gradient problem by introducing additional (skip) connections across time and/or layers. have shown that skipping state updates in RNN shrinks the effective computation graph and thereby helps to learn longer-range dependencies. introduce a residual connection between LSTM layers. propose a gated feedback RNN that extends the stacked RNN architecture with extra connections. An obvious disadvantage of such an architecture are the extra computation and memory costs of the additional connections. Moreover, the authors only report for rather shallow networks up to 3 layers. Despite the described efforts, it remains challenging to train deep RNNs. have proposed Recurrent Highway Networks (RHN) that combine LSTMs and highway networks to train deeper architectures. RHN are popular and perform well on language modelling tasks, but they are still prone to exploding gradients, as illustrated in our experiments. (a) propose a restricted RNN where all interactions are removed between neurons in the hidden state of a layer. This appears to greatly reduce the exploding gradient problem (allowing up to 21 layers), at the cost of a much lower representation power per layer. To process image sequence data, computer vision systems often rely on Convolutional LSTMs (convLSTM,). But while very deep CNNs are very effective and now standard , stacks of more than a few convLSTMs do not train well. In practice, shallow versions are preferred, for instance (b) use a single layer for action recognition, and use two layers to recognize for hand gestures (combined with a deeper feature extractor without recursion). We note that attempts to construct a deep counterpart to the Kalman filter can also be interpreted as recurrent networks (; ;). These provide a probabilistic, generative perspective on RNNs, but are even more complicated to train. It is at this point unclear how the basic units of these architectures could be stacked into a deep, multi-layer representation. A RNN cell is a non-linear transformation that maps the input signal x t at time t and the hidden state of the previous time step t − 1 to the current hidden state h t: with W the trainable parameters of the cell. The input sequences have an overall length of T, which can be variable. It depends on the task whether the relevant target prediction, for which also the loss L should be computed, is the final state h T, the complete sequence of states {h t}, or a single sequence label, typically defined as the average When stacking multiple RNN cells on top of each other, one passes the hidden state of the lower level l − 1 as input to the next-higher level l, see Fig. 1, which in mathematical terms corresponds to the recurrence relation h Temporal unfolding leads to a two-dimensional lattice with depth L and length T, as in Fig. 1. In this computation diagram, the forward pass runs from left to right and from bottom to top. Gradients flow in opposite direction: at each cell the gradient w.r.t. the loss arrives at the output gate and is used to compute the gradient w.r.t. (i) the weights, (ii) the input, and (iii) the previous hidden state. The latter two gradients are then propagated through the respective gates to the preceding cells in time and depth. In the following, we investigate how the magnitude of these gradients changes across the lattice. The analysis, backed up by numerical simulations, shows that common RNN cells are biased towards attenuating or amplifying the gradients, and thus prone to destabilising the training of deep recurrent networks. At a single cell in the lattice the gradient w.r.t. the trainable weights are where ∂h l t ∂w denotes the Jacobian matrix and g h l t is a column vector containing the partial derivatives of the loss w.r.t. the cell's output (hidden) state. From the equation, it becomes clear that the Jacobian acts as a "gain matrix" on the gradients, and should on average preserve their magnitude to prevent them from vanishing or exploding. By expanding the gradient g h l t we obtain the recurrence for propagation, from which we get the two Jacobians where D x denotes a diagonal matrix with the elements of vector x as diagonal entries. Ideally, we would like to know the expected values of the two matrices' singular values. Unfortunately, there is no easy way to derive closed-form analytical expressions for them, but we can compute them for a fixed, representative point. Perhaps the most natural and illustrative choice is to set h l−1 t = h l t−1 = b = 0, and to further choose orthogonal weight matrices W h and W x, a popular initialisation strategy. Since the derivative tanh = 1, the singular values of all matrices in Eq. are equal to 1 in this configuration. and g h l t+1 we expect to obtain a gradient g h l ) over all time steps. As the gradients flow back through time and layers, for a network of vanilla RNN units they get amplified; for LSTM units they get attenuated; whereas the proposed STAR unit approximately preserves their magnitude. in the time direction, but cannot help the flow through the layers. Again the numerical simulation support our hypothesis, as can be seen in Fig. 2. The LSTM gradients propagate relatively well backward through time, but vanish quickly towards shallower layers. We refer to the appendix for further numerical analysis, e.g., LSTMs with only a forget gate, and GRUs. Here, we briefly draw some connections between our analysis and the empirical of , who propose a gated feedback RNN (GFRNN) that extends the stacked RNN architecture with extra connections between adjacent layers. Empirically, GFRNN improves a 3-layer LSTM, but degrades the vanilla RNN performance. We conjecture that this might be due to the extra connections strengthening the gradient propagation. According to our findings, the additional gradient flow would benefit the LSTM, by bolstering the dwindling gradients; whereas for the vRNN, where the initial gradients are already too high, the added flow might be counterproductive. Building on the analysis above, we introduce a novel RNN cell designed to avoid vanishing or exploding gradients as much as possible. We start from the Jacobian matrix of the LSTM cell and examine in more detail which design features are responsible for the low singular values. In equation 8 we see that every multiplication with tanh non-linearities (D tanh ), gating functions (D σ ), and with their derivatives can only ever decrease the singular values of W, since all this terms are always <1. The effect is particularly pronounced for the sigmoid and its derivative, |σ (·)| ≤ 0.25 and E[|σ (x)|] = 0.5 for zero-mean, symmetric distribution of x. In particular, the output gate o l t is a sigmoid and plays a major role in shrinking the overall gradients, as it multiplicatively affects all parts of both Jacobians. As a first measure, we thus propose to remove the output gate. A secondary consequence of this measure is that now h l t and c l t carry the same information (the hidden state becomes an element-wise non-linear transformation of the cell state). To avoid this duplication and further simplify the design, we transfer the tanh non-linearity to the hidden state and remove the cell state altogether. As a final modification, we also remove the input gate i l t from the architecture. We have empirically observed that the presence of the input gate does not significantly improve performance, moreover, it actually harms the training for deeper networks. This empirical observation is in line with the of van der , who show that removing the input and output gates does not greatly affect the performance of LSTMs. More formally, our proposed STAR cell in the l-th layer takes the input h l−1 t (in the first layer, x t) at time t and non-linearly projects it to the space where the hidden vector h l lives, equation 10. Furthermore, the previous hidden state and the new input are combined into the gating variable k l t (equation 11). k l t is our analogue of the forget gate and controls how the information from previous hidden state and the new input are fused into a new hidden state. One could also intuitively interpret k l t as a sort of "Kalman gain": if it is large, the new observation is deemed reliable and dominates; otherwise the previous hidden state is conserved. The complete dynamics of the STAR unit is given by the expressions These equations lead to the following Jacobian matrices: Coming back to our previous analysis for state zero and orthogonal weight matrices, each of the two Jacobians now has singular values equal to 0.5. I.e., they lie between the vRNN cell and the LSTM cell, and when added together roughly preserve the gradient magnitude. We repeat the same numerical simulations as above for the STAR cell, and find that it indeed maintains healthy gradient magnitudes throughout most of the deep RNN, see Fig. 2. In the next section, we show also on real datasets that deep RNNs built from STAR units can be trained to a significantly greater depth. As a final remark, the proposed modifications mean that the STAR architecture requires significantly less memory. With the same input and the same capacity in the hidden state, it reduces the memory footprint to <40% of a classical LSTM and even uses slightly less memory than a recurrent highway net. A more detailed comparison is given in the appendix. We evaluate the performance of several well-known RNN baselines as well as that of the proposed STAR cell on five different sequence modelling tasks with three different datasets: sequential versions of MNIST, which are a popular common testbed for recurrent networks; the more realistic TUM dataset, where time series of intensities observed in satellite images shall be classified into different agricultural crops; and Jester, for hand gesture recognition with convolutional RNNs. The recurrent units we compare include the vRNN, the LSTM, the LSTM with only a forget gate (van der), the RHN, and the proposed STAR. The experimental protocol is similar for all tasks: For each RNN variant, we train multiple versions with different depth (number of layers). For each variant and each depth, we report the performance of the model with the lowest validation loss. Classification performance is measured by the rate of correct predictions (top-1 accuracy). Throughout, we use the orthogonal initialisation for weight matrices. Code and trained models (in Tensorflow), as well as code for the simulations (in PyTorch), will be released. Training and network details for each experiment can be found in the appendix. 97.0% 82.0% 100 11k uRNN 95.1% 91.4% 512 9k FC uRNN 96.9% 94.1% 512 270k Soft ortho 94.1% 91.4% 128 18k AntisymRNN 98 The first experiment uses the MNIST dataset . The 28×28 grey-scale images of handwritten digits are flattened into 784×1 vectors, and the 784 values are sequentially presented to the RNN. After seeing all pixels, the model predicts the digit. The second task, pMNIST, is more challenging. Before flattening the images, the pixels are shuffled with a fixed random permutation, turning correlations between spatially close pixels into non-local long-range dependencies. The model needs to remember those dependencies between distance parts of the sequence to classify the digit correctly. Fig. 3a shows the average gradient norms per layer at the start of training, for 12-layer networks built from different RNN cells. Like in the simulations above, the propagation through the network increases the gradients for the vRNN and shrinks them for the LSTM. As the optimisation proceeds, we find that STAR remains stable, whereas all other units see a rapid decline of the gradients already within the first epoch, except for RHN, where the gradients explode, see Fig. 3b. Consequently, STAR is the only unit for which a 12-layer model can be trained, as also confirmed by the evolution of the training loss, Fig. 3c. Fig. 4 confirms that stacking into deeper architectures does benefit RNNs (except for vRNN); but it increases the risk of a catastrophic training failure. STAR is significantly more robust in that respect and can be trained up to a depth of 20 layers. On the comparatively easy and saturated MNIST data, the performance is comparable that of a successfully trained LSTM (at depth 2-8 layers, LSTM training already often fails; the displayed accuracies are averaged only over successful training runs). In this experiment, the models are evaluated on a more realistic sequence modelling problem. The task is to classify agricultural crop types using sequences of satellite images, exploiting the fact that different crops have different growing patterns over the season. The input is a time series of 26 multi-spectral Sentinel-2A satellite images with a ground resolution of 10 m, collected over a 102 km x 42 km area north of Munich, Germany between December 2015 and August 2016 (Rußwurm & Körner, 2017). The input data points for the classifier are patches of 3×3 pixels recorded in 6 spectral channels, flattened into 54×1 vectors. In the first task these vectors are sequentially presented to the RNN model, which outputs a prediction at every time step (note that for this task the correct answer can sometimes be "cloud", "snow", "cloud shadow" or "water", which are easier to recognise than many crops). In the second task, the model makes only one crop prediction for the complete sequence, via an additional layer that averages across time. From Fig. 5 we see that STAR outperforms all baselines and its again more robust to stacking. For the single-output task also the STAR network fails at 14 layers. We have not yet been able to identify the reason for this, possibly it is due to cloud cover that completely blanks out the signal over extended time windows and degrades the propagation. This experiment serves to evaluate the performance of different recurrent cells, and in particular the proposed STAR cell, in a convolutional RNN (see appendix for details about convolutional STAR). To that end, we use the 20BN-Jester dataset V1 (jes). Jester is a large collection of densely-labeled short video clips, where each clip contains a predefined hand gesture performed by a worker in front of a laptop camera or webcam. In total, the dataset includes 148'094 RGB video files of 27 types of gestures. The task is to classify which gesture is seen in a video. 32 consecutive frames of size 112×112 pixels are sequentially presented to the convolutional RNN. At the end, the model again predicts a gesture class via an averaging layer over all time steps. The outcome for convolutional RNNs is coherent with the previous , see Fig. 5c. Going deeper improves the performance of all three tested convRNNs. The improvement is strongest for convolutional STAR, and the best performance is reached at high depth (12 layers), where training the baselines mostly fails. In summary, the confirm both our intuition that depth is particularly useful for convolutional RNNs; and that STAR is more suitable for deeper architectures, where it achieves higher performance with better memory efficiency. We note that the in the shallow 1-2 layer setting the conventional LSTM performs a bit better than the two others, likely due to its larger capacity. We have investigated the problem of vanishing/exploding gradient in deep RNNs. In a first step, we analyse how the derivatives of the non-linear activation functions rescale the gradients as they propagate through the temporally unrolled network. From both, the theoretical analysis, and associated numerical simulations, we find that standard RNN cells do not preserve the gradient magnitudes during backpropagation, and therefore, as the depth of the network grows, the risk that the gradients vanish or explode increases. In a second step, we have proposed a new RNN cell, termed the STAckable Recurrent unit, which better preserves gradients through deep architectures and facilitates their training. An extensive evaluation on three popular datasets confirms that STAR units can be stacked into deeper architectures than other RNN cells. We see two main directions for future work. On the one hand, it would be worthwhile to develop a more formal and thorough mathematical analysis of the gradient flow, and perhaps even derive rigorous bounds for specific cell types, that could, in turn, inform the network design. On the other hand, it appears promising to investigate whether the analysis of the gradient flows could serve as a basis for better initialisation schemes to compensate the systematic influences of the cells structure, e.g., gating functions, in the training of deep RNNs. C TRAINING DETAILS C.1 PIXEL-BY-PIXEL MNIST Following Tallec & Ollivier, chrono initialisation is applied for the bias term of k, b k. The basic idea is that k should not be too large; such that the memory h can be retained over longer time intervals. The same initialisation is used for the input and forget bias of the LSTM and the RHN and for the forget bias of LSTMw/f. For the final prediction, a feedforward layer with softmax activation converts the hidden state to a class label. The numbers of hidden units in the RNN layers are set to 128. All networks are trained for 100 epochs with batch size 100, using the Adam optimizer with learning rate 0.001, β 1 = 0.9 and β 2 = 0.999. For both tasks we use the same training schedule. Again a feedforward layer is appended to the RNN output to obtain a prediction. The numbers of hidden units in the RNN layers is set to 128. All networks are trained for 30 epochs with batch size 500, using Adam with learning rate 0.001 and β 1 = 0.9 and β 2 = 0.999. Throughout, convolution kernels are of size 3×3. Each convolutional RNN layer has 64 filters. A shallow CNN is used to convert the hidden state to a label, with 4 layers that have filter depths 128, 128, 256 and 256, respectively. All models are fitted with stochastic gradient descent (SGD) with momentum (β = 0.9). The batch size is set to 8, the learning rate starts at 0.001 and decays polynomially to 0.000001 over a total of 30 epochs. L2-regularisation with weight 0.00005 is applied to all parameters.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We analyze the gradient propagation in deep RNNs and from our analysis, we propose a new multi-layer deep RNN.

Represent the TLDR for finding the corresponding scientific article

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.