Question: Represent the article for finding a single-sentence TLDR

Question: We address the efficiency issues caused by the straggler effect in the recently emerged federated learning, which collaboratively trains a model on decentralized non-i.i.d. (non-independent and identically distributed) data across massive worker devices without exchanging training data in the unreliable and heterogeneous networks. We propose a novel two-stage analysis on the error bounds of general federated learning, which provides practical insights into optimization. As a , we propose a novel easy-to-implement federated learning algorithm that uses asynchronous settings and strategies to control discrepancies between the global model and delayed models and adjust the number of local epochs with the estimation of staleness to accelerate convergence and resist performance deterioration caused by stragglers. Experiment show that our algorithm converges fast and robust on the existence of massive stragglers. Distributed machine learning has received increasing attention in recent years, e.g., distributed stochastic gradient descent (DSGD) approaches and the well-known parameter server paradigm (; ; 2014). However, these approaches always suffer from communication overhead and privacy risk . Federated learning (FL) (Konečnỳ et al., 2016) is proposed to alleviate the above issues, where a subset of devices are randomly selected, and training data in devices are locally kept when training a global model, thus reducing communication and protecting user privacy. Furthermore, FL approaches are dedicated to a more complex context with 1) non-i.i.d. (Non-independent and identically distributed), unbalanced and heterogeneous data in devices, 2) constrained computing resources with unreliable connections and unstable environments (; Konečnỳ et al., 2016). Typically, FL approaches apply weight averaging methods for model aggregation, e.g., FedAvg and its variants (; ; ; ;). Such methods are similar to the synchronous distributed optimization domain. However, synchronous optimization methods are costly in synchronization , and they are potentially inefficient due to the synchrony even when collecting model updates from a much smaller subset of devices (b). Besides, waiting time for slow devices (i.e., stragglers or stale workers) is inevitable due to the heterogeneity and unreliability as mentioned above. The existence of such devices is proved to affect the convergence of FL . To address this problem, scholars propose asynchronous federated learning (AFL) methods (a; ;) that allow model aggregation without waiting for slow devices. However, asynchrony magnifies the straggler effect because 1) when the server node receives models uploaded by the slow workers, it probably has already updated the global model for many times, and 2) real-world data are usually heavy-tailed in distributed heterogeneous devices, where the rich get richer, i.e., the straggler effect accumulates when no adjustment operations in stale workers, and eventually it affects the convergence of the global model. Furthermore, dynamics in AFL brings more challenges in parameter tuning and speed-accuracy trade-off, and the guidelines for designing efficient and stale-robust algorithms in this context are still missing. Contributions Our main contributions are summarized as follows. We first establish a new twostage analysis on federated learning, namely training error decomposition and convergence analysis. To the best of our knowledge, it is the first analysis based on the above two stages that address the optimization roadmap for the general federated learning entirely. Such analysis provides insight into designing efficient and stale-robust federated learning algorithms. By following the guidelines of the above two stages, we propose a novel FL algorithm with asynchronous settings and a set of easy-to-implement training strategies. Specifically, the algorithm controls model training by estimating the model consistency and dynamically adjusting the number of local epochs on straggle workers to reduce the impact of staleness on the convergence of the global model. We conduct experiments to evaluate the efficiency and robustness of our algorithm on imbalanced and balanced data partitions with different proportions of straggle worker nodes. Results show that our approach converges fast and robust on the existence of straggle worker nodes compared to the state-of-the-art solutions. Related Work Our work is targeting the AFL and staleness resilience approaches in this context. Straggler effect (also called staleness) is one of the main problems in the similar asynchronous gradient descent (Async-SGD) approaches, which has been discussed by various studies and its remedies have been proposed (; ; ; ; ; ; ;). However, these works are mainly targeting the distributed Async-SGD scenarios, which is different from FL as discussed in the previous section. Existing FL solutions that address the straggler effect are mainly consensus-based. Consensus mechanisms are introduced where a threshold metric (i.e., control variable) is computed, and only the workers who satisfy this threshold are permitted to upload their model (; ;). Thus it significantly reduces the number of communications and updates model without waiting for straggle workers. However, current approaches are mainly focusing on synchronized FL. Xie et al. (2019a) propose an AFL algorithm which uses a mixing hyperparameter to adaptively control the trade-off between the convergence speed and error reduction on staleness. However, this work and above mentioned FL solutions only consider the staleness caused by network delay instead of imbalanced data size in each worker and only evaluate on equal size of local data, which is inconsistent with the real-world cases. Our approach is similar to (a), but instead we adaptively control the number of local epochs combined with the approximation of staleness and model discrepancy, and prove the performance guarantee on imbalanced data partitions. We illustrate our approach in the rest of this paper. We first summarize the general form of FL. Generally, an FL system consists of M distributed worker nodes (e.g., mobile phones) and a server node. The goal is training a global model across these worker nodes without uploading local data. Each worker node employs the same machine learning model, and an optimizer (e.g., stochastic gradient descent) to iteratively optimize the loss function of the local model. At t-th communication round, the server node uses an aggregation operator (e.g., averaging) to aggregate the local models uploaded by worker nodes, and broadcasts the aggregated global model to workers. We use mi to present local data points in worker node i, where m i is the size of data points in this worker. The whole dataset χ = i X (i), where i ∈ {1, 2, 3, ..., M}. We assume that X (i) X (j) = ∅ for i = j, and apparently, the total size of data m = M i=1 m i. We denote the model in worker node i by ω i ∈ R d, and the objective function of worker node i by where g(·) is the user-defined aggregation function, and ξ t is a vector which describes the settings of activated workers, such as worker ID, number of local epochs, and the learning rate. Here and thereafter, we use g(ω t) to represent g(ω t, ξ t) for convenience. We denote update term as h(·), a userdefined function which represents the model parameter differences between the collected models from activated worker nodes and previous global model, and Here τ i is the time when worker node i received the global model In this section, we aim to design an efficient and robust FL algorithm. To do so, we first establish a twostage analysis, and finally, propose our new FL algorithm by combining the insights provided by the two stages. Stage 1: Traning Error Decomposition. We first discuss the main errors of the general FL. We assume that each worker node has a local optimal model ω * i = arg min F i (ω). Then at the communication round t, we define the global error as where · is L 2 norm. For worker node i, two terms in the right-hand side of inequality 7 respectively represent 1) initialization and local error: the error between the local model at communication round t and the optimal local model (the well known empirical risk). Here, the initialization error (i.e., the error between the initial model and local model at communication round t) partially contributes to the first term. 2) local-global error: the error between optimal local models and optimal global solution, which is a constant given a specific learning task. Figure 1 illustrates these errors. Usually, the error between the initial model and the optimal global model is greater than the local-global error, and thus at the early stage of training, the first term is greater than the second term in the right-hand side of inequality 7. Therefore, reducing the initialization error and the local error at the beginning of model training can reduce the global error ω t − ω *. Afterward, when initialization and local error is minimized, the local-global error dominates the global error. However, as we mentioned previously, the local-global error is a constant that can not be solved directly. Therefore, we need a more sophisticated analysis to reach ω * since 7 is no longer appropriate to guide the optimization other than the early stage of FL training. Following the above analysis, we analyze the convergence bounds of the general FL (Eq. 6) on the rest of the training stages other than the early stage. Stage 2: Convergence Analysis. First, we make the following assumptions on the objective functions: Assumption 1. Smoothness. For all i in {1, 2, 3, ..., M} and given constant β, the objective function F (ω) and F i (ω) are β-smooth, i.e., Assumption 2. The first and second moment conditions. The objective function F (ω) and the aggregation operation g(ω t) satisfy the following: 2. E(·) is abbreviation of E ξt (·) which denotes the expected value w.r.t. the distribution of the random variable ξ t given t. Assumption 3. Strong convexity. For all i in {1, 2, 3, ..., M} and given constant c, the objective function F (ω) and F i (ω) are c-strong convex, i.e., Theorem 1. Convergence for strongly-convex problems. When c and β in assumption 1 and 3 satisfy c ≤ β, we can set the step size η t =η, where, and L G = 1 + δ 2 G. Withη, the upper error bound of global model satisfies: The proof of theorem 1 is provided in appendix A.1. Theorem 1 gives an error bound for the general form of model aggregation without assuming that g(ω t) should come from ∇F (ω t). Note that the scalars δ and δ G are equal to 1 when g(ω t) is the unbiased estimation of ∇F (ω t). However, current convergence bound in theorem 1 is too loose, and it can be further optimized by introducing controlled local epoch settings. We assume that ∇F i (ω Then we can extend theorem 1 with the local epochĒ. Theorem 2. Convergence with selected local epoch. We use δ 0 and L 0 to represent the scalar δ and L in assumption 2 whenĒ = 1, and all worker nodes are assumed to participate in model training, then under 9, 8 can be rewritten as The theorem 2 gives us the error bound of FL with the selected number of local epochs for stronglyconvex problems. The proof of theorem 2 is provided in appendix A.2. The right-hand side of theorem 2 implies the dynamics of hyper-parameters in local models for efficiency and robustness trade-off. In a general FL algorithm, e.g., FedAvg, the model settings in worker nodes are always predefined and kept fixed, which may lead to a higher variance. We now discuss such dynamics and practical insights on designing efficient and robust FL algorithms. Selection of local epochs. We discuss how to reduce the global error and communication round simultaneously for general FL. From the second term of the right-hand side of 10, we can see that theorem 2 yields to linear convergence when E( . In this condition, to quickly reduce the global error, we can reduce the second term of the right-hand side of 10 by increasing the local epochĒ while reducing the communication round t. Therefore, we can dynamically assign each worker with a bigger number of local epoch while reducing the communication round. Asynchronous acceleration with stragglers. We discuss why asynchronous strategies are needed in FL. We rearrange 10 as: When t increases, and we fixĒ andη, the global error only depends on L 0 . L 0 can be controlled by sampling more worker nodes within a communication round. Specifically, we compare n-worker participation with M -worker participation for model aggregation at the server node. When we select n workers out of M, L 0 increases according to assumption 2(c) since the variance increases. Thus, to get the same precision, we decreaseη, while it significantly slows the convergence speed. However, in practice, waiting for all the workers to be ready is time-consuming. Thus, we can introduce asynchronous mechanisms in model aggregation without waiting for all workers. Robust training on straggle workers. We discuss how to reduce the global error for FL on the existence of stragglers. As we mentioned above, asynchronous strategies can accelerate model training by reducing the waiting time at each communication round. However, the straggler effect is magnified by asynchrony, as discussed in section 1. Stale workers accumulate their staleness, which increases the variances and affects the convergence of the global model. A practical strategy to tame such effect is increasing the number of local epoch under the considerations that when the distributions of local data are far away from the global data, we use more epochs to train from the local data. However, the divergence of these local epoch numbers between stale and non-stale workers may affect variance adversely, and we can adjust the number of local epoch with the normalized epochs from all workers to reduce such variance. Theorem 3. Convergence for non-convex problems With the Assumption 1 and 2, we can select a step size The expected error bound satisfies: Theorem 3 is similar to theorem 2 that the first term of the right-hand side of does not decrease by iterative training. Note that the above remarks are also applicable to theorem 3. We provide proof of theorem 3 in appendix A.3. Proposed Algorithm. Under the guidance of the above analysis and the practical insights discussed above, we propose a fast and stale-robust AFL algorithm. Algorithm 1 and 2 illustrate the processes in worker nodes and the server node, respectively. H(t) is a predefined function at communication round t which determines how long should the server node waiting for the updated models from workers. H(t) can be used to control the accuracy-speed trade-off. The training processes on the server node can be divided into two stages, i.e., the initial stage and the converging stage. We switch the stages by estimating the consistency of model updates. Set t ← t + 1. During ∆t time, receive the triplet (ω i, τ i, E i) from any worker i. Update ω t with ω i with τ i = t − 1 using 5. Broadcast (ω t, t) to each worker. Calculate U using 13. 9: until U ≤ 0.1 10: Broadcast start flag to each worker. 11: repeat // the converging stage. Set t ← t + 1, ∆t ← H(t). During ∆t time, receive the triplet (ω i, τ i, E i) from any worker i. Update ω t by 6 and 14. Set E i ← mean(E) * s. 5: end if 6: Set τ i ← t, E i ← 0. 7: for e in 1, 2, 3,..., E i do 8: Randomly divide X (i) with batch size B i. Update ω i by using opt for each batch. Send triplet (ω i, τ i, E i) to the server. 14: e ← e − 1. end if 18: end for Definition 1. Update consistency. The model update consistency of n worker nodes is the similarities between worker models at communication round t, i.e., U is consistent with the global error in inequality 7, and in algorithm 1 we empirically set 0.1 as the threshold to switch from the initial stage and the converging stage of the global model training. At the initial stage of global model training, we use a bigger local epoch to accelerate training time as discussed above, and repeat this process until U ≤ 0.1. After the initial stage, we define the update term as ϕi ϕ is the above mentioned normalized local epochĒ with, and ϕ is the regularization term where ϕ = n i=1 ϕ i. Finally, we define a stale-related penalty function of ϕ i as: Here, ω t+1 f resh is the average model of worker nodes with τ i = t. The key processes of worker nodes are 1) estimating its staleness level, and 2) assign the number of local epoch using mean(E) in the received triplet from the server node and the previously estimated staleness level. In the next section, we evaluate the performance of our algorithms. We evaluate the performance of our approach on both imbalanced and balanced data partitions with the existence of stale worker nodes. Experiment Settings. We conduct experiments on Fashion-MNIST and CIFAR-10 to test the accuracy of our approach on 100 simulated workers, where 60 workers are stale. We use 55,000 on Fashion-MNIST and 50,000 on CIFAR-10 for training and 10,000 for testing. normalization is used in the data preprocessing. We conduct all experiments on CPU devices. We use a light-weight convolutional neural network (CNN) model, which is suitable for mobile edge devices. It has 4 convolutional layers that use 3 × 3 kernels with the size of 32, 64, 64, 128. Rectified linear unit (ReLU) is used in each convolutional layer, and every two convolutional layers are followed by a 2 × 2 max-pooling layer and a dropout of 50%. Finally, we use a 512-unit dense layer with ReLU and a dropout of 50% and an output layer with softmax. We use an SGD optimizer with a learning rate of 0.01. We set batch size as 50, and the initial number of local epochsĒ as 50. We randomly split the data size in each worker node ranging from 2 to 2122 with a standard deviation of 480 on CIFAR-10, and 9 to 2157 with a standard deviation of 540 on Fashion-MNIST. For the balanced cases we randomly assign each worker with 500 samples. The communication speed of nodes is divided into ten levels ranging from 100 milliseconds to 1 second, and the 60 stale workers are assigned with bigger levels. Finally, we set H(t) = 0.4s. We compare the performance of our proposed method with four approaches: 1) FedAvg (synchronized). We set the sampling rate C = 0.1 FedProx (synchronized). We set C = 0.1, µ = 1 as the best parameters provided in their paper. Results and Analysis. Figure 2 shows the performance of our proposed algorithm and four baselines. Our method converge faster compared to all the baselines, and the convergence is promised with 60% stale workers. Furthermore, the whole upload times of our method do not increase with the same level of accuracy. From the experiment on Fashion-MNIST, we can see that our method has the same accuracy level on test data compared with synchronized approach such as FedAvg. We can also see that on imbalanced data partitions (i.e., more realistic FL scenarios), our method is faster and more stable compared to other baselines. Finally, we can clearly see the stage transition from the initial training stage to the converging stage (e.g., the transitions in imbalanced cases in figure 2(b) and (d)), which validates the efficiency of our approach. Figure 3 shows the performance of our method with different proportion of stale nodes in 1,000 global communication rounds. Our method outperforms the AFL baseline (i.e., FedAsync) in both accuracy and loss, and when the proportion of stale workers is less than 80%, our method outperforms the synchronized FL baseline (i.e., FedAvg). In this paper, we propose a new two-stage analysis on federated learning, and inspired by such analysis, we propose a novel AFL algorithm that accelerates convergence and resists performance deterioration caused by stragglers simultaneously. Experimental show that our approach converges two times faster than baselines, and it can resist the straggler effect without sacrificing accuracy and communication. As a byproduct, our approach improves the generalization ability of neural network models. We will theoretically analyze it in future work. Besides, while not the focus of our work, security and privacy are essential concerns in federated learning, and as the future work, we can apply various security methods to our approach. Furthermore, besides the stale- We respectively test the performance with 20%, 60%, 80%, and 90% of stale workers. The green dotted line is FedAvg which waits all selected workers. resistance ability, the discrepancy estimation in our method also has the potential ability to resist malicious attacks to the worker nodes such as massive Byzantine attacks, which has been addressed in (; ; Muñoz-González et al., 2019). We will analyze and evaluate such ability in future work. A.1 PROOF OF THEOREM 1 Lemma 1. Under the assumption 1, we can get: Proof: Under the assumption 1, for any ω and ω, we have: Then using 18, we have: Taking expectations in 19 w.r.t the distribution of ξ t, we complete the proof. Lemma 2. Under the assumption 1 and 2, we can get: Proof: Using assumption 2(b) and 2(c), we have: Then using 21, assumption 2(b) and lemma 1, we have: We can easily get Lemma 2 by rearranging 22. Then we prove Theorem 1 under the assumption 1, 2, 3. First, we define Function F is a quadratic model relevant toω. Then it has the minimal value when all the partial derivatives are 0. That is Then, when we selectω =ω for 23, we get the minimal of 23, which is From assumption 3, we have which is equivalent to Then from Lemma 2, when a fixedη ≤ δ βL G is selected, we have: And using 26, we have Subtracting F (ω *) from both sides and moving F (ω t) from left to right, we get Taking the whole expectations and rearranging 29, we obtain Substracting the constantη βL 2cδ from both sides of 30, we have The left hand side of 31 is a geometric series with common ratio 1 − η t δc, then we complete the proof. We first prove 9. Assume ∇F i (ω Let ∇F i (ω t i) = a and ν∇F i (ω Since a+b a > 1 and ν ≤ 1, we have h(ν) min = h = 0, and a ≈ b when η t is small. We know that the smaller ν is, the smaller Res t i is. Then we consider the situation that E i = E = 1, and define ∇F (ω t) E(g E=1 (ω t)) ≥ δ 0 ∇F (ω t) 2 and Then sum all the form of 38 from 1 to t. We have Besides, we can easily understand that F inf ≤ E(F (ω t+1)), because F inf is the minimal value of F. Then we have By rearrange 40, we have Dividing t from both sides of 41, we get Then using equation 9, we complete the proof. We conduct additional experiments to evaluate stale-robustness of our algorithm on CIFAR-10 based on the settings in section 4. We visualize the impact of different staleness levels at different communication rounds with cosine angles (i.e., discrepancies) between the update terms (i.e., update directions of local models) of stale workers and fresh workers in figure 4. The show that our method (in the first row) effectively adjusts the update direction of the reversed stale nodes while angles of stale nodes reverse with FedAvg compared to our algorithm, which shows the robustness of our method. Figure 4: Impact visualization of different levels of staleness using cosine angles between the update terms defined in section 2 of fresh nodes (40 out of 100) and stale nodes (60 out of 100 worker nodes) on CIFAR-10 at different communication round. The blue numbers represent the staleness levels by using the differences of version numbers of models between the stale nodes and the fresh nodes. E.g., the staleness level is 10 at this communication round means that the fresh nodes has updated 10 more versions compared to the stale nodes.

We propose an efficient and robust asynchronous federated learning algorithm on the existence of stragglers

represent the short summary for finding the corresponding scientific article (long):

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.


Question: represent the scientific article to find a short tldr.

Question: Long Short-Term Memory (LSTM) units have the ability to memorise and use long-term dependencies between inputs to generate predictions on time series data. We introduce the concept of modifying the cell state (memory) of LSTMs using rotation matrices parametrised by a new set of trainable weights. This addition shows significant increases of performance on some of the tasks from the bAbI dataset. In the recent years, Recurrent Neural Networks (RNNs) have been successfully used to tackle problems with data that can be represented in the shape of time series. Application domains include Natural Language Processing (NLP) (translation BID12, summarisation BID9, question answering and more), speech recogition BID5 BID3 ), text to speech systems BID0, computer vision tasks BID13 BID16, and differentiable programming language interpreters BID10 BID11 ).An intuitive explanation for the success of RNNs in fields such as natural language understanding is that they allow words at the beginning of a sentence or paragraph to be memorised. This can be crucial to understanding the semantic content. Thus in the phrase "The cat ate the fish" it is important to memorise the subject (cat). However, often later words can change the meaning of a senstence in subtle ways. For example, "The cat ate the fish, didn't it" changes a simple statement into a question. In this paper, we study a mechanism to enhance a standard RNN to enable it to modify its memory, with the hope that this will allow it to capture in the memory cells sequence information using a shorter and more robust representation. One of the most used RNN units is the Long Short-Term Memory (LSTM) BID7. The core of the LSTM is that each unit has a cell state that is modified in a gated fashion at every time step. At a high level, the cell state has the role of providing the neural network with memory to hold long-term relationships between inputs. There are many small variations of LSTM units in the literature and most of them yield similar performance BID4.The memory (cell state) is expected to encode information necessary to make the next prediction. Currently the ability of the LSTMs to rotate and swap memory positions is limited to what can be achieved using the available gates. In this work we introduce a new operation on the memory that explicitly enables rotations and swaps of pairwise memory elements. Our preliminary tests show performance improvements on some of the bAbI tasks compared with LSTM based architectures. In this section we introduce the idea of adding a new set of parameters for the RNN cell that enable rotation of the cell state. The following subsection shows how this is implemented in the LSTM unit. One of the key innovations of LSTMs was the introduction of gated modified states so that if the gate neuron i is saturated then the memory c i (t − 1) would be unaltered. That is, c i (t − 1) ≈ c i (t) with high accuracy. The fact that the amplification factor is very close to 1 prevents the memory vanishing or exploding over many epochs. To modify the memory, but retain an amplification factor of 1 we take the output after appling the forget and add gates (we call it d t), and apply a rotation matrix U to obtain a modified memory c t = Ud t. Note that, for a rotation matrix U T U = I so that d t = c t.We parametrise the rotation by a vector of angles DISPLAYFORM0 where W rot is a weight matrix and b rot is a bias vector which we learn along with the other parameters. x is the vector of our concatenated inputs (in LSTMs given by concatenating the input for the current timestep with the output from the previous time step).A full rotation matrix is parametrisable by n(n − 1)/2 parameters (angles). Using all of these would introduce a huge number of weights, which is likely to over-fit. Instead, we have limited ourselves to considering rotations between pairs of inputs d i (t) and d i+1 (t). Exploring more powerful sets of rotations is currently being investigated. Our rotation matrix is a block-diagonal matrix of 2D rotations DISPLAYFORM1 where the cell state is of size n. Our choice of rotations only needs n/2 angles. In this section we show how to add memory rotation to the LSTM unit. The rotation is applied after the forget and add gates and before using the current cell state to produce an output. The RotLSTM equations are as follows: DISPLAYFORM0 DISPLAYFORM1 where W {f,i,o,rot,c} are weight matrices, b {f,i,o,rot,c} are biases (Ws and bs learned during training), h t−1 is the previous cell output, h t is the output the cell produces for the current timestep, similarly c t−1 and c t are the cell states for the previous and current timestep, • is element-wise multiplication and [·, ·] is concatenation. U as defined in Equation 2, parametrised by u t. Figure 1 shows a RotLSTM unit in detail. Assuming cell state size n, input size m, the RotLSTM has n(n + m)/2 extra parameters, a 12.5% increase (ignoring biases). Our expectation is that we can decrease n without harming performance and the rotations will enforce a better representation for the cell state. To empirically evaluate the performance of adding the rotation gate to LSTMs we use the toy NLP dataset bAbI with 1000 samples per task. The bAbI dataset is composed of 20 different tasks of various difficulties, starting from easy questions based on a single supporting fact (for example: DISPLAYFORM0 x is the concatenation of h t−1 and x t in the diagram (green and blue lines). Note that this differs from a regular LSTM by the introduction of the network producing angles u t and the rotation module marked U. In the diagram input size is 4 and cell state size is 3.John is in the kitchen. Where is John? A: Kitchen) and going to more difficult tasks of reasoning about size (example: The football fits in the suitcase. The box is smaller than the football. Will the box fit in the suitcase? A: yes) and path finding (example: The bathroom is south of the office. The bathroom is north of the hallway. How do you go from the hallway to the office? A: north, north). A summary of all tasks is available in Table 2. We are interested in evaluating the behaviour and performance of rotations on RNN units rather than beating state of the art. We compare a model based on RotLSTM with the same model based on the traditional LSTM. All models are trained with the same hyperparameters and we do not perform any hyperparameter tuning apart from using the sensible defaults provided in the Keras library and example code BID2.For the first experiment we train a LSTM and RotLSTM based model 10 times using a fixed cell state size of 50. In the second experiment we train the same models but vary the cell state size from 6 to 50 to assess whether the rotations help our models achieve good performance with smaller state sizes. We only choose even numbers for the cell state size to make all units go through rotations. The model architecture, illustrated in FIG0, is based on the Keras example implementation 1. This model architecture, empirically, shows better performance than the LSTM baseline published in. The input question and sentences are passed thorugh a word embedding layer (not shared, embeddings are different for questions and sentences). The question is fed into an RNN which produces a representation of the question. This representation is concatenated to every word vector from the story, which is then used as input to the second RNN. Intuitively, this helps the second RNN (Query) to focus on the important words to answer the question. The output of the second RNN is passed to a fully connected layer with a softmax activation of the size of the dictionary. The answer is the word with the highest activation. The categorical cross-entropy loss function was used for training. All dropout layers are dropping 30% of the nodes. The train-validation dataset split used was 95%-5%. The optimizer used was Adam with learning rate 0.001, no decay, β 1 = 0.9, β 2 = 0.999, = 10 −8. The training set was randomly shuffled before every epoch. All models were trained for 40 epochs. After every epoch the model performance was evaluated on the validation and training sets, and every 10 epochs on the test set. We set the random seeds to the same number for reproducibility and ran the experiments 10 times with 10 different random seeds. The source code is available at https://goo.gl/ Eopz2C 2. In this subsection we compare the the performance of models based on the LSTM and RotLSTM units on the bAbI dataset. Applying rotations on the unit memory of the LSTM cell gives a slight improvement in performance overall, and significant improvements on specific tasks. Results are shown in TAB0. The most significant improvements are faster convergence, as shown in Figure 3, and requiring smaller state sizes, illustrated in FIG2.On tasks 1 (basic factoid), 11 (basic coreference), 12 (conjunction) and 13 (compound coreference) the RotLSTM model reaches top performance a couple of epochs before the LSTM model consistently. The RotLSTM model also needs a smaller cell state size, reaching top performance at state size 10 to 20 where the LSTM needs 20 to 30. The top performance is, however, similar for both models, with RotLSTM improving the accuracy with up to 2.5%.The effect is observed on task 18 (reasoning about size) at a greater magnitude where the RotLSTM reaches top performance before epoch 20, after which it plateaus, while the LSTM model takes 40 epochs to fit the data. The training is more stable for RotLSTM and the final accuracy is improved by 20%. The RotLSTM reaches top performance using cell state 10 and the LSTM needs size 40. Similar performance increase for the RotLSTM (22.1%) is observed in task 5 (three argument relations), reaching top performance around epoch 25 and using a cell state of 50. Task 7 (counting) shows a similar behaviour with an accuracy increase of 14% for RotLSTM.Tasks 4 (two argument relations) and 20 (agent motivation) show quicker learning (better performance in the early epochs) for the RotLSTM model but both models reach their top performance after the same amount of traning. On task 20 the RotLSTM performance reaches top accuracy using state size 10 while the LSTM incremetally improves until using state size 40 to 50.Signs of overfitting for the RotLSTM model can be observed more prominently than for the LSTM model on tasks 15 (basic deduction) and 17 (positional reasoning).Our models, both LSTM and RotLSTM, perform poorly on tasks 2 and 3 (factoid questions with 2 and 3 supporting facts, respectively) and 14 (time manipulation). These problem classes are solved very well using models that look over the input data many times and use an attention mechanism that allows the model to focus on the relevant input sentences to answer a question BID14 BID8. Our models only look at the input data once and we do not filter out irrelevant information. A limitation of the models in our experiments is only applying pairwise 2D rotations. Representations of past input can be larger groups of the cell state vector, thus 2D rotations might not fully exploit the benefits of transformations. In the future we hope to explore rotating groups of elements and multi-dimensional rotations. Rotating groups of elements of the cell state could potentially also force the models to learn a more structured representation of the world, similar to how forcing a model to learn specific representations of scenes, as presented in BID6, yields semantic representations of the scene. Rotations also need not be fully flexible. Introducing hard constraints on the rotations and what groups of parameters can be rotated might lead the model to learn richer memory representations. Future work could explore how adding such constraints impacts learning times and final performance on different datasets, but also look at what constraints can qualitatively improve the representation of long-term dependencies. In this work we presented prelimiary tests for adding rotations to simple models but we only used a toy dataset. The bAbI dataset has certain advantages such as being small thus easy to train many models on a single machine, not having noise as it is generated from a simulation, and having a wide range of tasks of various difficulties. However it is a toy dataset that has a very limited vocabulary and lacks the complexity of real world datasets (noise, inconsistencies, larger vocabularies, more complex language constructs, and so on). Another limitation of our evaluation is only using text, specifically question answering. To fully evaluate the idea of adding rotations to memory cells, in the future, we aim to look into incorporating our rotations on different domains and tasks including speech to text, translation, language generation, stock prices, and other common problems using real world datasets. Tuning the hyperparameters of the rotation models might give better insights and performance increases and is something we aim to incorporate in our training pipeline in the future. A brief exploration of the angles produced by u and the weight matrix W rot show that u does not saturate, thus rotations are in fact applied to our cell states and do not converge to 0 (or 360 degress). A more in-depth qualitative analysis of the rotation gate is planned for future work. Peeking into the activations of our rotation gates could help understand the behaviour of rotations and to what extent they help better represent long-term memory. A very successful and popular mutation of the LSTM is the Gated Recurrent Unit (GRU) unit BID1. The GRU only has an output as opposed to both a cell state and an output and uses fewer gates. In the future we hope to explore adding rotations to GRU units and whether we can obtain similar . We have introduced a novel gating mechanism for RNN units that enables applying a parametrised transformation matrix to the cell state. We picked pairwise 2D rotations as the transformation and shown how this can be added to the popular LSTM units to create what we call RotLSTM. Figure 3: Accuracy comparison on training, validation (val) and test sets over 40 epochs for LSTM and RotLSTM models. The models were trained 10 times and shown is the average accuracy and in faded colour is the standard deviation. Test set accuracy was computed every 10 epochs. We trained a simple model using RotLSTM units and compared them with the same model based on LSTM units. We show that for the LSTM-based architetures adding rotations has a positive impact on most bAbI tasks, making the training require fewer epochs to achieve similar or higher accuracy. On some tasks the RotLSTM model can use a lower dimensional cell state vector and maintain its performance. Significant accracy improvements of approximatively 20% for the RotLSTM model over the LSTM model are visible on bAbI tasks 5 (three argument relations) and 18 (reasoning about size).

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Adding a new set of weights to the LSTM that rotate the cell memory improves performance on some bAbI tasks.

represent the tldr for finding the corresponding scientific article:

Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.


Question: Represent the scientific article to find a short TLDR

Question: We address the problem of marginal inference for an exponential family defined over the set of permutation matrices. This problem is known to quickly become intractable as the size of the permutation increases, since its involves the computation of the permanent of a matrix, a #P-hard problem. We introduce Sinkhorn variational marginal inference as a scalable alternative, a method whose validity is ultimately justified by the so-called Sinkhorn approximation of the permanent. We demonstrate the efectiveness of our method in the problem of probabilistic identification of neurons in the worm C.elegans Let P ∈ R n×n be a binary matrix representing a permutation of n elements (i.e. each row and column of P contains a unique 1). We consider the distribution over P defined as where A, B F is the Frobenius matrix inner product, log L is a parameter matrix and Z L is the normalizing constant. Here we address the problem of marginal inference, i.e. computing the matrix of expectations ρ:= E(P). This problem is known to be intractable since it requires access to Z L, also known as the permanent of L, and whose computation is known to be a #P-hard problem To overcome this difficulty we introduce Sinkhorn variational marginal inference, which can be computed efficiently and is straightforward to implement. Specifically, we approximate ρ as S(L), the Sinkhorn operator applied to L . S(L) is defined as the (infinite) successive row and column normalization of L (;, a limit that is known to in a doubly stochastic matrix . In section 2 we argue the Sinkhorn approximation is sensible, and in section 3 we describe the problem of probabilistic inference of neural identity in C.elegans and demonstrate the Sinkhorn approximation produces the best . Our argument bases on the well-known relation between marginal inference and the normalizing constant , valid for exponential families. Specifically, (1.1) defines an exponential family with sufficient statistic P and parameter log L. By virtue of Theorem 3.4 in: where M is the marginal polytope (here, the Birkhoff polytope, the set of doubly stochastic matrices) and A * (µ) is the dual function log Z L, i.e. Moreover, for a given L, µ(L) achieving the supremum in (2.1) is exactly the matrix of marginals, µ(L) = ρ L and the dual function A * (µ(L)) coincides with the negative entropy of (1.1). Then, marginal inference of ρ L and computation of the permanent Z L = perm(L) are linked by the optimization problem in (2.2). As in any generic variational inference scheme , we obtain an approximate ρ by replacing the variational representation of Z L in (2.1), by a different, more tractable optimization problem. Typically, the quality of the approximated ρ depends on how tight is the approximation to Z L. Our approximation is based on replacing the intractable dual function A * (µ) by a component-wise entropy, and whose solution is exactly S(L). In detail, the following variational representation holds ): By using the component-wise entropy in (2.1) we obtain an approximation of the normalizing constant, that we call as the Sinkhorn permanent , perm S (L). In the following proposition we provide bounds for this approximation. The following bounds hold We note the Sinkhorn approximation has recently been proposed independently . However, there the approximation is proposed rather heuristically, without any appeal to a theoretical framework. Additionally, the so-called Bethe variational inference method is a rather general rationale for obtaining variational approximations in graphical models, where the dual function A * (µ) is approximated by the value it would take if the underlying Markov random field had a tree structure . This approximation has successfully been applied to permutations (; ; ;), where the corresponding approximate marginal B(L) is computed through belief propagation , enjoying also better theoretical guarantees than the Sinkhorn approximation. Indeed, for the Bethe approximation of the permanent, perm B (·) the following bounds are known (; However, there are also important computational differences. A single iteration of the Sinkhorn algorithms corresponds to a row and column normalization, but the message computations in the belief propagation-like routine for the Bethe approximation are more complex. Explicit formulae of such Sinkhorn and Bethe iterations are available in Appendix C. Fig 1 (b) shows that in practice the Bethe approximation also produces better permanent approximations, confirming theoretical predictions. We considered the simple case where n = 8 and the permanent and marginal can be computed by enumeration, so comparisons with ground truth are possible. However, and quite interestingly, in many cases (see Figs 1(a) and A.1(a) in the Appendix) the Sinkhorn approximation produced qualitatively better marginals, putting more mass on more non-zero entries than the Bethe approximation, regardless of possibly worse permanents. Additionally, we observed that for moderate n the Sinkhorn approximation scaled better. For example, if n = 710, each Bethe iteration took on average 0.035 seconds, while each Sinkhorn iteration took only 0.0027 seconds (see Fig A. 2 in the Appendix for details). Comparison of Bethe and Sinkhorn approximations. 1,000 submatrices of size n = 8 were randomly sampled from the C.elegans dataset described in section 3. (a) Examples of a (log) true marginal matrix ρ along with Sinkhorn and Bette approximation. The rightmost plot is a histogram of the log permanent across the samples. (b) Differences between approximate and true log permanent (left) and mean absolute errors of log marginals (right) for our two approximations. We considered additional 1,000'random' submatrices made by uniformly sampling entries between the minimum and maximum values of each C.elegans submatrix Finally, we note that sampling-based methods may be also used for marginal inference. Indeed, quite sophisticated samplers have been proposed to show polynomial approximability of the permanent ; however, their practical appeal is limited. In section 3 we show that an elementary MCMC sampler failed to produce sensible marginal inferences at reasonable time. The worm C.elegans is a unique species since their nervous system is stereotypical; i.e., the number of neurons (roughly, 300) and the connections between those neurons remain unchanged from animal to animal. Recent advances in neurotechnology have enabled whole brain imaging so that the long-standing fundamental question about how the activity in the worm brain relates to its behavior in the world can be now studied and settled. However, before that, a technical problem has to be solved: given volumetric images of the worm neurons have to be identified; that is, canonical labels (names) must be assigned to each. We applied our methodology for such probabilistic neural identification in the context of NeuroPAL , a multicolor C.elegans transgene where neuron colors were designed to facilitate neural identification (see Fig 1 for an example). Specifically, given n observed neurons represented as vectors in R 6 (position and color), we aim to estimate the matrix of marginal ρ such that ρ k,i is the probability that observed neuron k is identified with the canonical identity i. These probabilities are relevant as they provide uncertainty estimates for model predictions, giving a much more complete picture than point estimates (e.g. a permutation found via maximum likelihood). We consider a gaussian model for each canonical neuron, whose parameters (µ k, Σ k) are inferred beforehand from previously annotated worms (see for details). Let π denote the permutation so that π(k) is the canonical index of the k − th observed neuron. Then, the likelihood of observing data Y = (y k) writes as: (3.1) Suppose a flat prior is assumed over P. Then, it is plain to verify that equation (3.1) induces a posterior over P that has the form of (1.1), with L defined as Figure 2: A worm's head displaying the deterministic coloring scheme identical across all NeuroPAL worms, with neuron names (determined by a human) over each neuron. In the context of NeuroPAL we consider a downstream task involving the computation of the approximate probabilistic neural identifies ρ. Specifically, in this task a human is asked to manually label the neurons for which the model estimates are the most uncertain; i.e., the rows of ρ that are closest to the uniform distribution. As the human progressively annotates neurons this uncertainty resolves and the corresponding model update lead to an increases in identification accuracy for the remaining neurons. Ideally the human will only require a few annotations to reach a high accuracy, and therefore, as a proxy for approximation quality we measure how much faster accuracy increases in comparison to simple baselines; e.g., where at each time a neuron is randomly chosen. Results are shown in Fig 3, and further details are described in the Appendix. We considered several alternatives: i) Sinkhorn approximation, ii) Bethe approximation, iii) MCMC, iv) the random baseline described above, v) a naive baseline where uncertainty estimates are made by scaling only the rows of the likelihood matrix, i.e., without imposing any one-to-one assignment structure, and vi) a'ground truth', the protocol where the labels that are chosen are the ones where the model makes a wrong prediction (this oracle cannot be realized in practice). Results of Sinkhorn and Bethe approximations are similar but the former slightly better, presumably a consequence of more accurate estimates of low probability marginals (see Figs 1(a) and A.1(a)). They both are substantially better than any baseline other than the oracle. Contrarily, we see MCMC does not provide better than the naive baseline, suggesting lack of convergence for chain lengths leading to computational times comparable to the ones of approximated methods. We have introduced the Sinkhorn approximation for marginal inference, and our it is a sensible alternative to sampling, and it may provide faster, simpler and more accurate approximate marginals than the Bethe approximation, despite typically leading to worse permanent approximations. We leave for future work a thorough analysis of the relation between quality of permanent approximation and corresponding marginals. Also, it can be verified that S(L) = diag(x)Ldiag(y), where diag(x), diag(y) are some positive vectors x, y turned into diagonal matrices (Peyré et al., 2019). Then, Additionally, we obtain the (log) Sinkhorn approximation of the permanent of L, perm S (L), by evaluating S(L) in the problem it solves, (2.3). By simple algebra and using the fact that S(L) is a doubly stochastic matrix we see that By combining the last three displays we obtain from which the follows. We used the dataset described in. This consists on ten NeuroPAL worm heads with available human labels, and with number of neurons n ranging from 180 to 195. Each of these worms is summarized through a n × n log-likelihood matrix L computed with the methods described in (, Supplemental Information). For both the Sinkhorn and Bethe approximation we used 200 iterations. These values led to the computation times described in Fig 1, and preliminary showed were sufficient to ensure convergence (that is, none of the would change dramatically for a larger number of iterations). For the MCMC sampler we used the method described in. We used 100 chains of length 1000, and for each of them considered we took as samples of the multiples of 10 starting from iteration 500 on. All were obtained on a desktop computer with an Intel Xeon W-2125 processor. r e t u r n np. exp (logP) Bethe approximation The following is an efficient log-space implementation of the message passing algorithm described in (, Lemma 29), which was subsequently simplified by. The parameter eps is introduced for numerical estability..., 710, a number of 1, 000 submatrices of size n were randomly drawn from the ten available log likelihood C.elegans matrices (see text on Appendix B, indexes were drawn with replacement). Error bars are omitted because they were too small to be noticed.

New methodology for variational marginal inference of permutations based on Sinkhorn algorithm, applied to probabilistic identification of neurons

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the scientific article to find a short tldr

Question: The robustness of neural networks to adversarial examples has received great attention due to security implications. Despite various attack approaches to crafting visually imperceptible adversarial examples, little has been developed towards a comprehensive measure of robustness. In this paper, we provide theoretical justification for converting robustness analysis into a local Lipschitz constant estimation problem, and propose to use the Extreme Value Theory for efficient evaluation. Our analysis yields a novel robustness metric called CLEVER, which is short for Cross Lipschitz Extreme Value for nEtwork Robustness. The proposed CLEVER score is attack-agnostic and is computationally feasible for large neural networks. Experimental on various networks, including ResNet, Inception-v3 and MobileNet, show that (i) CLEVER is aligned with the robustness indication measured by the $\ell_2$ and $\ell_\infty$ norms of adversarial examples from powerful attacks, and (ii) defended networks using defensive distillation or bounded ReLU indeed give better CLEVER scores. To the best of our knowledge, CLEVER is the first attack-independent robustness metric that can be applied to any neural network classifiers. Recent studies have highlighted the lack of robustness in state-of-the-art neural network models, e.g., a visually imperceptible adversarial image can be easily crafted to mislead a well-trained network BID28 BID9 BID3. Even worse, researchers have identified that these adversarial examples are not only valid in the digital space but also plausible in the physical world BID17 BID8. The vulnerability to adversarial examples calls into question safety-critical applications and services deployed by neural networks, including autonomous driving systems and malware detection protocols, among others. In the literature, studying adversarial examples of neural networks has twofold purposes: (i) security implications: devising effective attack algorithms for crafting adversarial examples, and (ii) robustness analysis: evaluating the intrinsic model robustness to adversarial perturbations to normal examples. Although in principle the means of tackling these two problems are expected to be independent, that is, the evaluation of a neural network's intrinsic robustness should be agnostic to attack methods, and vice versa, existing approaches extensively use different attack as a measure of robustness of a target neural network. Specifically, given a set of normal examples, the attack success rate and distortion of the corresponding adversarial examples crafted from a particular attack algorithm are treated as robustness metrics. Consequently, the network robustness is entangled with the attack algorithms used for evaluation and the analysis is limited by the attack capabilities. More importantly, the dependency between robustness evaluation and attack approaches can cause biased analysis. For example, adversarial training is a commonly used technique for improving the robustness of a neural network, accomplished by generating adversarial examples and retraining the network with corrected labels. However, while such an adversarially trained network is made robust to attacks used to craft adversarial examples for training, it can still be vulnerable to unseen attacks. Motivated by the evaluation criterion for assessing the quality of text and image generation that is completely independent of the underlying generative processes, such as the BLEU score for texts BID25 and the INCEPTION score for images BID27, we aim to propose a comprehensive and attack-agnostic robustness metric for neural networks. Stemming from a perturbation analysis of an arbitrary neural network classifier, we derive a universal lower bound on the minimal distortion required to craft an adversarial example from an original one, where the lower bound applies to any attack algorithm and any p norm for p ≥ 1. We show that this lower bound associates with the maximum norm of the local gradients with respect to the original example, and therefore robustness evaluation becomes a local Lipschitz constant estimation problem. To efficiently and reliably estimate the local Lipschitz constant, we propose to use extreme value theory BID6 for robustness evaluation. In this context, the extreme value corresponds to the local Lipschitz constant of our interest, which can be inferred by a set of independently and identically sampled local gradients. With the aid of extreme value theory, we propose a robustness metric called CLEVER, which is short for Cross Lipschitz Extreme Value for nEtwork Robustness. We note that CLEVER is an attack-independent robustness metric that applies to any neural network classifier. In contrast, the robustness metric proposed in BID11, albeit attack-agnostic, only applies to a neural network classifier with one hidden layer. We highlight the main contributions of this paper as follows:• We propose a novel robustness metric called CLEVER, which is short for Cross Lipschitz Extreme Value for nEtwork Robustness. To the best of our knowledge, CLEVER is the first robustness metric that is attack-independent and can be applied to any arbitrary neural network classifier and scales to large networks for ImageNet.• The proposed CLEVER score is well supported by our theoretical analysis on formal robustness guarantees and the use of extreme value theory. Our robustness analysis extends the in BID11 from continuously differentiable functions to a special class of non-differentiable functions -neural+ networks with ReLU activations.• We corroborate the effectiveness of CLEVER by conducting experiments on state-of-theart models for ImageNet, including ResNet BID10, Inception-v3 BID29 and MobileNet . We also use CLEVER to investigate defended networks against adversarial examples, including the use of defensive distillation BID23 and bounded ReLU BID34. Experimental show that our CLEVER score well aligns with the attack-specific robustness indicated by the 2 and ∞ distortions of adversarial examples. One of the most popular formulations found in literature for crafting adversarial examples to mislead a neural network is to formulate it as a minimization problem, where the variable δ ∈ R d to be optimized refers to the perturbation to the original example, and the objective function takes into account unsuccessful adversarial perturbations as well as a specific norm on δ for assuring similarity. For instance, the success of adversarial examples can be evaluated by their cross-entropy loss BID28 BID9 or model prediction BID2. The norm constraint on δ can be implemented in a clipping manner BID18 or treated as a penalty function BID2. The p norm of δ, defined as δ p = (DISPLAYFORM0 for any p ≥ 1, is often used for crafting adversarial examples. In particular, when p = ∞, δ ∞ = max i∈{1, ...,d} |δ i | measures the maximal variation among all dimensions in δ. When p = 2, δ 2 becomes the Euclidean norm of δ. When p = 1, δ 1 = p i=1 |δ i | measures the total variation of δ. The state-of-the-art attack methods for ∞, 2 and 1 norms are the iterative fast gradient sign method (I-FGSM) BID9 BID18, Carlini and Wagner's attack (CW attack) BID2, and elastic-net attacks to deep neural networks (EAD) BID4, respectively. These attacks fall into the category of white-box attacks since the network model is assumed to be transparent to an attacker. Adversarial examples can also be crafted from a black-box network model using an ensemble approach BID20, training a substitute model, or employing zeroth-order optimization based attacks BID5. Since the discovery of vulnerability to adversarial examples BID28, various defense methods have been proposed to improve the robustness of neural networks. The rationale for defense is to make a neural network more resilient to adversarial perturbations, while ensuring the ing defended model still attains similar test accuracy as the original undefended network. Papernot et al. proposed defensive distillation BID23, which uses the distillation technique BID12 and a modified softmax function at the final layer to retrain the network parameters with the prediction probabilities (i.e., soft labels) from the original network. BID34 showed that by changing the ReLU function to a bounded ReLU function, a neural network can be made more resilient. Another popular defense approach is adversarial training, which generates and augments adversarial examples with the original training data during the network training stage. On MNIST, the adversarially trained model proposed by BID21 can successfully defend a majority of adversarial examples at the price of increased network capacity. Model ensemble has also been discussed to increase the robustness to adversarial examples BID30 BID19. In addition, detection methods such as feature squeezing BID33 and example reforming BID22 can also be used to identify adversarial examples. However, the CW attack is shown to be able to bypass 10 different detection methods BID1. In this paper, we focus on evaluating the intrinsic robustness of a neural network model to adversarial examples. The effect of detection methods is beyond our scope. BID28 compute global Lipschitz constant for each layer and use their product to explain the robustness issue in neural networks, but the global Lipschitz constant often gives a very loose bound. BID11 gave a robustness lower bound using a local Lipschitz continuous condition and derived a closed-form bound for a multi-layer perceptron (MLP) with a single hidden layer and softplus activation. Nevertheless, a closed-form bound is hard to derive for a neural network with more than one hidden layer. BID31 utilized terminologies from topology to study robustness. However, no robustness bounds or estimates were provided for neural networks. On the other hand, works done by BID7; BID15 b); BID14 focus on formally verifying the viability of certain properties in neural networks for any possible input, and transform this formal verification problem into satisfiability modulo theory (SMT) and large-scale linear programming (LP) problems. These SMT or LP based approaches have high computational complexity and are only plausible for very small networks. Intuitively, we can use the distortion of adversarial examples found by a certain attack algorithm as a robustness metric. For example, BID0 proposed a linear programming (LP) formulation to find adversarial examples and use the distortions as the robustness metric. They observe that the LP formulation can find adversarial examples with smaller distortions than other gradient-based attacks like L-BFGS BID28. However, the distortion found by these algorithms is an upper bound of the true minimum distortion and depends on specific attack algorithms. These methods differ from our proposed robustness measure CLEVER, because CLEVER is an estimation of the lower bound of the minimum distortion and is independent of attack algorithms. Additionally, unlike LP-based approaches which are impractical for large networks, CLEVER is computationally feasible for large networks like Inception-v3. The concept of minimum distortion and upper/lower bound will be formally defined in Section 3. In this section, we provide formal robustness guarantees of a classifier in Theorem 3.2. Our robustness guarantees are general since they only require a mild assumption on Lipschitz continuity of the classification function. For differentiable classification functions, our are consistent with the main theorem in BID11 but are obtained by a much simpler and more BID11 ) is in fact a special case of our analysis. We start our analysis by defining the notion of adversarial examples, minimum p distortions, and lower/upper bounds. All the notations are summarized in TAB0. Definition 3.1 (perturbed example and adversarial example). Let DISPLAYFORM0 DISPLAYFORM1 An adversarial example is a perturbed example x a that changes c(x 0). A successful untargeted attack is to find a x a such that c(x a) = c(x 0) while a successful targeted attack is to find a x a such that c(x a) = t given a target class t = c(x 0). Definition 3.2 (minimum adversarial distortion ∆ p,min). Given an input vector x 0 of a classifier f, the minimum p adversarial distortion of x 0, denoted as ∆ p,min, is defined as the smallest ∆ p over all adversarial examples of x 0. Definition 3.3 (lower bound of ∆ p,min). Suppose ∆ p,min is the minimum adversarial distortion of DISPLAYFORM2, is defined such that any perturbed examples of x 0 with δ p ≤ β L are not adversarial examples. Definition 3.4 (upper bound of ∆ p,min). Suppose ∆ p,min is the minimum adversarial distortion of x 0. An upper bound of ∆ p,min, denoted by β U where β U ≥ ∆ p,min, is defined such that there exists an adversarial example of x 0 with δ p ≥ β U.The lower and upper bounds are instance-specific because they depend on the input x 0. While β U can be easily given by finding an adversarial example of x 0 using any attack method, β L is not easy to find. β L guarantees that the classifier is robust to any perturbations with δ p ≤ β L, certifying the robustness of the classifier. Below we show how to derive a formal robustness guarantee of a classifier with Lipschitz continuity assumption. Specifically, our analysis obtains a lower bound of DISPLAYFORM3 Lemma 3.1 (Lipschitz continuity and its relationship with gradient norm (Paulavičius &Žilinskas, 2006) ). Let S ⊂ R d be a convex bounded closed set and let h(x): S → R be a continuously differentiable function on an open set containing S. Then, h(x) is a Lipschitz function with Lipschitz constant L q if the following inequality holds for any x, y ∈ S: DISPLAYFORM4 where DISPLAYFORM5 ) is the gradient of h(x), and DISPLAYFORM6 Given Lemma 3.1, we then provide a formal guarantee to the lower bound β L. Theorem 3.2 (Formal guarantee on lower bound β L for untargeted attack). Let x 0 ∈ R d and f: R d → R K be a multi-class classifier with continuously differentiable components f i and let c = argmax 1≤i≤K f i (x 0) be the class which f predicts for x 0. For all δ ∈ R d with DISPLAYFORM7 argmax 1≤i≤K f i (x 0 + δ) = c holds with DISPLAYFORM8 is a lower bound of minimum distortion. The intuitions behind Theorem 3.2 is shown in FIG0 with an one-dimensional example. The function value g(x) = f c (x) − f j (x) near point x 0 is inside a double cone formed by two lines passing (x 0, g(x 0)) and with slopes equal to ±L q, where L q is the (local) Lipschitz constant of g(x) near x 0. In other words, the function value of g(x) around x 0, i.e. g(x 0 + δ) can be bounded by g(x 0), δ and the Lipschitz constant L q. When g(x 0 + δ) is decreased to 0, an adversarial example is found and the minimal change of δ is DISPLAYFORM9 Lq. The complete proof is deferred to Appendix A. DISPLAYFORM10 is the Lipschitz constant of the function involving cross terms: f c (x) − f j (x), hence we also call it cross Lipschitz constant following BID11.To distinguish our analysis from BID11, we show in Corollary 3.2.1 that we can obtain the same in BID11 by Theorem 3.2. In fact, the analysis in BID11 ) is a special case of our analysis because the authors implicitly assume Lipschitz continuity on f i (x) when requiring f i (x) to be continuously differentiable. They use local Lipschitz constant (L q,x0) instead of global Lipschitz constant (L q) to obtain a tighter bound in the adversarial perturbation δ. DISPLAYFORM11. By Theorem 3.2, we obtain the bound in BID11 ): DISPLAYFORM12 An important use case of Theorem 3.2 and Corollary 3.2.1 is the bound for targeted attack: Corollary 3.2.2 (Formal guarantee on β L for targeted attack). Assume the same notation as in Theorem 3.2 and Corollary 3.2.1. For a specified target class j, we have δ p ≤ min DISPLAYFORM13 In addition, we further extend Theorem 3.2 to a special case of non-differentiable functions -neural networks with ReLU activations. In this case the Lipchitz constant used in Lemma 3.1 can be replaced by the maximum norm of directional derivative, and our analysis above will go through. Lemma 3.3 (Formal guarantee on β L for ReLU networks).3 Let h(·) be a l-layer ReLU neural network with W i as the weights for layer i. We ignore bias terms as they don't contribute to gradient. DISPLAYFORM14 is the one-sided directional direvative, then Theorem 3.2, Corollary 3.2.1 and Corollary 3.2.2 still hold. In this section, we provide an algorithm to compute the robustness metric CLEVER with the aid of extreme value theory, where CLEVER can be viewed as an efficient estimator of the lower bound β L and is the first attack-agnostic score that applies to any neural network classifiers. Recall in Section 3 2 proof deferred to Appendix B 3 proof deferred to Appendix C we show that the lower bound of network robustness is associated with g(x 0) and its cross Lipschitz constant L j q,x0, where g(DISPLAYFORM0 is readily available at the output of a classifier and L j q,x0 is defined as max x∈Bp(x0,R) ∇g(x) q. Although ∇g(x) can be calculated easily via back propagation, computing L j q,x0 is more involved because it requires to obtain the maximum value of ∇g(x) q in a ball. Exhaustive search on low dimensional x in B p (x 0, R) seems already infeasible, not to mention the image classifiers with large feature dimensions of our interest. For instance, the feature dimension d = 784, 3072, 150528 for MNIST, CIFAR and ImageNet respectively. One approach to compute L j q,x0 is through sampling a set of points x (i) in a ball B p (x 0, R) around x 0 and taking the maximum value of ∇g(x (i) ) q. However, a significant amount of samples might be needed to obtain a good estimate of max ∇g(x) q and it is unknown how good the estimate is compared to the true maximum. Fortunately, Extreme Value Theory ensures that the maximum value of random variables can only follow one of the three extreme value distributions, which is useful to estimate max ∇g(x) q with only a tractable number of samples. It is worth noting that although BID32 also applied extreme value theory to estimate the Lipschitz constant. However, there are two main differences between their work and this paper. First of all, the sampling methodology is entirely different. BID32 calculates the slopes between pairs of sample points whereas we directly take samples on the norm of gradient as in Lemma 3.1. Secondly, the functions considered in BID32 are only one-dimensional as opposed to the high-dimensional classification functions considered in this paper. For comparison, we show in our experiment that the approach in BID32, denoted as SLOPE in Table 3 and FIG3, perform poorly for high-dimensional classifiers such as deep neural networks.4.1 ESTIMATE L j q,x0 VIA EXTREME VALUE THEORY When sampling a point x uniformly in B p (x 0, R), ∇g(x) q can be viewed as a random variable characterized by a cumulative distribution function (CDF). For the purpose of illustration, we derived the CDF for a 2-layer neural network in Theorem D.1. 4 For any neural networks, suppose we have n samples {∇g(x (i) ) q }, and denote them as a sequence of independent and identically distributed (iid) random variables Y 1, Y 2, · · ·, Y n, each with CDF F Y (y). The CDF of max{Y 1, · · ·, Y n}, denoted as F n Y (y), is called the limit distribution of F Y (y). Fisher-TippettGnedenko theorem says that F n Y (y), if exists, can only be one of the three family of extreme value distributions -the Gumbel class, the Fréchet class and the reverse Weibull class. Theorem 4.1 (Fisher-Tippett-Gnedenko Theorem). If there exists a sequence of pairs of real numbers (a n, b n) such that a n > 0 and lim n→∞ F n Y (a n y + b n) = G(y), where G is a non-degenerate distribution function, then G belongs to either the Gumbel class (Type I), the Fréchet class (Type II) or the Reverse Weibull class (Type III) with their CDFs as follows: DISPLAYFORM1 Fréchet class (Type II): DISPLAYFORM2 if y ≥ a W, where a W ∈ R, b W > 0 and c W > 0 are the location, scale and shape parameters, respectively. Theorem 4.1 implies that the maximum values of the samples follow one of the three families of distributions. If g(x) has a bounded Lipschitz constant, ∇g(x (i) ) q is also bounded, thus its limit distribution must have a finite right end-point. We are particularly interested in the reverse Weibull class, as its CDF has a finite right end-point (denoted as a W). The right end-point reveals the upper limit of the distribution, known as the extreme value. The extreme value is exactly the unknown local cross Lipschitz constant L j q,x0 we would like to estimate in this paper. To estimate L j q,x0, we first generate N s samples of x (i) over a fixed ball B p (x 0, R) uniformly and independently in each batch with a total of N b batches. We then compute ∇g(x (i) ) q and store the maximum values of each batch in set S. Next, with samples in S, we perform a maximum likelihood estimation of reverse Weibull distribution parameters, and the location estimateâ W is used as an estimate of L j q,x0. Given an instance x 0, its classifier f (x 0) and a target class j, a targeted CLEVER score of the classifier's robustness can be computed via g(x 0) and L j q,x0. Similarly, untargeted CLEVER scores can be computed. With the proposed procedure of estimating L j q,x0 described in Section 4.1, we summarize the flow of computing CLEVER score for both targeted attacks and un-targeted attacks in Algorithm 1 and 2, respectively. Algorithm 1: CLEVER-t, compute CLEVER score for targeted attack Input: a K-class classifier f (x), data example x 0 with predicted class c, target class j, batch size N b, number of samples per batch N s, perturbation norm p, maximum perturbation R Result: CLEVER Score µ ∈ R + for target class DISPLAYFORM0 Algorithm 2: CLEVER-u, compute CLEVER score for un-targeted attack Input: Same as Algorithm 1, but without a target class j Result: CLEVER score ν ∈ R + for un-targeted attack DISPLAYFORM1 We conduct experiments on CIFAR-10 (CIFAR for short), MNIST, and ImageNet data sets. For the former two smaller datasets CIFAR and MNIST, we evaluate CLEVER scores on four relatively small networks: a single hidden layer MLP with softplus activation (with the same number of hidden units as in BID11), a 7-layer AlexNet-like CNN (with the same structure as in BID2), and the 7-layer CNN with defensive distillation BID23 ) (DD) and bounded ReLU BID34 ) (BReLU) defense techniques employed. For ImageNet data set, we use three popular deep network architectures: a 50-layer Residual Network BID10 ) (ResNet-50), Inception-v3 BID29 and MobileNet . They were chosen for the following reasons: (i) they all yield (close to) state-of-theart performance among equal-sized networks; and (ii) their architectures are significantly different with unique building blocks, i.e., residual block in ResNet, inception module in Inception net, and depthwise separable convolution in MobileNet. Therefore, their diversity in network architectures is appropriate to test our robustness metric. For MobileNet, we set the width multiplier to 1.0, achieving a 70.6% accuracy on ImageNet. We used public pretrained weights for all ImageNet models 5.In all our experiments, we set the sampling parameters N b = 500, N s = 1024 and R = 5. For targeted attacks, we use 500 test-set images for CIFAR and MNIST and use 100 test-set images for ImageNet; for each image, we evaluate its targeted CLEVER score for three targets: a random target class, a least likely class (the class with lowest probability when predicting the original example), and the top-2 class (the class with largest probability except for the true class, which is usually the easiest target to attack). We also conduct untargeted attacks on MNIST and CIFAR for 100 test-set images, and evaluate their untargeted CLEVER scores. Our experiment code is publicly available 6. Figure 3. If the p-value is greater than 0.05, the null hypothesis cannot be rejected, meaning that the underlying data samples fit a reverse Weibull distribution well. Figure 3 shows that all numbers are close to 100%, validating the use of reverse Weibull distribution as an underlying distribution of gradient norm samples empirically. Therefore, the fitted location parameter of reverse Weibull distribution (i.e., the extreme value),â W, can be used as a good estimation of local cross Lipschitz constant to calculate the CLEVER score. The exact numbers are shown in TAB5 in Appendix E. All numbers for each model are close to 100%, indicating S fits reverse Weibull distributions well. We apply the state-of-the-art white-box attack methods, iterative fast gradient sign method (I-FGSM) BID9 BID18 and Carlini and Wagner's attack (CW) BID2, to find adversarial examples for 11 networks, including 4 networks trained on CIFAR, 4 networks trained on MNIST, and 3 networks trained on ImageNet. For CW attack, we run 1000 iterations for ImageNet and CIFAR, and 2000 iterations for MNIST, as MNIST has shown to be more difficult to attack BID4. Attack learning rate is individually tuned for each model: 0.001 for Inception-v3 and ResNet-50, 0.0005 for MobileNet and 0.01 for all other networks. For I-FGSM, we run 50 iterations and choose the optimal ∈ {0.01, 0.025, 0.05, 0.1, 0.3, 0.5, 0.8, 1.0} to achieve the smallest ∞ distortion for each individual image. For defensively distilled (DD) networks, 50 iterations of I-FGSM are not sufficient; we use 250 iterations for CIFAR-DD and 500 iterations for MNIST-DD to achieve a 100% success rate. For the problem to be non-trivial, images that are classified incorrectly are skipped. We report 100% attack success rates for all the networks, and thus the average distortion of adversarial examples can indicate the attack-specific robustness of each network. For comparison, we compute the CLEVER scores for the same set of images and attack targets. To the best of our knowledge, CLEVER is the first attack-independent robustness score that is capable of handling the large networks studied in this paper, so we directly compare it with the attack-induced distortion metrics in our study. We evaluate the effectiveness of our CLEVER score by comparing the upper bound β U (found by attacks) and CLEVER score, where CLEVER serves as an estimated lower bound, β L. Table 3 compares the average 2 and ∞ distortions of adversarial examples found by targeted CW and I-FGSM attacks and the corresponding average targeted CLEVER scores for 2 and ∞ norms, and FIG3 visualizes the for ∞ norm. Similarly, Table 2 compares untargeted CW and I-FGSM attacks with untargeted CLEVER scores. As expected, CLEVER is smaller than the distortions of adversarial images in most cases. More importantly, since CLEVER is independent of attack algorithms, the reported CLEVER scores can roughly indicate the distortion of the best possible attack in terms of a specific p distortion. The average 2 distortion found by CW attack is close to the 2 CLEVER score, indicating CW is a strong 2 attack. In addition, when a defense mechanism (Defensive Distillation or Bounded ReLU) is used, the corresponding CLEVER scores are consistently increased (except for CIFAR-BReLU), indicating that the network is indeed made more resilient to adversarial perturbations. For CIFAR-BReLU, both CLEVER scores and p norm of adversarial examples found by CW attack decrease, implying that bound ReLU is an ineffective defense for CIFAR. CLEVER scores can be seen as a security checkpoint for unseen attacks. For example, if there is a substantial gap in distortion between the CLEVER score and the considered attack algorithms, it may suggest the existence of a more effective attack that can close the gap. Since CLEVER score is derived from an estimation of the robustness lower bound, we further verify the viability of CLEVER per each example, i.e., whether it is usually smaller than the upper bound found by attacks. TAB4 shows the percentage of inaccurate estimations where the CLEVER score is larger than the distortion of adversarial examples found by CW and I-FGSM attacks in three ImageNet networks. We found that CLEVER score provides an accurate estimation for most of the examples. For MobileNet and Resnet-50, our CLEVER score is a strict lower bound of these two attacks for more than 96% of tested examples. For Inception-v3, the condition of strict lower bound Table 2: Comparison between the average untargeted CLEVER score and distortion found by CW and I-FGSM untargeted attacks. DD and BReLU represent Defensive Distillation and Bounded ReLU defending methods applied to the baseline CNN network. Table 3: Comparison of the average targeted CLEVER scores with average ∞ and 2 distortions found by CW, I-FSGM attacks, and the average scores calculated by using the algorithm in BID32 (denoted as SLOPE) to estimate Lipschitz constant. DD and BReLU denote Defensive Distillation and Bounded ReLU defending methods applied to the CNN network. We did not include SLOPE in ImageNet networks because it has been shown to be ineffective even for smaller networks.(a) avergage ∞ distortion of CW and I-FGSM targeted attacks, and CLEVER and SLOPE estimation. Some very large SLOPE estimates (in parentheses) exceeding the maximum possible ∞ distortion are reported as 1. Random Target Top-2 Target CW I-FGSM CLEVER SLOPE CW I-FGSM CLEVER BID32. SLOPE significantly exceeds the distortions found by attacks, thus it is an inappropriate estimation of lower bound β L.is worse (still more than 75%), but we found that in these cases the attack distortion only differs from our CLEVER score by a fairly small amount. In Figure 5 we show the empirical CDF of the gap between CLEVER score and the 2 norm of adversarial distortion generated by CW attack for the same set of images in TAB4. In Figure 6, we plot the 2 distortion and CLEVER scores for each DISPLAYFORM0 0% 0% 0% 2% 0% 0% 0% 0% 0% 0% 0% Resnet-50 4% 0% 0% 0% 2% 0% 0% 0% 1% 0% 0% 0% Inception-v3 25% 0% 0% 0% 23% 0% 0% 0% 15% 0% 0% 0% (a individual image. A positive gap indicates that CLEVER (estimated lower bound) is indeed less than the upper bound found by CW attack. Most images have a small positive gap, which signifies the near-optimality of CW attack in terms of 2 distortion, as CLEVER suffices for an estimated capacity of the best possible attack. In Figure 7, we vary the number of samples (N b = 50, 100, 250, 500) and compute the 2 CLEVER scores for three large ImageNet models, Inception-v3, ResNet-50 and MobileNet. We observe that 50 or 100 samples are usually sufficient to obtain a reasonably accurate robustness estimation despite using a smaller number of samples. On a single GTX 1080 Ti GPU, the cost of 1 sample (with N s = 1024) is measured as 2.9 s for MobileNet, 5.0 s for ResNet-50 and 8.9 s for Inception-v3, thus the computational cost of CLEVER is feasible for state-of-the-art large-scale deep neural networks. Additional figures for MNIST and CIFAR datasets are given in Appendix E. In this paper, we propose the CLEVER score, a novel and generic metric to evaluate the robustness of a target neural network classifier to adversarial examples. Compared to the existing robustness evaluation approaches, our metric has the following advantages: (i) attack-agnostic; (ii) applicable to any neural network classifier; (iii) comes with strong theoretical guarantees; and (iv) is computationally feasible for large neural networks. Our extensive experiments show that the CLEVER score well matches the practical robustness indication of a wide range of natural and defended networks. A PROOF OF THEOREM 3.2Proof. According to Lemma 3.1, the assumption that g(DISPLAYFORM0 Let x = x 0 + δ and y = x 0 in, we get DISPLAYFORM1 When g(x 0 + δ) = 0, an adversarial example is found. As indicated by, DISPLAYFORM2 no adversarial examples can be found: DISPLAYFORM3 Finally, to achieve argmax 1≤i≤K f i (x 0 + δ) = c, we take the minimum of the bound on δ p in (A) over j = c. I.e. if DISPLAYFORM4, the classifier decision can never be changed and the attack will never succeed. B PROOF OF COROLLARY 3.2.1Proof. By Lemma 3.1 and let g = f c − f j, we get L j q,x0 = max y∈Bp(x0,R) ∇g(y) q = max y∈Bp(x0,R) ∇f j (y) − ∇f c (y) q, which then gives the bound in Theorem 2.1 of BID11. DISPLAYFORM5 Proof. For any x, y, let d = y−x y−x p be the unit vector pointing from x to y and r = y − x p. Define uni-variate function u(z) = h(x + zd), then u = h(x) and u(r) = h(y) and observe that D + h(x + zd; d) and D + h(x + zd; −d) are the right-hand and left-hand derivatives of u(z), we have DISPLAYFORM6 For ReLU network, there can be at most finite number of points in z ∈ (0, r) such that g (z) does not exist. This can be shown because each discontinuous z is caused by some ReLU activation, and there are only finite combinations. Let 0 = z 0 < z 1 < · · · < z k−1 < z k = 1 be those points. Then, using the fundamental theorem of calculus on each interval separately, there existsz i ∈ (z i, z i−1) for each i such that Proof. The j th output of a one-hidden-layer neural network can be written as DISPLAYFORM7 DISPLAYFORM8 where σ(z) = max(z, 0) is ReLU activation function, W and V are the weight matrices of the first and second layer respectively, and w r is the r th row of W. Thus, we can compute g(x) and ∇g(x) q below:. The red dash line encloses the ball B 2 (x 0, R 1) and the blue dash line encloses a larger ball B 2 (x 0, R 2). If we draw samples uniformly within the balls, the probability of ∇g(x) 2 = y is proportional to the intersected volumes of the ball and the regions with ∇g(x) 2 = y. DISPLAYFORM9 As illustrated in FIG7, the hyperplanes w r x + b r = 0, r ∈ {1, . . ., U} divide the d dimensional spaces R d into different regions, with the interior of each region satisfying a different set of inequality constraints, e.g. w r+ x + b r+ > 0 and w r− x + b r− < 0. Given x, we can identify which region it belongs to by checking the sign of w r x + b r for each r. Notice that the gradient norm is the same for all the points in the same region, i.e. for any x 1, x 2 satisfying I(w r x 1 + b r) = I(w r x 2 + b r) ∀r, we have ∇g(x 1) q = ∇g(x 2) q. Since there can be at most M = d i=0 U i different regions for a d-dimensional space with U hyperplanes, ∇g(x) q can take at most M different values. Therefore, if we perform uniform sampling in a ball B p (x 0, R) centered at x 0 with radius R and denote ∇g(x) q as a random variable Y, the probability distribution of Y is discrete and its CDF is piece-wise constant with at most M pieces. Without loss of generality, assume there are M 0 ≤ M distinct values for Y and denote them as m, m,..., m (M0) in an increasing order, the CDF of Y, denoted as F Y (y), is the following: E.1 PERCENTAGE OF EXAMPLES HAVING P VALUE > 0.05 TAB5 shows the percentage of examples where the null hypothesis cannot be rejected by K-S test, indicating that the maximum gradient norm samples fit reverse Weibull distribution well. Figure 3. FIG10 shows the 2 CLEVER score with different number of samples (N b = 50, 100, 250, 500) for MNIST and CIFAR models. For most models except MNIST-BReLU, reducing the number of samples only change CLEVER scores very slightly. For MNIST-BReLU, increasing the number of samples improves the estimated lower bound, suggesting that a larger number of samples is preferred. In practice, we can start with a relatively small N b = a, and also try 2a, 4a, · · · samples to see if CLEVER scores change significantly. If CLEVER scores stay roughly the same despite increasing N b, we can conclude that using N b = a is sufficient. DISPLAYFORM10

We propose the first attack-independent robustness metric, a.k.a CLEVER, that can be applied to any neural network classifier.

Represent the short summary for finding the corresponding scientific article (long).

We revisit the simple idea of pruning connections of DNNs through $\ell_1$ regularization achieving state-of-the-art results on multiple datasets with theoretic guarantees.