Question: Represent the scientific article to find a short TLDR

Question: Abstract In this work, we describe a set of rules for the design and initialization of well-conditioned neural networks, guided by the goal of naturally balancing the diagonal blocks of the Hessian at the start of training. We show how our measure of conditioning of a block relates to another natural measure of conditioning, the ratio of weight gradients to the weights. We prove that for a ReLU-based deep multilayer perceptron, a simple initialization scheme using the geometric mean of the fan-in and fan-out satisfies our scaling rule. For more sophisticated architectures, we show how our scaling principle can be used to guide design choices to produce well-conditioned neural networks, reducing guess-work. The design of neural networks is often considered a black-art, driven by trial and error rather than foundational principles. This is exemplified by the success of recent architecture random-search techniques , which take the extreme of applying no human guidance at all. Although as a field we are far from fully understanding the nature of learning and generalization in neural networks, this does not mean that we should proceed blindly. In this work we define a scaling quantity γ l for each layer l that approximates the average squared singular value of the corresponding diagonal block of the Hessian for layer l. This quantity is easy to compute from the (non-central) second moments of the forward-propagated values and the (non-central) second moments of the backward-propagated gradients. We argue that networks that have constant γ l are better conditioned than those that do not, and we analyze how common layer types affect this quantity. We call networks that obey this rule preconditioned neural networks, in analogy to preconditioning of linear systems. As an example of some of the possible applications of our theory, we: • Propose a principled weight initialization scheme that can often provide an improvement over existing schemes; • Show which common layer types automatically in well-conditioned networks; • Show how to improve the conditioning of common structures such as bottlenecked residual blocks by the addition of fixed scaling constants to the network (Detailed in Appendix E). We will use the multilayer perceptron (i.e. a classical feed-forward deep neural network) as a running example as it is the simplest non-trivial deep neural network structure. We use ReLU activation functions, and use the following notation for layer l of L (following): x l+1 = ReLU (y l), where W l is a n out l × n in l matrix of weights, b l is the bias vector, y l the preactivation vector and x l is the input activation vector for the layer. The quantities n fan-in of the layer respectively. We also denote the gradient of a quantity with respect to the loss (i.e. the back-propagated gradient) with the prefix ∆. We initially focus on the least-squares loss. Additionally, we assume that each bias vector is initialized with zeros unless otherwise stated. Our proposed approach focuses on the singular values of the diagonal blocks of the Hessian. In the case of a multilayer perceptron network (MLP) network, each diagonal block corresponds to the weights from a single weight matrix W l or bias vector b l. This block structure is used by existing approaches such as K-FAC and variants (; ; ;), which correct the gradient step using estimates of secondorder information. In contrast, our approach modifies the network to improve the Hessian without modifying the step. Estimates of the magnitude of the singular values σ i (G l) of the diagonal blocks G 1,...,G L of the Hessian G provide information about the singular values of the full matrix. Proposition 1. Let G l be the lth diagonal block of a real symmetric matrix G: n × n. Then for all i = 1... n: We can use this simple bound to provide some insight into the conditioning of the full matrix: Corollary 2. Let S = {s 1, . . .,} be the union of the sets of singular values of the diagonal blocks G 1,..., G l of a real symmetric matrix G: n × n. Then the condition number κ(G) = σ max (G)/σ min (G) is bounded as: In particular, a Hessian matrix with a very large difference between the singular values of each block must be ill-conditioned. This provides strong motivation for balancing the magnitude of the singular values of each diagonal block, the goal of this work. Although ideally, we would like to state the converse, that a matrix with balanced blocks is well conditioned, we can not make such a statement without strong assumptions on the off-diagonal behavior of the matrix. We use the average squared singular value of each block as a proxy for the full spectrum, as it is particularly easy to estimate in expectation. Although the minimum and maximum for each block would seem like a more natural quantity to work with, we found that any such bounds tend to be too pessimistic to reflect the behavior of the actual singular values. When using the ReLU activation function, as we consider in this work, a neural network is no longer a smooth function of its inputs, and the Hessian becomes ill-defined at some points in the parameter space. Fortunately, the spectrum is still well-defined at any twice-differentiable point, and this gives a local measure of the curvature. ReLU networks are typically twice-differentiable almost everywhere. We assume this throughout the remainder of this work. Our analysis will proceed with batch-size 1 and a network with k outputs. We consider the network at initialization, where weights are centered, symmetric and i.i.d random variables and biases are set to zero. ReLU networks have a particularly simple structure for the Hessian with respect to any set of activations, as the network's output is a piecewise-linear function g fed into a final layer consisting of a loss. This structure in greatly simplified expressions for diagonal blocks of the Hessian with respect to the weights. We will consider the output of the network as a composition two functions, the current layer g, and the remainder of the network h. We write this as a function of the weights, i.e. f (W l) = h(g(W l)). The dependence on the input to the network is implicit in this notation, and the network below layer l does not need to be considered. h(y l) be the Hessian of h, the remainder of the network after application of layer l (recall y l = W l x l). Let J l be the Jacobian of y l with respect to W l. The Jacobian has shape Given these quantities, the diagonal block of the Hessian corresponding to W l is equal to: The lth diagonal block of the (Generalized) Gauss-Newton matrix G . We discuss this decomposition further in Appendix A.1. We use the notation E[X 2] for any matrix or vector X to denote the expectation of the element-wise non-central second moment. Proposition 3. (The GR scaling) Under the assumptions outlined in Section 3.2, the average squared singular value of G l is equal to the following quantity, which we call the GR scaling for MLP layers:. We define a "balanced" or "preconditioned" network as one in which γ l is equal for all l (full derivation in Appendix A). Balancing this theoretically derived GR scaling quantity in a network will produce an initial optimization problem for which the blocks of the Hessian are expected to be approximately balanced with respect to their average squared singular value. Due to the large number of approximations needed for this derivation, which we discuss further in the next section, we don't claim that this theoretical approximation is accurate, or that the blocks will be closely matched in practice. Rather, we make the lesser claim that a network with very disproportionate values of γ l between layers is likely to have convergence difficulties during the early stages of optimization due to Cor. 2. To check the quality of our approximation, we computed the ratio of the convolutional version of the GR scaling equation (Equation 1) to the actual E[(G l r) 2 ] product for a strided (rather than max-pooled, see Table 1) LeNet model, where we use random input data and a random loss (i.e. for outputs y we use y T Ry for an i.i.d normal matrix R), with batch-size 1024, and 32 × 32 input images. The are shown in Figure 2 for 100 sampled setups; there is generally good agreement with the theoretical expectation. The following strong assumptions are used in the derivation of the GR scaling: (A1) The input and target values are drawn element-wise i.i.d from a centered symmetric distribution with known variance. (A2) The Hessian of the remainder of the network above each block, with respect to the output, has Frobenius norm much larger than 1. More concretely, we assume that all but the highest order terms that are polynomial in this norm may be dropped. (A3) All activations, pre-activations and gradients are independently distributed element-wise. In practice due to the mixing effect of multiplication by random weight matrices, only the magnitudes of these quantities are correlated, and the effect is small for wide networks due to the law of large numbers. Independence assumptions of this kind are common when approximating second-order methods; the block-diagonal variant of K-FAC makes similar assumptions for instance. Assumption A2 is the most problematic of these assumptions, and we make no claim that it holds in practice. However, we are primarily interested in the properties of blocks and their scaling with respect to each other, not their absolute scaling. Assumption A2 in very simple expressions for the scaling of the blocks without requiring a more complicated analysis of the top of the network. Similar theory can be derived for other assumptions on the output structure, such as the assumption that the target values are much smaller than the outputs of the network. We provide further motivation for the utility of preconditioning by comparing it to another simple quantity of interest. Consider at network initialization, the ratio of the (element-wise non-central) second moments of each weight-matrix gradient to the weight matrix itself: This ratio approximately captures the relative change that a single SGD step with unit step-size on W l will produce. We call this quantity the weight-to-gradient ratio. A network with constant ν l is also well-behaved under weight-decay, as the ratio of weight-decay second moments to gradient second moments will stay constant throughout the network, keeping the push-pull of gradients and decay constant across the network. Remarkably, the weight-to-gradient ratio ν l turns out to be equivalent to the GR scaling for MLP networks: Proposition 4. (Appendix 8) ν l is equal to the GR scaling γ l for i.i.d mean-zero randomly-initialized multilayer perceptron layers under the independence assumptions of Appendix 3.2. The concept of GR scaling may be extended to scaled convolutional layers y l = α l Conv W l (x l) + b l with scaling factor α l, kernel width k l, batch-size b, and output resolution ρ l × ρ l. A straight-forward derivation gives expressions for the convolution weight and biases of: This requires an assumption of independence of the values of activations within a channel that is not true in practice, so γ l tends to be further away from empirical estimates for convolutional layers than for non-convolutional layers, although it is still a useful guide. The effect of padding is also ignored here. Sequences of convolutions are well-scaled against each other along as the kernel size remains the same. The scaling of layers involving differing kernel sizes can be corrected using the alpha parameter (Appendix E), and more generally any imbalance between the conditioning of layers can be fixed by modifying α l while at the same time changing the initialization of W l so that the forward variance remains the same as the unmodified version. This adjusts γ l while leaving all other γ the same. For ReLU networks with a classical multilayer-perceptron (i.e. non-convolutional, non-residual) structure, we show in this section that initialization using i.i.d mean-zero random variables with (non-central) second moment inversely proportional to the geometric mean of the fans: for some fixed constant c, in a constant GR scaling throughout the network. Proposition 5. Let W 0: m × n and W 1: p × m be weight matrices satisfying the geometric initialization criteria of Equation 2, and let b 0, b 1 be zero-initialized bias parameters. Then consider the following sequence of two layers where x 0 and ∆y 1 are i.i.d, mean 0, uncorrelated and symmetrically distributed: Proof. Note that the ReLU operation halves both the forward and backward (non-central) second moments, due to our assumptions on the distributions of x 0 and ∆y 1. So: Consider the first weight-gradient ratio, using E[∆W Under our assumptions, back-propagation to ∆x 1 in E[∆x Now consider the second weight-gradient ratio: Under our assumptions, applying forward propagation gives E[y 0], and so from Equation 3 we have: which matches Equation 4, so ν 0 = ν 1. Remark 6. This relation also holds for sequences of (potentially) strided convolutions, but only if the same kernel size is used everywhere and circular padding is used. The initialization should be modified to include the kernel size, changing the expression to c/ k Under review as a conference paper at ICLR 2020 The most common approaches are the Kaiming and Xavier initializations. The Kaiming technique for ReLU networks is actually one of two approaches: For the feed-forward network above, assuming random activations, the forward-activation variance will remain constant in expectation throughout the network if fan-in initialization of weights is used, whereas the fan-out variant maintains a constant variance of the back-propagated signal. The constant factor 2 in the above expressions corrects for the variance-reducing effect of the ReLU activation. Although popularized by , similar scaling was in use in early neural network models that used tanh activation functions . These two principles are clearly in conflict; unless n in l = n out l, either the forward variance or backward variance will become non-constant, or as more commonly expressed, either explode or vanish. No prima facie reason for preferring one initialization over the other is provided. Unfortunately, there is some confusion in the literature as many works reference using Kaiming initialization without specifying if the fan-in or fan-out variant is used. The Xavier initialization is the closest to our proposed approach. They balance these conflicting objectives using the arithmetic mean: to "... approximately satisfy our objectives of maintaining activation variances and back-propagated gradients variance as one moves up or down the network". This approach to balancing is essentially heuristic, in contrast to the geometric mean approach that our theory directly guides us to. We can use the same proof technique to compute the GR scaling for the bias parameters in a network. Our update equations change to include the bias term: y l = W l x l + b l, with b l assumed to be initialized at zero. We show in Appendix D that:. It is easy to show using the techniques of Section 6 that the biases of consecutive layers have equal GR scaling as long as geometric initialization is used. However, unlike in the case of weights, we have less flexibility in the choice of the numerator. Instead of allowing all weights to be scaled by c for any positive c, we require that c = 2, so that: Proposition 7 It is traditional to normalize a dataset before applying a neural network so that the input vector has mean 0 and variance 1 in expectation. This principle is rarely quested in modern neural networks, even though there is no longer a good justification for its use in modern ReLU based networks. In contrast, our theory provides direct guidance for the choice of input scaling. We show that the (non-central) second moment of the input affects the GR scaling of bias and weight parameters differently and that they can be balanced by careful choice of the initialization. Consider the GR scaling values for the bias and weight parameters in the first layer of a ReLU-based multilayer perceptron network, as considered in previous sections. We assume the data is already. We can cancel terms to find the value of E x 2 0 that makes these two quantities equal: In common computer vision architectures, the input planes are the 3 color channels, and the kernel size is k = 3, giving E x 2 0 ≈ 0.2. Using the traditional variance-one normalization will in the effective learning rate for the bias terms being lower than that of the weight terms. This will in potentially slower learning of the bias terms than for the input scaling we propose. A neural network's behavior is also very sensitive to the (non-central) second moment of the outputs. For a convolutional network without pooling layers (but potentially with strided dimensionality reduction), if geometric-mean initialization is used the activation (non-central) second moments are given by: The application of a sequence of these layers gives a telescoping product: We potentially have independent control over this (non-central) second moment at initialization, as we can insert a fixed scalar multiplication factor at the end of the network that modifies it. This may be necessary when adapting a network architecture that was designed and tested under a different initialization scheme, as the success of the architecture may be partially due to the output scaling that happens to be produced by that original initialization. We are not aware of any existing theory guiding the choice of output variance at initialization for the case of log-softmax losses, where it has a non-trivial effect on the back-propagated signals, although output variances of 0.01 to 0.1 appear to work well. The output variance should always be checked and potentially corrected when switching initialization schemes. Consider a network where γ l is constant throughout. We may add a layer between any two existing layers without affecting this conditioning, as long as the new layer maintains the activation-gradient (non-central) second-moment product: and dimensionality; this follows from Equation 1. For instance, adding a simple scaling layer of the form x l+1 = 2x l doubles the (non-central) second moment during the forward pass and doubles the backward (non-central) second moment during back-propagation, which maintains this product: When spatial dimensionality changes between layers we can see that the GR scaling is no longer maintained just by balancing this product, as γ depends directly on the square of the spatial dimension. Instead, a pooling operation that changes the forward and backward signals in a way that counteracts the change in spatial dimension is needed. The use of stride-2 convolutions, as well as average pooling, in the correct scaling, but other common types of spatial reduction generally do not. It is particularly interesting to note that the evolution in state-of-the-art architectures corresponds closely to a move from poorly scaled building blocks to well-scaled ones. Early shallow architectures like LeNet-5 used tanh nonlinearities, which were replaced by the (well-scaled) ReLU, used for instance in the seminal AlexNet architecture . AlexNet and the latter VGG architectures made heavy use of max-pooling and reshaping before the final layers, both operations which have been replaced in modern fully-convolutional architectures with (well-scaled) striding and average-pooling respectively. The use of large kernel sizes is also in decline. The AlexNet architecture used kernel sizes of 11, 5 and 3, whereas modern ResNet architectures only use 7, 3 and 1. Furthermore, recent research has shown that replacing the single 7x7 convolution used with a sequence of three 3x3 convolutions improves performance . We considered a selection of dense and moderate-sparsity multi-class classification datasets from the LibSVM repository, 26 in total. The same model was used for all datasets, a non-convolutional ReLU network with 3 weight layers total. The inner two layer widths were fixed at 384 and 64 nodes respectively. These numbers were chosen to in a larger gap between the optimization methods, less difference could be expected if a more typical 2× gap was used. Our are otherwise generally robust to the choice of layer widths. For every dataset, learning rate and initialization combination we ran 10 seeds and picked the median loss after 5 epochs as the focus of our study (The largest differences can be expected early in training). Learning rates in the range 2 1 to 2 −12 (in powers of 2) were checked for each dataset and initialization combination, with the best learning rate chosen in each case based off of the median of the 10 seeds. Training loss was used as the basis of our comparison as we care primarily about convergence rate, and are comparing identical network architectures. Some additional details concerning the experimental setup and which datasets were used is available in the Appendix. Table 1 shows that geometric initialization is the most consistent of the initialization approaches considered. It has the lowest loss, after normalizing each dataset, and it is never the worst of the 4 methods on any dataset. Interestingly, the fan-out method is most often the best method, but consideration of the per-dataset plots (Appendix F) shows that it often completely fails to learn for some problems, which pulls up its average loss and in it being the worst for 9 datasets. Testing an initialization method on modern computer vision problems is problematic due to the heavy architecture search, both automated and manual, that is behind the current best methods. This search will fit the architecture to the initialization method, in a sense, so any other initialization is at a disadvantage compared to the one used during architecture search. This is further complicated by the prevalence of BatchNorm which is not handled in our theory. Instead, to provide a clear comparison we use an older network with a large variability in kernel sizes, the AlexNet architecture. This architecture has a large variety of filter sizes (11, 5, 3, linear), which according to our theory will affect the conditioning adversely, and which should highlight the differences between the methods. We found that a network with consistent kernel sizes through-out showed only negligible differences between the initialization methods. The network was modified to replace max-pooling with striding as max-pooling is not well-scaled by our theory (further details in Appendix F). Following Section 6.4, we normalize the output of the network at initialization by running a single batch through the network and adding a fixed scaling factor to the network to produce output standard deviation 0.05. For our preconditioned variant, we added alpha correction factors following Section 5 in conjunction with geometric initialization, and compared this against other common initialization methods. We tested on CIFAR-10 following the standard practice as closely as possible, as detailed in Appendix F. We performed a geometric learning rate sweep over a power-of-two grid. Results are shown in Figure 3 for an average of 10 seeds for each initialization. Preconditioning improves training loss over all other initialization schemes tested, although only by a small margin. Although not a panacea, by using the scaling principle we have introduced, neural networks can be designed with a reasonable expectation that they will be optimizable by stochastic gradient methods, minimizing the amount of guess-and-check neural network design. As a consequence of our scaling principle, we have derived an initialization scheme that automatically preconditions common network architectures. Most developments in neural network theory attempt to explain the success of existing techniques post-hoc. Instead, we show the power of the scaling law approach by deriving a new initialization technique from theory directly. we then combine the simplifications from Equations 9, 10, and 11 to give:. Standard ReLU classification and regression networks have a particularly simple structure for the Hessian with respect to the input, as the network's output is a piecewise-linear function g feed into a final layer consisting of a convex log-softmax operation, or a least-squares loss. This structure in the Hessian with respect to the input being equivalent to its Gauss-Newton approximation. The Gauss-Newton matrix can be written in a factored form, which is used in the analysis we perform in this work. We emphasize that this is just used as a convenience when working with diagonal blocks, the GN representation is not an approximation in this case. The (Generalized) Gauss-Newton matrix G is a positive semi-definite approximation of the Hessian of a non-convex function f, given by factoring f into the composition of two functions where h is convex, and g is approximated by its Jacobian matrix J at x, for the purpose of computing G: The GN matrix also has close ties to the Fisher information matrix , providing another justification for its use. Surprisingly, the Gauss-Newton decomposition can be used to compute diagonal blocks of the Hessian with respect to the weights W l as well as the inputs . To see this, note that for any activation y l, the layers above may be treated in a combined fashion as the h in a f (W l) = h(g(W l)) decomposition of the network structure, as they are the composition of a (locally) linear function and a convex function and thus convex. In this decomposition g(W l) =W l x l + b l is a function of W l with x l fixed, and as this is linear in W l, the Gauss-Newton approximation to the block is thus not an approximation. We make heavy use of the equations for forward propagation and backward propagation of second moments, under the assumption that the weights are uncorrelated to the activations or gradients. For a convolution with input channels n in, output channels n out, and square k × k kernels, these formulas are (recall our notation for the second moments is element-wise for vectors and matrices): C THE WEIGHT GRADIENT RATIO IS EQUAL TO GR SCALING FOR MLP MODELS Proposition 8. The weight-gradient ratio ν l is equal to the GR scaling γ l for i.i.d mean-zero randomly-initialized multilayer perceptron layers under the independence assumptions of Appendix 3.2. Proof. To see the equivalence, note that under the zero-bias initialization, we have from and so: The gradient of the weights is given by ∆W ij = ∆y li x lj and so its second moment is: Combining these quantities gives:. We consider the case of a convolutional neural network with spatial resolution ρ × ρ for greater generality. Consider the Jacobian of y l with respect to the bias. It has shape J Each row corresponds to a y l output, and each column a bias weight. As before, we will approximate the product of G with a random i.i.d unit variance vector r: The structure of J b l is that each block of ρ 2 rows has the same set of 1s in the same column. Only a single 1 per row. It follows that: The calculation of the product of R l with J b l r is approximated in the same way as in the weight scaling calculation. For the J bT product, note that there is an additional ρ 2 as each column has ρ 2 non-zero entries, each equal to 1. Combining these three quantities gives:. Proposition 9. Consider the setup of Proposition 5, with the addition of biases: As long as the weights are initialized following Equation 7 and the biases are initialized to 0, we have that We will include c = 2 as a variable as it clarifies it's relation to other quantities. We reuse some calculations from Proposition 5. Namely that: Plugging these into the definition of γ b 0:. b 1, we require the additional quantity: Again plugging this in:. There has been significant recent interest in training residual networks without the use of batchnormalization or other normalization layers . In this section, we explore the modifications that are necessary to a network for this to be possible and show how to apply our preconditioning principle to these networks. The building block of a ResNet model is the residual block: where F is a composition of layers. Unlike classical feedforward architectures, the pass-through connection in an exponential increase in the variance of the activations in the network as the depth increases. A side effect of this is the output of the network becomes exponentially more sensitive to the input of the network as depth increases, a property characterized by the Lipschitz constant of the network . This exponential dependence can be reduced by the introduction of scaling constants s l to each block: The introduction of these constants requires a modification of the block structure to ensure constant conditioning between blocks. A standard bottleneck block, as used in the ResNet-50 architecture, has the following form: In this notation, C 0 is a 1 × 1 convolution that reduces the number of channels 4 fold, C 1 is a 3 × 3 convolution with equal input and output channels, and C 2 is a 1 × 1 convolution at increases the number of channels back up 4 fold to the original input count. If we introduce a scaling factor s l to each block l, then we must also add conditioning multipliers β l to each convolution to change their GR scaling, as we described in Section 5. The correct scaling constant depends on the scaling constant of the previous block. A simple calculation gives the equation: The initial β 0 and s 0 may be chosen arbitrarily. If a flat s l = s is used for all l, then we may use β l = 1. The block structure including the β l factors is: x 1 = ReLU(y 0), x 2 = ReLU(y 1), The weights of each convolution must then be initialized with the standard deviation modified such that the combined convolution-scaling operation gives the same output variance as would be given if the geometric-mean initialization scheme is used without extra scaling constants. For instance, the initialization of the C 0 convolution must have standard deviation scaled down by dividing by 1 β so that the multiplication by 1 β during the forward pass in the correct forward variance. The 1/ √ 3 factor is an α correction that corrects for change in kernel shape for the middle convolution. The variance at initialization must be scaled to correct for the α factor also. Since the initial convolution in a ResNet-50 model is also not within a residual block, it's GR scaling is different from the convolutions within residual blocks. Consider the composition of a non-residual followed by a residual block, without max-pooling or ReLUs for simplicity of exposition: Without loss of generality, we assume that E x 2 0 = 1, and assume a single channel input and output. Our goal is to find a constant α, so that γ 0 = γ 1. Recall that when using α scaling factors we must initialize C 0 so that the variance of y 0 is independent of the choice of α. Our scaling factor will also depend on the kernel sizes used in the two convolutions, so we must include those in the calculations. From Equation 1, the GR scaling for C 0 is Note that E[∆y For the residual convolution, we need to use a modification of the standard GR equation due to the residual branch. The derivation of γ for non-residual convolutions assumes that the remainder of the network above the convolution responds linearly (locally) with the scaling of the convolution, but here due to the residual connection, this is no longer the case. For instance, if the weight were scaled to zero, the output of the network would not also become zero (recall our assumption of zero-initialization for bias terms). This can be avoided by noting that the ratio E[∆y] in the GR scaling may be computed further up the network, as long as any scaling in between is corrected for. In particular, we may compute this ratio at the point after the residual addition, as long as we include the factor s 4 1 to account for this. So we in fact have: We now equate γ 0 = γ 1: Therefore to ensure that γ 0 = γ 1 we need:. A similar calculation applies when the residual block is before the non-residual convolution, as in the last layer linear in the ResNet network, giving a scaling factor for the linear layer (effective kernel size 1) of: To prevent the from being skewed by the number of classes and the number of inputs affecting the output variance, the logit output of the network was scaled to have standard deviation 0.05 after the first minibatch evaluation for every method, with the scaling constant fixed thereafter. LayerNorm was used on the input to whiten the data. Weight decay of 0.00001 was used for every dataset. To aggregate the losses across datasets we divided by the worst loss across the initializations before averaging. LIBSVM PLOTS Figure 4 shows the interquartile range (25%, 50% and 75% quantiles) of the best learning rate for each case. Following standard practice, training used random augmentations consisting of horizontal flips and random crops to 32x32, as well as normalization to the interval [-1,+1]. We used SGD with momentum 0.9, a learning rate schedule of decreases at 150 and 225 epochs, and no weight decay. The network architecture is the following sequence, with circular "equal" padding used and ReLU nonlinearities after each convolution: 1. 11x11 stride-1 convolution with 3 input and 64 output channels, 2. 5x5 stride-2 convolution with 64 input and 192 output channels, 3. 3x3 stride-2 convolution with 192 input and 384 output channels,

Represent the short summary for finding the corresponding scientific article (long):

A theory for initialization and scaling of ReLU neural network layers

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.


Question: Represent the scientific article to find a short TLDR

Question: We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT , we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction—membership classification and API watermarking—which while successful against some adversaries can also be circumvented by more clever ones. Machine learning models represent valuable intellectual property: the process of gathering training data, iterating over model design, and tuning hyperparameters costs considerable money and effort. As such, these models are often only indirectly accessible through web APIs that allow users to query a model but not inspect its parameters. Malicious users might try to sidestep the expensive model development cycle by instead locally reproducing an existing model served by such an API. In these attacks, known as "model stealing" or "model extraction" (; Tramèr et al., 2016), the adversary issues a large number of queries and uses the collected (input, output) pairs to train a local copy of the model. Besides theft of intellectual property, extracted models may leak sensitive information about the training data (Tramèr et al., 2016) or be used to generate adversarial examples that evade the model served by the API . With the recent success of contextualized pretrained representations for transfer learning, NLP APIs based on ELMo and BERT have become increasingly popular . Contextualized pretrained representations boost performance and reduce sample complexity , and they typically require only a shallow task-specific network-sometimes just a single layer as in BERT. While these properties are advantageous for representation learning, we hypothesize that they also make model extraction easier. In this paper, we demonstrate that NLP models obtained by fine-tuning a pretrained BERT model can be extracted even if the adversary does not have access to any training data used by the API provider. In fact, the adversary does not even need to issue well-formed queries: our experiments show that extraction attacks are possible even with queries consisting of randomly sampled sequences of words coupled with simple task-specific heuristics (Section 3). This contrasts with prior work, which for large-scale attacks requires at minimum that the adversary can access a small amount of semantically-coherent data relevant to the task (; ; a; ;). Extraction performance improves further by using randomly-sampled sentences and paragraphs from Wikipedia (instead of random word sequences) as queries (Section 4). These attacks are cheap; our most expensive attack cost around $500, estimated using rates of current API providers. Step 1: Attacker randomly samples words to form queries and sends them to victim BERT model Step 2: Attacker fine-tunes their own BERT on these queries using the victim outputs as labels Figure 1: Overview of our model extraction setup for question answering. 1 An attacker first queries a victim BERT model, and then uses its predicted answers to fine-tune their own BERT model. This process works even when passages and questions are random sequences of words as shown here. We perform a fine-grained analysis of the randomly-generated queries to shed light on why they work so well for model extraction. Human studies on the random queries show that despite their effectiveness in extracting good models, they are mostly nonsensical and uninterpretable, although queries closer to the original data distribution seem to work better for extraction (Section 5.1). Furthermore, we discover that pretraining on the attacker's side makes model extraction easier (Section 5.2). Finally, we study the efficacy of two simple defenses against extraction -membership classification (Section 6.1) and API watermarking (Section 6.2) -and find that while they work well against naïve adversaries, they fail against more clever ones. We hope that our work spurs future research into stronger defenses against model extraction and, more generally, on developing a better understanding of why these models and datasets are particularly vulnerable to such attacks. We relate our work to prior efforts on model extraction, most of which have focused on computer vision applications. Because of the way in which we synthesize queries for extracting models, our work also directly relates to zero-shot distillation and studies of rubbish inputs to NLP systems. Model extraction attacks have been studied both empirically (Tramèr et al., 2016; a;) and theoretically , mostly against image classification APIs. These works generally synthesize queries in an active learning setup by searching for inputs that lie close to the victim classifier's decision boundaries. This method does not transfer to text-based systems due to the discrete nature of the input space. 2 The only prior work attempting extraction on NLP systems is , who adopt pool-based active learning to select natural sentences from WikiText-2 and extract 1-layer CNNs for tasks expecting single inputs. In contrast, we study a more realistic extraction setting with nonsensical inputs on modern BERT-large models for tasks expecting pairwise inputs like question answering. Our work is related to prior work on data-efficient distillation, which attempts to distill knowledge from a larger model to a small model with access to limited input data or in a zeroshot setting . However, unlike the model extraction setting, these methods assume white-box access to the teacher model to generate data impressions. Rubbish inputs, which are randomly-generated examples that yield high-confidence predictions, have received some attention in the model extraction literature. Prior work (Tramèr et al., 2016) reports successful extraction on SVMs and 1-layer networks using i.i.d noise, but no prior work has scaled this idea to deeper neural networks for which a single class tends to dominate model predictions on most noise inputs . Unnatural text inputs have previously been shown to produce overly confident model predictions , break translation systems , and trigger disturbing outputs from text generators . In contrast, here we show their effectiveness at training models that work well on real NLP tasks despite not seeing any real examples during training. What is BERT? We study model extraction on BERT, Bidirectional Encoder Representations from Transformers . BERT-large is a 24-layer transformer , f bert,θ, which converts a word sequence x = (x 1, ..., x n) of length n into a high-quality sequence of vector representations v = (v 1, ..., v n). These representations are contextualized -every vector v i is conditioned on the whole sequence x. BERT's parameters θ * are learnt using masked language modelling on a large unlabelled corpus of natural text. The public release of f bert,θ * revolutionized NLP, as it achieved state-of-the-art performance on a wide variety of NLP tasks with minimal task-specific supervision. A modern NLP system for task T typically leverages the fine-tuning methodology in the public BERT repository: 3 a task-specific network f T,φ (generally, a 1-layer feedforward network) with parameters φ expecting v as input is used to construct a composite function g T = f T,φ • f bert,θ. The final parameters φ T, θ T are learned end-to-end using training data for T with a small learning rate ("fine-tuning"), with φ initialized randomly and θ initialized with θ *. Description of extraction attacks: Assume g T (the "victim model") is a commercially available black-box API for task T. A malicious user with black-box query access to g T attempts to reconstruct a local copy g T (the "extracted model"). Since the attacker does not have training data for T, they use a task-specific query generator to construct several possibly nonsensical word sequences {x i} m 1 as queries to the victim model. The ing dataset {x i, g T (x i)} m 1 is used to train g T. Specifically, we assume that the attacker fine-tunes the public release of f bert,θ * on this dataset to obtain g T. 4 A schematic of our extraction attacks is shown in Figure 1. We extract models on four diverse NLP tasks that have different kinds of input and output spaces: binary sentiment classification using SST2 , where the input is a single sentence and the output is a probability distribution between positive and negative; ternary natural language inference (NLI) classification using MNLI , where the input is a pair of sentences and the output is a distribution between entailment, contradiction and neutral; extractive question answering (QA) using SQuAD 1.1 , where the input is a paragraph and question and the output is an answer span from the paragraph; and boolean question answering using BoolQ , where the input is a paragraph and question and the output is a distribution between yes and no. Query generators: We study two kinds of query generators, RANDOM and WIKI. In the RANDOM generator, an input query is a nonsensical sequence of words constructed by sampling 5 a Wikipedia vocabulary built from WikiText-103 . In the WIKI setting, input queries are formed from actual sentences or paragraphs from the WikiText-103 corpus. We found these generators insufficient by themselves to extract models for tasks featuring complex interactions between different parts of the input space (e.g., between premise and hypothesis in MNLI or question and paragraph in SQuAD). Hence, we additionally apply the following task-specific heuristics: • MNLI: since the premise and hypothesis often share many words, we randomly replace three words in the premise with three random words to construct the hypothesis. • SQuAD / BoolQ: since questions often contain words in the associated passage, we uniformly sample words from the passage to form a question. We additionally prepend a question starter word (like "what") to the question and append a? symbol to the end. For more details on the query generation, see Appendix A.3. Representative example queries and their outputs are shown in Table 1. More examples are provided in Appendix A.5. First, we evaluate our extraction procedure in a controlled setting where an attacker uses an identical number of queries as the original training dataset (Table 2); afterwards, we investigate different query budgets for each task (Table 3). We provide commercial cost estimates for these query budgets using the Google Cloud Platform's Natural Language API calculator. 6 We use two metrics for eval- Table 3: Development set accuracy of various extracted models on the original development set at different query budgets expressed as fractions of the original dataset size. Note the high accuracies for some tasks even at low query budgets, and diminishing accuracy gains at higher budgets. uation: Accuracy of the extracted models on the original development set, and Agreement between the outputs of the extracted model and the victim model on the original development set inputs. In our controlled setting (Table 2), our extracted models are surprisingly accurate on the original development sets of all tasks, even when trained with nonsensical inputs (RANDOM) that do not match the original data distribution. 7 Accuracy improves further on WIKI: extracted SQuAD models recover 95% of original accuracy despite seeing only nonsensical questions during training. While extracted models have high accuracy, their agreement is only slightly better than accuracy in most cases. Agreement is even lower on held-out sets constructed using the WIKI and RANDOM sampling scheme. On SQuAD, extracted WIKI and RANDOM have low agreements of 59.2 F1 and 50.5 F1 despite being trained on identically distributed data. This indicates poor functional equivalence between the victim and extracted model as also found by. An ablation study with alternative query generation heuristics for SQuAD and MNLI is conducted in Appendix A.4. Classification with argmax labels only: For classification datasets, we assumed the API returns a probability distribution over output classes. This information may not be available to the adversary in practice. To measure what happens when the API only provides argmax outputs, we re-run our WIKI experiments for SST2, MNLI and BoolQ with argmax labels and present our in Table 2 (WIKI-ARGMAX). We notice a minimal drop in accuracy from the corresponding WIKI experiments, indicating that access to the output probability distribution is not crucial for model extraction. Hence, hiding the full probability distribution is not a viable defense strategy. Query efficiency: We measure the effectiveness of our extraction algorithms with varying query budgets, each a different fraction of the original dataset size, in Table 3. Even with small query budgets, extraction is often successful; while more queries is usually better, accuracy gains quickly diminish. Approximate costs for these attacks can be extrapolated from Table 2. These bring many natural questions to mind. What properties of nonsensical input queries make them so amenable to the model extraction process? How well does extraction work for these tasks without using large pretrained language models? In this section, we perform an analysis to answer these questions. Previously, we observed that nonsensical input queries are surprisingly effective for extracting NLP models based on BERT. Here, we dig into the properties of these queries in an attempt to understand why models trained on them perform so well. Do different victim models produce the same answer when given a nonsensical query? Are some of these queries more representative of the original data distribution than others? Did our task-specific heuristics perhaps make these nonsensical queries "interpretable" to humans in some way? We specifically examine the RANDOM and WIKI extraction configurations for SQuAD in this section to answer these questions. Do different victim models agree on the answers to nonsensical queries? We train five victim SQuAD models on the original training data with identical hyperparameters, varying only the random seed; each achieves an F1 of between 90 and 90.5. Then, we measure the average pairwise F1 ("agreement") between the answers produced by these models for different types of queries. As expected, the models agree very frequently when queries come from the SQuAD training set (96.9 F1) or development set (90.4 F1). However, their agreement drops significantly on WIKI queries (53.0 F1) and even further on RANDOM queries (41.2 F1). Are high-agreement queries closer to the original data distribution? While these indicate that on average, victim models tend to be brittle on nonsensical inputs, it is possible that highagreement queries are more useful than others for model extraction. To measure this, we sort queries from our 10x RANDOM and WIKI datasets according to their agreement and choose the highest and lowest agreement subsets, where subset size is a varying fraction of the original training data size (Figure 2). We observe large F1 improvements when extracting models using high-agreement subsets, consistently beating random and low-agreement subsets of identical sizes. This shows that agreement between victim models is a good proxy for the quality of an input-output pair for extraction. Measuring this agreement in extracted models and integrating this observation into an active learning objective for better extraction is an interesting direction that we leave to future work. Are high-agreement nonsensical queries interpretable to humans? Prior work has shown deep neural networks can leverage non-robust, uninterpretable features to learn classifiers. Our nonsensical queries are not completely random, as we do apply task-specific heuristics. Perhaps as a of these heuristics, do high-agreement nonsensical textual inputs have a human interpretation? To investigate, we asked three human annotators 9 to answer twenty SQuAD questions from each of the WIKI and RANDOM subsets that had unanimous agreement among victim models, and twenty original SQuAD questions as a control. On the WIKI subset, annotators matched the victim models' answer exactly 23% of the time (33 F1). Similarly, a 22% exact match (32 F1) was observed on RANDOM. In contrast, annotators scored significantly higher on original SQuAD questions (77% exact match, 85 F1 against original answers). Interviews with the annotators revealed a common trend: annotators used a word overlap heuristic (between the question and paragraph) to select entities as answer spans. While this heuristic partially interprets the extraction data's signal, most of the nonsensical question-answer pairs remain mysterious to humans. More details on inter-annotator agreement are provided in Appendix A.6. So far we assumed that the victim and the attacker both fine-tune a pretrained BERT-large model. However, in practical scenarios, the attacker might not have information about the victim architecture. What happens when the attacker fine-tunes a different base model than the victim? What if the attacker extracts a QA model from scratch instead of fine-tuning a large pretrained language model? Here, we examine how much the extraction accuracy depends on the pretraining setup. BERT comes in two different sizes: the 24 layer BERT-large and the 12 layer BERT-base. In Table 4, we measure the development set accuracy on MNLI and SQuAD when the victim and attacker use different configurations of these two models. Accuracy is always higher when the attacker starts from BERT-large, even when the victim was initialized with BERT-base. Additionally, given a fixed attacker architecture, accuracy is better when the victim uses the same model (e.g., if the attacker starts from BERT-base, they will have better if the victim also used BERT-base). This is reminiscent of similar discussion in Tramèr et al. What if we train from scratch? Finetuning BERT seems to give attackers a significant headstart, as only the final layer of the model is randomly initialized and the BERT parameters start from a good representation of language. To measure the importance of fine-tuning from a good starting point, we train a QANet model on SQuAD with no contextualized pretraining. This model has 1.3 million randomly initialized parameters at the start of training. Table 5 shows that QANet achieves high accuracy when original SQuAD inputs are used (TRUE X) with BERT-large labels, indicating sufficient model capacity. However, the F1 significantly degrades when training on nonsensical RANDOM and WIKI queries. The F1 drop is particularly striking when compared to the corresponding rows in Table 2 (only 4.5 F1 drop for WIKI). This reinforces our finding that better pretraining allows models to start from a good representation of language, thus simplifying extraction. Having established that BERT-based models are vulnerable to model extraction, we now shift our focus to investigating defense strategies. An ideal defense preserves API utility (b) while remaining undetectable to attackers ; furthermore, it is convenient if the defense does not require re-training the victim model. Here we explore two defenses that satisfy these properties but are also only effective against a class of weak adversaries. Our first defense uses membership inference, which determines whether a classifier was trained on a particular input point , to identify nonsensical out-ofdistribution inputs or adversarial examples that are unlikely to be issued by a legitimate user. When such inputs are detected, the API issues a random output instead of the model's predicted output, which eliminates the extraction signal. We treat membership inference as a binary classification problem, constructing datasets for MNLI and SQuAD by labeling their original training and validation examples as real and WIKI extraction examples as fake. We use the logits in addition to the final layer representations of the victim model as input features to train the classifier, as model confidence scores and rare word representations are useful for membership inference . Table 6 shows that these classifiers transfer well to a balanced development set with the same distribution as their training data (WIKI). They are also robust to the query generation process: accuracy remains high on auxiliary test sets where fake examples are either RANDOM (described in Section 3) or SHUFFLE, in which the word order of real examples is shuffled. An ablation study on the input features of the classifier is provided in Appendix A.7. Another defense against extraction is watermarking , in which a tiny fraction of queries are chosen at random and modified to return a wrong output. These "watermarked queries" and their outputs are stored on the API side. Since deep neural networks have the ability to memorize arbitrary information (;, this defense anticipates that extracted models will memorize some of the watermarked queries, leaving them vulnerable to post-hoc detection if they are deployed publicly. We evaluate watermarking on MNLI (by randomly permuting the predicted probability vector to ensure a different argmax output) and SQuAD (by returning a Table 8 : Results on watermarked models. Dev Acc represents the overall development set accuracy, WM Label Acc denotes the accuracy of predicting the watermarked output on the watermarked queries and Victim Label Acc denotes the accuracy of predicting the original labels on the watermarked queries. A watermarked WIKI has high WM Label Acc and low Victim Label Acc. single word answer which has less than 0.2 F1 overlap with the actual output). For both tasks, we watermark just 0.1% of all queries to minimize the overall drop in API performance. Table 8 shows that extracted models perform nearly identically on the development set (Dev Acc) with or without watermarking. When looking at the watermarked subset of the training data, however, non-watermarked models get nearly everything wrong (low WM Label Acc%) as they generally predict the victim model's outputs (high Victim Label Acc%), while watermarked models behave oppositely. Training with more epochs only makes these differences more drastic. Limitations: Watermarking works, but it can only be used after an attack has been carried out. Importantly, it assumes that an attacker will deploy an extracted model publicly with black-box query access and is thus irrelevant if the attacker instead keeps the model private. Furthermore, an attacker who anticipates watermarking might take steps to prevent detection, including differentially private training on extraction data ; fine-tuning or re-extracting an extracted model with different queries ; or issuing random outputs on queries exactly matching inputs in the extraction data. We study model extraction attacks against NLP APIs that serve BERT-based models. These attacks are surprisingly effective at extracting good models with low query budgets, even when an attacker uses nonsensical input queries. Our show that fine-tuning large pretrained language models simplifies the process of extraction for an attacker. Unfortunately, existing defenses against extraction, while effective in some scenarios, are generally inadequate, and further research is necessary to develop defenses robust in the face of adaptive adversaries who develop counter-attacks anticipating simple defenses. Other interesting future directions that follow from the in this paper include 1) leveraging nonsensical inputs to improve model distillation on tasks for which it is difficult to procure input data; 2) diagnosing dataset complexity by using query efficiency as a proxy; 3) further investigation of the agreement between victim models as a method to identify proximity in input distribution and its incorporation into an active learning setup for model extraction. We provide a distribution of agreement between victim SQuAD models on RANDOM and WIKI queries in Figure 3. In this paper, we have used the cost estimate from Google Cloud Platform's Calculator. 10 The Natural Language APIs typically allows inputs of length up to 1000 characters per query (https: //cloud.google.com/natural-language/pricing). To calculate costs for different datasets, we counted input instances with more than 1000 characters multiple times. Since Google Cloud did not have APIs for all tasks we study in this paper, we extrapolated the costs of the entity analysis and sentiment analysis APIs for natural language inference (MNLI) and reading comprehension (SQuAD, BoolQ). We believe this is a reasonable estimate since every model studied in this paper is a single layer in addition to BERT-large (thereby needing a similar number of FLOPs for similar input lengths). It is hard to provide a widely applicable estimate for the price of issuing a certain number of queries. Several API providers allow a small budget of free queries. An attacker could conceivably set up multiple accounts and collect extraction data in a distributed fashion. In addition, most APIs are implicitly used on webpages -they are freely available to web users (such as Google Search or Maps). If sufficient precautions are not taken, an attacker could easily emulate the HTTP requests used to call these APIs and extract information at a large scale, free of cost ("web scraping"). Besides these factors, API costs could also vary significantly depending on the computing infrastructure involved or the revenue model of the company deploying them. Given these caveats, it is important to focus on the relatively low costs needed to extract datasets rather than the actual cost estimates. Even complex text generation tasks like machine translation and speech recognition (for which Google Cloud has actual API estimates) are relatively inexpensive. It costs -$430.56 to extract Switchboard LDC97S62 , a large conversational speech recognition dataset with 300 hours of speech; $2000.00 to issue 1 million translation queries, each having a length of 100 characters. In this section we provide more details on the input generation algorithms adopted for each dataset. (SST2, RANDOM) -A vocabulary is built using wikitext103. The top 10000 tokens (in terms of unigram frequency in wikitext103) are preserved while the others are discarded. A length is chosen from the pool of wikitext-103 sentence lengths. Tokens are uniformly randomly sampled from the top-10000 wikitext103 vocabulary up to the chosen length. (SST2, WIKI) -A vocabulary is built using wikitext103. The top 10000 tokens (in terms of unigram frequency in wikitext103) are preserved while the others are discarded. A sentence is chosen at random from wikitext103. Words in the sentence which do not belong to the top-10000 wikitext103 vocabulary are replaced with words uniformly randomly chosen from this vocabulary. (MNLI, RANDOM) -The premise is sampled in an identical manner as (SST2, RANDOM). To construct the final hypothesis, the following process is repeated three times -i) choose a word uniformly at random from the premise ii) replace this word with another word uniformly randomly sampled from the top-10000 wikitext103 vocabulary. (MNLI, WIKI) -The premise is sampled in a manner identical to (SST2, WIKI). The hypothesis is sampled in a manner identical (MNLI, RANDOM). (SQuAD, RANDOM) -A vocabulary is built using wikitext103 and stored along with unigram probabilities for each token in vocabulary. A length is chosen from the pool of paragraph lengths in wikitext103. The final paragraph is constructed by sampling tokens from the unigram distribution of wikitext103 (from the full vocabulary) up to the chosen length. Next, a random integer length is chosen from the range. Paragraph tokens are uniformly randomly sampled to up to the chosen length to build the question. Once sampled, the question is appended with a? symbol and prepended with a question starter word chosen uniformly randomly from the list [A, According, After, Along, At, By, During, For, From, How, In, On, The, To, What, What's, When, Where, Which, Who, Whose, Why]. (SQuAD, WIKI) -A paragraph is chosen at random from wikitext103. Questions are sampled in a manner identical to (SQuAD, RANDOM). (BoolQ, RANDOM) -identical to (SQuAD, RANDOM). We avoid appending questions with? since they were absent in BoolQ. Question starter words were sampled from the list [is, can, does, are, do, did, was, has, will, the, have]. (BoolQ, WIKI) -identical to (SQuAD, WIKI). We avoid appending questions with? since they were absent in BoolQ. The question starter word list is identical to (BoolQ, RANDOM). In this section we study some additional query generation heuristics. In Table 11, we compare numerous extraction datasets we tried for SQuAD 1.1. Our general findings are -i) RANDOM works much better when the paragraphs are sampled from a distribution reflecting the unigram frequency in wikitext103 compared to uniform random sampling ii) starting questions with common question starter words like "what" helps, especially with RANDOM schemes. We present a similar ablation study on MNLI in Table 12. Our general findings parallel recent work studying MNLI -i) when the lexical overlap between the premise and hypothesis is too low (when they are independently sampled), the model almost always predicts neutral or contradiction, limiting the extraction signal from the dataset; ii) when the lexical overlap is too high (hypothesis is shuffled version of premise), the model generally predicts entailment leading to an unbalanced extraction dataset; iii) when the premise and hypothesis have a few different words (edit-distance 3 or 4), datasets tend to be balanced and have strong extraction signal; iv) using frequent words (top 10000 wikitext103 words) tends to aid extraction. More examples have been provided in Table 13. For our human studies, we asked fifteen human annotators to annotate five sets of twenty questions. Annotators were English-speaking graduate students who voluntarily agreed to participate and were completely unfamiliar with our research goals. Three annotators were used per question set. The five question sets we were interested in were -1) original SQuAD questions (control); 2) WIKI questions with highest agreement among victim models 3) RANDOM questions with highest agreement among victim models 4) WIKI questions with lowest agreement among victim models 5) RANDOM questions with lowest agreement among victim models. In Table 10 we show the inter-annotator agreement. Notice that average pairwise F1 (a measure of inter-annotator agreement) follows the order original SQuAD >> WIKI, highest agreement > RANDOM, highest agreement ∼ WIKI, lowest agreement > RANDOM, lowest agreement. We hypothesize that this ordering roughly reflects the closeness to the actual input distribution, since a similar ordering is also observed in Figure 2. Individual annotation scores have been shown below. In this section we run an ablation study on the input features for the membership classifier. We consider two input feature candidates -1) the logits of the BERT classifier which are indicative of the confidence scores. 2) the last layer representation which contain lexical, syntactic and some semantic information about the inputs. We present our in Table 9. Our ablation study indicates that the last layer representations are more effective than the logits in distinguishing between real and fake inputs. However, the best in most cases are obtained by using both feature sets. Table 9: Ablation study of the membership classifiers. We measure accuracy on an identically distributed development set (WIKI) and differently distributed test sets (RANDOM, SHUFFLE). Note the last layer representations tend to be more effective in classifying points as real or fake.

Represent the TLDR for finding the corresponding scientific article

Outputs of modern NLP APIs on nonsensical text provide strong signals about model internals, allowing adversaries to steal the APIs.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.