Question: Represent the text to find a very short summary (1-2 sentences max):

Question: In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM, which uses DropConnect on hidden-to-hidden weights, as a form of recurrent regularization. Further, we introduce NT-ASGD, a non-monotonically triggered (NT) variant of the averaged stochastic gradient method (ASGD), wherein the averaging trigger is determined using a NT condition as opposed to being tuned by the user. Using these and other regularization strategies, our ASGD Weight-Dropped LSTM (AWD-LSTM) achieves state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2. We also explore the viability of the proposed regularization and optimization strategies in the context of the quasi-recurrent neural network (QRNN) and demonstrate comparable performance to the AWD-LSTM counterpart. The code for reproducing the is open sourced and is available at https://github.com/salesforce/awd-lstm-lm. Effective regularization techniques for deep learning have been the subject of much research in recent years. Given the over-parameterization of neural networks, generalization performance crucially relies on the ability to regularize the models sufficiently. Strategies such as dropout BID33 and batch normalization BID13 have found great success and are now ubiquitous in feed-forward and convolutional neural networks. Naïvely applying these approaches to the case of recurrent neural networks (RNNs) has not been highly successful however. Many recent works have hence been focused on the extension of these regularization strategies to RNNs; we briefly discuss some of them below. A naïve application of dropout BID33 to an RNN's hidden state is ineffective as it disrupts the RNN's ability to retain long term dependencies BID40. BID7 propose overcoming this problem by retaining the same dropout mask across multiple time steps as opposed to sampling a new binary mask at each timestep. Another approach is to regularize the network through limiting updates to the RNN's hidden state. One such approach is taken by BID31 wherein the authors drop updates to network units, specifically the input gates of the LSTM, in lieu of the units themselves. This is reminiscent of zoneout BID20 where updates to the hidden state may fail to occur for randomly selected neurons. Instead of operating on the RNN's hidden states, one can regularize the network through restrictions on the recurrent matrices as well. This can be done either through restricting the capacity of the matrix BID0 BID39 BID14 or through element-wise interactions (; BID32 .Other forms of regularization explicitly act upon activations such as batch normalization BID13, recurrent batch normalization BID4, and layer normalization BID1 . These all introduce additional training parameters and can complicate the training process while increasing the sensitivity of the model. In this work, we investigate a set of regularization strategies that are not only highly effective but which can also be used with no modification to existing LSTM implementations. The weightdropped LSTM applies recurrent regularization through a DropConnect mask on the hidden-tohidden recurrent weights. Other strategies include the use of randomized-length backpropagation through time (BPTT), embedding dropout, activation regularization (AR), and temporal activation regularization (TAR).As no modifications are required of the LSTM implementation these regularization strategies are compatible with black box libraries, such as NVIDIA cuDNN, which can be many times faster than naïve LSTM implementations. Effective methods for training deep recurrent networks have also been a topic of renewed interest. Once a model has been defined, the training algorithm used is required to not only find a good minimizer of the loss function but also converge to such a minimizer rapidly. The choice of the optimizer is even more important in the context of regularized models since such strategies, especially the use of dropout, can impede the training process. Stochastic gradient descent (SGD), and its variants such as Adam BID18 and RMSprop BID36 are amongst the most popular training methods. These methods iteratively reduce the training loss through scaled (stochastic) gradient steps. In particular, Adam has been found to be widely applicable despite requiring less tuning of its hyperparameters. In the context of word-level language modeling, past work has empirically found that SGD outperforms other methods in not only the final loss but also in the rate of convergence. This is in agreement with recent evidence pointing to the insufficiency of adaptive gradient methods BID38.Given the success of SGD, especially within the language modeling domain, we investigate the use of averaged SGD (AvSGD) BID29 which is known to have superior theoretical guarantees. AvSGD carries out iterations similar to SGD, but instead of returning the last iterate as the solution, returns an average of the iterates past a certain, tuned, threshold T. This threshold T is typically tuned and has a direct impact on the performance of the method. We propose a variant of AvSGD where T is determined on the fly through a non-monotonic criterion and show that it achieves better training outcomes compared to SGD. We refer to the mathematical formulation of the LSTM, DISPLAYFORM0 where DISPLAYFORM1 are weight matrices, x t is the vector input to the timestep t, h t is the current exposed hidden state, c t is the memory cell state, and is element-wise multiplication. Preventing overfitting within the recurrent connections of an RNN has been an area of extensive research in language modeling. The majority of previous recurrent regularization techniques have acted on the hidden state vector h t−1, most frequently introducing a dropout operation between timesteps, or performing dropout on the update to the memory state c t. These modifications to a standard LSTM prevent the use of black box RNN implementations that may be many times faster due to low-level hardware-specific optimizations. We propose the use of DropConnect BID37 on the recurrent hidden to hidden weight matrices which does not require any modifications to an RNN's formulation. As the dropout operation is applied once to the weight matrices, before the forward and backward pass, the impact on training speed is minimal and any standard RNN implementation can be used, including inflexible but highly optimized black box LSTM implementations such as NVIDIA's cuDNN LSTM.By performing DropConnect on the hidden-to-hidden weight matrices DISPLAYFORM2 within the LSTM, we can prevent overfitting from occurring on the recurrent connections of the LSTM. This regularization technique would also be applicable to preventing overfitting on the recurrent weight matrices of other RNN cells. As the same weights are reused over multiple timesteps, the same individual dropped weights remain dropped for the entirety of the forward and backward pass. The is similar to variational dropout, which applies the same dropout mask to recurrent connections within the LSTM by performing dropout on h t−1, except that the dropout is applied to the recurrent weights. DropConnect could also be used on the non-recurrent weights of the LSTM [W i, W f, W o] though our focus was on preventing overfitting on the recurrent connection. SGD is among the most popular methods for training deep learning models across various modalities including computer vision, natural language processing, and reinforcement learning. The training of deep networks can be posed as a non-convex empirical risk minimization problem DISPLAYFORM0 where f i is the loss function for the i th data point, w are the weights of the network, and the expectation is taken over the data. In this context, given a sequence of learning rates, γ k, SGD iteratively takes steps of the form DISPLAYFORM1 where the subscript denotes the iteration number and the∇ denotes a stochastic gradient that may be computed on a minibatch of data points. SGD demonstrably performs well in practice and also possesses several attractive theoretical properties such as linear convergence BID2, saddle point avoidance BID28 and better generalization performance BID10. For the specific task of neural language modeling, traditionally SGD without momentum has been found to outperform other algorithms such as momentum SGD BID34, Adam BID18, Adagrad BID5 and RMSProp BID36 ) by a statistically significant margin. Motivated by this observation, we investigate averaged SGD (AvSGD) to further improve the training process. AvSGD has been analyzed in depth theoretically and many surprising have been shown including its asymptotic second-order convergence BID29 BID22. AvSGD takes steps identical to equation FORMULA4 but instead of returning the last iterate as the solution, returns DISPLAYFORM2 where K is the total number of iterations and T < K is a user-specified averaging trigger. Despite its theoretical appeal, AvSGD has found limited practical use in training of deep networks. This may be in part due to unclear tuning guidelines for the learning-rate schedule γ k and averaging trigger T. If the averaging is triggered too soon, the efficacy of the method is impacted, and if it is triggered too late, many additional iterations may be needed to converge to the solution. In this section, we describe a non-monotonically triggered variant of AvSGD (NT-AvSGD), which obviates the need for tuning T. Further, the algorithm uses a constant learning rate throughout the experiment and hence no further tuning is necessary for the decay scheduling. Ideally, averaging needs to be triggered when the SGD iterates converge to a steady-state distribution BID22. This is roughly equivalent to the convergence of SGD to a neighborhood around a solution. In the case of SGD, certain learning-rate reduction strategies such as the stepwise strategy analogously reduce the learning rate by a fixed quantity at such a point. A common strategy employed in language modeling is to reduce the learning rates by a fixed proportion when the performance of the model's primary metric (such as perplexity) worsens or stagnates. Along the same lines, one could make a triggering decision based on the performance of the model on the Algorithm 1 Non-monotonically Triggered AvSGD (NT-AvSGD) Inputs: Initial point w 0, learning rate γ, logging interval L, non-monotone interval n. DISPLAYFORM3 while stopping criterion not met do 3:Compute stochastic gradient∇f (w k) and take SGD step. if mod(k, L) = 0 and T = 0 then 5:Compute validation perplexity v. if t > n and v > min l∈{0,···,t−n−1} DISPLAYFORM0 Append v to logs 10: DISPLAYFORM1 end if 12: DISPLAYFORM2 validation set. However, instead of averaging immediately after the validation metric worsens, we propose a non-monotonic criterion that conservatively triggers the averaging when the validation metric fails to improve for multiple cycles; see Algorithm 1. Given that the choice of triggering is irreversible, this conservatism ensures that the randomness of training does not play a major role in the decision. Analogous strategies have also been proposed for learning-rate reduction in SGD BID16.While the algorithm introduces two additional hyperparameters, the logging interval L and nonmonotone interval n, we found that setting L to be the number of iterations in an epoch and n = 5 worked well across various models and data sets. As such, we use this setting in all of our NTAvSGD experiments in the following section and demonstrate that it achieves better training outcomes as compared to SGD. In addition to the regularization and optimization techniques above, we explored additional regularization techniques that aimed to improve data efficiency during training and to prevent overfitting of the RNN model. Given a fixed sequence length that is used to break a data set into fixed length batches, the data set is not efficiently used. To illustrate this, imagine being given 100 elements to perform backpropagation through with a fixed backpropagation through time (BPTT) window of 10. Any element divisible by 10 will never have any elements to backprop into, no matter how many times you may traverse the data set. Indeed, the backpropagation window that each element receives is equal to i mod 10 where i is the element's index. This is data inefficient, preventing 1 10 of the data set from ever being able to improve itself in a recurrent fashion, and ing in 8 10 of the remaining elements receiving only a partial backpropagation window compared to the full possible backpropagation window of length 10.To prevent such inefficient data usage, we randomly select the sequence length for the forward and backward pass in two steps. First, we select the base sequence length to be seq with probability p and seq 2 with probability 1 − p, where p is a high value approaching 1. This spreads the starting point for the BPTT window beyond the base sequence length. We then select the sequence length according to N (seq, s), where seq is the base sequence length and s is the standard deviation. This jitters the starting point such that it doesn't always fall on a specific word divisible by seq or seq 2. From these, the sequence length more efficiently uses the data set, ensuring that when given enough epochs all the elements in the data set experience a full BPTT window, while ensuring the average sequence length remains around the base sequence length for computational efficiency. During training, we rescale the learning rate depending on the length of the ing sequence compared to the original specified sequence length. The rescaling step is necessary as sampling arbitrary sequence lengths with a fixed learning rate favors short sequences over longer ones. This linear scaling rule has been noted as important for training large scale minibatch SGD without loss of accuracy BID8 and is a component of unbiased truncated backpropagation through time BID35. In standard dropout, a new binary dropout mask is sampled each and every time the dropout function is called. New dropout masks are sampled even if the given connection is repeated, such as the input x 0 to an LSTM at timestep t = 0 receiving a different dropout mask than the input x 1 fed to the same LSTM at t = 1. A variant of this, variational dropout BID7, samples a binary dropout mask only once upon the first call and then to repeatedly use that locked dropout mask for all repeated connections within the forward and backward pass. While we propose using DropConnect rather than variational dropout to regularize the hidden-tohidden transition within an RNN, we use variational dropout for all other dropout operations, specifically using the same dropout mask for all inputs and outputs of the LSTM within a given forward and backward pass. Each example within the minibatch uses a unique dropout mask, rather than a single dropout mask being used over all examples, ensuring diversity in the elements dropped out. Following BID7, we employ embedding dropout. This is equivalent to performing dropout on the embedding matrix at a word level, where the dropout is broadcast across all the word vector's embedding. The remaining non-dropped-out word embeddings are scaled by 1 1−pe where p e is the probability of embedding dropout. As the dropout occurs on the embedding matrix that is used for a full forward and backward pass, this means that all occurrences of a specific word will disappear within that pass, equivalent to performing variational dropout on the connection between the one-hot embedding and the embedding lookup. Weight tying BID12 BID30 shares the weights between the embedding and softmax layer, substantially reducing the total parameter count in the model. The technique has theoretical motivation BID12 ) and prevents the model from having to learn a one-to-one correspondence between the input and output, ing in substantial improvements to the standard LSTM language model. In most natural language processing tasks, both pre-trained and trained word vectors are of relatively low dimensionality-frequently between 100 and 400 dimensions in size. Most previous LSTM language models tie the dimensionality of the word vectors to the dimensionality of the LSTM's hidden state. Even if reducing the word embedding size was not beneficial in preventing overfitting, the easiest reduction in total parameters for a language model is reducing the word vector size. To achieve this, the first and last LSTM layers are modified such that their input and output dimensionality respectively are equal to the reduced embedding size. L 2 -regularization is often used on the weights of the network to control the norm of the ing model and reduce overfitting. In addition, L 2 decay can be used on the individual unit activations and on the difference in outputs of an RNN at different time steps; these strategies labeled as activation regularization (AR) and temporal activation regularization (TAR) respectively BID25. AR penalizes activations that are significantly larger than 0 as a means of regularizing the network. Concretely, AR is defined as DISPLAYFORM0 where m is the dropout mask, L 2 (·) = · 2, h t is the output of the RNN at timestep t, and α is a scaling coefficient. TAR falls under the broad category of slowness regularizers BID11 BID6 BID21 BID15 which penalize the model from producing large changes in the hidden state. Using the notation from AR, TAR is defined as DISPLAYFORM1 where β is a scaling coefficient. As in BID25, the AR and TAR loss are only applied to the output of the final RNN layer as opposed to being applied to all layers. For evaluating the impact of these approaches, we perform language modeling over a preprocessed version of the Penn Treebank (PTB) BID27 ) and the WikiText-2 (WT2) data set. The Penn Treebank data set has long been a central data set for experimenting with language modeling. The data set is heavily preprocessed and does not contain capital letters, numbers, or punctuation. The vocabulary is also capped at 10,000 unique words, quite small in comparison to most modern datasets, which in a large number of out of vocabulary (OoV) tokens. WikiText-2 is sourced from curated Wikipedia articles and is approximately twice the size of the PTB data set. The text is tokenized and processed using the Moses tokenizer BID19, frequently used for machine translation, and features a vocabulary of over 30,000 words. Capitalization, punctuation, and numbers are retained in this data set. All experiments use a three-layer LSTM model with 1150 units in the hidden layer and an embedding of size 400. The loss was averaged over all examples and timesteps. All embedding weights were uniformly initialized in the interval [−0.1, 0.1] and all other weights were initialized between DISPLAYFORM0, where H is the hidden size. For training the models, we use the NT-AvSGD algorithm discussed in the previous section for 750 epochs with L equivalent to one epoch and n = 5. We use a batch size of 80 for WT2 and 40 for PTB. Empirically, we found relatively large batch sizes (e.g., 40-80) performed better than smaller sizes (e.g., 10-20) for NT-AvSGD. After completion, we run AvSGD with T = 0 and hot-started w 0 as a fine-tuning step to further improve the solution. For this fine-tuning step, we terminate the run using the same non-monotonic criterion detailed in Algorithm 1.We carry out gradient clipping with maximum norm 0.25 and use an initial learning rate of 30 for all experiments. We use a random BPTT length which is N with probability 0.95 and N with probability 0.05. The values used for dropout on the word vectors, the output between LSTM layers, the output of the final LSTM layer, and embedding dropout where (0.4, 0.3, 0.4, 0.1) respectively. For the weight-dropped LSTM, a dropout of 0.5 was applied to the recurrent weight matrices. For WT2, we increase the input dropout to 0.65 to account for the increased vocabulary size. For all experiments, we use AR and TAR values of 2 and 1 respectively, and tie the embedding and softmax weights. These hyperparameters were chosen through trial and error and we expect further improvements may be possible if a fine-grained hyperparameter search were to be conducted. In the , we abbreviate our approach as AWD-LSTM for AvSGD Weight-Dropped LSTM. We present the single-model perplexity for both our models (AWD-LSTM) and other competitive models in Table 1 and 2 for PTB and WT2 respectively 1. On both data sets we improve the state-of-the-art, with our vanilla LSTM model beating the state of the art by approximately 1 unit on PTB and 0.1 units on WT2.In comparison to other recent state-of-the-art models, our model uses a vanilla LSTM. BID41 propose the recurrent highway network, which extends the LSTM to allow multiple hidden state updates per timestep. BID42 use a reinforcement learning agent to generate an RNN cell tailored to the specific task of language modeling, with the cell far more complex than the LSTM.Independently of our work, BID23 apply extensive hyperparameter search to an LSTM based language modeling implementation, analyzing the sensitivity of RNN based language models to hyperparameters. Unlike our work, they use a modified LSTM, which caps the input gate i t to be min(1 − f t, i t), use Adam with β 1 = 0 rather than SGD or AvSGD, use skip connections between LSTM layers, and use a black box hyperparameter tuner for exploring models and settings. Of particular interest is that their hyperparameters were tuned individually for each data set compared to our work which shared almost all hyperparameters between PTB and WT2, including the embedding and hidden size for both data sets. Due to this, they used less model parameters than our model and found shallow LSTMs of one or two layers worked best for WT2.Like our work, BID23 find that the underlying LSTM architecture can be highly effective compared to complex custom architectures when well tuned hyperparameters are used. The approaches used in our work and BID23 may be complementary and would be worth exploration. In past work, pointer based attention models have been shown to be highly effective in improving language modeling BID9. Given such substantial improvements to the underlying neural language model, it remained an open question as to how effective pointer augmentation may be, especially when improvements such as weight tying may act in mutually exclusive ways. The neural cache model BID9 can be added on top of a pre-trained language model at negligible cost. The neural cache stores the previous hidden states in memory cells and then uses a simple convex combination of the probability distributions suggested by the cache and the language model for prediction. The cache model has three hyperparameters: the memory size (window) for the cache, the coefficient of the combination (which determines how the two distributions are mixed), and the flatness of the cache distribution. All of these are tuned on the validation set once a trained language model has been obtained and require no training by themselves, making it quite inexpensive to use. The tuned values for these hyperparameters were (2000, 0.1, 1.0) for PTB and (3785, 0.1279, 0.662) for WT2 respectively. In TAB1, we show that the model further improves the perplexity of the language model by as much as 6 perplexity points for PTB and 11 points for WT2. While this is smaller than the gains reported in BID9, which used an LSTM without weight tying, this is still a substantial drop. Given the simplicity of the neural cache model, and the lack of any trained components, these suggest that existing neural language models remain fundamentally lacking, failing to capture long term dependencies or remember recently seen words effectively. To understand the impact the pointer had on the model, specifically the validation set perplexity, we detail the contribution that each word has on the cache model's overall perplexity in TAB3. We compute the sum of the total difference in the loss function value (i.e., log perplexity) between the LSTM-only and LSTM-with-cache models for the target words in the validation portion of the WikiText-2 data set. We present for the sum of the difference as opposed to the mean since the latter undesirably overemphasizes infrequently occurring words for which the cache helps significantly and ignores frequently occurring words for which the cache provides modest improvements that cumulatively make a strong contribution. The largest cumulative gain is in improving the handling of <unk> tokens, though this is over 11540 instances. The second best improvement, approximately one fifth the gain given by the <unk> tokens, is for Meridian, yet this word only occurs 161 times. This indicates the cache still helps significantly even for relatively rare words, further demonstrated by Churchill, Blythe, or Sonic. The cache is not beneficial when handling frequent word categories, such as punctuation or stop words, for which the language model is likely well suited. These observations motivate the design of a cache framework that is more aware of the relative strengths of the two models. Several architectures for learning sequential data based on convolutions, instead of recurrences, have been proposed recently. We briefly mention experiments on the same language modeling us- ing quasi-recurrent neural networks (QRNNs) instead of LSTMs; we label this setup the AWD-QRNN. As in the case of AWD-LSTM, we regularize the network through weight, embedding and variational dropouts along with variable sequence lengths, weight tying, AR and TAR. The networks were designed such that they had the same number of parameters as their LSTM counterparts and were trained using NT-AvSGD. Despite the same size of the network, QRNNs were 2 − 4× faster per epoch as compared to their LSTM counterparts and required fewer epochs to converge. We report the in TAB4. As is evident from the table, the QRNN model achieves comparable to the LSTM suggesting the generality of the proposed regularization techniques. Interestingly, the hyperparameter values for the various regularization components, including the optimization procedure, needed minimal changes from the LSTM to the QRNN models for competitive performance. For full details and hyperparameters, refer to the released code. In The first two variants deal with the optimization of the language models while the rest deal with the regularization. For the model using SGD with learning rate reduced by 2 using the same nonmonotonic fashion, there is a significant degradation in performance. This stands as empirical evidence regarding the benefit of averaging of the iterates. Using a monotonic criterion instead also hampered performance. Similarly, the removal of the fine-tuning step expectedly also degrades the performance. This step helps improve the estimate of the minimizer by resetting the memory of the previous experiment. While this process of fine-tuning can be repeated multiple times, we found little benefit in repeating it more than once. The removal of regularization strategies paints a similar picture; the inclusion of all of the proposed strategies was pivotal in ensuring state-of-the-art performance. The most extreme perplexity jump was in removing the hidden-to-hidden LSTM regularization provided by the weight-dropped LSTM. Without such hidden-to-hidden regularization, perplexity rises substantially, up to 11 points. This is in line with previous work showing the necessity of recurrent regularization in state-of-the-art models BID7 BID12.We also experiment with static sequence lengths which we had hypothesized would lead to inefficient data usage. This also worsens the performance by approximately one perplexity unit. Next, we experiment with reverting to matching the sizes of the embedding vectors and the hidden states. This significantly increases the number of parameters in the network (to 43M in the case of PTB and 70M for WT2) and leads to degradation by almost 8 perplexity points, which we attribute to overfitting in the word embeddings. While this could potentially be improved with more aggressive regularization, the computational overhead involved with substantially larger embeddings likely outweighs any advantages. Finally, we experiment with the removal of embedding dropout, AR/TAR and weight decay. In all of the cases, the model suffers a perplexity increase of 2-6 points which we hypothesize is due to insufficient regularization in the network. In this work, we discuss regularization and optimization strategies for neural language models. We propose the weight-dropped LSTM, a strategy that uses a DropConnect mask on the hidden-tohidden weight matrices, as a means to prevent overfitting across the recurrent connections. Further, we investigate the use of averaged SGD with a non-monontonic trigger for training language models and show that it outperforms SGD by a significant margin. We investigate other regularization strategies including the use of variable BPTT length and achieve a new state-of-the-art perplexity on the PTB and WikiText-2 data sets. Our models outperform custom-built RNN cells and complex regularization strategies that preclude the possibility of using optimized libraries such as the NVIDIA cuDNN LSTM. We explore the use of a neural cache in conjunction with our proposed model and show that this further improves the performance, thus attaining an even lower state-of-the-art perplexity. We also explore the viability of using the proposed regularization and optimization strategies in the context of a quasi-recurrent neural network (QRNN) and demonstrate comparable performance to the LSTM counterpart. While the regularization and optimization strategies proposed are demonstrated on the task of language modeling, we anticipate that they would be generally applicable across other sequence learning tasks.

represent the short summary for finding the corresponding scientific article (long)

Effective regularization and optimization strategies for LSTM-based language models achieves SOTA on PTB and WT2.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the article for finding a single-sentence TLDR

Question: Various methods of measuring unit selectivity have been developed with the aim of better understanding how neural networks work. But the different measures provide divergent estimates of selectivity, and this has led to different regarding the conditions in which selective object representations are learned and the functional relevance of these representations. In an attempt to better characterize object selectivity, we undertake a comparison of various selectivity measures on a large set of units in AlexNet, including localist selectivity, precision, class-conditional mean activity selectivity (CCMAS), network dissection, the human interpretation of activation maximization (AM) images, and standard signal-detection measures. We find that the different measures provide different estimates of object selectivity, with precision and CCMAS measures providing misleadingly high estimates. Indeed, the most selective units had a poor hit-rate or a high false-alarm rate (or both) in object classification, making them poor object detectors. We fail to find any units that are even remotely as selective as the'grandmother cell' units reported in recurrent neural networks. In order to generalize these , we compared selectivity measures on a few units in VGG-16 and GoogLeNet trained on the ImageNet or Places-365 datasets that have been described as'object detectors'. Again, we find poor hit-rates and high false-alarm rates for object classification. There have been recent attempts to understand how neural networks (NNs) work by analyzing hidden units one-at-a-time using various measures such as localist selectivity , class-conditional mean activity selectivity (CCMAS) , precision , network dissection (a), and activation maximization (AM) . These measures are all taken to provide evidence that some units respond highly selectively to categories of objects under some conditions. Not only are these findings surprising given the widespread assumption that NNs only learn highly distributed and entangled representations, they raise a host of questions, including the functional importance of these selective representations (b), the conditions in which they are learned (e.g.,), and the relation between these representations and the selective neurons observed in cortex . To answer these question, it is necessary to have a better understanding of what these metrics actually measure, and how they relate to one another. Accordingly, we directly compare these measures of selectivity on the same set of units as well as adopt standard signal-detection measures in an attempt to provide better measures of single-unit selectivity to object category. In addition, to provide a more intuitive assessment of selectivity, we report jitterplots for a few of the most selective units that visually display how the unit responds to the different image categories. We focus on AlexNet trained on ImageNet because many authors have studied the selectivity of single hidden units in this model using a range of quantitative (a; and qualitative (; ;) methods. But we also compare different selectivity measures on specific units in VGG-16 and GoogLeNet trained on the the ImageNet and Places-365 datasets that were characterized by Zhou et al. (2018a) as "object detectors" based on their Network Dissection method (a). Our main findings are: 1. The precision and CCMAS measures are misleading with near-maximum selectivity scores associated with units that strongly respond to many different image categories. By contrast, the signal-detection measures more closely capture the level of selectivity displayed in the jitterplots (Sec. 3.1). 2. Units with interpretable AM images do not correspond to highly selective representations (Sec. 3.2). 3. The Network Dissection method also provides a misleading measure for "object detectors" (Sec. 3.3). In one line of research, Bowers et al. (2014; assessed the selectivity of single hidden units in recurrent neural networks (RNNs) designed to model human short-term memory. They reported many'localist' or'grandmother cell' units that were 100% selective for specific letters or words, where all members of the selective category were more active than and disjoint from all non-members, as can be shown in jitterplots (see Fig. 1 for a unit selective to the letter 'j'). The authors argued that the network learned these representations in order to co-activate multiple letters or words at the same time in short-term memory without producing ambiguous blends of overlapping distributed patterns (the so-called 'superposition catastrophe'). Consistent with this hypothesis, localist units did not emerge when the model was trained on letters or words one-at-a-time (see Fig. 1 for an example of a non-selective unit). In parallel, researchers have reported selective units in the hidden layers of various CNNs trained to classify images into one of multiple categories (; ; ;), for a review see. For example, assessed the selectivity of units in the pool5 layer of two CNNs trained to classify images into 1000 objects and 205 scene categories, respectively. They reported many highly selective units that they characterized as'object detectors' in both networks. reported that CNNs trained on CIFAR-10 and ImageNet learned many highly selective hidden units, with CCMAS scores approaching the maximum of 1.0. These later findings appear to be inconsistent with who failed to observe selective representations in fully connected NNs trained on stimuli one-at-a-time (see Fig. 1), but the measures of selectivity that have been applied across studies are different, and accordingly, it is difficult to directly compare . A better understanding of the relation between selectivity measures is vital given that different measures are frequently used to address similar issues. For example, both the human interpretability of generated images and localist selectivity have been used to make claims about'grandmother cells', but it is not clear whether they provide similar insights into unit selectivity. Similarly, based on their precision metric, claim that the object detectors learned in CNNs play an important role in identifying specific objects, whereas challenge this based on their finding that units with high CCMAS measures were not especially important in the performance of their CNNs and concluded: "...it implies that methods for understanding neural networks based on analyzing highly selective single units, or finding optimal inputs for single units, such as activation maximization may be misleading". This makes a direct comparison between selectivity measures all the more important. In order to directly compare and have a better understanding of the different selectivity measures we assessed localist, precision, and CCMAS selectivity of the conv5, fc6, and fc7 of AlexNet trained on ImageNet, and in addition, we employed a range of signal detection methods on these units, namely, recall with 100% and 95% precision, maximum informedness, specificity at maximum informedness, and recall (also called sensitivity) at maximum informedness, and false alarm rates at maximum informedness (described in Sec. 2). We also assessed the selectivity of a few units in VGG-16 and GoogLeNet models trained on the ImageNet and Places-365 dataset that were highly selective according to the Network Dissection method (a). We show that the precision and CCMAS measures often provide misleadingly high estimates of object selectivity compared to other measures, and we do not find any units that can be reasonably described as'object detectors' given that the most selective units show a low hit-rate or a high false-alarm rate (or both) when classifying images. At best, the most selective units in CNNs are sensitive to some unknown feature that is weakly associated with the class in question. . Top middle: jitterplot of a non-selective unit 160 found in an RNN trained on words one-at-a-time from . Top right: Activation maximization image of unit conv5 9 AlexNet that resembles a lighthouse . Bottom: highest-activation images for a'lamp' detector with 84% precision in the layer conv5 of AlexNet; from . In addition to these quantitative measures and jitterplots we assessed selectivity with a common qualitative measure, namely, human interpretation of images generated by a state-of-the-art activation maximization (AM) method . AM images are generated to strongly activate individual units, and some of them are interpretable by humans (e.g., a generated image that looks like a lighthouse, see Fig. 1). For the first time, we systematically evaluated the interpretability of the AM images and compare these ratings with the selectivity measures for corresponding units. We show that the few hidden units with interpretable AM images are not highly selective. Network and Dataset All ∼1.3M photos from the ImageNet ILSVRC 2012 dataset were cropped to 277 × 277 pixels and classified by the pre-trained AlexNet CNN shipped with Caffe , ing in 721,536 correctly classified images. Once classified, the images are not re-cropped nor subject to any changes. We analyzed the fully connected (fc) layers: fc6 and fc7 (4096 units), and the top convolutional layer conv5 which has 256 filters. We only recorded the activations of correctly classified images. The activation files are stored in.h5 format and will be available at http://anonymizedForReview. We randomly selected 233 conv5, 2738 fc6, 2239 fc7 units for analysis. Localist selectivity , we define a unit to be localist for class A if the set of activations for class A was higher and disjoint with those of ¬A. Localist selectivity is easily depicted with jitterplots in which a scatter plot for each unit is generated (see Figs. 1 and 3). Each point in a plot corresponds to a unit's activation in response to a single image, and only correctly classified images are plotted. The level of activation is coded along the x-axis, and an arbitrary value is assigned to each point on the y-axis. Precision Precision refers to the proportion of items above some threshold from a given class. The precision method of finding object detectors involves identifying a small subset of images that most strongly activate a unit and then identifying the critical part of these images that are responsible for driving the unit. took the 60 images that activated a unit the most strongly and asked independent raters to interpret the critical image patches (e.g., if 50 of the 60 images were labeled as 'lamp', the unit would have a precision index of 50/60 or 83%; see Fig. 1). Object detectors were defined as units with a precision score > 75%: they reported multiple such detectors. Here, we approximate this approach by considering the 60 images that most strongly activate a given unit and assess the highest percentage of images from a given output class. introduced a selectivity index called the Class-conditional Mean Activation Selectivity (CCMAS). The CCMAS for class A compares the mean activation of all images in class A, µ A, with the mean activation of all images not in class A, µ ¬A, and is given by: Here, we assessed class selectivity for the highest mean activation class. We harnessed an activation maximization method called Plug & Play Generative Networks in which an image generator network was used to generate images (AM images) that highly activate a unit in a target network. We used the public code released by and their default hyperparameters. 1 We generated 100 separate images that maximally activated each unit in the conv5, fc6, and fc8 layers of AlexNet and asked participants to judge whether they could identify any repeating objects, animals, or places in images in a behavioral experiment (Sec. 3.2). Readers can test themselves at: https://research.sc/ participant/login/dynamic/63907FB2-3CB9-45A9-B4AC-EFFD4C4A95D5 Recall with perfect and 95% precision Recall with perfect and 95% precision are related to localist selectivity except that they provide a continuous rather than discrete measure. For recall with perfect precision we identified the image that activated a given unit the most and counted the number of images from the same class that were more active than all images from all other classes. We then divided this by the total number of correctly identified images from this class. A recall with a perfect precision score of 1 is equivalent to a localist representation. Recall with a 95% precision allows 5% false alarms. Maximum informedness Maximum informedness identifies the class and threshold where the highest proportion of images above the threshold and the lowest proportion of images below the threshold are from that class . The informedness is computed for each class at each threshold, with the highest value selected. Informedness summarises the diagnostic performance of unit for a given class at a certain threshold based on the recall [True Positives / (True Positives + False Negatives)] and specificity [True Negatives / (True Negatives + False Positives)] in the formula [informedness = recall + specificity − 1] . Sensitivity or Recall at Maximum Informedness For the threshold and class selected by Maximum Informedness, recall (or hit-rate) is the proportion of items from the given class that are above the threshold. Also known as true postive rate. Specificity at Maximum Informedness For the threshold and class selected by Maximum Informedness, the proportion of items that are not from the given class that are below the threshold. Also known as true negative rate. False Alarm Rate at Maximum Informedness For the threshold and class selected by Maximum Informedness, the proportion of items that are not from the given class that are above the threshold. Network Dissection To assess the selectivity of a unit in the Network Dissection technique, Zhou et al. (2018a) compute the Intersection over Union (IoU) of an annotated input image L c, for the set of all'concepts' c and a spatial activation map, M k, of where a unit k is. A unit k is taken as a detector for concept c if its IoU k,c exceeds a pre-defined threshold T. See Zhou et al. (2018a) for more details. The from the various of selectivity measures applied to the conv5, fc6, and fc7 layers of AlexNet are displayed in Fig. 2a -i. We did not plot the localist selectivity as there were no localist'grandmother units'. The first point to note is that multiple units in the fc6 and fc7 layers had near 100% precision scores and multiple units had CCMAS scores approaching 1. For example, in layer fc7, we found 14 units with a precision > 0.9, and 1487 units with a CCMAS > 0.9. The second point is that other measures provided much reduced estimates of selectivity. For example, the unit with the highest recall with a perfect precision score was only.08 (unit 255 responding to images of Monarch butterflies), and the unit with the top maximum informedness score (unit 3290 also responding to images of Monarch butterflies with a score of 0.91) had a false alarm rate above its optimal threshold > 99% (indeed the minimum false alarm rate was 0.96). To illustrate the contrasting measures of selectivity consider unit fc6 1199 depicted in Fig. 3 Figure 2: Different selectivity measures across the conv5, fc6, and fc7 layers of AlexNet. Red-line: median of data, top and bottom of box edges is the 25 th and 75 th percentile, whiskers extend to extreme edges of distribution not considered outliers and red crosses are outliers. Green points and dashed lines are the means of the distributions with standard errors. The high levels of selectivity observed with the precision and CCMAS measures are in stark contrast with the low levels of selectivity observed with the recall with perfect precision and high false-alarm rates at maximum informedness. scores show this is a mischaracterisation of this unit given that the false alarm rate at maximum informedness was greater than 99% and the modal response to Monarch butterflies was zero. What level of selectivity is required before a unit can be considered an'object detector' for a given category? In the end, this is a terminological point. On an extreme view, one might limit the term to the'grandmother units' that categorize objects with perfect recall and specificity, or alternatively, it might seem reasonable to describe a unit as a detector for a specific object category if there is some threshold of activation that supports more hits than misses (the unit is strongly activated by the majority of images from a given category), and at the same time, supports more hits than false alarms (the unit is strongly activated by items from the given category more often than by items from other categories). Or perhaps a lower standard could be defended, but in our view, the term "object detector" suggests a higher level of selectivity than 8% recall at perfect precision. That said, our show that some units respond strongly to some (unknown) features that are weakly correlated with an object category. For instance, unit fc6 1199 is responding to features that occur more frequently in Monarch Butterflies than other categories. This can also be seen in a recent ablation study in which removing the most selective units tended to impair the CNN's performance in identifying the corresponding object categories more than other categories (b). But again, the pattern of performance is not consistent with the units being labeled'object detectors'. Although the high precision score suggests that this unit is a butterfly detector this is misleading given there are butterfly images over the entire activation range (including 0). Activation Maximization is one of the most commonly used interpretability methods for explaining what a single unit has learned in many artificial CNNs and even biological neural networks (see for a survey). Our behavioral experiment provides the first quantitative assessment of AM images and compares AM interpretability to other selectivity measures. We generated 100 AM images images for every unit in the layers conv5, fc6, and fc8 in AlexNet, as in , and displayed them as 10 × 10-image panels. A total of 3,299 image panels were used in the experiment (995 fc8, 256 conv5, and 2048 randomly selected fc6 image panels) and were divided into 64 counterbalanced lists for testing. To assess the interpretability for these units as object detectors, 333 paid volunteers were asked to look at image panels and asked if the images had an object / animal or place in common. If the answer was'yes', they were asked to write down a generic name for that object (e.g. "fish" rather than "goldfish"). Analyses of common responses was done for any units where over 80% of humans agreed there was an object present. The are summarized in Table 1. Not surprisingly, the AM images for output fc8 units are the most human-recognizable as objects across the AlexNet layers (71.2%; Table 1a). In addition, when they were given a consistent interpretation, they almost always (95.4%; Table 1d) match the corresponding ImageNet category. By contrast, less than 5% of units in conv5 or fc6 were associated with consistently interpretable images (Table 1b), and the interpretations only weakly matched the category associated with the highest-activation images or CCMAS selectivity (Table 1d -e). Apart from showing that there are few interpretable units in the hidden layers of AlexNet, our findings show that the interpretability of images does not imply a high level of selectivity given the signal-detection Figure 4: Example AM images that were either judged by all participants to contain objects (a-c) or to be uninterpretable as objects (d-f). The human label for unit conv5 183 (a) was'dogs'; the most active image was of a'flat-coated retriever'; CCMAS class was'monitor'. For fc6 319 (b), subjects reported'green peppers' or'apples' (all classified as the same broad class in our analysis); both the most active item and CCMAS class were'Granny Smith apples'. For fc8 969 (c), humans suggested'beverage' or'drink'; both the most active item and CCMAS class were'eggnog'. (Fig. 2d-h). See Fig. 4 for an example of the types of images that participants rated as objects or non-objects. Thus far we have assessed the selectivity of hidden units in AlexNet and shown that no units can reasonably be characterized as object detectors despite the high precision and CCMAS scores of some units. This raises the question as to whether more recent CNNs learn object detector units. In order to address this we display jitterplots for three units that have the highest IoU scores according to the Network Dissection for the category BUS in (a) GoogLeNet trained on ImageNet, (b) GoogLeNet trained on Places-365, and (c) VGG-16 trained on Places-365, respectively (a). Models trained on the Places-365 dataset learn to categorize images into scenes (e.g., bedrooms, kitchens, etc.) rather than into object categories, and nevertheless, Zhou et al. (2018a) reported more object detectors in the former models. We illustrate the selectivity of the BUS category because it is an output category in ImageNet so we can easily plot the jitterplots for these units. As was the case with AlexNet, the jitterplots show that the most selective units show some degree of selectivity, with the BUS images more active on average compared to non-Buses, and the percentage of nonzero activations for BUS higher than the non-BUS categories (see tables A3 -A5 in the appendix for summary of more units). But the units are no more selective than the units we observed in AlexNet. Indeed, the precision measure of selectivity for the first units is 0.0, with none of the units having a precision of.75 that was the criterion of object detectors by , and CCMAS scores for first two units were roughly similar to the mean CCMAS score for AlexNet units in conv 5 (and much lower than the mean in fc6 and fc7). The most selective VGG-16 unit trained on Places-365 has lower precision and CCMAS scores than the Monarch Butterfly unit depicted in Figure 3. So again, different measures of selectivity provide support different , and even the most selective units are far from the selective units observed in recurrent networks as reported in Figure 1a. See tables A3 -A5 in the appendix for more details about these three units. Our central finding is that different measures of single-unit selectivity for objects support very different when applied to the same units in AlexNet. In contrast with the precision and CCMAS measures that suggest some highly selective units for objects in layers conv5, fc6, and fc7, the recall with perfect precision and false alarm rates at maximum informedness show low levels of selectivity. Indeed, the most selective units have a poor hit-rate or a high false-alarm rate (or both) for identifying an object class. The same outcome was observed with units in VGG-16 and GoogLeNet trained on either ImageNet or the Places-365 dataset. Not only do the different measures provide very different assessments of selectivity, the precision, CCMAS, and Network Dissection measures provide highly misleading estimates of selectivity that have led to mistaken . For example, unit fc6 1199 in AlexNet trained on ImageNet is considered an Monarch Butterfly detector according to with a precision score of 98% (and a CCMAS score of .93). But the jitterplot in Fig. 3 and signal detection scores (e.g., high false alarm rate at maximum informedness) show this is a mischaracterisation of this unit. In the same way, the Network Dissection method identified many object detectors in VGG-16 and GoogLeNet CNNs, but the jitterplots in Fig. 5 (and precision scores) show that this is unjustified. For additional problems with the CCMAS score see Figure 5 in Appendix C. Similarly, the images generated by Activation Maximization also provided a misleading estimate of selectivity given that interpretable images were associated with very low selectivity scores. This has led to confusions that have delayed theoretical progress. For example, describing single units in CNNs as "object detectors" in response to high precision measures (Zhou et al.) suggests similar types of representations are learned in CNNs and RNNs. Indeed, we are not aware of anyone in the machine learning community who has even considered the hypothesis that selectivity is reduced in CNNs compared RNNs. Our findings highlight the contrasting . What should be made of the finding that localist representations are sometimes learned in RNNs (units with perfect specificity and recall), but not in AlexNet and related CNNs? The failure to observe localist units in the hidden layers of these CNNs is consistent with's claim that these units emerge in order to support the co-activation of multiple items at the same time in short-term memory. That is, localist representations may be the solution to the superposition catastrophe, and these CNNs only have to identify one image at a time. The pressure to learn highly selective representations in response to the superposition constraint may help explain the reports of highly selective neurons in cortex given that the cortex needs to co-activate multiple items at the same time in order to support short-term memory . Note, the RNNs that learned localist units were very small in scale compared to CNNs we have studied here, and accordingly, it is possible that the contrasting reflect the size of the networks rather than the superposition catastrophe per se. Relevant to this issue a number of authors have reported the existence of selective units in larger RNNs with long-short term memory (LSTM) units (; ; ;). use the term'grandmother cell' to describe the units they observed. It will be interesting to apply our measures of selectivity to these larger RNNs and see whether these units are indeed'grandmother units'. It should also be noted that there are recent reports of impressively selective representations in Generative Adversarial Networks and Variational Autoencoders where the superposition catastrophe is not an issue. Again, it will be interesting to assess the selectivity of these units according to signal detection measures in order to see whether there are additional computational pressures to learn highly selective or even grandmother cells. We will be exploring these issues in future work. One hundred generated images were made for every unit in layers conv5, fc6 and fc8 in AlexNet, as in , and displayed as 10x10 image panels (figures A4 and Figures A2 and A3). A total of 3,299 image panels were used in the experiment (995 fc8, 256 conv5, and 2048 randomly selected fc6 image panels) and were divided into 64 counterbalanced lists of 51 or 52 (4 conv5, 15 or 16 fc8 and 32 fc6). 51 of the lists were assigned to 5 participants and 13 lists were assigned to 6 participants. To test the interpretability of these units, paid volunteers were asked to look at image panels and asked if the images had an object / animal or place in common. If the answer was'yes', they were asked to name that object simply (i.e. fish rather than goldfish). Analyses of common responses was carried out for any units where over 80% of humans agreed there was an object present, by reading the human responses and comparing them to both each other and to the output classes. Agreement was taken if the object was the same rough class. For example,'beer','glass', and'drink' were all considered to be in agreement in the general object of'drink', and in agreement with both the classes of'wine glass' and'beer' as these classes were also general drink classes (this is an actual example, most responses were more obvious and required far less interpretation than that). Participants were given six practice trials, each with panels of 20 images before starting the main experiment. Practice trials included images that varied in their interpretability. Some examples of the 10x10 grids of activation maximisation images that were presented to participants are shown in Figures A2, A3 and A4. Figure A2 shows an example from conv5 that human participants agreed had no obvious object in common (although there are repeated shape motifs, the participants were specifically asked for objects, and not abstract concepts like shape or color. Figure A3 is also from the conv5 and was judged by participants as some images containing 'dogs'. Figure A4 is the AM images for the supposed 'butterfly detector' unit example discussed in the paper. Figure A2 : Example activation maximisation images for unit conv5.65. These images were judged by humans to not contain any interpretable objects in common (although the reader may agree that there are some shape and colour similarities in the images). Figure A3: Example activation maximisation images for unit conv5.183. These images were judged by humans to contain some interpretable images, in this case, of the type'dogs'. Figure A4: Example activation maximisation images for unit fc6.1199. Whilst there are some butterfly wing shapes in these images, there are not obvious butterflies. N.B. the second highest activating class for this unit is ladybirds, and there are some orange round shapes that could conceivably be ladybug-alikes. B FURTHER DATA ON THE SELECTIVITY MEASURES ACROSS ALEXNET Table A1 gives the highest values of CCMAS and precision for each layer in AlexNet. It is worth noting that the highest CCMAS score of all hidden units was.94 (fc7.31), which at first glance suggests that this unit is close to'perfect' selectivity. However, this unit only has low a precision score of 11%. In other words, although the mean activation for the given class is very high relative to the mean of all other activations (high CCMAS), the proportion of items from that class in the 100 most active items is low (low precision). See appendix Sec. C for discussion of how this occurs and Fig. A5(a) for an illustrative example. Top Fig. 3. Table A2 shows positive correlations between four of the selectivity measures used. There are moderate positive correlations between precision and CCMAS; and precision and Recall at 95% precision. The other correlations between selectivity measures have weak positive correlations. All four selectivity measures are negatively correlated with the number of classes present in the 100 most active items, that is, the more selective the unit, the fewer classes will be represented in the most active 100 items. The CCMAS measure is based on comparing the mean activation of a category with the mean activation for all other items, and this is problematic for a few reasons. First, in many units a large proportion of images do not activate a unit at all. For instance, our butterfly'detector' unit fc6.1199 has a high proportion of images with an activation of 0.0 (see figure 3). Indeed, the inset on the middle figure shows that the distribution can be better described by exponential-derived fits rather than a Gaussian. This means that the CCMAS selectivity is heavily influenced by the the proportion of images that have an activation value of zero (or close to zero). This can lead to very different estimates of selectivity for CCMAS and precision or localist selectivity, which are driven by the most highly activated items. In A5 we generate example data to highlight ways in which CCMAS score may be non-intuitive. In subplot (a) we demonstrate that a unit can have a CCMAS score of of 1.0 despite only a single item activating the unit. The point that we wish to emphasise is that a high CCMAS score does not necessarily imply selectivity for a given class, but might in fact relate to selectivity for a small subset of items from a given class, and this is especially true when a unit's activation is sparse (many items do not activate the unit). However, the reverse can also be true. In subplot (c) we demonstrate that a unit can have a very low CCMAS score of.06 despite all of the most active items being from the same class. In addition, if the CCMAS provided a good measure of a unit's class selectivity, then one should expect that a high measure of selectivity for one class would imply that the unit is not highly selective for other classes. However, the CCMAS score for the most selective category and the second most selective category (CCMAS2) were similar across the conv5, fc6 and fc7. layers, with the mean CCMAS scores.491,.844, and.848, and the CCMAS2 scores.464,.821,.831. For example, unit fc7.0 has a CCMAS of.813 for the class'maypole', and a CCMAS2 score of.808 for'chainsaw' (with neither of these categories corresponding 'orangutan' that had the highest precision of score of 14%). To investigate units claimed by Zhou et al. (2018a) to be object detectors, we focus on units from a single layer that are reported to be'bus detectors', that is, units with an IoU ≥.04. We used the first 100 images per class from the ImageNet 2012 dataset as our test data. There are three classes of bus in this dataset:'n04146614 school bus','n04487081 trolleybus, trolley coach, trackless trolley','n03769881 minibus', and this corresponded to 300 items out of 100000 images. Data for all bus unit detectors for VGG trained on places 365 are shown in table A3; for GoogLeNet trained on places 365 in table A4; and for GoogLeNet trained on ImageNet are shown in table A5. Note that for all units there are very few busses with activation at zero and that the mean activation for busses is higher than the mean activation for non-busses. However, all precision scores are all below.6, meaning that of the 100 items that most strongly activated the unit, at least 40 of them were not busses. Together these suggests that whilst these units demonstrate some sensitivity to busses, they show poor specificity for busses (e.g., high false-alarm rate). Table A5: Selectivity measures for GoogLeNet, trained on ImageNet, layer inception4e units units identified by Zhou et al. (2018a) as object detectors. Standard errors not shown for space, but were below ±2. A units is marked as correct if there was a single bus in the 4 example pictures on the website (http://netdissect.csail.mit.edu/dissect/googlenet_imagenet/), and false if not. This might suggest that the units were responding to'bus like' features in none bus objects.

Represent the TLDR for finding the corresponding scientific article.

Looking for object detectors using many different selectivity measures; CNNs are slightly selective , but not enough to be termed object detectors.

Represent the short summary for finding the corresponding scientific article (long)

Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.


Question: Represent the article for finding a single-sentence TLDR

Question: The folding structure of the DNA molecule combined with helper molecules, also referred to as the chromatin, is highly relevant for the functional properties of DNA. The chromatin structure is largely determined by the underlying primary DNA sequence, though the interaction is not yet fully understood. In this paper we develop a convolutional neural network that takes an image-representation of primary DNA sequence as its input, and predicts key determinants of chromatin structure. The method is developed such that it is capable of detecting interactions between distal elements in the DNA sequence, which are known to be highly relevant. Our experiments show that the method outperforms several existing methods both in terms of prediction accuracy and training time. DNA is perceived as a sequence over the letters {A,C,G,T}, the alphabet of nucleotides. This sequence constitutes the code that acts as a blueprint for all processes taking place in a cell. But beyond merely reflecting primary sequence, DNA is a molecule, which implies that DNA assumes spatial structure and shape. The spatial organization of DNA is achieved by integrating ("recruiting") other molecules, the histone proteins, that help to assume the correct spatial configuration. The combination of DNA and helper molecules is called chromatin; the spatial configuration of the chromatin, finally, defines the functional properties of local areas of the DNA BID9.Chromatin can assume several function-defining epigenetic states, where states vary along the genome BID12. The key determinant for spatial configuration is the underlying primary DNA sequence: sequential patterns are responsible for recruiting histone proteins and their chemical modifications, which in turn give rise to or even define the chromatin states. The exact configuration of the chromatin and its interplay with the underlying raw DNA sequence are under active research. Despite many enlightening recent findings (e.g. BID6 ; BID11, comprehensive understanding has not yet been reached. Methods that predict chromatin related states from primary DNA sequence are thus of utmost interest. In machine learning, many prediction methods are available, of which deep neural networks have recently been shown to be promising in many applications BID17 . Also in biology deep neural networks have been shown to be valuable (see BID3 for a review).Although DNA is primarily viewed as a sequence, treating genome sequence data as just a sequence neglects its inherent and biologically relevant spatial configuration and the ing interaction between distal sequence elements. We hypothesize that a deep neural network designed to account for long-term interactions can improve performance. Additionally, the molecular spatial configuration of DNA suggests the relevance of a higher-dimensional spatial representation of DNA. However, due to the lack of comprehensive understanding with respect to the structure of the chromatin, sensible suggestions for such higher-dimensional representations of DNA do not exist. One way to enable a neural net to identify long-term interactions is the use of fully connected layers. However, when the number of input nodes to the fully connected layer is large, this comes with a large number of parameters. We therefore use three other techniques to detect long-term interactions. First, most convolutional neural networks (CNNs) use small convolution filters. Using larger filters already at an early stage in the network allows for early detection of long-term interactions without the need of fully connected layers with a large input. Second, a deep network similar to the ResNet BID14 or Inception BID27 network design prevents features found in early layers from vanishing. Also, they reduce the size of the layers such that the final fully connected layers have a smaller input and don't require a huge number of parameters. Third, we propose a novel kind of DNA representation by mapping DNA sequences to higher-dimensional images using space-filling curves. Space-filling curves map a 1-dimensional line to a 2-dimensional space by mapping each element of the sequence to a pixel in the 2D image. By doing so, proximal elements of the sequence will stay in close proximity to one another, while the distance between distal elements is reduced. The space-filling curve that will be used in this work is the Hilbert curve which has several advantages. (i): [Continuity] Hilbert curves optimally ensure that the pixels representing two sequence elements that are close within the sequence are also close within the image BID4 BID1. (ii): [Clustering property] Cutting out rectangular subsets of pixels (which is what convolutional filters do) yields a minimum amount of disconnected subsequences BID20. (iii): If a rectangular subimage cuts out two subsequences that are disconnected in the original sequence, chances are maximal that the two different subsequences are relatively far apart (see our analysis in Appendix A).The combination of these points arguably renders Hilbert curves an interesting choice for representing DNA sequence as two-dimensional images. (i) is a basic requirement for mapping short-term sequential relationships, which are ubiquitous in DNA (such as codons, motifs or intron-exon structure).(ii) relates to the structure of the chromatin, which -without all details being fully understood -is tightly packaged and organized in general. Results from BID10 indicate that when arranging DNA sequence based on Hilbert curves, contiguous areas belonging to identical chromatin states cover rectangular areas. In particular, the combination of (i) and (ii) motivate the application of convolutional layers on Hilbert curves derived from DNA sequence: rectangular subspaces, in other words, submatrices encoding the convolution operations, contain a minimum amount of disconnected pieces of DNA. (iii) finally is beneficial insofar as long-term interactions affecting DNA can also be mapped. This in particular applies to so-called enhancers and silencers, which exert positive (enhancer) or negative (silencer) effects on the activity of regions harboring genes, even though they may be far apart from those regions in terms of sequential distance. Since Watson and Crick first discovered the double-helix model of DNA structure in 1953 BID31, researchers have attempted to interpret biological characteristics from DNA. DNA sequence classification is the task of determining whether a sequence S belongs to an existing class C, and this is one of the fundamental tasks in bio-informatics research for biometric data analysis (Z.). Many methods have been used, ranging from statistical learning BID30 to machine learning methods BID19. Deep neural networks BID17 form the most recent class of methods used for DNA sequence classification (R.R. BID25 BID26 BID33 BID3 .Both BID22 and BID15 use support vector machines (SVM) to predict chromatin state from DNA sequence features. While BID22 use the entire set of features as input to the SVM, BID15 use random forests to pre-select a subset of features that are expected to be highly relevant for prediction of chromatin state to use as input to the SVM. Only BID21 use a CCN as we do. There are two major differences between their approach and ours. First and foremost, the model architecture is different: the network in BID21 consists of two convolution layers followed by pooling layers, a fully connected layer and a sigmoid layer, while our model architecture is deeper, uses residual connections to reuse the learned features, has larger convolution filters and has small layers preceding the fully connected layers (see Methods). Second, while we use a space-filling curve to transform the sequence data into an image-like tensor, BID21 keep the sequential form of the input data. Apart from BID10, the only example we are aware of where Hilbert curves were used to map DNA sequence into two-dimensional space is from BID2, who demonstrated the power of Hilbert curves for visualizing DNA. Beyond our theoretical considerations, these last two studies suggest there are practical benefits of mapping DNA using Hilbert curves. Our contributions are twofold. First, we predict chromatin state using a CNN that, in terms of architecture, resembles conventional CNNs for image classification and is designed for detecting distal relations. Second, we propose a method to transform DNA sequence patches into two-dimensional image-like arrays to enhance the strengths of CNNs using space-filling curves, in particular the Hilbert curve. Our experiments demonstrate the benefits of our approach: the developed CNN decisively outperforms all existing approaches for predicting the chromatin state in terms of prediction performance measures as well as runtime, an improvement which is further enhanced by the convolution of DNA sequence to a 2D image. In summary, we present a novel, powerful way to harness the power of CNNs in image classification for predicting biologically relevant features from primary DNA sequence. We transform DNA sequences into images through three steps. First, we represent a sequence as a list of k-mers. Next, we transform each k-mer into a one-hot vector, which in the sequence being represented as a list of one-hot vectors. Finally, we create an image-like tensor by assigning each element of the list of k-mers to a pixel in the image using Hilbert curves. Each of the steps is explained in further detail below. From a molecular biology point of view, the nucleotides that constitute a DNA sequence do not mean much individually. Instead, nucleotide motifs play a key role in protein synthesis. In bioinformatics it is common to consider a sequence's k-mers, defined as the k-letter words from the alphabet {A,C,G,T} that together make up the sequence. In computer science the term q-gram is more frequently used, and is often applied in text mining BID29. As an example, the sequence TGACGAC can be transformed into the list of 3-mers {TGA, GAC, ACG, CGA, GAC} (note that these are overlapping). The first step in our approach is thus to transform the DNA sequence into a list of k-mers. Previous work has shown that 3-mers and 4-mers are useful for predicting epigenetic state BID22 BID15. Through preliminary experiments, we found that k = 4 yields the best performance: lower values for k in reduced accuracy, while higher values yield a high risk of overfitting. Only for the Splice dataset (see experiments) we used k = 1 to prevent overfitting, as this is a small dataset. In natural language processing, it is common to use word embeddings as GloVe or word2vec or onehot vectors BID13. The latter approach is most suitable for our method. Each element of such a vector corresponds to a word, and a vector of length N can thus be used to represent N different words. A one-hot vector has a one in the position that corresponds to the word the position is representing, and a zero in all other positions. In order to represent all k-mers in a DNA sequence, we need a vector of length 4 k, as this is the number of words of length k that can be constructed from the alphabet {A,C,G,T}. For example, if we wish to represent all 1-mers, we can do so using a one-hot vector of length 4, where A corresponds to Our next step is to transform the list of one-hot vectors into an image. For this purpose, we aim to assign each one-hot vector to a pixel. This gives us a 3-dimensional tensor, which is similar in shape to the tensor that serves as an input to image classification networks: the color of a pixel in an RGB-colored image is represented by a vector of length 3, while in our approach each pixel is represented by a one-hot vector of length 256.What remains now is to assign each of the one-hot vectors in the list to a pixel in the image. For this purpose, we can make use of space-filling curves, as they can map 1-dimensional sequences to a 2-dimensional surface preserving continuity of the sequence BID4 BID1. Various types of space-filling curves are available. We have compared the performance of several such curves, and concluded that Hilbert curves yield the best performance (Appendix A). This corresponds with our intuition: the Hilbert curve has several properties that are advantageous in the case of DNA sequences, as discussed in the introduction section. The Hilbert curve is a well-known space-filling curve that is constructed in a recursive manner: in the first iteration, the curve is divided into four parts, which are mapped to the four quadrants of a square FIG1. In the next iteration, each quadrant is divided into four sub-quadrants, which, in a similar way, each hold 1/16 th of the curve FIG1. The quadrants of these sub-quadrants each hold 1/64 th of the curve, etc FIG1.By construction, the Hilbert curve yields a square image of size 2 n × 2 n, where n is the order of the curve (see FIG1). However, a DNA sequence does not necessarily have 2 n * 2 n k-mers. In order to fit all k-mers into the image, we need to choose n such that 2 n * 2 n is at least the number of k-mers in the sequence, and since we do not wish to make the image too large, we pick the smallest such n. In many cases, a large fraction of the pixels then remains unused, as there are fewer k-mers than pixels in the image. By construction, the used pixels are located in upper half of the image. Cropping the picture by removing the unused part of the image yields rectangular images, and increases the fraction of the image that is used FIG1 ).In most of our experiments we used sequences with a length of 500 base pairs, which we convert to a sequence of 500 -4 + 1 = 497 4-mers. We thus need a Hilbert curve of order 5, ing in an image of dimensions 2 5 × 2 5 × 256 = 32 × 32 × 256 (recall that each pixel is assigned a one-hot vector of length 256). Almost half of the ing 1024 pixels are filled, leaving the other half of the image empty which requires memory. We therefore remove the empty half of the image and end up with an image of size 16 × 32 × 256.The data now has the appropriate form to input in our model. Modern CNNs or other image classification systems mainly focus on gray-scale images and standard RGB images, ing in channels of length 1 or 3, respectively, for each pixel. In our approach, each pixel in the generated image is assigned a one-hot vector representing a k-mer. For increasing k, the length of the vector and thus the image dimension increases. Here, we use k = 4 ing in 256 channels, which implies that each channel contains very sparse information. Due to the curse of dimensionality standard network architectures applied to such images are prone to severe overfitting. Here, we design a specific CNN for the kind of high dimensional image that is generated from a DNA sequence. The architecture is inspired by ResNet BID14 and Inception BID27. The network has L layers and each layer implements a non-linear function F l (x l) where l is the index of the hidden layer with output x l+1. The function F l (x l) consists of various layers such as convolution (denoted by c), batch normalization (bn), pooling (p) and non-linear activation function (af).The first part of the network has the objective to reduce the sparseness of the input image FIG2, and consists of the consecutive layers [c, bn, c, bn, af, p]. The main body of the network enhances the ability of the CNN to extract the relevant features from DNA space-filling curves. For this purpose, we designed a specific Computational Block inspired by the ResNet residual blocks BID14. The last part of the network consists of 3 fully-connected layers, and softmax is used to obtain the output classification label. The complete model is presented in TAB1, and code is available on Github (https://github.com/Bojian/Hilbert-CNN/tree/master). A simplified version of our network with two Computational Blocks is illustrated in FIG2.Computation Block. In the Computation Block first the outputs of two Residual Blocks and one identity mapping are summed, followed by a bn and an af layer FIG2 ). In total, the computational block has 4 convolutional layers, two in each Residual Block (see FIG3 . The Residual Block first computes the composite function of five consecutive layers, namely [c, bn, af, c, bn], followed by the concatenation of the output with the input tensor. The residual block concludes with an af. Implementation details. Most convolutional layers use small squared filters of size 2, 3 and 4, except for the layers in the first part of the network, where large filters are applied to capture long range features. We use Exponential Linear Units (ELU, BID8) as our activation function af to reduce the effect of gradient vanishing: preliminary experiments showed that ELU preformed significantly better than other activation functions (data not shown). For the pooling layers p we used Average pooling. Average pooling outperformed Max pooling in terms of prediction accuracy by more than 2% in general, as it reduces the high variance of the sparse generated images. Cross entropy was used as the loss function. We test the performance of our approach using ten publicly available datasets from BID23. The datasets contain DNA subsequences with a length of 500 base pairs. Each sequence is labeled either as "positive" or "negative", indicating whether or not the subsequence contains regions that are wrapped around a histone protein. The ten datasets each consider a different type of histone protein, indicated by the name of the dataset. Details can be found in TAB2.A randomly chosen 90% of the dataset is used for training the network, 5% is used for validation and early stopping, and the remaining (5%) is used for evaluation. We train the network using the AdamOptimizer BID16 1. The learning rate is set to 0.003, the batch-size was set to 300 samples and the maximum number of epochs is 10. After each epoch the level of generalization is measured as the accuracy obtained on the validation set. We use early stopping to prevent overfitting. To ensure the model stops at the correct time, we combine the GL α measurement BID24 of generalization capability and the No-Improvement-In-N-Steps(Nii-N) method BID24. For instance, Nii-2 means that the training process is terminated when generalization capability is not improved in two consecutive epochs. We compare the performance of our approach, referred to as HCNN, to existing models. One of these is the support vector machine (SVM) model by BID15, for which are available in their paper. Second, in tight communication with the authors, we reconstructed the Seq-CNN model presented in BID21 (the original software was no longer available), see Appendix C for detailed settings. Third, we constructed the commonly used LSTM, where the so-called 4-mer profile of the sequence is used as input. A 4-mer profile is a list containing the number of occurrences of all 256 4-mers of the alphabet {A,C,G,T} in a sequence. Preliminary tests showed that using all 256 4-mers ed in overfitting, and including only the 100 most frequent 4-mers is sufficient. Details of the LSTM architecture can be found in TAB6 in Appendix C.In order to assess the effect of using a 2D representation of the DNA sequence in isolation, we compare HCNN to a neural network using a sequential representation as input. We refer to this model as seq-HCNN. As in HCNN, the DNA sequence is converted into a list of kmer representing one-hot vectors, though the mapping of the sequence into a 2D image is omitted. The network architecture is a "flattened" version of the one used in HCNN: for example, a 7×7 convolution filter in HCNN is transformed to a 49×1 convolution filter in the 1D-sequence model. As a summary of model size, the Seq-CNN model contains 1.1M parameters, while both HCNN and seq-HCNN have 961K parameters, and the LSTM has 455K parameters. In order to test whether our method is also applicable to DNA sequence classification tasks other than chromatin state prediction only, we performed additional tests on the splice-junction gene sequences dataset from BID18. Most of the DNA sequence is unused, and splice-junctions refer to positions in the genetic sequence where the transition from an unused subsequence (intron) to a used subsequence (exon) or vice versa takes place. The dataset consists of DNA subsequences of length 61, and each of the sequences is known to be an intron-to-exon splice-junction, an exon-to-intron splice junction or neither. As the dataset is relatively small, we used 1-mers instead of 4-mers. Note that the sequences are much shorter than for the other datasets, ing in smaller images (dimensions 8 × 8 × 4). The show that SVM and Seq-CNN were both outperformed by HCNN and seq-HCNN; LSTM shows poor performance. HCNN and seq-HCNN show similar performance in terms of prediction accuracy, though HCNN shows more consistent over the ten folds indicating that using a 2D representation of the sequence improves robustness. Furthermore, HCNN yields better performance than seq-HCNN in terms of precision, recall, AP and AUC (Table 5). It thus enables to reliably vary the tradeoff between recall and false discoveries. HCNN outperforms all methods in terms of training time (Table 4).The good performance of HCNN observed above may either be attributable to the conversion from DNA sequence to image, or to the use of the Hilbert curve. In order to address this question, we adapted our approach by replacing the Hilbert curve with other space-filling curves and compared their prediction accuracy. Besides the Hilbert curve, other space-filling curves exist BID20 ) (see Appendix A). In Figure 4, we compare the performance of our model with different mapping strategies in various datasets as displayed. We find that the images generated by the spacefilling Hilbert curve provide the best accuracy on most datasets and the 1-d sequence performs worst. In this paper we developed a CNN that outperforms the state-of-the-art for prediction of epigenetic states from primary DNA sequence. Indeed, our methods show improved prediction accuracy and training time compared to the currently available chromatin state prediction methods from Pahm TAB1 in BID15. In the splice dataset, Seq-CNN performed best when using 4-mers, while for HCNN and seq-HCNN 1-mers yielded the best performance. Figure 4: HCNN with different mapping strategies by BID21 and thus yields a huge number of parameters in the fully connected layer. In HCNN on the other hand the number of nodes is strongly reduced before introducing a fully connected layer. Third, the use of a two-dimensional input further enhances the model's capabilities of incorporating long-term interactions. We showed that seq-HCNN and HCNN are not only capable of predicting chromatin state, but can also predict the presence or absence of splice-junctions in DNA subsequences. This suggests that our approach could be useful for DNA sequence classification problems in general. Hilbert curves have several properties that are desirable for DNA sequence classification. The intuitive motivation for the use of Hilbert curves is supported by good when comparing Hilbert curves to other space-filling curves. Additionally, Hilbert curves have previously been shown to be useful for visualization of DNA sequences BID2 ).The main limitation of Hilbert curves is their fixed length, which implies that the generated image contains some empty spaces. These spaces consume computation resources; nevertheless, the 2D representation still yields reduced training times compared to the 1D-sequence representation, presumably due to the high degree of optimization for 2D inputs present in standard CNN frameworks. Given that a substantial part of the improvements in performance rates are due to our novel architecture, we plan on investigating the details of how components of the architecture are intertwined with improvements in prediction performance in more detail. We also plan to further investigate why Hilbert curves yield the particular advantages in terms of robustness and false discovery control we have observed here. As noted before, long-term interactions are highly relevant in DNA sequences. In this section we consider these long-term interactions in four space-filling curves: the reshape curve, the snake curve, the Hilbert curve and the diag-snake curve. See FIG4 for an illustration. As can be seen in FIG4, mapping a sequence to an image reduces the distance between two elements that are far from one another in the sequence, while the distance between nearby elements does not increase. Each of the curves does have a different effect on the distance between faraway elements. In order to assess these differences, we use a measure that is based on the distance between two sequence elements as can be observed in the image. We denote this distance by L C (x,y) where x, y ∈ S, with S the sequence and C the curve under consideration. Then for the sequence S = {A,B,C,D,E, · · ·,P} we obtain• L seq (A,P) = 15 for the sequence;• L reshape (A,P) = 3 √ 2 for the reshape curve;• L snakesnake (A,P) = 3 for the snake curve;• L diag−snake (A,P) = 3 √ 2 for the diagonal snake curve.• L hilbert (A,P) = 3 for the Hilbert curve;We now introduce the following measure: DISPLAYFORM0 where the ∆(C) is the set of the weighted distances between all pairs of the elements in the sequence. Here, ∆(C) is a set containing the distance between any two sequence elements, weighted by their distance in the sequence: DISPLAYFORM1 Note that a low max(∆(C)) relative to mean(∆(C)) implies that long-term interactions are strongly accounted for, so a high Γ(C) is desirable. Γ(C) is evaluated for the four space-filling curves as well as the sequence representation for sequences of varying lengths. The show that the Hilbert curve yields the highest values for Γ(C) 6 and thus performs best in terms of retaining long-distance interactions. Table 6: Γ(C) for four space-filling curves, evaluated in sequences of varying lengths. B FROM SEQUENCE TO IMAGE FIG5 shows the conversion from DNA sequence to image. Accuracy is one of the most intuitive performance measurements in deep learning and machine learning. We therefore optimized the hyperparameters such as the network architecture and learning rate based on maximum accuracy. The hyperparameters are optimized through random search BID5 and using general principles from successful deep learning strategies and the following intuition. First, as our main goal was to capture long-term interactions, we chose a large kernel size for the first layer, for which various values were attempted and 7x7 gave the best performance. As is common practice in deep learning, we then opted for a smaller kernel size in the following layer. Second, in order to limit the number of parameters, we made use of residual blocks inspired by ResNet BID14 and Inception BID27. Finally, we applied batch normalization to prevent overfitting.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

A method to transform DNA sequences into 2D images using space-filling Hilbert Curves to enhance the strengths of CNNs

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.