Fast Parametric Learning with Activation Memorization
Jack W Rae Chris Dyer Peter Dayan Timothy P Lillicrap

[
Fast Parametric Learning with Activation Memorization


Jack W Raedm,ucl
Chris Dyerdm
Peter Dayangatsby
Timothy P Lillicrapdm,ucl


dmDeepMind, London, UK
uclCoMPLEX, Computer Science, University College London, London, UK
gatsbyGatsby Computational Neuroscience Unit, University College London, UK


Jack W Raejwrae@google.com
Timothy P Lillicrapcountzero@google.com

Language Modeling, Softmax, Hebbian, Classification, Machine Learning, ICML

0.3in
]




Neural networks trained with backpropagation often struggle to identify classes that have been observed a small number of times. In applications where most class labels are rare, such as language modelling, this can become a performance bottleneck. One potential remedy is to augment the network with a fast-learning non-parametric model which stores recent activations and class labels into an external memory. We explore a simplified architecture where we treat a subset of the model parameters as fast memory stores. This can help retain information over longer time intervals than a traditional memory, and does not require additional space or compute.
In the case of image classification, we display faster binding of novel classes on an Omniglot image curriculum task.
We also show improved performance for word-based language models on news reports (GigaWord), books (Project Gutenberg) and Wikipedia articles (WikiText-103) — the latter achieving a state-of-the-art perplexity of 29.2.







§ INTRODUCTION

Neural networks can be trained to classify discrete outputs by appending a softmax output layer. This is a linear map projecting the d-dimensional hidden output of the network to m outputs, where m is the number of distinct classes. A softmax operator <cit.> is then applied to produce a probability distribution over classes. The parameters in this softmax layer are typically optimized with the network's parameters by gradient descent. 

We can think of the weights in the softmax layer θ∈ℝ^m × d as a set of m vectors θ[i];   i = 1, …, m that each correspond to a given class. When trained with a supervised loss, such as cross-entropy, each step of gradient descent pulls the parameter θ[y], corresponding to the class label y, towards having a greater inner product with the network output h, and pushes all other parameters θ[j]  ,   j ≠ y towards having a smaller inner product with h.

One shortcoming of neural network classifiers trained with backpropagation is that they require many input examples for a given class in order to predict it with reasonable accuracy. That is, many positive class examples and optimization steps are required to pull θ[i] towards a point in space where class i can then be recognized. While the learner will have many opportunities to organize θ[i] parameters associated with frequent classes, infrequent class parameters will be poorly estimated. In domains where new classes are frequently introduced, or large-scale classification problems where some classes are very infrequently observed, this estimation problem is potentially quite serious.

One approach to speed up learning, which has received revived interest, is meta-learning. Here, meta-learning refers to algorithms which learn to produce  or manipulate learning algorithms <cit.>, and it operates by learning over a distribution of tasks or datasets. A meta-learner applies knowledge from the global distribution of tasks to produce or optimize algorithms which specialize to a given task instance. Meta-learning of neural networks has seen promising results for applications such as parameter optimization <cit.> and classification <cit.>. For classification, the networks are augmented with a differentiable external memory, and are trained with many rounds of data — with class labels permuted between episodes.

Meta-learning can be very powerful for few-shot learning in cases where there is a set of similar prior data to meta-learn over, however it may not be practical for standalone datasets. For example, if one wants to model the grammar of computer code, it is unclear that a meta-learning system trained over natural language will be useful. Also memory-based meta-learning requires backpropagating from the read time to the original write time, which is not well suited to applications where writes and reads are separated by long time steps of conditional computation. In the case of modelling language, for example, infrequent words will not occur for large time intervals — rendering memory-based meta-learning challenging.

The task of statistical language modelling itself is interesting to investigate issues of binding new or infrequent classes, because most classes (words) are infrequent <cit.> and new classes naturally emerge over time. Recent approaches to improve neural language models have involved augmenting the network with a non-parametric cache, which stores past hidden activations h_t-n, … , h_t-1 and corresponding labels, y_t-n, …, y_t-1 <cit.>. Attention over this cache provides better modelling of infrequent words that occur in a recent context, including previously unknown words <cit.>. However there is a diminishing return to increasing the cache size <cit.>, and once rare words fall outside the recent context the boost in predictive performance expires.



Motivated from these memory systems, we explore a very simple optimization procedure where the network accumulates activations h_t directly into the softmax layer weights θ[y_t] when a class y_t has been seen a small number of times, and uses gradient descent otherwise. Accumulating or smoothing network activations into the weights actually corresponds to the well-known Hebbian learning update rule W[i, j] ←1/n∑_t = 1^n x_t^i x_t^j  <cit.> in the special case of classification on the output layer, where W, x_t^i, x_t^j correspond to θ, h_t, y_t respectively. We see that mixing the two rules provides better initial representations and can also preserve these representations for much longer time spans. This is because memorized activations for one class are not competing for space with activations from other (more frequent, say) classes — unlike a conventional external memory. In this sense, the parameters become an instance of a quickly updated compressed memory, we explore this idea in Section <ref>

We demonstrate this model adapts quickly to novel classes in a simple image classification task using handwritten characters from Omniglot <cit.>. We then show it improves overall test perplexity for two medium-scale language modelling corpora, WikiText103 (wikipedia articles) <cit.> and Project Gutenberg  (books) [Project Gutenberg. (n.d.). Retrieved January 2, 2018, from www.gutenberg.org] alongside a large-scale corpus GigaWord v5 (news articles) <cit.>. By splitting accuracy over word frequency buckets, we see improved perplexity for less frequent words. 




§ BACKGROUND












 §.§ Memory




There has been recent interest in models which store past hidden activations through time h_1, h_2, …, h_t-1 into a memory matrix and query the contents with a differentiable attention mechanism. This has been applied to  machine translation <cit.>, program induction <cit.>, and question answering <cit.>. Memory-augmented neural networks have also been successfully applied to language modelling <cit.> to facilitate the learning of unknown words, capture the tendency for globally rare words to be repeated in close proximity, and to quickly adapt the network to contextually relevant prior text <cit.>.

There are many variants of how to read from memory and mix this information with the network computations. One approach is to retrieve hidden activations and mix these with network activations in latent space <cit.>. Another approach is a classic mixture model, as shown in Figure <ref>; the output probability distribution can be obtained by interpolating the probabilities p_p, p_np from the parametric model and memory respectively.

For intuition we briefly explain a particular architecture, the Neural Cache <cit.>, whose operation is related to our model. The cache is a store of the last n hidden activations along with their corresponding target output (next word) from a trained parametric language model, such as the Long Short Term Memory (LSTM) <cit.>. The conditional probability of a word w occurring is proportional to the sum over kernalized inner product similarities between the current hidden state h_t and past hidden states when word w occurred.


    p_c(w | h_t) ∝∑_i = t - n^t - 1 e^h_t^T h_i 𝕀{y_i = w}


Where 𝕀{p} = 1 if p is true, 0 otherwise. This is then interpolated with the parametric language model using a fixed hyper-parameter, swept over during validation. Although the cache is of fixed size n, it can be defined to be very large with sparse attention and efficient data-structures <cit.>.



 §.§ Language modelling

We can model a sequence of text as the product of conditional word probabilities,


    p(w_1, w_2, …, w_t) = ∏_i = 1^t p(w_i | w_1, w_2, … , w_i-1)


which are estimated separately. Traditional n-gram models take frequency-based estimates of these conditional probabilities with truncated contexts p_n = p(w_i | w_i-n, …, w_i-1) and smooth between them to estimate the full conditional probability, p(w_i | w_1, …, w_i-1) = ∑_j=1^n λ_j p_j. A popular approach is Kneser-Ney smoothing <cit.>. More recently, neural language models such as LSTMs and convolutional neural networks directly model the conditional probabilities through sequence-to-sequence training and achieve state-of-the-art performance in many established benchmarks <cit.>.



§ MODEL



We propose , a modification of the traditional softmax layer with an updated learning rule. contains the same linear map from the hidden state to the output vocabulary, but learns by smoothing hidden activations into the weight parameters for novel classes whilst concurrently applying gradient descent. This is to facilitate faster binding of novel classes, and improve learning of infrequent classes. We note this corresponds to a learning rule that transitions from Hebbian learning to gradient descent, and we will show that the combination of the two learning rules works better than either one in isolation.

Many of the features of are motivated from memory systems, and the theory of complementary learning systems in the brain <cit.>. During training, the weights corresponding to a given class will initially correspond to a compressed[The memory is compressed because multiple activations corresponding to the same class are smoothed into one vector, instead of being stored separately.] episodic memory store — with new activations memorized and older activations eventually forgotten. 

The parameters of the softmax layer are treated both as regular slow-adapting network parameters through which gradients flow to the rest of the network, and fast-adapting memory slots which are updated sparsely without altering the rest of the network. In comparison to an external memory, the advantage of is that it is simple to implement and requires almost no additional space or computation.

We will describe the learning rule in detail, and contrast the conditional probabilities from to those generated by a non-parametric cache. We also generalize the memorization procedure in Section <ref> as an instance of a secondary fast-learning overfitting procedure with respect to a euclidean objective, and explore several promising variant objective functions. 


 §.§ Update Rule



Given the weights of a linear projection θ∈ℝ^d × m in the final softmax layer of a network, we calculate the gradient descent update with respect to a cross-entropy loss,


    θ̂_t + 0.5[i] ←θ_t[i] - α  (p_i - 1)   h_t     i = y_t 
    θ_t[i] - α  p_i   h_t     i ≠ y_t


where p_i = e^h_t^T θ_i / ∑_j=1^n e^h_t^T θ_j is the probability output from the softmax, and α is the learning rate. In practice the gradient descent update θ̂_t + 0.5 can be calculated with adaptive optimizers, such as RMSProp <cit.>. This is interpolated with the previous layer's hidden activation h_t for the active class y_t,


    θ_t+1[i] ←λ_t   h_t + (1 - λ_t )  θ̂_t+0.5[i]     i = y_t 
    θ̂_t+0.5[i]     i ≠ y_t   ,


as illustrated in Figure <ref>. When λ_t = 1 this corresponds to the rule θ_t+1← h_t ·𝕀{y_t} where 𝕀{y_t}∈ [0, 1]^m is a one-hot target vector. In this case Hebbian update rule, W_ij← x_i x_j for x_i = h_t the hidden output and x_j = 𝕀{y_t} the target. Naturally when λ = 0 this is gradient descent, and so we see is mixture of the two learning rules. All remaining parameters in the model are optimized with gradient descent as usual. 

When mixing the two learning rules, we would like to benefit from fast initial learning of classes that have not been seen many times, along with stable consolidation of frequently seen classes. As such we do not want λ_t to be constant, but instead something that is eventually annealed to zero. We add an additional counter array ∈̧ℤ^m which counts class occurrences, and propose an annealing function of

    λ_t = max(1   / [̧y_t],  γ)  · 𝕀{[̧y_t] < T}

where γ,  T are tuning parameters. T is the number of class occurrences before switching completely to gradient descent and γ is the minimum activation mixing parameter. Although heuristic, we found this worked well in practice vs. a constant λ or pure annealing λ_t = 1 / c[y_t]. If training from scratch, we suggest setting γ = 1 / N_min and T = N_min× (# epochs until convergence) where N_min is the minimum number of occurrences of any class in a training epoch. This is to ensure we smooth over many class examples in a given epoch, and the memorization of activations continues until the representation of h_t stabilizes. We describe the full algorithm in Algorithm <ref>, including details for training with minibatches.




The final layer trains with a two-speed dynamic. For some training steps the full network will be optimized slowly via gradient descent as usual (when frequently-encountered classes are observed), and for other time steps a sparse subset of parameters will rapidly change. The remaining network parameters are optimized with gradient descent.

It is worth noting that simply increasing the learning rate of the softmax layer, or running multiple steps of optimization on rare class inputs, would not achieve the same effect. The value θ[y_t] would indeed be pulled towards a large inner product with h_t, however neighbouring parameters θ[i] ;   i ≠ y_t would be pushed towards a large negative inner product with h_t and this could lead to catastrophic forgetting of previously consolidated classes. Instead we allow gradient descent to slowly push neighbouring parameters away, and thus disambiguate similar classes in a gradual fashion.



 §.§ Relation to cache models


We can consider the weights constructed from the above optimization procedure as a compressed memory storing historic activations. We contrast the output probabilities of with those produced from a non-parametric cache model.

Recall the conditional probability of a class, w, given a cache of previous activations (<ref>). If we set I_w(j) to be the time step of j-th most recent occurrence of w, then we can re-write the cache probability,


    p_c(w | h_t)    ∝∑_i = t - n^t - 1 e^h_t^T h_i𝕀{y_i = w}
        = ∑_j = 1^N_w e^ g(j)   h_t^T h_I_w(j)


where g(j) = -∞ if  j < t - n  and  1 otherwise, is a weighting function which places uniform weight to the attention over classes in the past n time steps. However if we wish to characterize infrequent classes, we may want a weighting scheme with a larger time horizon that has a smooth decay.

If we modified the cache to have infinite memory capacity and used a geometric weighting scheme to decay the contribution of the j-th most recent activation corresponding to the given class, e.g. g(j) = λ  (1 - λ)^j-1, then the resulting conditional probability is,


    p̃_c (w | h_t) ∝∑_j = 1^N_w e^λ  (1 - λ)^j-1  h_t^T h_I_w(j)


where N_w is the total number of occurrences of class w. Let us now consider the conditional probability from for class w, where w has been observed less than T times. If θ has not received large gradients from the occurrence of nearby neighboring classes, and we fix λ_t = λ over time, then (<ref>) gives


    θ_i ≈∑_j = 1^N_wλ  (1 - λ)^j-1 h_I_w(j)  ,


plugging this into our softmax conditional probability,


    p_θ(w | h_t) ∝ e^h_t^T θ_w   ≈ e^h_t^T ∑_j=1^N_wλ  (1 - λ)^j-1 h_I_w(j)
        = ∏_j=1^N_w e^λ  (1 - λ)^j-1 h_t^T h_I_w(j)  .


we see the parametric actually becomes a proxy for the conditional probability output by the non-parametric infinite cache model p̃_c. Past activations now have a geometric contribution to the probability, versus the cache's arithmetic reduction (<ref>). This form is useful because we can compute p_sm much more efficiently than p̃_c and it does not require storing the entire history of past activations.



 §.§ Alternate Objective Functions


We briefly discuss a generalization of the update by casting it as an overfitting procedure to an inner objective function. Recall equation (<ref>) for parameters corresponding to the active class, 


    θ_t+1[i] ←λ_t   h_t + (1 - λ_t)  θ̂_t+0.5[i].


We can re-phrase this as smoothing θ̂_t+0.5[i] with the trivial solution to a euclidean objective function, which we overfit to. 


    θ_t+1[i] ←λ w^* + (1 - λ)  θ̂_t+0.5[i]  
    
        w^* ←_w -||w - h_t||_2


From this perspective we are performing a two-level optimization procedure. The outer optimization loop is the mixture of gradient descent and exponential smoothing, and the inner optimization loop determines a good value for w^* based on the activation h_t and the current parameters.

We consider several other objective functions that are more expensive to compute, but may be preferable to a simple Euclidean distance. Notably, switching to inner product similarity (), and also incorporating a cost to parameter similarity () to push w^* towards h_t but away from neighbouring parameters — to avoid confusion or interference with other classes. As we keep neighbouring parameters fixed, we hope to avoid the catastrophic forgetting typically associated with model overfitting. We list the set of objectives considered,


    w^* ←_w   g(w) 
        g_L2(w) = -||w - h_t||_2 
        g_IP(w) = w^T h_t 
        g_SVM(w) = w^T h_t - -1em ∑_θ_j ∈_k(h_t)-1em ξ  w^T θ_j  · 𝕀(w^T θ_j > ϵ) 
        g_Smax(w)  = e^w^T h_t / ∑_θ_j ∈_k(h_t) e^w^T θ_j


where _k(h_t) refers to the k nearest parameters to the activation h_t that do not correspond to y_t, the class label. Including all M parameters in θ_t would make the inner optimization loop very slow, so we choose a sparse subset k ≪ M. These are all optimized under the hard norm constraint ||w||_2 < 10 with gradient descent for multiple steps, typically 20, at a given point in training. 



§ RESULTS



 §.§ Image Curriculum


We apply to the problem of image classification. We create a simple curriculum task using Omniglot data <cit.>, where a subset of classes (30) are initially provided, and 5 new classes are added when test performance exceeds a threshold (60%). Although this is a toy setup, it allows us to investigate the basic properties of fast class binding without other confounding factors, found in real-world problems.

Omniglot contains handwritten characters from 50 alphabets, totalling 1,623 unique character classes. There are 20 examples per class. We partition the first 5 examples per class to a test set, and assign the rest for training.

We use the same architectural setup as Matching Networks <cit.> where the images are re-sized to 28 × 28 and a 4 layer convolutional neural network is used. Each layer has 64 filters, 3× 3 convolutions, batch normalization, ReLU activations, and 2 × 2 max pooling. Each channel maps the input to a scalar, so the resulting hidden size is 64. All weight parameter in the softmax are initialized with Glorot initialization <cit.>. Models were trained with 20% dropout on the final layer and a small amount of data augmentation was applied to training examples (rotation ∈ [-30, 30], translation) to avoid over-fitting. Otherwise the models quickly plateau on a low level. For the update, we store the pristine hidden activation pre-dropout. Unlike many one-shot Omniglot papers, we do not train in a meta-learning setup — namely, labels are not shuffled between episodes.

We trained the convnet classifier with RMSProp and swept over learning rates α∈ [1e-4, 4e-2] to find the fastest-learning baseline softmax model. A value of α = 8e-3 was the largest learning rate to provide stable learning (see Figure <ref> in Appendix <ref>). We then compared the regular softmax layer with , both placed on the deep convnet.

If we inspect the number of steps spent on each level averaged over 10 seeds, we see in Figure <ref> that the model is noticeably more data efficient after 80 total classes. Although it is still far from one-shot, there is a 1-2X data efficiency gain on average. In Appendix <ref>, Figure <ref> we show the progression of the curriculum in terms of the number of classes shown versus training steps. progresses through the curriculum at identical speed to the softmax until around 1,000 steps of training, from then on it begins to bind new classes and complete the each level faster.







 §.§ Language Modelling

We would like to evaluate in the context of a large-scale classification task, where some classes are infrequently observed. Word-level language modelling is an ideal fit because it satisfies both criteria, and there are established performance benchmarks. Some large-scale language modelling corpora require the use of efficient softmax approximations, such as the adaptive softmax <cit.> or hierarchical softmax <cit.> due to the very large vocabulary size. To reduce confounding factors, we restrict ourselves to applications where the full softmax can be used. We investigate two medium-sized corpora, WikiText-103  which contains just over 100M tokens derived from Wikipedia articles <cit.>, and Gutenberg which contains a subset of open-access texts from Project Gutenberg listed in Appendix <ref>. The idea is that Wikipedia articles should cover factual information, where the style of writing is somewhat consistent and named entities may appear across many articles; whereas books should be more self-contained (unique named entities) and stylistically different. We also consider a very large corpus, GigaWord v5, which is a collection of articles from eight press associations exceeding a decade's worth of global news.



  §.§.§ Model details

For WikiText-103 we swept over LSTM hidden sizes {1024, 2048, 4096}, no. LSTM layers {1, 2}, embedding dropout {0, 0.1, 0.2, 0.3}, use of layer norm <cit.> {True, False}, and whether to share the input/output embedding parameters {True, False} totalling 96 parameters. A single-layer LSTM with 2048 hidden units with tied embedding parameters and an input dropout rate of 0.3 was selected, and we used this same model configuration for the other language corpora. We trained the models on 8 P100 Nvidia GPUs by splitting the batch size into 8 sub-batches, sending them to each GPU and summing the resulting gradients. The total batch size used was 512 and a sequence length of 100 was chosen. We did not pass the state of the LSTM between sequences during training, however the state is passed during evaluation. 





  §.§.§ WikiText-103

The WikiText-103 corpus contains 267,735 unique words and each word occurs at least three times in the training set. We take the best LSTM parameter configuration (described above) as a baseline, and compare it to an identical model where the final layer is replaced with . We swept over the insertion limit parameter T ∈{100, 500, 1000} and discount factor γ∈{0.05, 0.1, 0.25} using the validation set. We found T = 500,  γ = 0.25 worked best, achieving a test perplexity of 34.3 on this dataset (Table <ref>). Inspecting the validation curves in Figure <ref> we see the initially hampers validation performance, until around 2–3B training tokens have been consumed. This makes sense, as storing activations from prior layers of the network is only an effective strategy once the network has rich intermediate representations of its inputs. Inspecting Table <ref> we see the test perplexity broken down by word frequency, the gain in overall performance is obtained from less frequent vocabulary.

We also investigate the model evaluated dynamically on the test using (a) a Neural Cache <cit.> and (b) Memory-based Parameter Adaptation (MbPA) <cit.>. Hyper-parameter details for these models are detailed in Appendix <ref>. The cache reduces the test perplexity by 1.6 for the LSTM and 4.4 for LSTM + . The addition of MbPA reaches a test perplexity of 29.2 which is, to the authors' knowledge, state-of-the-art at time of writing.






  §.§.§ Gutenberg

Books provide several different linguistic challenges to articles. The style of writing is intentionally varied between authors, and named entities can be wholly fictional — confined to a single text. We extract a subset of English-language books from the corpus, strip the Gutenberg headers and tokenize the text (Appendix <ref>). We select a dataset of comparable size to WikiText-103; 2042 books in total with 2017 training books (175,181,505 tokens), 12 validation books (609,545 tokens), and 13 test books (526,646 tokens) — see Appendix <ref> for full details. We select all words that occur at least five times in the training set, a total vocabulary of 242,621 and map the remainder to an unk token.


We use the same LSTM hyper-parameters as those chosen from the wikipedia sweep, and compare against with T=100, T=500 and γ = 0.1. Figure <ref> in Appendix <ref> shows the validation performance after 15B steps of training, equating to roughly 80 epochs and 6 days of training with 8 P100s training synchronously. After approximately 4B steps of training the softmax performance is surpassed, and this gap widens even up to 15B steps to a gap of 2-3 points in perplexity. Similar to WikiText-103, we see in Table <ref> the gain in perplexity is more pronounced over less frequent words.




 §.§ GigaWord v5

We evaluate on a large-scale language modelling corpus. GigaWord is interesting because it is a vast collection of news articles, and there is a natural temporal order. We pre-process the dataset (Appendix <ref>), select all articles from 2000–2009 for the training set, and test on all articles from 2010. The total number of training tokens is 4.0B and the total number of test tokens is 260M. The total unique tokens (after pre-processing) for the training set reaches 6M, however for parity with the other experiments we choose a vocabulary size of 250K. We use the same LSTM hyper-parameters and hyper-parameters, and train the model for 6B steps, after which the models plateau in evaluation performance. We observe a 9.8-point drop in perplexity, from 53.5 to 43.7, illustrated in Table <ref>. 



 §.§ Alternate Objective Functions

We test out some of the alternate inner objective functions described in Section <ref>. These are described in (<ref>), and  the inner objective functions include Euclidean, Inner Product, SVM, (sparse) Softmax. These could be applied to any of the described experiments, we chose the WikiText-103 language modelling task because it is more comparable to prior work.

Although more expressive objective functions appear promising, in practice we see (Figure <ref> in Appendix <ref>) that validation performance is roughly equivalent between all inner objective functions. This suggests the network activation h_t naturally do not land too close to other class parameters, and the norm of activations is not too large or small, in comparison to the model parameters θ. The latter may be due to the use of layer normalization from the LSTM.



§ RELATED WORK

Few-shot classification has been investigated in a meta-learning setup with a mixture model of a parametric neural network and a non-parametric memory <cit.>. Here, a subset of classes are used with permuted labels per episode, activations are stored to memory, and gradients are passed through the memory. This allows the network to shape its activations to be conducive to accurate retrieval and classification. Here, we do not meta-learn the activations stored into network parameters and instead rely on their representation being rich enough from regular parametric training. We do this to avoid backpropagating through time to the point of writing to memory, as the parameters may contain memories stored millions of time steps ago in the case of rare words.

In natural language processing memory-augmented models have been shown to improve the modelling of unknown words and adaptation to new domains <cit.>. However in these works the memory is typically small and models the recent past. During evaluation the test activations and corresponding labels are stored in memory, and the model is evaluated dynamically — adapting to the test data on the fly. Whilst dynamic evaluation provides insights into domain transfer, it is limited in applicability as the model may not receive ground-truth labels when launched into production.

More recent work has investigated methods of memorizing and searching over the training set to enhance performance <cit.>. These approaches typically require complex engineering to efficiently index this memory store. Part of the benefit of is implementation simplicity.

Prior literature on the softmax operator for language modelling computational efficiency <cit.> or tricks such as smoothing across many softmax layers <cit.>. However these do not focus on increasing the data-efficiency or faster learning of infrequent classes.

Other architectures have been considered for fast learning, such as the `fast weights' auto-associative memory <cit.>. This focuses on fast adaptation to recent information that persists over a short window of time. The LEABRA architecture <cit.> contains a mixture of contrastive Hebbian learning (GENEREC) <cit.> and gradient descent for fast and slow learning, however this cognitively-inspired model has not been shown to scale to large-scale classification problems. 



§ DISCUSSION

This paper explores one way in which we can achieve fast parametric learning in neural networks, and preserve this knowledge over time. We show that activation memorization is useful for vision in the binding of newly introduced classes, beating a well tuned adaptive learning rate optimizer, RMSProp. 

For language we show improvement in the modelling of text with an extensive vocabulary. In the latter we show the model beats a very strong LSTM benchmark on three stylistically different corpora, and achieves state of the art on WikiText-103. This is achieved with effectively no additional compute or memory resources. Breaking down perplexity over word frequency bucket, we see that less frequent words are better modelled, as hypothesized. We suggest that could be applied to any classification domain with infrequent classes, or non-stationary data. It may also be useful in quickly adapting a pre-trained classifier to a new task / set of classes — however this is beyond the scope of our initial investigation.

It would also be interesting to explore activation memorization deeper within the network, and thus in more general scenarios to classification. In this case, there is no direct feedback from a ground-truth class label and the update rule would not necessarily be an instance of Hebbian learning. A natural first step would be to generalize the ideas to large-scale softmax operators that are internal to the network — such as attention over a large memory.



§ ACKNOWLEDGEMENTS


The authors would like to thank Gabor Melis, Greg Wayne, Oriol Vinyals, Pablo Sprechmann, Siddhant Jayakumar, Charles Blundell, Koray Kavukcuoglu, Shakir Mohamed, Adam Santoro, Phil Blunsom, Felix Hill, Angeliki Lazaridou, James Martens, Ozlem Aslan, Guillaume Desjardins, and Chloe Hillier for their comments and assistance during the course of this project. Peter Dayan is currently on a leave of absence at Uber Technologies; Uber was not involved in this study.




icml2018










§ APPENDIX




§ LANGUAGE MODELLING




 §.§ WikiText-103





  §.§.§ Dynamic Evaluation Parameters

For the Neural Cache, we swept over the hyper-parameters:


  * Softmax inverse temperature: θ_cache∈{0.1, 0.2, 0.3}

  * Cache output interpolation: λ_cache∈{0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35}

  * Cache size n_cache∈{1000, 5000, 8000, 9000, 10000}

and chose θ_cache = 0.3,  λ_cache = 0.1,   n_cache = 10000 by sweeping over the validation set. 

For the mixture of Neural Cache and MbPA we swept over the same cache parameters, alongside:



  * MbPA output interpolation: λ_mbpa∈{0.02, 0.04, 0.06, 0.08, 0.10},

  * Number of neighbours retrieved from memory: K ∈{512, 1024},

  * Number of MbPA steps: n_mbpa∈{1, 2 }


and selected λ_mbpa = 0.04, λ_cache = 0.1, θ_cache = 0.3, K = 1024, n_mbpa = 1, n_cache = 10000. We also selected the MbPA learning rate α_lr = 0.3, and the L2-regularization β_mbpa = 0.5 on the MbPA-modified parameters. The memory size for MbPA was chosen to be equal to the cache size.




 §.§ Gutenberg






  §.§.§ Splits


We downloaded a subset of books (listed below) from Project Gutenberg on January 2, 2018 from a mirror site (<https://www.gutenberg.org/MIRRORS.ALL>). We selected 2042 English-language books under the /1 subdirectory. Each book has a unique id, we shuffled the books and split out a reasonably sized train, validation and test set. The book ids are listed below for these splits.

Test (13 books, 526,646 tokens):



Validation (12 books, 609,545 tokens):



Train (2017 books, 175,181,505 tokens)









 §.§ GigaWord







 §.§ Alternate Objective Functions








 §.§ Data pre-processing


For Project Gutenberg and GigaWord v5 we used a very simple python script to pre-process and tokenize the data using NLTK. We post the Gutenberg script here for ease of reproduction. The GigaWord v5 script excludes the Project Gutenberg-specific selection of start / end markers to extract the text. The NLTK library [<https://www.nltk.org/>] is used to split out sentence and word tokens, the resulting text contains lower-case text with one sentence per line.








§ OMNIGLOT








