Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We argue that symmetry is an important consideration in addressing the problem of systematicity and investigate two forms of symmetry relevant to symbolic processes. We implement this approach in terms of convolution and show that it can be used to achieve effective generalisation in three toy problems: rule learning, composition and grammar learning. Convolution has been an incredibly effective element in making Deep Learning successful. Applying the same set of filters across all positions in an image captures an important characteristic of the processes that generate the objects depicted in them, namely the translational symmetry of the underlying laws of nature. Given the impact of these architectures, researchers are increasingly interested in finding approaches that can be used to exploit further symmetries , such as rotation or scale. Here, we will investigate symmetries relevant to symbolic processing. We show that incorporating symmetries derived from symbolic processes into neural architectures allows them to generalise more robustly on tasks that require handling elements and structures that were not seen at training time. Specifically, we construct convolution-based models that outperform standard approaches on the rule learning task of , a simplified form of the SCAN task and a simple context free language learning task. Symbolic architectures form the main alternative to conventional neural networks as models of intelligent behaviour, and have distinct characteristics and abilities. Specifically, they form representations in terms of structured combinations of atomic symbols. Their power comes not from the atomic symbols themselves, which are essentially arbitrary, but from the ability to construct and transform complex structures. This allows symbolic processing to happen without regard to the meaning of the symbols themselves, expressed in the formalist's motto as If you take care of the syntax, the semantics will take care of itself . From this point of view, thought is a form of algebra (James, 1890; Boole, 1854) in which formal rules operate over symbolic expressions, without regard to the values of the variables they contain . As a consequence, those values can be processed systematically across all the contexts they occur in. So, for example, we do not need to know who Socrates is or even what mortal means in order to draw a valid from All men are mortal and Socrates is a man. However, connectionist approaches have been criticised as lacking this systematicity. claimed that neural networks lack the inherent ability to model the fact that cognitive capacities always exhibit certain symmetries, so that the ability to entertain a given thought implies the ability to entertain thoughts with semantically related contents. Thus, understanding these symmetries and designing neural architectures around them may enable us to build systems that demonstrate this systematicity. However, the concept of systematicity has itself drawn scrutiny and criticism from a range of researchers interested in real human cognition and behaviour. argue that the definition is too vague. Nonetheless, understanding the symmetries of symbolic processes is likely to be fruitful in itself, even where human cognition fails to fully embody that idealisation. We investigate two kinds of symmetry, relating to permutations of symbols and to equivalence between memory slots. The relation between symbols and their referents is, in principle, arbitrary, and any permutation of this correspondence is therefore a symmetry of the system. More simply, the names we give to things do not matter, and we should be able to get equivalent whether we call it rose or trandafir, as long as we do so consistently. Following on from that, a given symbol should be treated consistently wherever we find it. This can be thought of as a form of symmetry over the various slots within the data structures, such as stacks and queues, where symbols can be stored. We explore these questions using a number of small toy problems and compare the performance of architectures with and without the relevant symmetries. In each case, we use convolution as the means of implementing the symmetry, which, in practical terms, allows us to rely only on standard deep learning components. In addition, this approach opens up novel uses for convolutional architectures, and suggests connections between symbolic processes and spatial representations. The first symmetry to be considered is the one arising from the fact that the correspondence between referring atomic symbols and their referents is entirely arbitrary, which entails that permutations of these symbols are symmetries of the system. In fact, use this permutation invariance to define the difference between the logical and non-logical parts of a language. This symmetry, in which any symbol is as good as any other, is anathema to the sort of problem that neural nets are typically applied to, in which the inputs are not arbitrary names but specific measurement values, e.g. images or medical records. In that case, effective learning requires discovering the appropriate differentiations, and would be sabotaged by randomly permuting the inputs. Nonetheless, the work of suggest that this symmetry may be relevant to human cognition, even for infants as young as 7 months. In these experiments, the infants were habituated to sequences of syllables which obeyed a simple rule, such as ABB (e.g. la ti ti) or ABA (e.g. la ti la). Subsequent testing on novel stimuli showed they were able generalise this rule to syllables not present in the training stimuli (e.g. wo fe fe or wo fe wo). In other words the representation of the learned rule allowed it to be abstracted from the particular training stimuli and applied to any syllable. One interpretation is that the infants were treating the stimuli symbolically, in that one syllable was as good as any other, and that as a consequence their behaviour was symmetric under replacement of the syllables. were unable to obtain the same behaviour from a recurrent network architecture, because the statistical regularities it learned were linked to the specific syllables seen at training time and so generalisation to unseen syllables was not achieved. Here we show that this problem can be solved by imposing a symmetry on the architecture, that corresponds to weight sharing between syllables. Practically, this is implemented as a one dimensional convolution of width one, followed by maxpooling across all syllables, and a softmax to produce output probabilities. The input consists of a 12 × 3 array of binary values representing the 12 syllables and 3 time steps, with convolution treating the syllables as positions and the time steps as channels. The output of the convolution has two channels, which after pooling become the logits for the binary outputs. Note that this the opposite of how convolutions are most frequently used in application to language sequences. In that case, invariance to time translations is achieved by weight sharing across time steps, with the representation of each symbol being encoded in the channels. Here, in contrast, weight sharing happens between symbols and temporal information is encoded in the channels. This means our model is not invariant to translations in time, but is instead invariant to permutations of symbols. Figure 1 shows the input sequence wo fe wo being processed by this architecture. The input is first encoded as activation in the first and third channels at the wo position, and activation in the second channel at the fe position. Convolution reduces these three channels down to two, and pooling projects this down to a pair of logits corresponding to the ABB and ABA categories. The training and test inputs are taken from the paper, and we train the model to distinguish ABB sequences from ABA sequences. We also train a multi-layer perceptron and a recurrent net on the same data, with the recurrence happening over the time dimension. The architecture applied to the rule learning task of , consisting of convolution followed by max-pooling and softmax. The on the test set in Table 1 show that neither the multi-layer perceptron nor the recurrent network learn a rule that generalises effectively to unseen syllables. However, the weight sharing in the convolutional net requires that the same function is applied to each syllable, giving perfect generalisation. The filter, being applied at every position, cannot discriminate between syllables, and instead can only respond to the information about temporal structure in the channels. So, for example, in the case of the sequence wo fe wo, the input channels at the wo position take the values 101, representing the fact that the same token occurs in the first and third temporal slots. This is very similar to what suggest is being learned by the infants: algebra-like rules that represent relationships between placeholders (variables), such as'the first item X is the same as the third item Y'. Thus, by imposing a symmetry on the network, we learn functions that are sensitive to an abstract structure rather than the specific raw syllables in the input. Thus, convolution provides a straightforward solution to this long-standing problem using the simple expedient of sharing weights between seen and unseen syllables. Moreover, we can make further use of the insight that symmetry allows us to abstract structure away from the particular content it contains. 3 investigated the systematicity of recurrent networks in terms of a task they call SCAN. This requires translating a sequence of instructions -such as turn left and jump twice after walk -into a sequence of actions -such as LTURN WALK JUMP JUMP. They found that while a sequence-to-sequence architecture could achieve near perfect scores when the test instances closely resembled the training data, performance broke down when generalisation outside the training distribution was required. In particular, the network struggled to translate sequences that were longer than those in the training data and those that contained novel combinations of words, e.g. jump twice when both jump and twice had been seen, but their combination had not. A full solution for the SCAN task requires methods for handling a number of tricky problems in a systematic manner, e.g. parsing the instruction sequence, composing items, reversing the order of time Hidden Units Output Units jump two JUMP JUMP The encoder-decoder architecture applied to the composition task. For each input word, two embeddings are learned, one of which is concatenated with the current hidden state and the other forms the filter in a convolution applied to that input. In the decoder, actions are predicted by taking a convolution of the hidden state, and recurrence between time steps is also a convolution. constituents linked by after. Instead, we focus on a single problem: that of learning to compose an instruction such as jump with a modifier such as twice in a way that generalises systematically. Specifically, we construct a dataset in which the inputs are two word instructions drawn from a vocabulary of 10 commands (jump, walk, run, . . .) and 10 modifiers (one, two, three, . . .). Each instruction is to be translated into the corresponding action repeated the appropriate number of times, e.g. jump four → JUMP JUMP JUMP JUMP. We randomly sample with replacement 1000 such translation pairs, choose three combinations and remove all instances of them from the training data and then exclusively test on these unseen pairings of command and modifier. For a network to behave systematically, it must learn to associate a modifier, e.g. four, with a structure, e.g. repeat the same action four times, that applies to all actions indiscriminately. Following the lesson learned in the previous experiment, we might expect that the structural information might be best represented in the channels and filter of a convolution, and the actions as positions which are then treated symmetrically. However, rather than hard-code this distinction into the encoder network innately, we allow a recurrent-convolutional architecture to discover this approach for itself. Each symbol is given two input representations, one of which is used as a filter in a width one convolution and the other of which feeds into the data that the convolution is applied to. Our intention is that the former should represent structural information (how many times to repeat) and the latter represent the content (which action) within that structure. As shown in Figure 2, information flows through the model from these inputs to the output logits through a series of convolutions, which impose a permutation symmetry on the function learned in training. This invariance to permutations of the output symbols should permit the model to learn representations of structure which abstract away from the particular actions seen during training. Within the recurrent-convolutional architecture, the hidden states, h t, consist of an array of units comprising 5 channels by 11 positions, and one dimensional convolutions of width one are used as the basis for both recurrence between time steps and also to project the output logits from the hidden states. In the encoder, two embeddings x and y are learned for every word, one of which is concatenated with the hidden state, to become the 6th channel, and the other is used as the filter in the convolution that produces the next hidden state. In the decoder, two convolutional filters are learned, one of which projects the 5 hidden channels down to a single channel, to form the output logits, and the other predicts the new hidden state from the old at each time step. The targets are encoded as one-hot values in an 11-dimensional vector (10 actions + pad), and the loss is the cross entropy with the predictions. We also add an L 1 regularizer to the loss. Appendix B describes this architecture in more mathematical detail. The in Table 2 demonstrates that only the architecture exploiting the convolutional symmetry generalizes systematically. The net learns to associate a command, e.g. jump, with an action, e.g. JUMP, and a modifier, e.g. two, with an abstract structure, e.g. repeat the same thing twice. This is possible because the symmetry across symbols allows a structure to be represented in a way that makes no reference to the particular symbols instantiating that structure. The structures considered in the previous section are extremely simple, having only short-range sequential dependencies. In contrast, the real grammars of natural languages produce long-range dependencies within hierarchical structures. Handling such structures, in which multiple dependencies are embedded within each other, will typically require some form of memory in order to keep track of the unresolved outer dependencies until the inner dependencies are completed. In this section we consider the role that symmetry plays in structuring this memory, again using convolution, which allows us to separate the contents of memory from the structure of how it is manipulated. In particular, we consider a reverse recall task that captures a key property of how such a memory has to operate. For example, in the sentence The racquet is actually very cheap the subject noun, racquet, and main verb, is, display number agreement. In this case, both are singular, but they could also have been plural, i.e. racquets and are. For this simple sentence, noun and verb are adjacent so the span of the dependency is minimal. However, we can, in principle, insert as much material as we like, and this syntactic connection persists. In particular, we can add a relative clause to the noun: The racquet that the tennis player uses is actually very cheap. Now another subject noun, player, and verb, uses, intervenes between the first pair, but the dependency, and in particular the number agreement, between racquet and is has to be maintained. This remains true even when we insert another relative clause: The racquet that the tennis player we are all in awe of uses is actually very cheap. In this last case, the subject, we, and verb, are, are both plural, and effective processing of the whole sentence requires that this should not disrupt processing of the outer singular dependencies. In other words, a language user must be able to maintain a trace of multiple open dependencies until the relevant material is encountered. Notably, in the case of a centre embedded construction such as the sentence above, recall has to happen in a last-in-first-out manner as processing descends through the hierarchy and then rises back out again. That is, the subjects in the sentence above -racquet, player, we -are matched to verbs in reverse order -are, uses, is. A common model for such structures are Context Free Grammars (CFG), which generate sentences in terms of production rules, such as those in Figure 3. These rules describe how the start symbol, S, is expanded into sequences of terminals symbols, such as adoda or ccbabdodbabcc. Each rule describes a substitution that can be applied to a single non-terminal symbol, i.e. S, A, B, C or D to yield a sequence of symbols. The context free aspect of such a grammar lies in the fact the substitutions are made without regard for the context around the original non-terminal symbol. In the case of the grammar described in Figure 3, the substitutions applied to the non-terminals A, B, C and D yield a new string with the same terminal at the beginning and the end, and the final rule inserts an o. As a consequence, the ing strings are palindromes with a single o at their centre. An equivalent model is a pushdown automata (PDA), in which hierarchical structure is handled by pushing symbols representing the outer structures onto a stack, until the interior structure is completed, and then popping symbols back off the stack to move outward in the hierarchy until no (a) Production rules for a simple CFG. Figure 3: A simple CFG, producing palindromic strings. more symbols remain. Crucially, the stack has a last-in-first-out structure that essentially returns items in the reverse order in which they were pushed onto it. In principle, such a system can handle sentences of unbounded length, containing arbitrarily long dependencies between constituents. In practice, however, language users struggle with nested structures more than two or three levels deep. Moreover, it is generally accepted that an ordinary CFG is not an accurate model of the grammatical structures that natural languages display. Instead, their grammars appear to be mildly context sensitive, as shown by evidence from Swiss German and Dutch . However, here we focus on CFGs and the ability of Recurrent Neural Nets to learn these structures. In particular, we investigate the ability of a Long Short Term Memory network to learn the simple palindromic language over the terminal symbols a, b, c, d and o, defined by the grammar in Figure 3. We train an LSTM containing 100 hidden units, on 100,000 examples of strings of length 15, 17, 19, 21, 23 and 25, and then perform an in-domain test on novel strings of the same length, and an out-of-domain test on longer strings of lengths 29, 33, and 37. We also retrain the LSTM after removing all strings which contain more than 4 tokens of the symbol a from the training set, and then test only on examples from the test set containing more than 4 tokens of the symbol a. The first row of Table 3 gives the for this evaluation in terms of the proportion of symbols after the central o symbol that were predicted correctly. The in-domain , in the first column, make it clear that the net has learned the structure of the grammar, and how to make accurate predictions, at least for sequences of lengths seen at training time. The second column, containing the for longer sequences, shows that generalisation outside the range of the training set is not robust. The third column indicates that the model has difficulty generalising to sequences where the nesting of A symbols is deeper than at training time, even though the actual length of sequences is unchanged. These failures of generalisation can be seen as symptoms of the same underlying problem. In particular, success on each of these out-of-domain tasks simply requires extending the application of the same rules. However, the problem arises because the network lacks the required concept of sameness. The LSTM cells form an unstructured memory resource, without any notion of two cells containing the same information. Nor can there be a meaning to the idea of applying the same rule to that information. The parameters for each cell are learned independently, and so each cell carries out its own isolated task. During training these cells do learn to behave as a coherent whole, achieving impressive in-domain performance, but there is no way for the model to apply the same rule to the n th item as it applied to the previous n − 1. To address this shortcoming, we propose to organize the memory cells into an ordered onedimensional stack structure, and to use convolutions to control the flow of information across timesteps, replacing the forget gate. The translational symmetry of this convolutional layer gives meaning to the idea of the same symbol being able to be stored in different cells and we use a filter As shown in Figure 4, each token in the input is given an embedding which is then concatenated with the current hidden state. Together these values form the inputs to units that control the flow of information into, out of and between the memory cells, as in a standard LSTM. In this case, however, the memory cells are a set of one channel one dimensional convolutional layers and the forget gate has been replaced with a set of width three filters that shape the recurrent flow of information. These filters are the softmax outputs of units driven by the concatenated input embeddings and hidden units, allowing the network to use the input context to control the memory cell stack. Information is written to and read out from only the bottom entries in the stack, using standard input and output gates. This architecture is described in more mathematical detail in Appendix C. Performance of this convolutional LSTM on the same evaluations is given in the second row of Table 3. There, all three columns show optimal performance on both the in-domain and out-ofdomain tasks, demonstrating the utility of the convolutional layer in helping the model to generalise robustly. Numerous authors have tackled the problem of replicating the rule learning behaviour studied by in a connectionist system, and give an extensive review. Many of these approaches rely on a specific training regime to obtain the desired behaviour, rather than our approach of modifying the architecture to embed the appropriate capacities innately. However, our core intention was to demonstrate the relevance of symmetry, with convolution being a convenient and transparent means to that end. The same end could conceivably be achieved purely through learning. The SCAN task of has also stimulated a number of responses which attempt to obtain the required systematicity. uses a meta learning approach, which employed a training regime that explicitly permuted the correspondence between symbols and their meaning, i.e. between instructions and actions. This could be seen as learning an invariance to these permutations, rather than specifying it innately in the form of convolution. , instead, take an approach based on learning separate semantic and syntactic representations for each word, which is comparable to our approach of learning two embeddings for each word. The recurrent PDA we describe in Section 4 is very similar to a number of other architectures. proposed a neural network pushdown automata. proposed architectures for a number of data structures: queues, dequeues and stacks. proposed a recurrent stack structure, which in practice, is almost equivalent to our proposal. However, none of these works discuss the role of symmetry or the connection to convolution. Symmetries beyond spatial translation have been discussed by a number of authors. propose a generalisation of convolution for arbitrary discrete symmetries, such as reflections and rotations. The role of invariances in disentangled representations is discussed by , and investigate the application of probabilistic symmetries to neural network architectures. Practical examples of symmetries supporting extrapolation and generalisation beyond the training set are discussed by. Permutation invariance is relevant to a number of representational strategies, such as bag-of-words approaches (e.g.) or Deep Sets . However, the relevant symmetry in these cases is usually over permutations on the order of inputs, e.g. a symmetry between wo fe fe and fe wo fe. In our case, the permutation is over the identity of the symbols, i.e. a symmetry between wo fe fe and la ti ti. One way to address the criticisms of distributed approaches raised by has been to focus on methods for binding and combining multiple representations (; ; ; ;) in order to handle constituent structure more effectively. Here, we instead examined the role of symmetry in the systematicity of how those representations are processed, using a few simple proof-of-concept problems. We showed that imposing a symmetry on the architecture was effective in obtaining the desired form of generalisation when learning simple rules, composing representations and learning grammars. In particular, we discussed two forms of symmetry relevant to the processing of symbols, corresponding respectively to the fact that all atomic symbols are essentially equivalent and the fact that any given symbol can be represented in multiple places, yet retain the same meaning. The first of these gives rise to a symmetry under permutations of these symbols, which allows generalisation to occur from one symbol to another. The second gives rise to a symmetry across memory locations, which allows generalisation from simple structures to more complex ones. On all the problems, we implemented the symmetries using convolution. From a practical point of view, this allowed us to build networks using only long-accepted components from the standard neural toolkit. From a theoretical point of view, however, this implementation decision draws a connection between the cognition of space and the cognition of symbols. The translational invariance of space is probably the most significant and familiar example of symmetry we encounter in our natural environment. As such it forms a sensible foundation on which to build an understanding of other symmetries. In fact, use invariances under various spatial transformations within geometry as a starting point for their definition of logical notion in terms of invariance under all permutations. Moreover, from an evolutionary perspective, it is also plausible that there are common origins behind the mechanisms that support the exploitation of a variety of different symmetries, including potentially spatial and symbolic. In addition, recent research supports the idea that cerebral structures historically associated with the representation of spatial structure, such as the hippocampus and entorhinal cortex, also play a role in representing more general relational structures . Thus, our use of convolution is not merely a detail of implementation, but also an illustration of how spatial symmetries might relate to more abstract domains. In particular, the recursive push down automata, discussed in Section 4, utilises push and pop operations that relate fairly transparently to spatial translations. Of course, a variety of other symmetries, beyond translations, are likely to be important in human cognition, and an important challenge for future research will be to understand how symmetries are discovered and learned empirically, rather than being innately specified. A common theme in our exploration of symmetry, was the ability it conferred to separate content from structure. Imposing a symmetry across symbols or memory locations, allowed us to abstract away from the particular content represented to represent the structure containing it. So, for example the grammar rule learned by our network on the syllable sequences of was able to generalise from seen to unseen syllables because it represented the abstract structure of ABB and ABA sequences, without reference to the particular syllables involved. We explored how this ability could also be exploited on composition and grammar learning tasks, but it is likely that there are many other situations where such a mechanism would be useful. Informally, a symmetry of a system is a mapping of the system onto itself which preserves the fundamental properties of the system. In the case of translation symmetry, we have input images, x, and output labels, y, and we want to learn a function, f (x), which predicts these labels, and is invariant to spatial translations, T. That is, we want f to obey f (T x) = f (x). Typically, we achieve this by composing two types of function: equivariant convolutions, c, and invariant poolings, p. Equivariant here means that the output from a translated input is itself the translation of the original output: c(T x) = T c(x). While invariant means the output is unchanged by input translations: When the width of the convolution is reduced to one, the function, c, becomes equivariant to all permutations, S, not just translations: c(Sx) = Sc(x). Permutation equivariance arises, for example, in formal logic, where the rules of deduction depend not on the particular names within an expression, but on its logical structure. So, Socrates is mortal follows from Socrates is a man and All men are mortal not because of the meaning of Socrates or mortal, but because the syllogism has the right form. Thus, if x represents the premises and y represents the , then the process of deduction d(x) = y should be equivariant under any substitution, S, of names. That is, we should be able to reach an equivalent even if we rename Socrates as Bob, and so d(Sx) = Sd(x). Symmetries also arise in computational processes. For example, if x is the state of some machine containing an addressable memory and running a program that references various addresses then we want the behaviour of the machine to be equivalent if we utilise a different set of memory addresses. That is, the state transition function, f , should be equivariant under permutations, S, of states, which apply equivalent permutations of the memory content and the addresses referenced in the program: f (Sx) = Sf (x). Our approach to the composition task is based on the encoder-decoder architecture shown in Figure 2. This is made more precise in the equations below. In the encoder, each token in the input is given an N-dimensional embedding, x and an M+1 × M dimensional embedding, y. The former is concatenated to the current N × M dimensional hidden state to become the M+1th channel of a vector, g. A convolution of width one is applied to this vector to generate the next hidden state, using the second embedding as filter. The decoder takes the final hidden state from the encoder and projects it down to a single channel, using a width one convolution. These are then the logits of a softmax output, o, predicting the next action. The next hidden state is also produced by a width one convolution, in this case maintaining the number of channels. An L 1 regularisation is added to the cross entropy between the predictions, o, and a one hot encoding, a, of the correct actions. Loss = H(a, o) + λ x |x| 1 + λ y |y| 1 For the composition task, we used N=11 and M=5. C CONTEXT FREE LANGUAGE LEARNING Figure 4 gives a visual overview of the architecture we apply to learning the simple palindromic language. This is essentially a modified LSTM in which the forget gate has been replaced with convolutional filters. We define this explicitly below. Each token in the input is given an M-dimensional embedding, x, which is concatenated with the current N-dimensional hidden state, h, to give a vector, g, representing the current context. This controls N single channel width three filters, f, each of which is the output of a softmax. f n,t = softmax (W f,n g t + b f,n) (As in a standard LSTM, values written to the cells are the outputs of tanh units gated by a sigmoid. And the output gate is also a sigmoid function. Recurrence between the cells in each of the N memory stacks is based on one-dimensional convolution and controlled by the filters f . c n,t = f n,t c n,t−1 The 0 th values in each stack are updated using the values of i. c n,t,0 = c n,t,0 + i t,n And the new hidden state are also read out from the 0 th values, gated by tanh units. h t+1,n = o t,n · tanh (c n,t,0) The final outputs, to predict a one hot vector representing the next symbol, apply a softmax to these new hidden states. The loss is the cross entropy over the second half of the sequence, and for this task M and N are both 10, with the memory stack having a depth of 20.

Represent the TLDR for finding the corresponding scientific article.

We use convolution to make neural networks behave more like symbolic systems.

A structured latent-variable approach that adds discrete control states within a standard autoregressive neural paradigm to provide arbitrary grounding of internal model decisions, without sacrificing any representational power of neural models.


Question: Represent the scientific article to find a short TLDR

Question: Key equatorial climate phenomena such as QBO and ENSO have never been adequately explained as deterministic processes. This in spite of recent research showing growing evidence of predictable behavior. This study applies the fundamental Laplace tidal equations with simplifying assumptions along the equator — i.e. no Coriolis force and a small angle approximation. The solutions to the partial differential equations are highly non-linear related to Navier-Stokes and only search approaches can be used to fit to the data.

Analytical Formulation of Equatorial Standing Wave Phenomena: Application to QBO and ENSO

We establish that the scaling laws derived in (Bora et al., 2017) are optimal or near-optimal in the absence of further assumptions.


Question: Represent the article for finding a single-sentence TLDR

Question: In the field of Continual Learning, the objective is to learn several tasks one after the other without access to the data from previous tasks. Several solutions have been proposed to tackle this problem but they usually assume that the user knows which of the tasks to perform at test time on a particular sample, or rely on small samples from previous data and most of them suffer of a substantial drop in accuracy when updated with batches of only one class at a time. In this article, we propose a new method, OvA-INN, which is able to learn one class at a time and without storing any of the previous data. To achieve this, for each class, we train a specific Invertible Neural Network to output the zero vector for its class. At test time, we can predict the class of a sample by identifying which network outputs the vector with the smallest norm. With this method, we show that we can take advantage of pretrained models by stacking an invertible network on top of a features extractor. This way, we are able to outperform state-of-the-art approaches that rely on features learning for the Continual Learning of MNIST and CIFAR-100 datasets. In our experiments, we are reaching 72% accuracy on CIFAR-100 after training our model one class at a time. A typical Deep Learning workflow consists in gathering data, training a model on this data and finally deploying the model in the real world . If one would need to update the model with new data, it would require to merge the old and new data and process a training from scratch on this new dataset. Nevertheless, there are circumstances where this method may not apply. For example, it may not be possible to store the old data because of privacy issues (health records, sensible data) or memory limitations (embedded systems, very large datasets). In order to address those limitations, recent works propose a variety of approaches in a setting called Continual Learning . In Continual Learning, we aim to learn the parameters w of a model on a sequence of datasets with the inputs x j i ∈ X i and the labels y j i ∈ Y i, to predict p(y * |w, x *) for an unseen pair (x *, y *). The training has to be done on each dataset, one after the other, without the possibility to reuse previous datasets. The performance of a Continual Learning algorithm can then be measured with two protocols: multi-head or single-head. In the multi-head scenario, the task identifier i is known at test time. For evaluating performances on task i, the set of all possible labels is then Y = Y i. Whilst in the single-head scenario, the task identifier is unknown, in that case we have Y = ∪ N i=1 Y i with N the number of tasks learned so far. For example, let us say that the goal is to learn MNIST sequentially with two batches: using only the data from the first five classes and then only the data from the remaining five other classes. In multi-head learning, one asks at test time to be able to recognize samples of 0-4 among the classes 0-4 and samples of 5-9 among classes 5-9. On the other hand, in single-head learning, one can not assume from which batch a sample is coming from, hence the need to be able to recognize any samples of 0-9 among classes 0-9. Although the former one has received the most attention from researchers, the last one fits better to the desiderata of a Continual Learning system as expressed in and (van de). The single-head scenario is also notoriously harder than its multi-head counterpart and is the focus of the present work. Updating the parameters with data from a new dataset exposes the model to drastically deteriorate its performance on previous data, a phenomenon known as catastrophic forgetting . To alleviate this problem, researchers have proposed a variety of approaches such as storing a few samples from previous datasets , adding distillation regularization , updating the parameters according to their usefulness on previous datasets , using a generative model to produce samples from previous datasets . Despite those efforts toward a more realistic setting of Continual Learning, one can notice that, most of the time, are proposed in the case of a sequence of batches of multiple classes. This scenario often ends up with better accuracy (because the learning procedure highly benefits of the diversity of classes to find the best tuning of parameters) but it does not illustrate the behavior of those methods in the worst case scenario. In fact, Continual Learning algorithms should be robust in the size of the batch of classes. In this work, we propose to implement a method specially designed to handle the case where each task consists of only one class. It will therefore be evaluated in the single-head scenario. Our approach, named One-versus-All Invertible Neural Networks (OvA-INN), is based on an invertible neural network architecture proposed by. We use it in a One-versus-All strategy: each network is trained to make a prediction of a class and the most confident one on a sample is used to identify the class of the sample. In contrast to most other methods, the training phase of each class can be independently executed from one another. The contributions of our work are: (i) a new approach for Continual Learning with one class per batch; (ii) a neural architecture based on Invertible Networks that does not require to store any of the previous data; (iii) state-of-the-art on several tasks of Continual Learning for Computer Vision (CIFAR-100, MNIST) in this setting. We start by reviewing the closest methods to our approach in Section 2, then explain our method in Section 3, analyse its performances in Section 4 and identify limitations and possible extensions in Section 5. Generative models Inspired by biological mechanisms such as the hippocampal system that rapidly encodes recent experiences and the memory of the neocortex that is consolidated during sleep phases, a natural approach is to produce samples of previous data that can be added to the new data to learn a new task. FearNet relies on an architecture based on an autoencoder, whereas Deep Generative Replay and Parameter Generation and Model Adaptation propose to use a generative adversarial network. Those methods present good but require complex models to be able to generate reliable data. Furthermore, it is difficult to assess the relevance of the generated data to conduct subsequent training iterations. Coreset-based models These approaches alleviate the constraint on the availability of data by allowing the storage of a few samples from previous data (which are called coreset). iCaRL and End-to-end IL store 2000 samples from previous batches and rely on respectively a distillation loss and a mixture of cross-entropy and distillation loss to alleviate forgetting. The authors of SupportNet have also proposed a strategy to select relevant samples for the coreset. Gradient Episodic Memory ensures that gradients computed on new tasks do not interfere with the loss of previous tasks. Those approaches give the best for single-head learning. But, similarly to generated data, it is not clear which data may be useful to conduct further training iterations. In this paper, we are challenging the need of the coreset for single-head learning. Distance-based models These methods propose to embed the data in a space which can be used to identify the class of a sample by computing a distance between the embedding of the sample and a reference for each class. Among the most popular, we can cite Matching Networks and Prototypical Networks , but these methods have been mostly applied to few-shot learning scenarios rather than continual. Regularization-based approaches These approaches present an attempt to mitigate the effect of catastrophic forgetting by imposing some constraints on the loss function when training subsequent classes. Elastic Weight Consolidation , Synaptic Intelligence and Memory Aware Synapses all seek to prevent the update of weights that were the most useful to discriminate between previous classes. Hence, it is possible to constrain the learning of a new task in such a way that the most relevant weights for the previous tasks are less susceptible to be updated. Learning without forgetting proposes to use knowledge distillation to preserve previous performances. The network is divided in two parts: the shared weights and the dedicated weights for each task. When learning a new task A, the data of A get assigned "soft" labels by computing the output by the network with the dedicated weight for each previous task. Then the network is trained with the loss of task A and is also constrained to reproduce the recorded output for each other tasks. , the authors propose to use an autoencoder to reconstruct the extracted features for each task. When learning a new task, the features extractor is adapted but has to make sure that the autoencoder of the other tasks are still able to reconstruct the extracted features from the current samples. While these methods obtain good for learning one new task, they become limited when it comes to learn several new tasks, especially in the one class per batch setting. Expandable models In the case of the multi-head setting, it has been proposed to use the previously learned layers and complete them with new layers trained on a new task. This strategy is presented in Progressive Networks . In order to reduce the growth in memory caused by the new layers, the authors of Dynamically Expandable Networks proposed an hybrid method which retrains some of the previous weights and add new ones when necessary. Although these approaches work very well in the case of multi-head learning, they can not be adapted to single-head and are therefore not included in benchmarks with OvA-INN. We investigate the problem of training several datasets in a sequential fashion with batches of only one class at a time. Most approaches of the state-of-the-art rely on updating a features extractor when data from a new class is available. But this strategy is unreliable in the special case we are interested in, namely batches of data from only one class. With few or no sample of negative data, it is very inefficient to update the weights of a network because the setting of deep learning normally involves vast amounts of data to be able to learn to extract valuable features. Without enough negative samples, the training is prone to overfit the new class. Recent works have proposed to rely on generative models to overcome this lack of data by generating samples of old classes. Nevertheless, updating a network with sampled data is not as efficient as with real data and, on the long run, the generative quality of early classes suffer from the multiple updates. Our approach consists in interpreting a Continual Learning problem as several out-of-distribution (OOD) detection problems. OOD detection has already been studied for neural networks and can be formulated as a binary classification problem which consists in predicting if an input x was sampled from the same distribution as the training data or from a different distribution ). Hence, for each class, we can train a network to predict if an input x is likely to have been sampled from the distribution of this class. The class with the highest confidence can be used as the prediction of the class of x. This training procedure is particularly suitable for Continual Learning since the training of each network does not require any negative sample. Using the same protocol as NICE , for a class i, it is possible to train a neural network f i to fit a prior distribution p and compute the exact log-likelihood l i on a sample x: To obtain the formulation of log-likelihood as expressed in Equation 1, the network f i has to respect some constraints discussed in Section 3.3. Keeping the same hypothesis as NICE, we consider the x n/2:n y 0:n/2 y n/2:n Figure 1: Forward pass in an invertible block. x is split in x 0:n/2 and x n/2:n. f 1 and f 2 can be any type of Neural Networks as long as the dimension of their output dimension is the same as their input dimension. In our experiments, we stack two of these blocks one after the other and use fully-connected feedforward layers for f 1 and f 2. case where p is a distribution with independent components p d: In our experiments, we considered p d to be standard normal distributions. Although, it is possible to learn the parameters of the distributions, we found experimentally that doing so decreases the . Under these design choices, the computation of the log-likelihood becomes: where β = −n log √ 2π is a constant term. Hence, identifying the network with the highest log-likelihood is equivalent to find the network with the smallest output norm. The neural network architecture proposed by NICE is designed to operate a change of variables between two density functions. This assumes that the network is invertible and respect some constraints to make it efficiently computable. An invertible block (see Figure 1) consists in splitting the input x into two subvectors x 1 and x 2 of equal size; then successively applying two (non necessarily invertible) networks f 1 and f 2 following the equation: and finally, concatenate y 1 and y 2. The inverse operation can be computed with: These invertible equations illustrate how Invertible Networks operate a bijection between their input and their output. We propose to specialize each Invertible Network to a specific class by training them to output a vector with small norm when presented with data samples from their class. Given a dataset X i of class i and an Invertible Network f i, our objective is to minimize the loss L: Once the training has converged, the weights of this network won't be updated when new classes will be added. At inference time, after learning t classes, the predicted class y * for a sample x is obtained by running each network and identifying the one with the smallest output: As it is common practice in image processing, one can also use a preprocessing step by applying a fixed pretrained features extractor beforehand. We compare our method against several state-of-the-art baselines for single-head learning on MNIST and CIFAR-100 datasets. Topology of OvA-INN Due to the bijective nature of Invertible Networks, their output size is the same as their input size, hence the only way to change their size is by changing the depth or by compressing the parameters of the intermediate networks f 1 and f 2. In our experiments, these networks are fully connected layers. To reduce memory footprint, we replace the square matrix of parameters W of size n × n by a product of matrices AB of sizes n × m and m × n (with a compressing factor for the first and second block m = 16 for MNIST and m = 32 for CIFAR-100). More details on the memory cost can be found in Appendix A. Regularization When performing learning one class at a time, the amount of training data can be highly reduced: only 500 training samples per class for CIFAR-100. To avoid overfitting the training set, we found that adding a weight decay regularization could increase the validation accuracy. More details on the hyperparameters choices can be found in Appendix B. Rescaling As ResNet has been trained on images of size 224×224, we rescale CIFAR-100 images to match the size of images from Imagenet. We start by considering the MNIST dataset , as it is a common benchmark that remains challenging in the case of single-head Continual Learning. Generative models: -Parameter Generation and Model Adaptation (PGMA) -Deep Generative Replay (DGR) Coreset-based models: -iCaRL -SupportNet For Parameter Generation and Model Adaptation (PGMA) and Deep Generative Replay (DGR), we report the from the original papers; whereas we use the provided code of SupportNet to compute the for iCaRL and SupportNet with the conventional architecture of two layers of convolutions with poolings and a fully connected last layer. We have also set the coreset size to s = 800 samples. 81.7 6,000k 2 by 2 SupportNet 89.9 940k 2 by 2 DGR 95.8 12,700k 2 by 2 iCaRL We report the average accuracy over all the classes after the networks have been trained on all batches (See Table 1). Our architecture does not use any pretrained features extractor common to every classes (contrarily to our CIFAR-100 experiment): each sample is processed through an Invertible Network, composed of two stacked invertible blocks. Our approach presents better than all the other reference methods while having a smaller cost in memory (see Appendix A) and being trained by batches of only one class. Also, our architecture relies on simple fully-connected layers (as parts of invertible layers) whilst the other baselines implement convolutional layers. We now consider a more complex image dataset with a greater number of classes. This allows us to make comparisons in the case of a long sequence of data batches and to illustrate the value of using a pretrained features extractor for Continual Learning. Distance-based model: -Nearest prototype: our implementation of the method consisting in computing the mean vector (prototype) of the output of a pretrained ResNet32 for each class at train time. Inference is performed by finding the closest prototype to the ResNet output of a given sample. Generative model: -FearNet : uses a pretrained ResNet48 features extractor. FearNet is trained with a warm-up phase. Namely, the network is first trained with the all the first 50 classes of CIFAR-100, and subsequently learns the next 50 classes one by one in a continual fashion. Coreset-based models: -iCaRL : retrains a ResNet32 architecture on new data with a distillation loss. -End-to-end IL : retrains a ResNet32 architecture on new data with a crossentropy together with distillation loss. The data is provided by batch of classes. When the training on a batch (D i) is completed, the accuracy of the classifier is evaluated on the test data of classes from all previous batches (D 1, ..., D i). We report the from the literature with various size of batch when they are available. OvA-INN uses the weights of a ResNet32 pretrained on ImageNet and never update them. FearNet also uses pretrained weights from a ResNet. iCaRL and End-to-End IL use this architecture but retrain it from scratch at the beginning and fine-tune it with each new batch. The performance of the Nearest prototype baseline proves that there is high benefit in using pretrained features extractor on this kind of dataset. quickly deteriorate compared to those using pretrained parameters. Even with larger batches of classes, the gap is still present. It can be surpising that at the end of its warm-up phase, FearNet still has an accuracy bellow OvA-INN, even though it has been trained on all the data available at this point. It should be noted that FearNet is training an autoencoder and uses its encoding part as a features extractor (stacked on the ResNet) before classifying a sample. This can diminish the discriminative power of the network since it is also constrained to reproduce its input (only a single autoencoder is used for all classes). To further understand the effect of an Invertible Network on the feature space of a sample, we propose to project the different features spaces in 2D using t-SNE . We project the features of the five first classes of CIFAR-100 test set (see Figure 3). Classes that are already well represented in a cluster with ResNet features (like violet class) are clearly separated from the clusters of Invertible Networks. Classes represented with ambiguity with ResNet features (like light green and red) are better clustered in the Invertible Network space. A limiting factor in our approach is the necessity to add a new network each time one wants to learn a new class. This makes the memory and computational cost of OvA-INN linear with the number of classes. Recent works in networks merging could alleviate the memory issue by sharing weights or relying on weights superposition . This being said, we showed that Ova-INN was able to achieve superior accuracy on CIFAR-100 class-by-class training than approaches reported in the literature, while using less parameters. Another constraint of using Invertible Networks is to keep the size of the output equal to the size of the input. When one wants to apply a features extractor with a high number of output channels, it can have a very negative impact on the memory consumption of the invertible layers. Feature Selection or Feature Aggregation techniques may help to alleviate this issue . Finally, we can notice that our approach is highly dependent on the quality of the pretrained features extractor. In our CIFAR-100, we had to rescale the input to make it compatible with ResNet. Nonetheless, recent research works show promising in training features extractors in very efficient ways . Because it does not require to retrain its features extractor, we can foresee better performance in class-by-class learning with OvA-INN as new and more efficient features extractors are discovered. As a future research direction, one could try to incorporate our method in a Reinforcement Learning scenario where various situations can be learned separately in a first phase (each situation with its own Invertible Network). Then during a second phase where any situation can appear without the agent explicitly told in which situation it is in, the agent could rely on previously trained Invertible Networks to improve its policy. This setting is closely related to Options in Reinforcement Learning. Also, in a regression setting, one can add a fully connected layer after an intermediate layer of an Invertible Network and use it to predict the output for the trained class. At test time, one only need to read the output from the regression layer of the Invertible Network that had the highest confidence. In this paper, we proposed a new approach for the challenging problem of single-head Continual Learning without storing any of the previous data. On top of a fixed pretrained neural network, we trained for each class an Invertible Network to refine the extracted features and maximize the loglikelihood on samples from its class. This way, we show that we can predict the class of a sample by running each Invertible Network and identifying the one with the highest log-likelihood. This setting allows us to take full benefit of pretrained models, which in very good performances on the class-by-class training of CIFAR-100 compared to prior works. channels with 5 × 5 kernel, a fully-connected layer with 100 channels applied on an input of size 7 × 7 and a final layer of 10 channels: S iCaRL,MNIST = 28 × 28 × 800 + (5 × 5 + 1) × 32 + (5 × 5 + 1) × 64 + (7 × 7 × 64 + 1) × 100 + (100 + 1) × 10 = 944406 Since every method rely on a ResNet32 (around 20M parameters) to compute their features (except FearNet which uses ResNet48). We do not count the features extractor in the memory consumption. OvA-INN uses 2 blocks with 2 layers (f 1 and f 2) for 100 classes. The weight matrix of each layer W is a product of two matrices A and B of size 256 × 32 and 32 × 256. The memory required is: S OvA-INN,CIFAR = (256 × 32 × 2 + 256) × 2 × 2 × 100 = 6656000 We use the default coreset size s = 2000 of iCaRL and End-to-End IL with each image of size 32 × 32: S iCaRL,CIFAR = 32 × 32 × 3 × 2000 = 6144000 Our implementation is done with Pytorch , using the Adam optimizer and a scheduler that reduces the learning rate by a factor of 0.5 when the loss stops improving. We use the resize transformation from torchvision with the default bilinear interpolation. C TASK-BY-TASK LEARNING We provide additional experimental on the multi-head learning of CIFAR100 with 10 tasks of 10 classes each. The training procedure of OvA-INN does not change from the usual single-head learning but, at test time, the evaluation is processed by batches of 10 classes. The accuracy score is the average accuracy over all 10 tasks. We report the from . Although our approach is able to match state-of-the-art in accuracy, it should be noticed that it is drastically more memory and time consuming than the other baselines. EWC 81.34 Progressive Networks 88.19 DEN 92.

Represent the TLDR for finding the corresponding scientific article

We propose to train an Invertible Neural Network for each class to perform class-by-class Continual Learning.

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.