Question: Represent the article for finding a single-sentence TLDR.

Question: When communicating, humans rely on internally-consistent language representations. That is, as speakers, we expect listeners to behave the same way we do when we listen. This work proposes several methods for encouraging such internal consistency in dialog agents in an emergent communication setting. We consider two hypotheses about the effect of internal-consistency constraints: 1) that they improve agents’ ability to refer to unseen referents, and 2) that they improve agents’ ability to generalize across communicative roles (e.g. performing as a speaker de- spite only being trained as a listener). While we do not find evidence in favor of the former, our show significant support for the latter. Emergent communication is the study of how linguistic protocols evolve when agents are tasked to cooperate. For example, agents engaged in a simple object retrieval task learn to communicate with one another in order to get the items they want. To date, work of this type has each agent assume a conversational role. Thus, agents are often trained only to speak or only to listen, or similarily trained to speak using a vocabulary disjoint from the vocabulary it is understands as a listener-e.g. speaking only to ask questions ("what color?") and listening only to comprehend the answer ("blue") ). These assumptions are misaligned with how we think about human communication, and with the way we'd like computational models to work in practice. As humans, not only can we easily shift between roles, we also know that there is inherent symmetry between these roles: we expect others to speak (or listen) similarly to the way we do, and we know that others expect the same of us. We test if dialog agents that incorporate the symmetry between themselves and their communicative partners learn more generalizable representations than those which do not. We introduce three modifications to the agents to encourage that they abide by the "golden rule": speak/listen as you would want to be spoken/listened to. Specifically, these modifications include self-play training objectives, shared embedding spaces, and symmetric decoding and encoding mechanisms that share parameters. We test two hypotheses about the effect of the proposed modifications on emergent communication: 1. Internal-consistency constraints improve agents' ability to generalize to unseen items-e.g. training on "red square" and "blue circle" and then testing on "blue square". 2. Internal-consistency constraints improve agents' ability to generalize across communicative roles-e.g. training on "blue" as a listener, and using "blue" as a speaker when testing. We evaluate the effect of each of the proposed modifications with two reference game datasets and two model architectures, an RNN model used by and a Transformer model. We find no evidence to support that internal-consistency improves generalization to unseen items (Hypothesis 1), but significant evidence that these proposed constraints enable models to generalize learned representations across communicative roles (Hypothesis 2), even in the case of where the agent receives no direct training in the target (test) role. All of our code and data are available at bit.ly/internal-consistency-emergent-communication. Notation. The space of possible references is parameterized by the number of attributes n f that describe each item (e.g. color) and the number of values n v each attribute can take (e.g.{red, blue}). Each item o is a bag-of-features vector o P t0, 1u N where N " n f¨nv. Each index o i is 1 if o expresses the ith feature value. The speaker produces a message with symbols from a vocabulary V with length L. For comparison, we use the best-performing setting |V| " 100 and L " 10 from previous work. Symbols in V are represented as 1-hot vectors. In each round of the reference game, we construct xC, r, ry where C is the context (set of item column vectors stacked into a matrix), r is a vector representing the referent, and r is the index of the referent in C. We uniformly sample k´1 items as distractors to form C " to 1,... o k´1 uYtru. The distractors are is sampled randomly each round (in every epoch). We begin with a general architecture and training objective to underly all of our models (Sections 3.1 and 3.2). We then introduce three modifications which can be used to encourage internallyconsistent representations: a self-play training objective, a shared embedding space, and a symmetric decoding and encoding mechanism with shared parameters (Section 3.3) 1. Agents contain four modules. Embedding modules 1) E item P R NˆD, E message P R |V|ˆD. E˚pxq embed items and messages. When speaking, the decoder module 2) consumes the embedded referent E item prq and produces a discrete message M P V L. Next, when listening, the encoder module 3) consumes embedded messages E message pM q P R WˆD and then produces a representation of the referentr P R D. Finally, a non-parametric pointing module 4) produces a distribution P pCq over the context by matrix multiplyingr with the embedded context E item pCq. The decoders emit one symbol at a time, auto-regressively producing fixed-length messages. The messages are discretized with the straight-through Gumbel Softmax as in. This converts a distribution to a one-hot vector while permitting gradients to flow through the operation and enables the agents to be optimized without reinforcement methods. The Recurrent model uses a LSTM decoder when speaking and a LSTM encoder when listening, as in. The Transformer model uses a Transformer Decoder when speaking and a Transformer Encoder to encode when listening. See Appendix A for implementation details and hyperparameters. Speaking S: r, θ Ñ M and listening L: C, M, θ Ñ P pCq are both functions where Lˆ|V| is a discrete-valued message with length L, P pCq is a distribution over the items in context, and θ are optimizable parameters. We optimize the parameters θ A, θ B of agents A, B over a dataset D to select each referent r from among the distractors in its context C by minimizing the negative log likelihood of selecting the correct referent in context (Eq 1). In our experiments, the speaker modules and the listener modules instantiate the function S and L respectively. We investigate three internal-consistency constraints, that encourage internally-consistent representations. Baseline agents consist of two separate sets of parameters, one for listening and one for speaking. For example, the baseline recurrent model consists of two recurrent models. This corresponds to the scenario where agents either speak or listen, but not both. We introduce a 1) self-play loss for both agents of the same form as Eq. 1, except the given agent fulfills both roles, encouraging it to speak/listen to others the same way it speaks/listens to itself. When we use the self-play training objective, we use it for both agents. Next, 2) shared embedding agents use the same item embeddings and the same message embeddings when listening and speaking. Finally, 3) symmetric encoding and decoding agents use the same parameters (but different mechanisms) to decode a message when speaking as it does to encode a message when listening. Parameters are only ever shared between roles within an agent, and never between agents. Our evaluation is based on the simple reference game 2 as described in Section 2, played across two datasets. The datasets, summarized in Table 1, target different aspects of lexical reference. The first, Visual Attributes for Concepts (CONCEPTS) , is derived from annotated images of actual items (animals, tools, etc). Thus, the data contains realistic co-occurance patterns: groups of attributes like has-head, has-mouth, and has-snout appears together several times, whereas has-seeds, has-mouth, made-of-metal never co-occur. The intuition is that a good lexicon will exploit the structure by developing words which refer to groups of frequently co-occurring attributes (e.g. "mammal") and will describe unseen referents in terms of these primitive concepts. The second dataset, SHAPES, is one we create ourselves in contrast with the CONCEPTS data. In SHAPES, items correspond to abstract shapes which are exactly describable by their attributes (e.g. blue, shaded, hexagon). All the attributes are independent and there is no co-occurence structure to exploit, so a good lexicon should ideally provide a means for uniquely specifying each attribute's value independently of the others. We present experimental aimed at testing the hypotheses stated in Section 1. To provide intuition, we frame experiments in terms of two agents, "Alice" (Agent A) and "Bob" (Agent B), who are taking turns asking for and fetching toys. We first test whether any of the proposed internal-consistency constraints improve the agents' ability to generalize to novel items-i.e. items which consist of unseen combinations of features. Here, we focus on the performance when models are trained and tested within the same communicative role. This corresponds to the setting that has typically been used in prior work on emergent communication: Alice always speaks in order to ask for toys, Bob always responds by fetching them, and the pair's success is evaluated in terms of Alice's ability to describe new toys such that Bob correctly gets them. For evaluation, we hold out a subset of the value combinations from each dataset to use for testing. For example, the agents might be trained to refer to trshape:circle, color:reds, rshape:square, color:bluesu and then tested on its ability to refer to rshape:circle, color:blues. We compute validation and test accuracies. Table 2: Results when agents are trained and tested in a single role, before any internal-consistency constraints. These scores are mean accuracy with 95% confidence range averaged over 5 runs over different test set samplings (the distractors change). one role, they can still impose internally consistent behavior across both roles. It is conceivable that doing so might improve performance even though each agent remains in a fixed role, either by providing the model with additional information about the task structure, or simply by acting as a regularizer. Thus, for completeness, we assess whether the internal-consistency constraints provide any advantage to the models in the vanilla emergent communication setting. Table 3 shows the effect of adding the self-play objective in the fixed-role setting, across architectures and datasets. The trends are mixed: it appears the additional signal only noises the baseline and symmetric models, whereas the shared embeddings models are able to leverage it effectively. Thus, the effect is not clear enough to establish conclusively that the internal-consistency constraints help the agents generalize in this fixed-role setting, and in fact it may hurt. Table 3: Performance on task of referring to/fetching unseen items for baseline model compared against models with the internal-consistency constraints. To highlight the difference of each constraint compared to the baseline performance, each delta compares the performance of the modified model to the baseline model. In this setting, we see no clear advantage to enforcing internalconsistency via self-play. These scores are mean accuracy with 95% confidence interval averaged over 5 runs over different test set samplings (the distractors change). We now look at whether internal-consistency improves the agents' ability to generalize linguistic knowledge across roles. For example, we can picture the following scenario: Alice is speaking to Bob, and asks for the "truck". Bob hands her the doll, and Alice replies negatively, indicating that what she actually wanted was the truck. Now, without additional direct supervision, when Bob wants the truck, will he know do use the word "truck"? Such a setting is particularly relevant in practical settings, for example when robotic agents must reach high accuracy despite only limited access to human interaction for training. We consider two versions of this setting, involving different levels of direct supervision (i.e. interaction with the other agent) as described below. Training in one role. Our first experimental setting assumes that Alice and Bob each only receive direct training in one role, e.g. Alice only ever speaks to Bob, so Alice only receives feedback on how she is performing as a speaker, and Bob on how he is performing as a listener. However, both Alice and Bob are able to practice in the opposite role via self-play. This setup is analogous to the experiment just discussed in Section 5.1.3. However, unlike before, Alice and Bob will be tested in the roles opposite of those in which they were trained. That is, if Alice was trained as a speaker, then she will be tested as a listener (on her ability to correctly identify items to which Bob refers). Training in both roles. In our second experimental setting, we assume Alice and Bob enjoy a healthy friendship, in which both take turns speaking and listening to each other, and thus both receive direct supervision in both roles. However, they do not necessarily receive equal training on every vocabulary item. Rather, there are some contexts in which Alice only speaks and other contexts in which she only listens. Intuitively, this corresponds to a scenario in which Alice speaks exclusively about food (while Bob listens), while Bob speaks exclusively about toys (while Alice listens). We are interested in testing how well Alice is able to speak about toys at test time. We use the SHAPES dataset 4 to create two training splits, each having the same attributes but covering disjoint sets of values. For example, the first training split (train-1) might have colorP{blue, red, yellow} whereas the second training split (train-2) has colorP{green, orange, purple}. We use train-1 to train Alice as speaker and Bob as listener and train-2 to train them in the reverse roles. We then report performance with Alice as listener and Bob as speaker using a test set that uses the same attribute values as train-1. Our for both training conditions are shown in Table 4. The baseline model (which includes no internal-consistency constraints) performs, unsurprisingly, at chance. Adding the self-play objective gives improvements across the board. Again, while seemingly straight forward, this has promising practical interpretations in settings in which a model has access to only a small amount of interaction. For example, a human may be willing to train a robot via speaking (pointing and naming items), but not patient enough to train it via listening (responding to the robot's noisy commands). In such a setting, the ability to massively augment performance via self-play is significant. In addition to the self-play objective, we see that enforcing shared-embedding spaces yields further significant performance gains (in the range of 30 percentage points in some cases). The symmetric constraints on top of self-play and shared embeddings seem to hurt performance in general. Baseline`Self-play`Shared Emb.`Symmetric Table 4: Performance for tasks that requires agents to generalize across roles-e.g. training on the word "blue" as a listener, but then having to produce "blue" as a speaker. " One Role" refers to when agents receive direct feedback in a single role (i.e. their training on the other roles is only via self-play). " Both Roles" refers to when agents receive direct feedback in both roles, but only see the test vocabulary in the role opposite that in which they are tested. To inspect the additive differences between the internal-consistency constraints, each delta compares the performance of the current column to the previous column. These scores are mean accuracy with 95% confidence range averaged over 5 with different test set samplings (the distractors change). Overall, when agents can be trained directly in the role in which they are tested, there is no clear evidence that adding internal-consistency constraints improves the ability of agents to generalize to new items. However, internal-consistency constraints improve performance significantly when agents have limited ability to train in a given role. Specifically, models which are equipped with selfplay training objectives and shared embedding spaces show superior ability to generalize learned representations across roles, and perform about as well as if they had been trained on the target role. In this section we provide additional analyses to highlight the effects of internal-consistency (in particular, self-play) on training efficiency and on the emerged protocol. Here, we use a smaller SHAPES dataset (see B.1), and reduce the vocabulary size and message length (|V| " 10, L " 3). Here we inspect if self-play supplants direct supervision between agents. We consider the setting in which Alice trains with full data as a speaker, but vary the amount of data she has access to as a listener. We then test Alice's performance as a listener (and vice-versa for Bob as a speaker). Fig. 2 shows the , with fraction of the full training data that Alice (Bob) sees as a speaker (listener) shown along the x-axis. The self-play models without direct supervision perform well: it appears their protocol transfers across roles with out drifting. This sheds some light on the performance drop between the "one role" and "two role" settings in Section 5.2.2. where the additional experience in the "two role" setting did not help. Fig. 2 shows that additional training in the primary role is unnecessary, so it is not surprising that training on disjoint features (train-2) is helpful. We measure whether self-play leads to better communication protocols in general. First, we measure the agents' speaking and listening capacities separately, using measures proposed by;. Specifically, positive signaling (S`) measures if the speaker's messages depend on the features of the referent, and positive listening (L`) measures if the listener's actions depend on the message 6. Table 5 shows that self-play improves the agents' communication as measured by accuracy as well as these orthogonal metrics. We also find that the model architectures and self-play impact the agents' lexicons. The recurrent models produces fewer unique messages than the transformer models (on average 110 versus 300), and often neglect to use all the vocabulary. Fig. 3 shows that self-play helps the recurrent model use more of the vocabulary, and leads to both the recurrent and transformer models to develop sparser mappings from symbols onto features. Work in emergent communication (; analyzes agents that develop a shared protocol by playing reference games . presented showing that computational models do not learn compositional protocols by default. Instead, the agents tend to develop brittle protocols that have trouble generalizing to novel items. Several approaches have been proposed which could encourage models to learn more generalizable representations of language, including compression , efficiency , memory constraints, pragmatic constraints , and positive biases to respond to other agents . Some work, like ours, assumes access to symbolic representations of referents and their attributes, whereas others' are set in pixel-based multi-agent games (; ;) or multi-agent grid-worlds . Our work also relates to a broader body of work on speaker-listener models, specifically pragmatically-informed models in which speakers reason recursively about listeners' beliefs (and vice-versa) . Such models have been used in applications such image captioning (; ;), and robotics (; ;), as well as in linguistics and psychology in order to explain complex linguistic inferences . Conceptually, our proposed internal-consistency constraints share something in common with these neural speaker-listener models developed outside of emergent communication. However, again, past work has tended to assume that a speaker's mental model of their listener is not necessarily consistent-in fact, it is often assumed explicitly to be inconsistent -with the way the speaker themself would behave as a listener. We note, however, that our proposed model architecture (because it lacks the recursion typical in other pragmatics models) is likely unable to handle the types of higher-level inferences (e.g. implicatures) targeted by the mentioned prior work on computational pragmatics, though this is an interesting avenue to explore. We propose three methods for encouraging dialog agents to follow "the golden rule": speak/listen to others as you would expect to be spoken/listened to. In the emergent communication setting, we find that the internal-consistency constraints do not systematically improve models' generalization to novel items, but both the self-play objective and shared embeddings significantly improve performance when agents are tested on roles they were not directly trained for. In fact, when trained in one role and tested on another, these internal-consistency constraints allow the agents to perform about as well as if they had been trained in the target role. We use the deep learning framework Pytorch 7 (v1.2.0) to implement our models (and Python 3.7.3). For reproducibility, all random seeds (random, numpy, torch) are arbitrarily set to 42. The general architecture, the four modules that comprise each agent are shown in Figure 4. The recurrent model decodes and encodes message as follows: to generate a message, the first input is the embedding of a SOS start-of-sentence symbol and the initial hidden state is set as the embedded referent (and the cell memory is all zeroes). From here, at each step, the outputted hidden state (P R D) is projected by the transposed word embeddings (E ⊺ message P R Dˆ|V |), and the next word is sampled from this ing distribution across the vocabulary. Moving forward, the next input is the embedding of the sampled word, and the hidden state and cell memory are set those emitted at the previous step. We produce words until the maximum length is reached. When encoding a message, a learned embedding is set to the first hidden state, and the input at each time step is the corresponding embedded word. The last hidden state is set as the encoding. In the symmetric variant of this model, the LSTM cell used for encoding and decoding is the same. This architecture underlies each model we use; only the implementation of the Decoder and Encoder modules vary between models. In the baseline models no parameters shared within an agent. In shared embedding models, the embeddings (purple) are shared across roles. In symmetric models, the encoder and decoder (pink) are shared across both roles. The blue modules are non-parametric. The transformer model decodes and encodes message as follows: to generate a message describing the referent auto-regressively, all the embeddings of the words produced so far M and the embedding of a NEXT symbol are concatenated together into a matrix X (Eq. 2). Next, the transformer decoder consumes this matrix and the referent embedding and produces a contextualized representation of the input matrix (Eq. 3). The last column vectorX:,W, which corresponds to the NEXT embedding, is the internal representation of the next word. The next word m is sampled from the projection of this representation with the transposed word embedding. More words are produced in this way until the maximum length message is formed. Producing the i`1th word (so M consists of the first i words), works as follows: To encode incoming messages when listening, all the embeddings of the words in the message plus the embedding of a ITEM symbol are concatenated together into a matrix X (Eq. 5). Then, the transformer encoder is used to produce a contextualized embeddingX (Eq. 6). The last column vectorX:,W, which corresponds to the ITEM embedding, is set as the message encoding. Note, W is the number of words in each message (and the length of M). X " TransformerEncoderpXq P R pW`1qˆD. r "X:,W A.3 SYMMETRIC AGENTS For the Symmetric Recurrent Model, a LSTM cell is shared between the encoder and decoder. Otherwise, the recurrent model is unchanged. For the Symmetric Transformer Model, Transformer Encoders and Transformer Decoders have different structures, so to share parameters between them, we have change either how the transformer agent speaks or how it listens. We opt to replace the Transformer Decoder with Transformer Encoder, and use it to decode messages in-place when speaking. The Symmetric Transformer uses the same mechanism for encoding messages when listening as the default transformer model (described directly above). However, it uses a Transformer Encoder when speaking instead of a Transformer Decode. When speaking, to produce the next symbol, the embeddings of the i words produced so far, the referent embedding, and the embedding of a NEXT symbol are concatenated together into a matrix X (Eq. 8). The Transformer Encoder then maps X to a contextualized representationX (Eq. 9). Finally, the column vector in X that corresponds to NEXT, is used to sample the next symbol: X " rE item prq; E message pM q; E ITEM s, A.4 HYPER PARAMETERS We uniformly sampled 25 hyperparameter configurations for each model architecture, experiment, and dataset split. In every case, we fixed the hidden size dimensionality and embedding dimensionality to be the same. We searched over three different learning rate schedulers: None (or no scheduler, ReduceLROnPlateau with a patience of 25, reduction factor of 0.1, and uses validation accuracy as its measure of progress, and CyclicLR rising from 0.00001 to the given learning rate over 500 batches and then declines towards 0.00001 for the rest of training (10,000 batches). This is similar to the Noam Update in. To save space, we relegate the hyper-parameter selections in our code bit.ly/internal-consistency-emergent-communicationsee /lib/hyperparamters.py. TRE takes two important hyperparameters, an error function and a composition function. We select the same choices as the author, for what amounts to the same task (producing a discrete message) as detailed in the original paper. This method requires structured feature representations, so we assume that the features in each item are entirely right-branching. The composition function is learned, and set the number of update steps to 1000. The original implementation is at https://github.com/jacobandreas/tre, and our modification is at bit.ly/internal-consistency-emergent-communication in the file./lib/compositionality.py. We simplify the SHAPES dataset in order to be able to empirically compute the positive listening and signaling scores, which requires iterating overall possible messages. In the original version of SHAPES this is impractical as there would be 50 10 possible messages. The details of this smaller version of SHAPES detailed in Table 9. We fix the settings for the recurrent and transformer models as we found that a majority of models across experiments used the same parameters. See Tables 10, 11. Furthermore, all in Sec. 6 are averaged over 5 arbitrary random seeds (both trained and tested). Positive listening was introduced by as Causal Influence of Communication, and the precursor to positive signaling was introduced by. We use the definitions modified for a one step game for both metrics: Positive Listening (L`). " D KL pP rpa|mq P r l paqq, Positive Speaking (S`). " Ipm, xq " Hpmq´Hpm|xq, where m is a given message, a are the actions the agent can take, x is state, D KL is the Kuller-bach divergence, I is the mutual information, H is the entropy. We can compute these quantities without sampling messages because the number of messages is tractable. In our setting, the game is a single step, and the average policy over actions independent of the message converges to the uniform distribution over actions, as the order of referents is randomized. Thus we have: L`" D KL pP rpa|mq U q. The positive speaking metric is computed over a sampling of the dataset (the distractors are random) as follows as in : S`" Hpmq´Hpm|xq, "´ÿ m πpmq log πpmq`E x r ÿ m πpm|xq log πpm|xqs, where the summations are over all possible messages, x is the given referent, and πpmq is empirically likelihood of the message being produced irrespective of x, and πpm|xq is the likelihood of the given message being produced given the referent x. We report empirical averages of L`and Sò ver all items in the dataset, also averaged over 5 arbitrary random seeds. Figs. 5, 6, 7, 8 show lexicons across random seeds. See Table 8 for additional in the vanilla setting for SHAPES-small that were elided for space.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Internal-consistency constraints improve agents ability to develop emergent protocols that generalize across communicative roles.

Represent the TLDR for finding the corresponding scientific article!

We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Neural networks (NNs) are able to perform tasks that rely on compositional structure even though they lack obvious mechanisms for representing this structure. To analyze the internal representations that enable such success, we propose ROLE, a technique that detects whether these representations implicitly encode symbolic structure. ROLE learns to approximate the representations of a target encoder E by learning a symbolic constituent structure and an embedding of that structure into E’s representational vector space. The constituents of the approximating symbol structure are defined by structural positions — roles — that can be filled by symbols. We show that when E is constructed to explicitly embed a particular type of structure (e.g., string or tree), ROLE successfully extracts the ground-truth roles defining that structure. We then analyze a seq2seq network trained to perform a more complex compositional task (SCAN), where there is no ground truth role scheme available. For this model, ROLE successfully discovers an interpretable symbolic structure that the model implicitly uses to perform the SCAN task, providing a comprehensive account of the link between the representations and the behavior of a notoriously hard-to-interpret type of model. We verify the causal importance of the discovered symbolic structure by showing that, when we systematically manipulate hidden embeddings based on this symbolic structure, the model’s output is also changed in the way predicted by our analysis. Finally, we use ROLE to explore whether popular sentence embedding models are capturing compositional structure and find evidence that they are not; we conclude by discussing how insights from ROLE can be used to impart new inductive biases that will improve the compositional abilities of such models. Certain AI tasks consist in computing a function ϕ that is governed by strict rules: e.g., if ϕ is the function mapping a mathematical expression to its value (e.g., mapping '19−2 * 7' to 5), then ϕ obeys the rule that ϕ(x + y) = sum(ϕ(x), ϕ(y)) for any expressions x and y. This rule is compositional: the output of a structure (here, x + y) is a function of the outputs of the structure's constituents (here, x and y). The rule can be stated with full generality once the input is assigned a symbolic structure giving its decomposition into constituents. For a fully-compositional task, completely determined by compositional rules, an AI system that can assign appropriate symbolic structures to inputs and apply appropriate compositional rules to these structures will display full systematic generalization: it will correctly process arbitrary novel combinations of familiar constituents. This is a core capability of symbolic AI systems. Other tasks, including most natural language tasks such as machine translation, are only partially characterizable by compositional rules because natural language is only partially compositional in nature. For example, if ϕ is the function that assigns meanings to English adjectives, it generally obeys the rule that ϕ(in-+ x) = not ϕ(x), (e.g., ϕ(inoffensive) = not ϕ(offensive)), yet there are exceptions: ϕ(inflammable) = ϕ(flammable). On these "partially-compositional" AI tasks, this strategy of compositional analysis has demonstrated considerable, but limited, generalization capabilities. Deep learning research has shown that Neural Networks (NNs) can display remarkable degrees of combinatorial generalization, often surpassing symbolic AI systems for partially-compositional tasks , and exhibit good generalization (although generally falling short of symbolic AI systems) on fully-compositional tasks (; a). Given that standard NNs have no obvious mechanisms for representing symbolic structures, parsing inputs into such structures, nor applying compositional symbolic rules to them, this success raises the question we address in this paper: How do NNs achieve such strong generalization on partially-compositional tasks, and good performance on fully-compositional tasks? An important step towards answering this question was reported in McCoy et al. (2019a), which showed that when trained on highly compositional tasks, standard NNs learned representations that were well approximated by symbolic structures (Sec. 2). Processing in these NNs assigns such representations to inputs and generates outputs that are governed by compositional rules stated over those representations. We refer to the networks to be analyzed as target NNs, because we will propose a new type of NN (in Sec. 4) -the Role Learner (ROLE) -which is used to analyze the target network, after first discussing related analysis methods in Sec. 3. In contrast with the analysis model of McCoy et al. (2019a), which relies on a hand-specified hypothesis about the underlying structure, ROLE automatically learns a symbolic structure that best approximates the internal representation of the target network. Automating the discovery of structural hypotheses provides two advantages. First, ROLE achieves success at analyzing networks for which it is not clear what the underlying structure is. We show this in Sec. 6, where ROLE successfully uncovers the symbolic structures learned by a seq2seq RNN trained on the SCAN task . Second, removing the need for hand-specified hypotheses allows the data to speak for itself, which simplifies the burden on the user, who only needs to provide input sequences and associated embeddings. We first consider fully-compositional (hence synthetic) tasks: a simple string-manipulation task in Sec. 5, and the richer SCAN task, which has been the basis of previous work on combinatorial generalization in NNs, in Sec. 6. Discovering symbolic structure within a model enables us to perform precise alterations to the internal representations in order to produce desired combinatorial alterations in the output (Sec. 6.3). Then, in Sec. 7, we turn briefly to partially-compositional tasks in NLP. In Sec. 8 we consider how what we have learned about standard NNs can suggest new inductive biases to strengthen compositionality in NN learning. We build on McCoy et al. (2019a), which introduced the analysis task DISCOVER (DISsecting COmpositionality in VEctor Representations): take a NN and, to the extent possible, find an explicitly-compositional approximation to its internal distributed representations. McCoy et al. (2019a) showed that, in GRU encoder-decoder networks performing simple, fullycompositional string manipulations, the medial encoding (between encoder and decoder) could be extremely well approximated, up to a linear transformation, by Tensor Product Representations (TPRs) , which are explicitly-compositional vector embeddings of symbolic structures. To represent a string of symbols as a TPR, the symbols in the string 337 might be parsed into three constituents {3 : pos1, 7 : pos3, 3 : pos2}, where posn is the role of n th position from the left edge of the string; other role schemes are also possible, such as roles denoting right-to-left position: {3 : third-to-last, 3 : second-to-last, 7 : last}. The embedding of a constituent 7: pos3 is e(7 : pos3) = e F ⊗ e R (pos3), where e R, e F are respectively a vector embedding of the roles and a vector embedding of the fillers of those roles: the digits. The embedding of the whole string is the sum of the embeddings of its constituents. In general, for a symbol structure S with roles {r k} that are respectively filled by the symbols {f k}, e TPR (S) = k e F (f k) ⊗ e R (r k). This work falls within the larger paradigm of using analysis techniques to interpret NNs (see for a recent survey), often including a focus on compositional structure (; ;). Two of the most popular analysis techniques are the behavioral and probing approaches. In the behavioral approach, a model is evaluated on a set of examples carefully chosen to require competence in particular linguistic phenomena (; ; ; ; ; b). This technique can illuminate behavioral shortcomings but says little about how the internal representations are structured, treating the model as a black box. In the probing approach, an auxiliary classifier is trained to classify the model's internal representations based on some linguistically-relevant distinction (; ; ; ;). In contrast with the behavioral approach, the probing approach tests whether some particular information is present in the model's encodings, but it says little about whether this information is actually used by the model. Indeed, in at least some cases models will fail despite having the necessary information to succeed in their representations, showing that the ability of a classifier to extract that information does not mean that the model is using it . DISCOVER bridges the gap between representation and behavior: It reveals not only what information is encoded in the representation, but also shows how that information is causally implicated in the model's behavior. Moreover, it provides a much more comprehensive window into the representation than the probing approach does; while probing extracts particular types of information from a representation (e.g., "does this representation distinguish between active and passive sentences?"), DISCOVER exhaustively decomposes the model's representational space. In this regard, DISCOVER is most closely related to the approaches of , Chrupała & , and , who also propose methods for discovering a complete symbolic characterization of a set of vector representations, and and , which also seek to extract more interpretable symbolic models that approximate neural network behavior. 1 produces a vector-space embedding of an input string of T symbols S = s 1 s 2... s T by producing a TPR T(S) and then passing it through a linear transformation W. ROLE is trained to approximate a pre-trained target string-encoder E. Given a set of N training strings {S,..., S (N) }, ROLE minimizes the total mean-squared error (MSE) between its output W T(S (i) ) and E's corresponding output, E(S (i) ). Figure 1: The role learning module. The role attention vector a t is encouraged to be one-hot through regularization; if a t were one-hot, the produced role embedding r t would correspond directly to one of the roles defined in the role matrix R. The LSTM can be unidirectional or bidirectional. ROLE is an extension of the Tensor-Product Encoder (TPE) introduced in McCoy et al. (2019a) (as the "Tensor Product Decomposition Network"), which produces a linearly-transformed TPR given a string of symbols and pre-assigned role labels for each symbol (see Appendix A.1 for details). Crucially, ROLE is not given role labels for the input symbols, but learns to compute them. More precisely, it learns a dictionary of n R d R -dimensional role-embedding vectors, R ∈ R dR×nR, and, for each input symbol s t, computes a soft-attention vector a t over these role vectors: the role vector assigned to s t is then the attention-weighted linear combination of role vectors, r t = R a t. ROLE simultaneously learns a dictionary of n F d F -dimensional symbol-embedding filler vectors F ∈ R dF×nF, the φ th column of which is f φ, the embedding of symbol type φ; φ ∈ 1,..., n F where n F is the size of the vocabulary of symbol types. The TPR generated by ROLE is thus T(S) = T t=1 f τ (st) ⊗ r t, where τ (s t) is symbol s t's type. Finally, ROLE learns a linear transformation W to map this TPR into R d, where d is the dimension of the representations of the encoder E it is learning to approximate. ROLE uses an LSTM to compute the role-assigning attentionvectors a t from its learned embedding F of the input symbols s t: at each t, the hidden state of the LSTM passes through a linear layer and then a softmax to produce a t (depicted in Figure 1). Since a TPR for a discrete symbol structure deploys a discrete set of roles specifying discrete structural positions, ideally a single role would be selected for each s t: a t would be one-hot. ROLE training therefore deploys regularization to bias learning towards one-hot a t vectors (based on the regularization proposed in , developed for the same purpose). See Appendix A.4 for the precise regularization terms that we used. Although the regularization can yield good , performing continuous gradient descent on a nearly discrete symbolic landscape can cause networks to get stuck in local optima where the learned symbolic structure is not ideal. For this reason, the performance at convergence can vary appreciably across multiple runs. It is important to note that, while we impose this regularization on ROLE, there is no explicit bias favoring discrete compositional representations in the target encoder E: any such structure that ROLE finds hidden in the representations learned by E must be the of biases implicit in the vanilla RNN-architecture of E when applied to its target task. We first apply ROLE to two target Tensor Product Encoder (TPE) models which are fully compositional by design. Since we know what role scheme each target model deploys, we can test how well ROLE learns these ground-truth roles. The TPEs are trained on the fully compositional task of autoencoding sequences of digits. We use two types of TPEs: one that uses a simple left-to-right role scheme (e.g., first, second, third) and one that uses a complex tree position role scheme (e.g., left child of the root of the tree, right child of the left child of the left child of the root of the tree), where the trees are generated from digit sequence inputs using a deterministic parsing algorithm (see Appendix A.2 for explanations and examples of the designed role schemes). The left-to-right TPE was paired with a unidirectional RNN decoder, while the tree-position TPE was paired with the tree-RNN decoder used by McCoy et al. (2019a). Both of these target models attained near-perfect performance on the autoencoding task (Table 1). Once the encoders are finished training, we extract the encoding for each sequence in the dataset and use this to train ROLE. See Appendices A.3 and A.5 for additional training details. Table 1 reports the approximation performance of ROLE in two ways. Substitution Accuracy is the proportion of the items for which the decoder produces the correct output string when it is fed the ROLE approximation. The V-Measure assesses the extent to which the clustering of the role vectors assigned by ROLE matches the ground truth role assignments. The ROLE approximation of the left-to-right TPE attained perfect performance, with a substitution accuracy of 100% and a V-Measure of 1.0, indicating that the role scheme it learned perfectly matched the ground truth. On the significantly more complex case of tree position roles, ROLE achieves 2 Let the t th LSTM hidden state be qt ∈ R H; let the output-layer weight-matrix have rows k ρ ∈ R H and let the columns of R be vρ ∈ R d R, with ρ = 1,..., nR. Then rt = R at = n R ρ=1 vρ softmax(k ρ qt): the of query-key attention (e.g.,) with query qt to a fixed external memory containing key-value pairs {(kρ, vρ)} n R ρ=1. essentially the same accuracy as the target encoder E and has considerable success at recovering the ground truth roles for the vectors it was analyzing. These show that, when a target model has a known fully compositional structure, ROLE can successfully find that structure. We have established that ROLE can uncover the compositional structure used by a model that is compositional by design. But, returning to our central question from Sec. 1, how can models without explicit compositional structure (namely, standard RNNs) still be as successful at fully compositional tasks as fully compositional models? Our hypothesis is that, though these models have no constraint forcing them to be compositional, they still have the ability to implicitly learn compositional structure. To test this hypothesis, we apply ROLE to a standard RNN-based seq2seq model trained on a fully compositional task. Because the RNN has no constraint forcing it to use TPRs, we do not know a priori whether there exists any solution that ROLE could learn; thus, if ROLE does learn anything it will be a significant empirical finding about how these RNNs operate. We consider the SCAN task , which was designed to test compositional generalization and systematicity. SCAN is a synthetic sequence-to-sequence mapping task, with an input sequence describing an action plan, e.g., jump opposite left, being mapped to a sequence of primitive actions, e.g., TL TL JUMP (see Sec. 6.3 for a complex example). We use TL to abbreviate TURN_LEFT, sometimes written LTURN; similarly, we use TR for TURN_RIGHT. The SCAN mapping is defined by a complete set of compositional rules (, Supplementary Fig. 7). For our target SCAN encoder E, we trained a standard GRU with one hidden layer of dimension 100 for 100,000 steps (batch-size 1) with a dropout of 0.1 on the simple train-test split. (The hidden dimension and dropout rate were determined by a limited hyper-parameter search; see Appendix A.6). E achieves 98.47% (full-string) accuracy on the test set. Thus E provides what we want: a standard RNN achieving near-perfect accuracy on a non-trivial fully compositional task. After training, we extract the final hidden embedding from the encoder for each example in the training and test sets. These are the encodings we attempt to approximate as explicitly compositional TPRs. We provide ROLE with 50 roles to use as it wants (additional training information is in Appendix A.7). We evaluate the substitution accuracy that this learned role scheme provides in three ways. The continuous method tests ROLE in the same way as it was trained, with input symbol s t assigned role vector r t = R a t. The continuous method does not use a discrete set of role vectors because the weighted sum that generates a t allows for continuously-valued weights. The remaining two methods test the efficacy of a truly discrete set of role vectors. First, in the snapped method, a t is replaced at evaluation time by the one-hot vector m t singling out role m t = arg max(a t): r t = R m t. This method serves the goal of enforcing the discreteness of roles, but it is expected to decrease performance because it tests ROLE in a different way than it was trained. Our final evaluation method, the discrete method, uses discrete roles without having such a train/test discrepancy; in this method, we use the one-hot vector m t to output roles for every symbol in the dataset and then train a TPE which does not learn roles but rather uses the one-hot vector m t as input during training. In this case, ROLE acts as an automatic data labeler, assigning a role to every input word. For comparison, we also train TPEs using a variety of discrete hand-crafted role schemes: left-to-right (LTR), right-to-left (RTL), bidirectional (Bi), tree position, Wickelrole (Wickel), and bag-of-words (BOW) (additional information provided in Appendix A.2). The substitution accuracy from these different methods is shown in Table 2. All of the predefined role schemes provide poor approximations, none surpassing 44.12% accuracy. The role scheme learned by ROLE does significantly better than any of the predefined role schemes: when tested with the basic, continuous role-attention method, the accuracy is 94.12%. The success of ROLE tells us two things. First, it shows that the target model's compositional behavior relies on compositional internal representations: it was by no means guaranteed to be the case that ROLE would be successful here, so the fact that it is successful tells us that the encoder has learned compositional representations. Second, it adds further validation to the efficacy of ROLE, because it shows that it can be a useful analysis tool in cases of significantly greater complexity than the autoencoding task. Analyzing the roles assigned by ROLE to the sequences in the SCAN training set, we created a symbolic algorithm for predicting which role will be assigned to a given filler. This is described in Appendix A.8.1 and discussed at some length in Appendix A.8.2. Though the algorithm was created based only on sequences in the SCAN training set, it is equally successful at predicting which roles will be assigned to test sequences, exactly matching ROLE's predicted roles for 98.7% of sequences. The details of this algorithm illuminate how the filler-role scheme encodes information relevant to the task. First, one of the initial facts that the decoder must determine is whether the sequence is a single command, a pair of commands connected by and, or a pair of commands connected by after; such a determination is crucial for knowing the basic structure of the output (how many actions to perform and in what order). We have found that role 30 is used for, and only for, the filler and, while role 17 is used in and only in sequences containing after (usually with after as the filler bound to role 17). Thus, the decoder can use these roles to tell which basic structure is in play: if role 30 is present, it is an and sequence; if role 17 is present, it is an after sequence; otherwise it is a single command. Once the decoder has established the basic syntactic structure of the output, it must then fill in the particular actions. This can be accomplished using the remaining roles, which mainly encode absolute position within a command. For example, the last word of a command before after (e.g., jump left after walk twice) is always assigned role 8, while the last word of a command after after (e.g., jump left after walk twice) is always assigned role 46. Therefore, once the decoder knows (based on the presence of role 17) that it is dealing with an after sequence, it can check for the fillers bound to roles 8 and 46 to begin to figure out what the two subcommands surrounding after look like. The identity of the last word in a command is informative because that is where a cardinality (i.e., twice or thrice) appears if there is one. Thus, by checking what filler is at the end of a command, the model can learn whether there is a cardinality present and, if so, which one. This description of how the decoding could take place does not necessarily match how it actually does take place; for example, it is likely that some of the steps we have described as occurring serially, for expository simplicity, actually occur in parallel. We leave for future work the question of which operations are actually being performed and how those operations are instantiated in an RNN. To address this causal question , we actively intervene on the constituent structure of the internal representations by replacing one constituent with another syntactically equivalent one 4, and see whether this produces the expected change in the output of the decoder. We take the encoding generated by the RNN encoder E for an input such as jump opposite left, subtract the vector embedding of the opposite constituent, add the embedding of the around constituent, and see whether this causes the output to change from the correct output for jump opposite left (TL TL JUMP) to the correct output for jump around left (TL JUMP TL JUMP TL JUMP TL JUMP). The roles in these constituents are determined by the algorithm of Appendix A.8. If changing a word leads other roles in the sequence to change (according to the algorithm), we update the encoding with those new roles as well. Such surgery can be viewed as based in a more general extension of the analogy approach used by for analysis of word embeddings. An example of applying a sequence of five such constituent surgeries to a sequence are shown in Figure 2 (left). The previous sections explored fully-compositional tasks where there is a strong signal for compositionality. In this section, we explore whether the representations of NNs trained on tasks that are only partially-compositional also capture compositional structure. Partially-compositional tasks are especially challenging to model because a fully-compositional model may enforce compositionality too strictly to handle the non-compositional aspects of the task, while a model without a compositional bias may not learn any sort of compositionality from the weak cues in the training set. We test four sentence encoding models for compositionality: InferSent , Skipthought , Stanford Sentiment Model (SST) , and SPINN . For each of these models, we extract the encodings for the SNLI premise sentences . We use the extracted embeddings to train ROLE with 50 roles available (additional training information provided in Appendix A.10). of cognitive representations must individually be causally efficacious in order for those constituents to provide an explanation of the compositionality of cognition . That TPRs meet the challenge of explaining compositionality was argued in Smolensky (1987; 1991). 4 We extract syntactic categories from the SCAN grammar (, Supplementary Fig. 6) by saying that two words belong to the same category if every occurrence of one could be grammatically replaced by the other. Based on our analysis in Appendix A.8, we do not replace occurrences of and and after since the presence of either of these words causes substantial changes in the roles assigned to the sequence. As a baseline, we also train TPEs that use pre-defined role schemes (additional training information in Appendix A.9). For all of the sentence embedding models except Skip-thought, ROLE with continuous attention provides the lowest mean squared error at approximating the encoding (Table 3). The BOW (bag-of-words) role scheme represents a TPE that does not use compositional structure by assigning the same role to every filler; for each of the sentence embedding models tested except for SST, performance is within the same order of magnitude as structure-free BOW. found that a bag-of-words model scores extremely well on Natural Language Inference despite having no knowledge of word order, showing that structure is not necessary to perform well on the sorts of tasks commonly used to train sentence encoders. Although not definitive, these provide no evidence that these sentence embedding models rely on compositional representations. TPRs provide NNs an aspect of the systematicity of symbolic computation by disentangling fillers and roles. The learner needs to learn to process fillers -providing, essentially, what each input constituent contributes to the output -and to process roles -essentially, how these contributions are used in the output. In this work, we used ROLE to interpret the workings of a target encoder E, and in future work, we plan to train ROLE in an end-to-end manner, either using it as the encoder itself, or using it to regularize a standard (e.g., RNN) encoder with a loss term that rewards learning compositional encodings that ROLE can approximate well. We will test whether such an explicit bias for compositionality allows networks to train faster, or with fewer parameters, and to achieve more systematic generalization. Recent work showed improvements in compositionality by separating out syntax and semantics with attention , and our suggest that ROLE can also disentangle syntax and semantics. The structured representations that TPRs provide allow us to structure the processing that occurs over these representations. In fully-compositional tasks, the processing of fillers and roles can be encoded (and hence in principle learned) independently. In partially-compositional tasks, the processing of fillers and roles may be approximately encoded independently, with key interactions when a task deviates from full compositionality. In this work, we showed how a hidden embedding can be factored into fillers and roles. Similarly, it is possible to factor a weight matrix into weights that process roles and weights that process fillers. An illustration of this with a simple example from the SCAN task is provided in Appendix A.11. We plan to explore whether providing networks with a bias favoring weight matrices that factor in this way can improve systematic generalization. We have introduced ROLE, a neural network that learns to approximate the representations of an existing target neural network E using an explicit symbolic structure. ROLE successfully discovers symbolic structure both in models that explicitly define this structure and in an RNN without explicit structure trained on the fully-compositional SCAN task. When applied to sentence embedding models trained on partially-compositional tasks, ROLE performs better than hand-specified role schemes but still provides little evidence that the sentence encodings represent compositional structure. Uncovering the latent symbolic structure of NN representations on fully-compositional tasks is a significant step towards explaining how they can achieve the level of compositional generalization that they do, and suggests types of inductive bias to improve such generalization for partially-compositional tasks. Figure 3: The Tensor Product Encoder architecture. The yellow circle is an embedding layer for the fillers, and the blue circle is an embedding layer for the roles. These two vector embeddings are combined by an outer product to produce the green matrix representing the TPR of the constituent. All of the constituents are summed together to produce the TPR of the sequence, and then a linear transformation is applied to resize the TPR to the target encoders dimensionality. ROLE replaces the role embedding layer and directly produces the blue role vector. We use six hand-specified role schemes as a baseline to compare the learned role schemes against. Examples of each role scheme are shown in Table 4. We trained two TPEs end-to-end with an RNN decoder for our target networks on the digit sequence tasks. The left-to-right (LTR) TPE used a left-to-right role scheme applied to each element in the input and was connected to a unidirectional GRU decoder. The tree TPE used tree positions for each element in the input and was connected to a tree GRU decoder; here, the digit strings were parsed as a binary tree by a deterministic algorithm given in McCoy et al. (2019a, App. C). The filler and role dimensions were both 20 for the LTR TPE. The filler dimension was 20 for the Tree TPE, and the role dimension was 120. We used a hidden size of 60 for the GRU decoders. We used a patience of 2 for early stopping. The left-to-right TPE achieves 100% accuracy on the test set and the tree TPE achieves 98.62% on the test set. Letting A = {a t} T t=1, the regularization term applied during ROLE training is R = λ(R 1 +R 2 +R 3), where λ is a regularization hyperparameter and: Since each a t from a softmax, its elements are positive and sum to 1. Thus the factors in R 1 (A) are all non-negative, so R 1 assumes its minimal value of 0 when each a t has binary elements; since these elements must sum to 1, such an a t must be one-hot. R 2 (A) is also minimized when each a t is one-hot because when a vector's L 1 norm is 1, its L 2 norm is maximized when it is one-hot. Although each of these terms individually favor one-hot vectors, empirically we find that using both terms helps the training process. In a discrete symbolic structure, each position can hold at most one symbol, and the final term R 3 in ROLE's regularizer R is designed to encourage this. In the vector s A = T t=1 a t, the ρ th element is the total attention weight, over all symbols in the string, assigned to the ρ th role: in the discrete case, this must be 0 (if no symbol is assigned this role) or 1 (if a single symbol is assigned this role). Thus R 3 is minimized when all elements of s are 0 or 1 (R 3 is similar to R 1, but with squared terms since we are no longer assured each element is at most 1). It is important to normalize each role embedding in the role matrix R so that small attention weights have correspondingly small impacts on the weighted-sum role embedding. Once the TPEs in Sec. A.3 were trained, we extracted the hidden embedding for each item in the training, dev, and test sets. For both ROLE models trained on the digit sequence task, we used a bidirectional 2-layer LSTM with filler dimension of 20, and regularization constant λ = 1. For training, we used the ADAM optimizer with a learning rate of.001, batch size 32, and an early stopping patience of 10. The ROLE model trained on the LTR TPE was given 20 roles each of dimension 20. The ROLE model trained on the Tree TPE was given 120 roles each of dimension 120. To train the standard RNN on SCAN, we ran a limited hyperparameter search similar to the procedure in. Since our goal was to produce a single embedding that captured the entire input sequence, we fixed the architecture to GRU with a single hidden layer. We did not train models with attention, since we wanted to investigate whether a standard RNN could capture compositionality. The remaining hyperparameters were hidden dimension and dropout. We ran a search over the hidden dimension sizes of 50, 100, 200, and 400 as well as dropout with a value of 0,.1, and.5 applied to the word embeddings and recurrent layer. Each network was trained with the ADAM optimizer and a learning rate of.001 for 100,000 steps with a batch-size of 1. The best performing network had a hidden dimension or 100 and dropout of.1. For the ROLE models trained to approximate the GRU encoder trained on SCAN, we used a filler dimension of 100, a role dimension of 50 with 50 roles available. For training, we used the ADAM optimizer with a learning rate of.001, batch size 32, and an early stopping patience of 10. The role assignment module used a bidirectional 2-layer LSTM . We performed a hyperparameter search over the regularization coefficient λ using the values in the set [.1, .02, .01]. The best performing value was.02, and we used this model in our analysis. The algorithm below characterizes our post-hoc interpretation of which roles the Role Learner will assign to elements of the input to the SCAN model. This algorithm was created by hand based on an analysis of the Role Learner's outputs for the elements of the SCAN training set. The algorithm works equally well on examples in the training set and the test set; on both datasets, it exactly matches the roles chosen by the Role Learner for 98.7% of sequences (20,642 out of 20,910). The input sequences have three basic types that are relevant to determining the role assignment: sequences that contain and (e.g., jump around left and walk thrice), sequences that contain after (e.g., jump around left after walk thrice), and sequences without and or after (e.g., turn opposite right thrice). Within commands containing and or after, it is convenient to break the command down into the command before the connecting word and the command after it; for example, in the command jump around left after walk thrice, these two components would be jump around left and walk thrice. • Sequence with and: -Elements of the command before and: * Last word: 28 * First word (if not also last word): 46 * opposite if the command ends with thrice: 22 * Direction word between opposite and thrice: 2 * opposite if the command does not end with thrice: 2 * Direction word after opposite but not before thrice: 4 * around: 22 * Direction word after around: 2 * Direction word between an action word and twice or thrice: 2 -Elements of the command before and: * First word: 11 * Last word (if not also the first word): 36 * Second-to-last word (if not also the first word): 3 * Second of four words: 24 -and: 30 • Sequence with after: -Elements of the command before after: * Last word: 8 * Second-to-last word: 36 * First word (if not the last or second-to-last word): 11 * Second word (if not the last or second-to-last word): 3 -Elements of the command after after: * Last word: 46 * Second-to-last word: 4 * First word if the command ends with around right: 4 * First word if the command ends with thrice and contains a rotation: 10 * First word if the command does not end with around right and does not contain both thrice and a rotation: 17 * Second word if the command ends with thrice: 17 * Second word if the command does not end with thrice: 10 -after: 17 if no other word has role 17 or if the command after after ends with around left; 43 otherwise • Sequence without and or after: -Action word directly before a cardinality: 4 -Action word before, but not directly before, a cardinality: 34 -thrice directly after an action word: 2 -twice directly after an action word: 2 -opposite in a sequence ending with twice: 8 -opposite in a sequence ending with thrice: 34 -around in a sequence ending with a cardinality: 22 -Direction word directly before a cardinality: 2 -Action word in a sequence without a cardinality: 46 -opposite in a sequence without a cardinality: 2 -Direction after opposite in a sequence without a cardinality: 26 -around in a sequence without a cardinality: 3 -Direction after around in a sequence without a cardinality: 22 -Direction directly after an action in a sequence without a cardinality: 22 To show how this works with an example, consider the input jump around left after walk thrice. The command before after is jump around left. left, as the last word, is given role 8. around, as the second-to-last word, gets role 36. jump, as a first word that is not also the last or second-to-last word gets role 11. The command after after is walk thrice. thrice, as the last word, gets role 46. walk, as the second-to-last word, gets role 4. Finally, after gets role 17 because no other elements have been assigned role 17 yet. These predicted outputs match those given by the Role Learner. We offer several observations about this algorithm. 1. This algorithm may seem convoluted, but a few observations can illuminate how the roles assigned by such an algorithm support success on the SCAN task. First, a sequence will contain role 30 if and only if it contains and, and it will contain role 17 if and only if it contains after. Thus, by implicitly checking for the presence of these two roles (regardless of the fillers bound to them), the decoder can tell whether the output involves one or two basic commands, where the presence of and or after leads to two basic commands and the absence of both leads to one basic command. Moreover, if there are two basic commands, whether it is role 17 or role 30 that is present can tell the decoder whether the input order of these commands also corresponds to their output order (when it is and in play, i.e., role 30), or if the input order is reversed (when it is after in play, i.e., role 17). With these basic structural facts established, the decoder can begin to decode the specific commands. For example, if the input is a sequence with after, it can begin with the command after after, which it can decode by checking which fillers are bound to the relevant roles for that type of command. It may seem odd that so many of the roles are based on position (e.g., "first word" and "second-to-last word"), rather than more functionally-relevant categories such as "direction word." However, this approach may actually be more efficient: Each command consists of a single mandatory element (namely, an action word such as walk or jump) followed by several optional modifiers (namely, rotation words, direction words, and cardinalities). Because most of the word categories are optional, it might be inefficient to check for the presence of, e.g., a cardinality, since many sequences will not have one. By contrast, every sequence will have a last word, and checking the identity of the last word provides much functionally-relevant information: if that word is not a cardinality, then the decoder knows that there is no cardinality present in the command (because if there were, it would be the last word); and if it is a cardinality, then that is important to know, because the presence of twice or thrice can dramatically affect the shape of the output sequence. In this light, it is unsurprising that the SCAN encoder has implicitly learned several different roles that essentially mean the last element of a particular subcommand. 2. The algorithm does not constitute a simple, transparent role scheme. But its job is to describe the representations that the original network produces, and we have no a priori expectation about how complex that process may be. The role-assignment algorithm implicitly learned by ROLE is interpretable locally (each line is readily expressible in simple English), but not intuitively transparent globally. We see this as a positive , in two respects. First, it shows why ROLE is crucial: no human-generated role scheme would provide a good approximation to this algorithm. Such an algorithm can only be identified because ROLE is able to use gradient descent to find role schemes far more complex than any we would hypothesize intuitively. This enables us to analyze networks far more complex than we could analyze previously, being necessarily limited to hand-designed role schemes based on human intuitions about how to perform the task. Second, when future work illuminates the computation in the original SCAN GRU seq2seq decoder, the baroqueness of the role-assignment algorithm that ROLE has shown to be implicit in the seq2seq encoder can potentially explain certain limitations in the original model, which is known to suffer from severe failures of systematic generalization outside the training distribution . It is reasonable to hypothesize that systematic generalization requires that the encoder learn an implicit role scheme that is relatively simple and highly compositional. Future proposals for improving the systematic generalization of models on SCAN can be examined using ROLE to test the hypothesis that greater systematicity requires greater compositional simplicity in the role scheme implicitly learned by the encoder. 3. While the role-assignment algorithm of A.8.1 may not be simple, from a certain perspective, it is quite surprising that it is not far more complex. Although ROLE is provided 50 roles to learn to deploy as it likes, it only chooses to use 16 of them (only 16 are ever selected as the arg max(a t); see Sec. 6.1). Furthermore, the SCAN grammar generates 20,910 input sequences, containing a total of 151,688 words (an average of 7.25 words per input). This means that, if one were to generate a series of conditional statements to determine which role is assigned to each word in every context, this could in theory require up to 151,688 conditionals (e.g., "if the filler is 'jump' in the context 'walk thrice after opposite left', then assign role 17"). However, our algorithm involves just 47 conditionals. This reduction helps explain how the model performs so well on the test set: If it used many more of the 151,688 possible conditional rules, it would completely overfit the training examples in a way that would be unlikely to generalize. The 47-conditional algorithm we found is more likely to generalize by abstracting over many details of the context. 4. Were it not for ROLE's ability to characterize the representations generated by the original encoder in terms of implicit roles, providing an equally complete and accurate interpretation of those representations would necessarily require identifying the conditions determining the activation level of each of the 100 neurons hosting those representations. It seems to us grossly overly optimistic to estimate that each neuron's activation level in the representation of a given input could be characterized by a property of the input statable in, say, two lines of roughly 20 words/symbols; yet even then, the algorithm would require 200 lines, whereas the algorithm in A.8.1 requires 47 lines of that scale. Thus, by even such a crude estimate of the degree of complexity expected for an algorithm describing the representations in terms of neuron activities, the algorithm we find, stated over roles, is 4 times simpler. For each sentence embedding model, we trained three randomly initialized TPEs for each role scheme and selected the best performing one as measured by the lowest MSE. For each TPE, we used the original filler embedding from the sentence embedding model. This filler dimensionality is 25 for SST, 300 for SPINN and InferSent, and 620 for Skipthought. We applied a linear transformation to the pre-trained filler embedding where the input size is the dimensionality of the pre-trained embedding and the output size is also the dimensionality of the pre-trained embedding. This linearly transformed embedding is used as the filler vector in the filler-role binding in the TPE. For each TPE, we use a role dimension of 50. Training was done with a batch size of 32 using the ADAM optimizer with a learning rate of.001. To generate tree roles from the English sentences, we used the constituency parser released in version 3.9.1 of Stanford CoreNLP . For each sentence embedding model, we trained three randomly initialized ROLE models and selected the best performing one as measured by the lowest MSE. We used the original filler embedding from the sentence embedding model (25 for SST, 300 for SPINN and InferSent, and 620 for Skipthought). We applied a linear transformation to the pre-trained filler embedding where the input size is the dimensionality of the pre-trained embedding and the output size is also the dimensionality of the pre-trained embedding. This linearly transformed embedding is used as the filler vector in the filler-role binding in the TPE. We also applied a similar linear transformation to the pre-trained filler embedding before input to the role learner LSTM. For each ROLE model, we provide up to 50 roles with a role dimension of 50. Training was done with a batch size of 32 using the ADAM optimizer with a learning rate of.001. We performed a hyperparameter search over the regularization coefficient λ using the values in the set {1, 0.1, 0.01, 0.001, 0.0001}. For SST, SPINN, InferSent and SST, respectively, the best performing network used λ = 0.001, 0.01, 0.001, 0.1. At the end of Sec. 8 we remarked: "the structured representation bias that TPRs provide allows us to also provide biases to structure the processing that occurs over these representations. In fullycompositional tasks, the processing of fillers and roles can be encoded (and hence in principle learned) independently.... In this work, we showed how a hidden embedding can be factored into fillers and roles. Similarly, it is possible to factor a weight matrix into weights that process roles and weights that process fillers." Here we exemplify this with SCAN. One of the compositional rules for the mapping ϕ defined by the SCAN task is: ϕ(x twice) = ϕ(x) ϕ(x); for example, ϕ(walk twice) = ϕ(walk) ϕ(walk) = WALK WALK. For the purposes of illustration, suppose that, given the input string x twice, a NN encoder produces a representation that is approximated by the TPR of a single-constituent structure in which the filler x is assigned the role "argument of twice": x: r 2 ce -arg. So the TPR encoding of jump twice is e(jump twice) = e(jump : r 2 ce -arg) = e F (jump) ⊗ e R (r 2 ce -arg), where e F, e R are the embedding functions for fillers and roles, respectively. Let us also suppose that the output string is encoded as a TPR with positional roles R i, so that WALK LOOK has filler:role bindings {WALK : R 1, LOOK : R 2} and so has TPR e ({WALK : R 1, LOOK : R 2}) = e F (WALK) ⊗ e R (R 1) + e F (LOOK) ⊗ e R (R 2). So to generalize correctly to the input jump twice, producing output JUMP JUMP, the system needs to learn two things: how to map the filler -ϕ F (jump) = JUMP -and how to map the role -ϕ R (r 2 ce -arg) = {R 1, R 2}. The desired mapping is ϕ: e(jump : r 2 ce -arg) → e(JUMP JUMP), that is, ϕ: e F (jump) ⊗ e R (r 2 ce -arg) → e F (JUMP) ⊗ e R (R 1) + e F (JUMP) ⊗ e R (R 2). We now show that this can be accomplished through the separate filler-and role-mappings ϕ F: e F (jump) → e F (JUMP) and ϕ R: e R (r 2 ce -arg) → e R (R 1) + e R (R 2). To do this, we show that if ϕ F, ϕ R are respectively computed over embedding vectors through weight matrices W F, W R in a NN, then the weight tensor W = W F ⊗ W R will correctly map the embedding of jump twice to the embedding of JUMP JUMP. )] = e F (JUMP)⊗[e R (R 1)+e R (R 2)] = e F (JUMP)⊗e R (R 1)+e F (JUMP)⊗e R (R 2) = e(JUMP JUMP), as desired. This suggests that greater compositional generalization might be achieved by future models that are explicitly biased to utilize TPRs as internal representations and explicitly biased to factor their processing weights into those that process fillers (independently of their roles) and those that process roles (independently of their fillers), as in the SCAN example with W = W F ⊗ W R above.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We introduce a new analysis technique that discovers interpretable compositional structure in notoriously hard-to-interpret recurrent neural networks.

Represent the TLDR for finding the corresponding scientific article

By taking inspiration from linguistics, specifically the Universal Grammar hypothesis, we learn language agnostic universal representations which we can utilize to do zero-shot learning across languages.