Question: Represent the article for finding a single-sentence TLDR

Question: Understanding procedural language requires anticipating the causal effects of actions, even when they are not explicitly stated. In this work, we introduce Neural Process Networks to understand procedural text through (neural) simulation of action dynamics. Our model complements existing memory architectures with dynamic entity tracking by explicitly modeling actions as state transformers. The model updates the states of the entities by executing learned action operators. Empirical demonstrate that our proposed model can reason about the unstated causal effects of actions, allowing it to provide more accurate contextual information for understanding and generating procedural text, all while offering more interpretable internal representations than existing alternatives. Understanding procedural text such as instructions or stories requires anticipating the implicit causal effects of actions on entities. For example, given instructions such as "add blueberries to the muffin mix, then bake for one half hour," an intelligent agent must be able to anticipate a number of entailed facts (e.g., the blueberries are now in the oven; their "temperature" will increase). While this common sense reasoning is trivial for humans, most natural language understanding algorithms do not have the capacity to reason about causal effects not mentioned directly in the surface strings BID12 BID7 BID14. The process is a narrative of entity state changes induced by actions. In each sentence, these state changes are induced by simulated actions and must be remembered. In this paper, we introduce Neural Process Networks, a procedural language understanding system that tracks common sense attributes through neural simulation of action dynamics. Our network models interpretation of natural language instructions as a process of actions and their cumulative effects on entities. More concretely, reading one sentence at a time, our model attentively selects what actions to execute on which entities, and remembers the state changes induced with a recurrent memory structure. In FIG0, for example, our model indexes the "tomato" embedding, selects the "wash" and "cut" functions and performs a computation that changes the "tomato" embedding so that it can reason about attributes such as its "SHAPE" and "CLEANLINESS".Our model contributes to a recent line of research that aims to model aspects of world state changes, such as language models and machine readers with explicit entity representations BID4 BID6, as well as other more general purpose memory network variants BID30 BID26 BID5 BID23. This worldcentric modeling of procedural language (i.e., understanding by simulation) abstracts away from the surface strings, complementing text-centric modeling of language, which focuses on syntactic and semantic labeling of surface words (i.e., understanding by labeling).Unlike previous approaches, however, our model also learns explicit action representations as functional operators (See FIG0 . While representations of action semantics could be acquired through an embodied agent that can see and interact with the world BID22, we propose to learn these representations from text. In particular, we require the model to be able to explain the causal effects of actions by predicting natural language attributes about entities such as "LOCATION" and "TEMPERATURE". The model adjusts its representations of actions based on errors it makes in predicting the ant state changes to attributes. This textual simulation allows us to model aspects of action causality that are not readily available in existing simulation environments. Indeed, most virtual environments offer limited aspects of the world -with a primary focus on spatial relations BID22 BID1 BID29 . They leave out various other dimensions of the world states that are implied by diverse everyday actions such as "dissolve" (change of "COMPOSITION") and "wash" (change of "CLEANLINESS").Empirical demonstrate that parametrizing explicit action embeddings provides an inductive bias that allows the neural process network to learn more informative context representations for understanding and generating natural language procedural text. In addition, our model offers more interpretable internal representations and can reason about the unstated causal effects of actions explained through natural language descriptors. Finally, we include a new dataset with fine-grained annotations on state changes, to be shared publicly, to encourage future research in this direction. The neural process network is an interpreter that reads in natural language sentences, one at a time, and simulates the process of actions being applied to relevant entities through learned representations of actions and entities. The main component of the neural process network is the simulation module (§2.5), a recurrent unit whose internals simulate the effects of actions being applied to entities. A set of V actions is known a priori and an embedding is initialized for each one, F = {f 1, ...f V}. Similarly, a set of I entities is known and an embedding is initialized for each one: E = {e 1, ... e I}. Each e i can be considered to encode information about state attributes of that entity, which can be extracted by a set of state predictors (§2.6). As the model reads text, it "applies" action embeddings to the entity vectors, thereby changing the state information encoded about the entities. For any document d, an initial list of entities I d is known and E d = {e i |i ∈ I d} ⊂ E entity state embeddings are initialized. As the neural process network reads a sentence from the document, it selects a subset of both F (§2.3) and E d (§2.4) based on the actions performed and entities affected in the sentence. The entity state embeddings are changed by the action and the new embeddings are used to predict end states for a set of state changes (§2.6). The prediction error for end states is backpropagated to the action embeddings, learning action representations that model the simulation of desired causal effects on entities. This process is broken down into five modules below. Unless explicitly defined, all W and b variables are parametrized linear projections and biases. We use the notation {e i} t when referring to the values of the entity embeddings before processing sentence s t. Given a sentence s t, a Gated Recurrent Unit encodes each word and outputs its last hidden vector as a sentence encoding h t. Given h t from the sentence encoder, the action selector (bottom left in Fig. 2 beets", both f wash and f cut must be selected. To account for multiple actions, we make a soft selection over F, yielding a weighted sum of the selected action embeddingsf t : DISPLAYFORM0 where MLP is a parametrized feed-forward network with a sigmoid activation and w p ∈ R V is the attention distribution over V possible actions ( §3.1). We compose the action embedding by taking the weighted average of the selected actions. Sentence Attention Given h t from the sentence encoder, the entity selector chooses relevant entities using a soft attention mechanism: DISPLAYFORM0 where W 2 is a bilinear mapping, e i0 is a unique key for each entity (§2.5), and d i is the attention weight for entity embedding e i. For example, in "wash and cut beets and carrots", the model should select e beet and e carrot. Recurrent Attention While sentence attention would suffice if entities were always explicitly mentioned, natural language often elides arguments or uses referent pronouns. As such, the module must be able to consider entities mentioned in previous sentences. Usingh t, the model computes a soft choice over whether to choose affected entities from this step's attention d i or the previous step's attention distribution. DISPLAYFORM1 where c ∈ R 3 is the choice distribution, a it−1 is the previous sentence's attention weight for each entity, a it is the final attention for each entity, and 0 is a vector of zeroes (providing the option to not change any entity). Prior entity attentions can propagate forward for multiple steps. Entity Memory A unique state embedding e i is initialized for every entity i in the document. A unique key to index each embedding e i0 is set as the initial value of the embedding BID4 BID17. After the model reads s t, it modifies {e i} t to reflect changes influenced by actions. At every time step, the entity memory receives the attention weights from the entity selector, normalizes them and computes a weighted average of the relevant entity state embeddings: DISPLAYFORM0 Applicator Given the action summary embeddingf t and the entity summary embeddingē t, the applicator (middle right in Fig. 2) applies the selected actions to the selected entities, and outputs the new proposal entity embedding k t. DISPLAYFORM1 where W 4 is a third order tensor projection. The vector k t is the new representation of the entityē t after the applicator simulates the action being applied to it. Entity Updater The entity updater interpolates the new proposal entity embedding k t and the set of current entity embeddings {e i} t: DISPLAYFORM2 yielding an updated set of entity embeddings {e i} t+1. Each embedding is updated proportional to its entity's unnormalized attention a i, allowing the model to completely overwrite the state embedding for any entity. For example, in the sentence "mix the flour and water," the embeddings for e f lour and e water must both be overwritten by k t because they no longer exist outside of this new composition. Given the new proposal entity embedding k t, the state predictor (bottom right in Fig. 2) predicts changes to the ing entity embedding k t along the following six dimensions: location, cookedness, temperature, composition, shape, and cleanliness. Discrete multi-class classifiers, one for each dimension, take in k t and predict a unique end state for their corresponding state change type: DISPLAYFORM0 For location changes, which require contextual information to predict the end state, k t is concatenated with the original sentence representation h t to predict the final state. In this work we focus on physical action verbs in cooking recipes. We manually collect a set of 384 actions such as cut, bake, boil, arrange, and place, organizing their causal effects along the following predefined dimensions: LOCATION, COOKEDNESS, TEMPERATURE, SHAPE, CLEANLI-NESS and COMPOSITION. The textual simulation operated by the model induces state changes along these dimensions by applying actions functions from the above set of 384. For example, cut entails a change in SHAPE, while bake entails a change in TEMPERATURE, COOKEDNESS, and even LO-CATION. We annotate the state changes each action induces, as well as the end state of the action, using Amazon Mechanical Turk. The set of possible end states for a state change can range from 2 for binary state changes to more than 200 (See Appendix C for details). For learning and evaluation, we use a subset of the Now You're Cooking dataset BID9. We chose 65816 recipes for training, 175 recipes for development, and 700 recipes for testing. For the development and test sets, crowdsourced workers densely annotate actions, entities and state changes that occur in each sentence so that we can tune hyperparameters and evaluate on gold evaluation sets. Annotation details are provided in Appendix C.3. The neural process network is trained by jointly optimizing multiple losses for the action selector, entity selector, and state change predictors. Importantly, our training scheme uses weak supervision because dense annotations are prohibitively expensive to acquire at a very large scale. Thus, we heuristically extract verb mentions from each recipe step and assign a state change label based on the state changes induced by that action (§3.1). Entities are extracted similarly based on string matching between the instructions and the ingredient list. We use the following losses for training: Action Selection Loss Using noisy supervision, the action selector is trained to minimize the cross-entropy loss for each possible action, allowing multiple actions to be chosen at each step if multiple actions are mentioned in a sentence. The MLP in the action selector (Eq. 1) is pretrained. Entity Selection Loss Similarly, to train the attentive entity selector, we minimize the binary cross-entropy loss of predicting whether each entity is affected in the sentence. State Change Loss For each state change predictor, we minimize the negative loglikelihood of predicting the correct end state for each state change. Coverage Loss An underlying assumption in many narratives is that all entities that are mentioned should be important to the narrative. We add a loss term that penalizes narratives whose combined attention weights for each entity does not sum to more than 1. DISPLAYFORM0 where a it is the attention weight for a particular entity at sentence t and I d is the number of entities in a document. S t=1 a it is upper bounded by 1. This is similar to the coverage penalty used in neural machine translation BID28. We evaluate our model on a set of intrinsic tasks centered around tracking entities and state changes in recipes to show that the model can simulate preliminary dynamics of the recipe task. Additionally, we provide a qualitative analysis of the internal components of our model. Finally, we evaluate the quality of the states encoded by our model on the extrinsic task of generating future steps in a recipe. In the tracking task, we evaluate the model's ability to identify which entities are selected and what changes have been made to them in every step. We break the tracking task into two separate evalua- Metrics In the entity selection test, we report the F1 score of choosing the correct entities in any step. A selected entity is defined as one whose attention weight a i is greater than 50% (§2.4). Because entities may be harder to predict when they have been combined with other entities (e.g., the mixture may have a new name), we also report the recall for selecting combined (CR) and uncombined (UR) entities. In the end state prediction test, we report how often the model correctly predicts the state change performed in a recipe step and the ant end state. This score is then scaled by the accuracy of predicting which entities were changed in that same step. We report the average F1 and accuracy across the six state change types. Baselines We compare our models against two baselines. First, we built a GRU model that is trained to predict entities and state changes independently. This can be viewed as a bare minimum network with no action representations or recurrent entity memory. The second baseline is a Recurrent Entity Network BID4 with changes to fit our task. First, the model can tie memory cells to a subset of the full list of entities so that it only considers entities that are present in a particular recipe. Second, the entity distribution for writing to the memory cells is re-used when we query the memory cells. The normalized weighted average of the entity cells is used as the input to the state predictors. The unnormalized attention when writing to each cell is used to predict selected entities. Both baselines are trained with entity selection and state change losses (§3.3).Ablations We report on six ablations. First, we remove the recurrent attention (Eq. 3). The model only predicts entities using the current encoder hidden state. In the second ablation, the model is trained with no coverage penalty (Eq. 9). The third ablation prunes the connection from the action selector w p to the entity selector (Eq. 2). We also explore not pretraining the action selector. Finally, we look at two ablations where we intialize the action embeddings with vectors from a skipgram model. In the first, the model operates normally, and in the second, we do not allow gradients to backpropagate to the action embeddings, updating only the mapping tensor W 4 instead (Eq. 6). The generation task tests whether our system can produce the next step in a recipe based on the previous steps that have been performed. The model is provided all of the previous steps as context. We report the combined BLEU score and ROUGE score of the generated sequence relative to the reference sequence. Each candidate sequence has one reference sentence. Both metrics are computed at the corpus-level. Also reported are "VF1", the F1 score for the overlap of the actions performed in the reference sequence and the verbs mentioned in the generated sequence, and "SF1", the F1 score for the overlap of end states annotated in the reference sequence and predicted by the generated sequences. End states for the generated sequences are extracted using the lexicon from Section 3.1 based on the actions performed in the sentence. Setup To apply our model to the task of recipe step generation, we input the context sentences through the neural process network and record the entity state vectors once the entire context has st Let cool. selected oats, sugar, flour, corn syrup, milk, vanilla extract, salt correct oats, sugar, flour, corn syrup, milk, vanilla extract, salt Good st−1 In a large saucepan over low heat, melt marshmallows.st Add sprinkles, cereal, and raisins, stir until well coated. selected marshmallows, cereal, raisins correct marshmallows, cereal, raisins, sprinkles Bad st−3 Ladle the barbecue sauce around the crust and spread. st−2 Add mozzarella, yellow cheddar, and monterey jack cheese. st−1 Next, add onion mixture and sliced chicken breast.st Top pizza with jalapeno peppers. selected jalapenos correct crust, sauce, mozzarella, cheddar, monterey jack, white onion, chicken, jalapenos Bad st−2 Combine 1 cup flour, salt, and 1 tbsp sugar. st−1 Cut in butter until mixture is crumbly, then sprinkle with vinegar.st Gather dough into a ball and press into bottom of 9 inch springform pan. selected butter, vinegar correct flour, salt, sugar, butter, vinegar TAB5: Examples of the model selecting entities for sentence s t. The previous sentences are provided as context in cases where they are relevant. been read (§2.5). These vectors can be viewed as a snapshot of the current state of the entities once the preceding context has been simulated inside the neural process network. We encode these vectors using a bidirectional GRU and take the final time step hidden state e I. A different GRU encodes the context words in the same way (yielding h T) and the first hidden state input to the decoder is computed using the projection function: DISPLAYFORM0 where • is the Hadamard product between the two encoder outputs. All models are trained by minimizing the negative loglikelihood of predicting the next word for the full sequence. Implementation details can be found in Appendix A. Baselines For the generation task, we use three baselines: a seq2seq model with no attention, an attentive seq2seq model BID0, and a similar variant as our NPN generator, except where the entity states have been computed by the Recurrent Entity Network (EntNet) baseline (§4.1). Implementation details for baselines can be found in Appendix B. Entity Selection As shown in Table 8, our full model outperforms all baselines at selecting entities, with an F1 score of 55.39%. The ablation study shows that the recurrent attention, coverage loss, action connections and action selector pretraining improve performance. Our success at predicting entities extends to both uncomposed entities, which are still in their raw forms (e.g., melt the butter → butter), and composed entities, in which all of the entities that make up a composition must be selected. For example, in a Cooking lasagna recipe, if the final step involves baking the prepared lasagna, the model must select all the entities that make up the lasagna (e.g., lasagna sheets, beef, tomato sauce). In as compositional entities (Ex. 1, 3), and elided arguments over long time windows (Ex. 2). We also provide examples where the model fails to select the correct entities because it does not identify the mapping between a reference construct such as "pizza" (Ex. 4) or "dough" (Ex. 5) and the set of entities that composes it, showcasing the difficulty of selecting the full set for a composed entity. State Change Tracking In Table 8, we show that our full model outperforms competitive baselines such as Recurrent Entity Networks BID4 and jointly trained GRUs. While the ablation without the coverage loss shows higher accuracy, we attribute this to the fact that it predicts a smaller number of total state changes. Interestingly, initializing action embeddings with skipgram vectors and locking their values shows relatively high performance, indicating the potential gains in using powerful pretrained representations to represent actions. Action Embeddings In our model, each action is assigned its own embedding, but many actions induce similar changes in the physical world (e.g.,"cut" and "slice"). After training, we compute the pairwise cosine similarity between each pair of action embeddings. In TAB6, we see that actions that perform similar functions are neighbors in embedding space, indicating the model has captured certain semantic properties of these actions. Learning action representations through the state changes they induce has allowed the model to cluster actions by their transformation functions. Entity Compositions When individual entities are combined into new constructs, our model averages their state embeddings (Eq. 5), applies an action embedding to them (Eq. 6), and writes them to memory (Eq. 7). The state embeddings of entities that are combined should be overwritten by the same new embedding. In Figure 3, we present the percentage increase in cosine similarity for state embeddings of entities that are combined in a sentence (blue) as opposed to the percentage increase for those that are not (red bars). While the soft attention mechanism for entity selection allows similarities to leak between entity embeddings, our system is generally able to model the compositionality patterns that from entities being combined into new constructs. Step Generation Our in TAB7 indicate that sequences generated using the neural process network entity states as additional input yield higher scores than competitive baselines. The entity states allow the model to predict next steps conditioned on a representation of the world being simulated by the neural process network. Additionally, the higher VF1 and SF1 scores indicate that the model is indeed using the extra information to better predict the actions that should follow the context provided. Example generations for each baselines from the dev set are provided in Table 6, Context Preheat oven to 425 degrees. Reference Melt butter in saucepan and mix in bourbon, thyme, pepper, and salt. NPN Melt butter in skillet. Seq2seq Lightly grease 4 x 8 baking pan with sunflower oil. Attentive Seq2seq Combine all ingredients and mix well. EntNet In a large bowl, combine flour, baking powder, baking soda, salt, and pepper. Context Pour egg mixture over caramelized sugar in cake pan. Place cake pan in large shallow baking dish. Bake for 55 minutes or until knife inserted into flan comes out clean. Reference Cover and chill at least 8 hours. NPN Refrigerate until ready to use. Seq2seq Serve at room temperature. Attentive Seq2seq Store in an airtight container. EntNet Store in an airtight container. Context Cut squash into large pieces and steam. Remove cooked squash from shells; Reference Measure 4 cups pulp and reserve remainder for another dish. NPN Drain. Seq2seq Mash pulp with a fork. Attentive Seq2seq Set aside. EntNet Set aside. Table 6: Examples of the model generating sentences compared to baselines. The context and reference are provided first, followed by our model's generation and then the baseline generationsshowing that the NPN generator can use information about ingredient states to reason about the most likely next step. The first and second examples are interesting as it shows that the NPN-aware model has learned to condition on entity state -knowing that raw butter will likely be melted or that a cooked flan must be refrigerated. The third example is also interesting because the model learns that cooked vegetables such as squash will sometimes be drained, even if it is not relevant to this recipe because the squash is steamed. The seq2seq and EntNet baselines, meanwhile, output reasonable sentences given the immediate context, but do not exhibit understanding of global patterns. Recent studies in machine comprehension have used a neural memory component to store a running representation of processed text BID30 BID26 BID5 BID23. While these approaches map text to memory vectors using standard neural encoder approaches, our model, in contrast, directly interprets text in terms of the effects actions induce in entities, providing an inductive bias for learning how to represent stored memories. More recent work in machine comprehension also sought to couple the memory representation with tracking entity states BID4. Our work seeks to provide a relatively more structured representation of domain-specific action knowledge to provide an inductive bias to the reasoning process. Neural Programmers BID20 have also used functions to simulate reasoning, by building a model to select rows in a database and applying operation on those selected rows. While their work explicitly defined the effect of a number of operations for those rows, we provide a framework for learning representations for a more expansive set of actions, allowing the model to learn representations for how actions change the state space. Works on instructional language studied the task of building discrete graph representations of recipes using probabilistic models BID8 BID19 BID18. We propose a complementary new model by integrating action and entity relations into the neural network architecture and also address the additional challenge of tracking the state changes of the entities. Additional work in tracking states with visual or multimodal context has focused on 1) building graph representations for how entities change in goal-oriented domains BID3 BID24 or 2) tracking visual state changes based on decisions taken by agents in environment simulators such as videos or games BID1 BID29 BID22. Our work, in contrast, models state changes in embedding space using only text-based signals to map real-world actions to algebraic transformations. We introduced the Neural Process Network for modeling a process of actions and their causal effects on entities by learning action transformations that change entity state representations. The model maintains a recurrent memory structure to track entity states and is trained to predict the state changes that entities undergo. Empirical demonstrate that our model can learn the causal effects of action semantics in the cooking domain and track the dynamic state changes of entities, showing advantages over competitive baselines. A TRAINING DETAILS OF OUR FULL MODEL AND ABLATIONS The hidden size of the instruction encoder is 100, the embedding sizes of action functions and entities are 30. We use dropout with a rate of 0.3 before any non-recurrent fully connected layers BID25. We use the Adam optimizer BID11 ) with a learning rate of.001 and decay by a factor of 0.1 if we see no improvement on validation loss over three epochs. We stop training early if the development loss does not decrease for five epochs. The batch size is 64. We use two instruction encoders, one for the entity selector, and one for the action selector. Word embeddings and entity embeddings are initialized with skipgram embeddings BID15;b) using a word2vec model trained on the training set. We use a vocabulary size of 7358 for words, and 2996 for entities. Gradients with respect to the coverage loss (Eq. 9) are only backpropagated in steps where no entity is annotated as being selected. To account for the false negatives in the training data due to the heuristic generation of the labels, gradients with respect to the entity selection loss are zeroed when no entity label is present. The hidden size of the context encoder is 200. The hidden size of the state vector encoder is 200. State vectors have dimensionality 30 (the same as in the neural process network). Dropout of 0.3 is used during training in the decoder. The context and state representations are projected jointly using an element-wise product followed by a linear projection BID10. Both encoders and the decoder are single layer. The learning rate is 0.0003 initially and is halved every 5 epochs. The model is trained with the Adam optimizer. Joint Gated Recurrent Unit The hidden state of the GRU is 100. We use a dropout with a rate of 0.3 before any non-recurrent fully connected layers. We use the Adam optimizer with a learning rate of.001 and decay by a factor of 0.1 if we see no improvement on validation loss over a single epoch. We stop training early if the development loss does not decrease for five epochs. The batch size is 64. We use encoders, one for the entity selector, and one for the state change predictors. Word embeddings are initialized with skipgram embeddings using a word2vec model trained on the training set. We use a vocabulary size of 7358 for words. Recurrent Entity Networks Memory cells are tied to the entities in the document. For a recipe with 12 ingredients, 12 entity cells are initialized. All hyperparameters are the same as the in the bAbI task from BID4. The learning rate start at 0.01 and is halved every 25 epochs. Entity cells and word embeddings are 100 dimensional. The encoder is a multiplicative mask initialized the same as in BID4. Intermediate supervision from the weak labels is provided to help predict entities. A separate encoder is used for computing the attention over memory cells and the content to write to the memory. Dropout of 0.3 is used in the encoders. The batch size is 64. We use a vocabulary size of 7358 for words, and 2996 for entities. Seq2seq The encoder and decoder are both single-layer GRUs with hidden size 200. We use dropout with probability 0.3 in the decoder. We train with the Adam optimizer starting with a learning rate 0.0003 that is halved every 5 epochs. The encoder is bidirectional. The model is trained to minimize the negative loglikelihood of predicting the next word. The encoder is the same as in the seq2seq baseline. A multiplicative attention between the decoder hidden state and the context vectors is used to compute the attention over the context at every decoder time step. The model is trained with the same learning rate, learning schedule and loss function as the seq2seq baseline. The model is trained in the same way as the NPN generator model in Appendix A.2 except that the state representations used as input are produced from by EntNet baseline described in Section 4.1 and Appendix B.1. We provide workers with a verb, its definition, an illustrative image of the action, and a set of sentences where the verb is mentioned. Workers are provided a checklist of the six state change types and instructed to identify which of them the verb causes. They are free to identify multiple changes. Seven workers annotate each verb and we assign a state change based on majority vote. Of the set of 384 verbs extracted, only 342 have a state change type identified with them. Of those, 74 entail multiple state change types. We give workers a verb, a state change type, and an example with the verb and ask them to provide an end state for the ingredient the verb is applied to in the example. We then use the answers to manually aggregate a set of end states for each state change type. These end states are used as labels when the model predicting state changes. For example, a LOCATION change might lead to an end state of "pan," "pot", or "oven." End states for each state change type are provided in Annotators are instructed to note any entities that undergo one of the six state changes in each step, as well as to identify new combinations of ingredients that are created. For example, the sentence "Cut the tomatoes and add to the onions" would involve a SHAPE change for the tomatoes and a combination created from the "tomatoes" and "onions". In a separate task, three workers are asked to identify the actions performed in every sentence of the development and test set recipes. If an action receives a majority vote that it is performed, it is included in the annotations. Table 8: Results for entity selection and state change selection on the development set when randomly dropping a percentage of the training labels

We propose a new recurrent memory architecture that can track common sense state changes of entities by simulating the causal effects of actions.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: There has been a recent trend in training neural networks to replace data structures that have been crafted by hand, with an aim for faster execution, better accuracy, or greater compression. In this setting, a neural data structure is instantiated by training a network over many epochs of its inputs until convergence. In many applications this expensive initialization is not practical, for example streaming algorithms --- where inputs are ephemeral and can only be inspected a small number of times. In this paper we explore the learning of approximate set membership over a stream of data in one-shot via meta-learning. We propose a novel memory architecture, the Neural Bloom Filter, which we show to be more compressive than Bloom Filters and several existing memory-augmented neural networks in scenarios of skewed data or structured sets. One of the simplest questions one can ask of a set of data is whether or not a given query is contained within it. Is q, our query, a member of S, our chosen set of observations? This set membership query arises across many computing domains; from databases, network routing, and firewalls. One could query set membership by storing S in its entirety and comparing q against each element. However, more space-efficient solutions exist. The original and most widely implemented approximate set membership data-structure is the Bloom Filter BID2. It works by storing sparse distributed codes, produced from randomized hash functions, within a binary vector. The Bloom-filter trades off space for an allowed false positive rate, which arises due to hash collisions. However its error is one-sided; if an element q is contained in S then it will always be recognized. It never emits false negatives. One can find Bloom Filters embedded within a wide range of production systems; from network security BID16, to block malicious IP addresses; databases, such as Google's Bigtable BID7, to avoid unnecessary disk lookups; cryptocurrency BID19, to allow clients to filter irrelevant transactions; search, such as Facebook's typeahead search BID0, to filter pages which do not contain query prefixes; and program verification BID13, to avoid recomputation over previously observed states. While the main appeal of Bloom Filters is favourable compression, another important quality is the support for dynamic updates. New elements can be inserted in O time. This is not the case for all approximate set membership data structures. For example, perfect hashing saves ≈ 40% space over Bloom Filters but requires a pre-processing stage that is polynomial-time in the number of elements to store BID12. Whilst the static set membership problem is interesting, it limits the applicability of the algorithm. For example, in a database application that is serving a high throughput of write operations, it may be intractable to regenerate the full data-structure upon each batch of writes. We thus focus on the data stream computation model BID27, where input observations are assumed to be ephemeral and can only be inspected a constant number of timesusually once. This captures many real-world applications: network traffic analysis, database query serving, and reinforcement learning in complex domains. Devising an approximate set membership data structure that is not only more compressive than Bloom Filters, but can be applied to either dynamic or static sets, could have a significant performance impact on modern computing applications. In this paper we investigate this problem using memory-augmented neural networks and meta-learning. We build upon the recently growing literature on using neural networks to replace algorithms that are configured by heuristics, or do not take advantage of the data distribution. For example, Bloom Filters are indifferent to the data distribution. They have near-optimal space efficiency when data is drawn uniformly from a universe set BID5 (maximal-entropy case) but (as we shall show) are sub-optimal when there is more structure. Prior studies on this theme have investigated compiler optimization BID11, computation graph placement, and data index structures such as b-trees BID22.In the latter work, BID22 explicitly consider the problem of static set membership. By training a neural network over a fixed S (URLs from Google's Transparency Report) with negative examples in the form of held-out URLs, they observe 36% space reduction over a conventional Bloom Filter 1. Crucially this requires iterating over the storage set S a large number of times to embed its salient information into the weights of a neural network classifier. For a new S this process would have to be repeated from scratch. Instead of learning from scratch, we draw inspiration from the few-shot learning advances obtained by meta-learning memory-augmented neural networks BID30 BID34. In this setup, tasks are sampled from a common distribution and a network learns to specialize to (learn) a given task with few examples. This matches very well to applications where many Bloom Filters are instantiated over different subsets of a common data distribution. For example, a Bigtable database usually contains one Bloom Filter per SSTable file. For a large table that contains Petabytes of data, say, there can be over 100, 000 separate instantiated data-structures which share a common row key format and query distribution. Meta-learning allows us to exploit this common redundancy. The main contributions of this paper are A new sparse memory-augmented neural network architecture, the Neural Bloom Filter, which learns to write to memory using a distributed write scheme, and An empirical evaluation of the Neural Bloom Filter meta-learned on one-shot approximate set membership problems of varying structure. We compare with the classical Bloom Filter alongside other memory-augmented neural networks such as the Differentiable Neural Computer and Memory Networks BID33. We find when there is no structure, that differentiates the query set elements and queries, the Neural Bloom Filter learns a solution similar to a Bloom Filter derivative (a Bloom-g filter BID28), but when there is a lot of structure the solution can be considerably more space-efficient. The problem of exact set membership is to state whether or not a given query q belongs to a set of n distinct observations S = {x 1, . . ., x n} where x i are drawn from a universe set U. By counting the number of distinct subsets of size n it can be shown that any such exact set membership tester requires at least log 2 |U | n bits of space. To mitigate the space dependency on |U |, which can be prohibitively large, one can relax the constraint on perfect correctness. Approximate set membership allows for a false positive rate of at most. Specifically we answer q ∈ A(S) where A(S) ⊇ S and p(q ∈ A(S) − S) ≤. It can be shown 2 the space requirement for approximate set membership of uniformly sampled observations is at least n log 2 bits BID5 which can be achieved with perfect hashing. So for a false positive rate of 1%, say, this amounts to 6.6 bits per element. In contrast to storing raw or compressed elements this can be a huge space saving, for example ImageNet images require 108 KB per image on average when compressed with JPEG, an increase of over four orders of magnitude. The Bloom Filter BID2 is a data structure which solves the dynamic approximate set membership problem with near-optimal space complexity. It assumes access to k uniform hash functions DISPLAYFORM0 m is a binary string of length m which is initialized to zero. Writes are performed by hashing an input x to k locations in M and setting the corresponding bits to 1, M [h i (x)] ← 1; i = 1,..., k. For a given query q the Bloom Filter returns true if all corresponding hashed locations are set to 1 and returns false otherwise: DISPLAYFORM1. This incurs zero false negatives, as any previously observed input must have enabled the corresponding bits in M, however there can be false positives due to hash collisions. To achieve a false positive rate of with minimal space one can set k = log 2 (1/) and m = n log 2 (1/) log 2 e, where e is Euler's number. The ing space is a factor of log 2 e ≈ 1.44 from the optimal static lower bound given by BID5. Recurrent neural networks such as LSTMs retain a small amount of memory via the recurrent state. However this is usually tied to the number of trainable parameters in the model. There has been recent interest in augmenting neural networks with a larger external memory. The method for doing so, via a differentiable write and read interface, was first popularized by the Neural Turing Machine (NTM) BID17 and its successor the Differentiable Neural Computer (DNC) in the context of learning algorithms, and by Memory Networks BID33 in the context of question answering. Memory Networks store embeddings of the input in separate rows of a memory matrix M. Reads are performed via a differentiable content-based addressing operation. Given a query embedding q we take some similarity measure D (e.g. cosine similarity, or negative euclidean distance) against each row in memory and apply a softmax to obtain a soft address vector a ∝ e D(q,M). A read is then a weighted sum over memory r ← a T M. The NTM and DNC use the same content-based read mechanism, but also learns to write. These models can arbitrate whether to write to slots in memory with similar content (content-based writes), temporally ordered locations, or unused memory. When it comes to capacity, there has been consideration to scaling both the DNC and Memory Networks to very large sizes using sparse read and write operations BID29 BID6. However another way to increase the capacity is to increase the amount of compression which occurs in memory. Memory Nets can create compressive representations of each input, but cannot compress jointly over multiple inputs because they are hard-wired to write one slot per timestep. The NTM and DNC can compress over multiple slots in memory because they can arbitrate writes across multiple locations, but in practice seem to choose very sharp read and write addresses. The Kanerva Machine BID36 tackles memory-wide compression using a distributed write scheme to jointly compose and compress its memory contents. The model uses content-based addressing over a separate learnable addressing matrix A, instead of the memory M, and thus learns where to write. We take inspiration from this scheme. One approach to learning set membership in one-shot would be to use a recurrent neural network, such as an LSTM or DNC. Here, the model sequentially ingests the N elements to store, answers a set of queries using the final state, and is trained by BPTT. Whilst this is a general training approach, and the model may learn a compressive solution, it does not scale well to larger number of elements. Even when N = 1000, backpropagating over a sequence of this length induces computational and optimization challenges. For larger values this quickly becomes intractable. Alternatively one could store an embedding of each element x i ∈ S in a slot-based Memory Network. This is more scalable as it avoids BPTT, because the gradients of each input can be calculated in parallel. However Memory Networks are not a space efficient solution (as shown in in Section 5) because there is no joint compression of inputs. We propose a memory model, the Neural Bloom Filter, that is both compressive and scalable. The network uses a purely additive write operation -i.e. no multiplicative gating or squashing -to update a real-valued memory matrix. An additive write operation can be seen as a continuous relaxation of the the Bloom Filter's logical OR write operation. The benefit of a purely additive write is that the ing memory does not depend upon the ordering of the input (as addition is commutative) and the gradient with respect to each input can be computed in parallel. The network addresses memory by classifying which memory slots to read or write to via a softmax, conditioned purely on the input. We can think of this as a continuous relaxation of the Bloom Filter's hash function. To make the addressing more efficient for larger memory sizes, we sparsify the softmax by preserving only the top k components. When using a sparse address, we find the network can fixate on a subset of rows. This observation is common to prior sparse addressing work BID31. We find sphering, often dubbed whitening, the addressing activations prior to the softmax (see Appendix F for an ablation) remedies this. Finally we make the final linear layer (denoted A) of the addressing softmax to be non-trainable. This is to ensure the number of trainable model parameters is independent of memory size. It also allows the memory to be resized at test-time. The full architecture depicted in FIG0 consists of a controller network which encodes the input to an embedding z ← f enc (x) and transforms this to a write word w ← f w (z) and a query s ← f q (z). An addressing network takes s and performs a sphering transformation to produce a query q with decorrelated dimensions. We used a moving average ZCA transformation. We choose ZCA over other decorrelation approaches as this worked well in practice, e.g. over PCA. We describe the exact moving zca transformation in full in Supplementary A.1. The address a ← σ k (q T A) is computed over memory via content-based attention between q and a non-learnable address matrix A. Here, σ k denotes a sparse softmax where the top k similarities are retained. The addressing matrix A ∼ N (0, I) is chosen to be a fixed sample of Gaussian random variables. It simply represents a linear map from queries to memory addresses; we keep it non-trainable to avoid the coupling of model parameters and memory capacity. A write is performed by running the controller and addressing network, and then additively writing w to the top k addresses in a, M t+1 ← M t + wa T. This scheme is inspired by the Bloom Filter, replacing a logical OR for an addition, and it ensures that we do not have to backpropagatethrough-time (BPTT) over sequential writes. A read is performed by also running the controller and addressing network and retrieving the top k addresses from M. These rows in memory are multiplied element-wise with the address weight r ← M a and are fed through an MLP with the residual inputs w and z. We found this to be more powerful than the conventional read operation r ← a T M used by the DNC and Memory Networks, as it allows for non-linear interactions between rows in memory at the time of read. An overview of the operations is described below. Controller network Write DISPLAYFORM0 To give an example network configuration, in our experiments we chose f enc to be a 3-layer CNN in the case of image input, and in the case of text input we choose f enc to be an LSTM with 128 hidden units. We chose f w and f q to be an MLP with a single hidden layer of size 128, and f out to be a 3-layer mlp with residual connections. We used leaky relus as the non-linearity. We further discuss the motivation for decorrelation / sphering in the addressing network, and the model's relation to uniform hashing in Appendix A.2. We also discuss how the model could be implemented for O(log m) time reads and writes, where m is the size of memory, and with an O network space overhead by avoiding the storage of A, the addressing matrix, in Appendix A.3. In this section we discuss space lower bounds for the approximate set membership problem when there is some structure to the storage or query set. This can help us formalize why and where neural networks may be able to beat classical lower bounds to this problem. The n log 2 (1/) lower bound from BID5 assumes that all subsets S ⊂ U of size n, and all queries q ∈ U have equal probability. Whilst it is instructive to bound this maximum-entropy scenario, which we can think of as'worst case', most applications of approximate set membership e.g. web cache sharing, querying databases, or spell-checking, involve sets and queries that are not sampled uniformly. For example, the elements within a given set may be highly dependent, there may be a power-law distribution over queries, or the queries and sets themselves may not be sampled independently. A more general space lower bound can be defined by an information theoretic argument from communication complexity BID37. Namely, approximate set membership can be framed as a two-party communication problem between Alice, who observes the set S and Bob, who observes a query q. They can agree on a shared policy Π in which to communicate. For given inputs S, q they can produce a transcript A S,q = Π(S, q) ∈ Z which can be processed g: et al. shows that the maximum transcript size is greater than the mutual information between the inputs and transcript: max S,q |A S,q | ≥ I(S, q; A S,q) = H(S, q) − H(S, q|A S,q). Thus we note problems where we may be able to use less space than the classical lower bound are cases where the entropy H(S, q) is small, e.g. our sets are highly non-uniform, or cases where H(S, q|A S,q) is large, which signifies that many query and set pairs can be solved with the same transcript. DISPLAYFORM0 Algorithm 1 Meta-Learning Training 1: while training iteration i < budget do Calculate query outputs DISPLAYFORM1 Calculate XE loss: DISPLAYFORM2 Calculate dL/dθ (backpropagate through queries and writes) 10:Update parameters θ i+1 ← Optimizer(θ i, dL/dθ) Our experiments aim to explore scenarios inspired by real-world scenarios where there are varying levels of structure in the storage sets S and queries q. We consider four neural architectures, the LSTM, DNC, Memory Network, and Neural Bloom Filter. An encoder architecture is shared across all models. For images we use a 3-layer convnet with a kernel size of 3 and 64 filters. For text we used a character LSTM with 128 hidden units. The training setup follows the memory-augmented meta-learning training scheme of BID34, only here the task is familiarity classification versus image classification. The network samples tasks which involve classifying familiarity for a given storage set in one-shot. Meta-learning occurs as a two-speed process, where the model quickly learns to recognize a given storage set S within a training episode via writing to a memory or state, and the model slowly learns to improve this fast-learning process by optimizing the model parameters θ over multiple tasks. We detail the training routine in Algorithm 1. DISPLAYFORM0 Failed at task for larger no. items For the RNN baselines (LSTM and DNC) the write operation corresponds to unrolling the network over the inputs and outputting the final state. For these models, the query network is simply an MLP classifier which receives the concatenated final state and query, and outputs a scalar logit. For the Memory Network the read and write operations are defined from the model, and for the Neural Bloom Filter the read and write operations are defined in Section 3. We compared the space (in bits) of the model's memory (or state) to a Bloom Filter at a given false positive rate. The false positive rate is measured empirically over a sample of queries for the learned models; for the Bloom Filter we employ the analytical false positive rate. Beating a Bloom Filter's space usage with the analytical false positive rate implies better performance for any given Bloom Filter library version (as actual Bloom Filter hash functions are not uniform), thus the comparison is fair. For each model we sweep over hyper-parameters relating to model size to obtain their smallest operating size at the desired false positive rate (for the full set, see Appendix D). Because the neural models can emit false negatives, we store these in a (ideally small) backup bloom filter, as proposed by BID22 BID25. We account for the space of this backup bloom filter, and add it to the space usage of the model's memory for parity (See Appendix B for further discussion). Thus the neural network must learn to output a small state in one-shot that can serve set membership queries at a given false positive rate, and emit a small enough number of false negatives such that the backup filter is also small -and the total size is considerably less than a Bloom Filter. We chose three simple set membership tasks that have a graded level of structure and use images as an input type, namely images from MNIST. The input modality is not of principle importance, one could have designed similar tasks with textual input for example, but it is interesting to see how the model operate on images before moving to text. We experiment with three different levels of inherent structure to the sampling of sets and queries, however crucially all problems are approximate set membership tasks that can be solved by a Bloom Filter. They do not require generalization or familiarity to the sensitivity of a particular precision. Class-based familiarity, each set of images is sampled with the constraint that they arise from the same randomly-sampled class. Non-uniform instance-based familiarity, the images are sampled without replacement from an exponential distribution. Uniform instance-based familiarity, where each subset contains images sampled uniformly without replacement. See Appendix E for further details on the task setup. In the class-based sampling task we see in FIG2 we see that the DNC, LSTM and Neural Bloom Filter are able to significantly outperform the classical Bloom Filter when images are sampled by class. The Memory Network is able to solve the task with a word size of only 2, however this corresponds to a far greater number of bits per element, 64 versus the Bloom Filter's 9.8 (to a total size of 4.8kb), and so the overall size was prohibitive. The DNC, LSTM, and Neural Bloom Filter are able to solve the task with 500 elements at 1.1kb, 217b, and 382b; a 4.3×, 22×, and 12× saving respectively. For the non-uniform sampling task in FIG2 we see the Bloom Filter is preferable for less than 500 stored elements, but is overtaken thereafter. At 1000 elements the DNC, LSTM, and Neural Bloom Filter consume 7.9kb, 7.7kb, and 6.8kb respectively which corresponds to a 17.6%, 19.7%, and 28.6% reduction over the 9.6kb Bloom Filter. In the uniform sampling task shown in FIG2, there is no structure to the sampling of S. The two architectures which rely on BPTT essentially fail to solve the task at some threshold of storage size. The Neural Bloom Filter solves it with 6.8kb (using a memory size of 50 and word size of 2). If the Neural Bloom Filter were trained at low precision -say 5 bits per element which is a demonstrated capacity of RNN activations BID10 -the overall space would be 5.8kb; still 20% larger than the Bloom Filter. The overall from these sets of experiments is that the classical Bloom Filter cannot be matched (or beaten) by a meta-learned memory-augmented neural network when there is no structure to the data, however in the case of imbalanced data, or highly dependent sets that share common attributes, we do see significant space savings. We wanted to understand whether the Neural Bloom Filter is able to learn'where' to store inputs in memory, as well as what to store. In other memory-augmented neural networks, such as the DNC and Memory Networks, there is no direct correspondence between the contents of an input and where it should be organized in memory. The Neural Bloom Filter has the ability do this, but it may not choose to in practice. We investigated this for the MNIST class-based familiarity task, giving the model 10 memory slots, each with a word size of 2 and a top k addressing of k = 3. We inspect three trained models; the full model, an ablated model where the write words are fixed to be a constant w ← 1, and an ablated model with w ← 1 and a linear read operator r ← a T M. We visualize the average write address and memory contents broken down by class. The full model, shown in FIG3 learns to place some classes in particular slots, e.g. class 1 → slot 5, however most are very distributed. Inspecting the memory contents it is clear the write word encodes a unique 2D token for each class. This solution bears resemblance with Bloom-g Filters BID28 where elements are spread across a smaller memory with the same hashing scheme as Bloom Filters, but a unique token is stored in each slot instead of a constant 1-bit value. With the model ablated to store only 1s in FIG3 we see it chooses to allocate semantic addressing codes for some classes (e.g. 0 and 1) however for other classes it uses a distributed address. E.g. for class 3 the model prefers to uniformly spread its writes across memory slot 1, 4, and 8. The model solution is similar to that of Bloom Filters, with distributed addressing codes as a solution -but no information in the written words themselves. By inspecting the determinant of the average addresses, we find they are not linearly separable. Thus the output MLP f out computes a non-linearly separation. When we force the read operation to be linear in FIG3, this is no longer possible and the network learns to produce linearly separable addresses for each class. This solution has a correspondence with perfect hashing as now each (unobserved) input class will map to a unique slot in memory. We look at a task inspired by database interactions. NoSQL databases, such as Bigtable and Cassandra, use a single string-valued row key, which is used to index the data. The database is comprised of a union of files (e.g. SSTables) storing contiguous row key chunks. Bloom Filters are used to determine whether a given query q lies within the stored set. We emulate this setup by constructing a universe of strings, that is alphabetically ordered, and by sampling contiguous ranges (to represent a given SSTable). We choose unique tokens in GigaWord v5 news corpus to comprise as a universe, which consists of 2.5M words. We train models with up to 200 unique strings in a set, and extrapolate to larger sizes. We evaluate the models on a held-out test set. We see in FIG4 that the neural architectures perform best for larger false positive rates. For a storage size of 200 we observe a 6× space reduction (200b vs 1.2kb) when using the Neural Bloom Filter over the classical Bloom Filter. For smaller false positive rates, e.g. 0.1% this gap reduces to a 20% reduction (2.5kb vs 3kb) which decreases during the extrapolation phase. Interestingly the performance degradation of the LSTM is quite smooth between in-sample set sizes (< 200) and out-of-sample sizes (> 200). We hypothesize the LSTM is smoothly forgetting earlier inputs. Finally, we train the Neural Bloom Filter with a storage size of 5000 and compare it to Bloom Filters and Cuckoo Filters in Table 1, where we see 3 − 40× space reduction. Here the LSTM was not able to learn the task; optimizing insertions via BPTT over a sequence of length 5000 did not in a remotely usable solution. A storage size of 5000 is still small, but it is relevant to the NOSQL database-scenario where SSTables (say) are typically kept small. E.g. if the stored values were of size 100kB per row, we would expect 5000 unique keys in an average Bigtable SSTable. We furthermore considered the latency and throughput of such a learned model in this database setting. Both the query and insertion latencies of the Neural Bloom Filter are considerably higher than a classical Bloom Filter (14ms vs 20ns) however the maximum throughput of the Bloom Filter (≈ 60K QPS) can be matched by the neural variant when run on a GPU (NVIDIA P6000). The same is not true of an LSTM, which peaks at a maximum insertion throughput of 4K even when the model is placed on the GPU -due to its sequential write scheme. See Appendix F.1 for more details on the speed benchmarks. The large database task validates the choice of simple write scheme for the Neural Bloom Filter. The additive write can be successfully optimized during training over larger storage sets as there is no BPTT, and it is an order of magnitude faster at evaluation time -even matching the throughput of a Bloom Filter. There have been a large number of Bloom Filter variants published; from Counting Bloom Filters which support deletions BID15, Bloomier Filters which store functions vs sets BID8, Compressed Bloom Filters which use arithmetic encoding to compress the storage set BID24, and Cuckoo Filters which use cuckoo hashing to reduce redundancy within the storage vector BID14. Although some of these variants focus on better compression, they do not achieve this by specializing to the data distribution. One of the few works which do address data-dependence are Weighted Bloom Filters BID4 BID35. They work by modulating the number of hash functions used to store or query each input, dependent on its storage and query frequency. This requires estimating a large number of separate storage and query frequencies. The Neural Bloom Filter in contrast is a more general solution to non-uniform S sets. The encoder may be able to learn the statistics of the data distribution during the meta-learning phase and represent these in its parametric weights. However it can also take advantage of dependent sets where the entropy H(S) may be low but the frequency of each element is uniform (e.g. such as row keys in a database). proposes a neurally-inspired set membership data-structure that works by replacing the randomized hash functions with a randomly-wired computation graph of OR and AND gates. The false positive rate is controlled analytically by modulating the number of gates and the overall memory size. However there is no learning or specialization to the data with this setup. BID3 investigates a learnable neural familiarity module, which serves as a biologically plausible model of familiarity mechanisms in the brain, namely within the perirhinal cortex. However this has not shown to be empirically effective at exact matching. BID22 consider the use of a neural network to classify the membership of queries to a fixed set S. Here the network itself is more akin to a perfect hashing setup where multiple epochs are required to find a succinct holistic representation of the set, which is embedded into the weights of the network. In their case this search is performed by gradient-based optimization. We emulate their experimental comparison approach but instead propose a memory architecture that represents the set as activations in memory, versus weights in a network. Mitzenmacher (2018a) discusses the benefits and draw-backs of a learned bloom filter; distinguishing the empirical false positive rate over the distribution of sets S versus the conditional false positive rate of the model given a particular set S. In this paper we focus on the empirical false positive rate because we wish to exploit redundancy in the data and query distribution. Mitzenmacher (2018b) considers a different way to combine classical and learned bloom filters; by'sandwiching' the learned model with a pre-filter classical bloom filter and a post-filter classical bloom filter. This is seen to be more efficient for learned models with a larger false positive rate. In many situations neural networks are not a suitable replacement to Bloom Filters and their variants. The Bloom Filter is robust to changes in data distribution, and adversarial attacks, because it delivers a bounded false positive rate for any sampled subset, unlike a neural network. However in this paper we consider the questions, "When might a neural network provide better compression than a Bloom Filter?" and "What kind of neural architecture is practical?". We see that a model which uses an external memory with an adaptable capacity, avoids BPTT with a feed-forward write scheme, and learns to address its memory, is the most promising option in contrast to popular memory models such as DNCs and LSTMs. We term this model the Neural Bloom Filter due to the analogous incorporation of a hashing scheme, commutative write scheme, and multiplicative read mechanism. The Neural Bloom Filter relies on settings where cost of learning to query is possible and will be a net benefit to a population of existing bloom filters. That is, because we rely on meta-learning, we need situations where we have an off-line dataset (both of stored elements and queries) that is similar enough to future data that we wish to store. In the case of a large database we think this is warranted, a database with 100, 000 separate set membership data structures will benefit from a single (or periodic) meta-learning training routine that can run on a single machine and sample from the currently stored data, generating a large number of efficient data-structures. We envisage the space cost of the network to be amortized by sharing it across many neural bloom filters, and the time-cost of executing the network to be offset by the continuous acceleration of dense linear algebra on modern hardware, and the ability to batch writes and queries efficiently. A promising future direction would be to investigate the feasibility of this approach in a production system. A.1 MOVING ZCAThe moving ZCA was computed by taking moving averages of the first and second moment, calculating the ZCA matrix and updating a moving average projection matrix θ zca. This is only done during training, at evaluation time θ zca is fixed. We describe the update below for completeness. DISPLAYFORM0 Calculate singular values DISPLAYFORM1 In practice we do not compute the singular value decomposition at each time step to save computational resources, but instead calculate it and update θ every T steps. We scale the discount in this case η = η/T. We can think of the decorrelation of s, along with the sparse content-based attention with A, as a hash function that maps s to several indices in M. For moderate dimension sizes of s (256, say) we note that the Gaussian samples in A lie close to the surface of a sphere, uniformly scattered across it. If q, the decorrelated query, were to be Gaussian then the marginal distribution of nearest neighbours rows in A will be uniform. If we chose the number of nearest neighbours k = 1 then this implies the slots in M are selected independently with uniform probability. This is the exact hash function specification that Bloom Filters assume. Instead we use a continuous (as we choose k > 1) approximation (as we decorrelate s → q vs Gaussianize) to this uniform hashing scheme, so it is differentiable and the network can learn to shape query representations. We discuss some implementation tricks that could be employed for a production system. One can avoid the linear-time addressing operation σ k (q T A) by using an approximate k-nearest neighbour index, such as locality-sensitive hashing, to extract the nearest neighbours in A. The use of an approximate nearest neighbour index has been empirically considered for scaling memory-augmented neural networks BID29 BID21 however this was used for attention on M directly. As M is dynamic the knn requires frequent re-building as memories are stored or modified. This architecture is simpler -A is fixed and so the approximate knn can be built once. To ensure the size of the network (which can be shared across many memory instantiations) is independent of the number of slots in memory m we can avoid storing A. Because it is a fixed sample of random variables that are generated from a deterministic random number generator we can instead store a set of integer seeds that can be used to re-generate the rows of A. We can let the i-th seed c i, say represented as a 16-bit integer, correspond to the set of 16 rows with indices 16i, 16i + 1,..., 16i + 15. If these rows need to be accessed, they can be regenerated on-the-fly by c i. The total memory cost of A is thus m bits, where m is the number of memory slots 3.Putting these two together it is possible to query and write to a Neural Bloom Filter with m memory slots in O(log m) time, where the network consumes O space. It is worth noting, however, the Neural Bloom Filter's memory is often much smaller than the corresponding classical Bloom Filter's memory, and in many of our experiments is even smaller than the number of unique elements to store. Thus dense matrix multiplication can still be preferable -especially due to its acceleration on GPUs and TPUs BID20 -and a dense representation of A is not inhibitory. As model 3 One can replace 16 with 32 if there are more than one million slots optimization can become application-specific, we do not focus on these implementation details and use the model in its simplest setting with dense matrix operations. For each task we compare the model's memory size, in bits, at a given false positive rate -usually chosen to be 1%. For our neural networks which output a probability p = f (x) one could select an operating point τ such that the false positive rate is. In all of our experiments the neural network outputs a memory (state) s which characterizes the storage set. Let us say SPACE(f,) is the minimum size of s, in bits, for the network to achieve an average false positive rate of. We could compare SPACE(f,) with SPACE(Bloom Filter,) directly, but this would not be a fair comparison as our network f can emit false negatives. To remedy this, we employ the same scheme as BID22 where we use a'backup' Bloom Filter with false positive rate δ to store all false negatives. When f (x) < τ we query the backup Bloom Filter. Because the overall false positive rate is + (1 −)δ, to achieve a false positive rate of at most α (say 1%) we can set = δ = α/2. The number of elements stored in the backup bloom filter is equal to the number of false negatives, denoted n f n. Thus the total space can be calculated, TOTAL SPACE(f,α) = SPACE(f, DISPLAYFORM0 2). We compare this quantity for different storage set sizes. For the MNIST experiments we used a 3-layer convolutional neural network with 64 filters followed by a two-layer feed-forward network with 64&128 hidden-layers respectively. The number of trainable parameters in the Neural Bloom Filter (including the encoder) is 243, 437 which amounts to 7.8Mb at 32-bit precision. We did not optimize the encoder architecture to be lean, as we consider it part of the library in a sense. For example, we do not count the size of the hashing library that an implemented Bloom Filter relies on, which may have a chain of dependencies, or the package size of TensorFlow used for our experiments. Nevertheless we can reason that when the Neural Bloom Filter is 4kb smaller than the classical, such as for the non-uniform instance-based familiarity in FIG2, we would expect to see a net gain if we have a collection of at least 1, 950 data-structures. We imagine this could be optimized quite significantly, by using 16-bit precision and perhaps using more convolution layers or smaller feed-forward linear operations. For the database experiments we used an LSTM character encoder with 256 hidden units followed by another 256 feed-forward layer. The number of trainable parameters in the Neural Bloom Filter 419, 339 which amounts to 13Mb. One could imagine optimizing this by switching to a GRU or investigating temporal convolutions as encoders. We swept over the following hyper-parameters, over the range of memory sizes displayed for each task. We computed the best model parameters by selecting those which ed in a model consuming the least space as defined in Appendix B. This depends on model performance as well as state size. For the class-based familiarity task, and uniform sampling task, the model was trained on the training set and evaluated on the test set. For the class-based task sampling, a class is sampled at random and S is formed from a random subset of images from that class. The queries q are chosen uniformly from either S or from images of a different class. For the non-uniform instance-based familiarity task we sampled images from an exponential distribution. Specifically we used a fix permutation of the training images, and from that ordering chose p(i th image) ∝ 0.999 i for the images to store. The query images were selected uniformly. We used a fixed permutation (or shuffle) of the images to ensure most probability mass was not placed on images of a certain class. I.e. by the natural ordering of the dataset we would have otherwise almost always sampled 0 images. This would be confounding task non-uniformity for other latent structure to the sets. Because the network needed to relate the image to its frequency of occurence for task, the models were evaluated on the training set. This is reasonable as we are not wishing for the model to visually generalize to unseen elements in the setting of this exact-familiarity task. We specifically want the network weights to compress a map of image to probability of storage. For the database task a universe of 2.5M unique tokens were extracted from GigaWord v5. We shuffled the tokens and placed 2.3M in a training set and 250K in a test set. These sets were then sorted alphabetically. A random subset, representing an SSTable, was sampled by choosing a random start index and selecting the next n elements, which form our set S. Queries are sampled uniformly at random from the universe set. Models are trained on the training set and evaluated on the test set. We see the benefit of sphering in Figure 5 where the converged validation performance ends up at a higher state. Investigating the memory fill rate, we see this is because the model is ignoring many more rows than is necessary. This is likely due to the network fixating on rows it has accessed with sparse addressing, and ignoring rows it has otherwise never touched -a phenomena noted in BID31. The model finds a local minima in continually storing and accessing the same rows in memory. The effect of sphering is that the query now appears to be Gaussian (up to the first two moments) and so the nearest neighbour in the address matrix A (which is initialized to Gaussian random variables) will be close to uniform. This in a more uniform memory access (as seen in Figure 5b) which significantly aids performance (as seen in Figure 5a). Figure 5: The effect of sphering the query vector on performance in the uniform MNIST sampling task. We provide a brief comparison of run-time (in terms of latency and max-throughput) between a classical Bloom Filter and a Neural Bloom Filter. In several applications of Bloom Filters, the actual query latency is not in the critical path of computation. For example, for a distributed database, the network latency and disk access latency for one tablet can be orders of magnitude greater than the in-memory latency of a Bloom Filter query. For this reason, we have not made run-time a point of focus in this study, and it is implicitly assumed that the neural network is trading off greater latency for less space. However it is worth checking whether run-time could be prohibitive. Table 3: Latency for a single query, and throughput for a batch of 10,000 queries. *Bloom Filter benchmark taken from'query-efficient' Bloom Filter BID9.We use the Neural Bloom Filter network architecture for the large database task (Table 1). The network uses an encoder LSTM with 256 hidden units over the characters, and feeds this through a 256 fully connected layer to encode the input. A two-layer 256-hidden-unit MLP is used as the query architecture. The memory and word size is 8 and 4 respectively, and so the majority of the compute is spent in the encoder and query network. We compare this with an LSTM containing 32 hidden units. We benchmark the single-query latency of the network alongside the throughput of a batch of queries, and a batch of inserts. The Neural Bloom Filter and LSTM is implemented in TensorFlow without any custom kernels or specialized code. We benchmark it on the CPU (Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz) and a GPU (NVIDIA Quadro P6000). We compare to empirical timing published in a query-optimized Bloom Filter variant BID9.We see in Table 3. that the query and insert latency of the Neural Bloom Filter sits at 5ms on the CPU, around 400× slower than the classical Bloom Filter. The LSTM requires roughly the same latency. However when multiple queries are received, the operations batch well and the model is only around 20× slower for both the Neural Bloom Filter and LSTM -as these can be batched in parallel on the CPU. If we are able to use a GPU, the throughput further increases to roughly parity with the classical Bloom Filter. This is a very rough comparison, however it leads to the that a Neural Bloom Filter could be deployed in scenarios with high query load without a catastrophic decrease in throughput. For insertions we see the same insertion throughput of ≈ 58K is maintained for the Neural Bloom Filter on the GPU as the model uses batchable insertion operations that can be parallelized. The use of a purely additive write was primarily chosen to avoid optimizing the model with backpropagationthrough-time, however here we see it is beneficial for computational efficiency at evaluation time. However here we see a large difference between the NBF and the LSTM, which uses a strictlysequential write scheme. Even with a powerful NVIDIA P6000 GPU the LSTM's maximum insertion throughput is 4.6K, over an order of magnitude lower. Thus we conclude that even if we were able to train an LSTM to perform exact set membership on the larger database task to a good space performance -which we were not able to do due to the difficulty of training an RNN over rich sequences of length 5000 -it would serve insertions at over an order of magnitude slower than a Bloom Filter, even with accelerated hardware.

represent the short summary for finding the corresponding scientific article (long)

We investigate the space efficiency of memory-augmented neural nets when learning set membership.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.