Most tasks in natural language processing can be cast into question answering (QA) problems over language input. We introduce the dynamic memory network (DMN), a neural network architecture which processes input sequences and questions, forms episodic memories, and generates relevant answers. Questions trigger an iterative attention process which allows the model to condition its attention on the inputs and the result of previous iterations. These results are then reasoned over in a hierarchical recurrent sequence model to generate answers. The DMN can be trained end-to-end and obtains state-of-the-art results on several types of tasks and datasets: question answering (Facebook's bAbI dataset), text classification for sentiment analysis (Stanford Sentiment Treebank) and sequence modeling for part-of-speech tagging (WSJ-PTB). The training for these different tasks relies exclusively on trained word vector representations and input-question-answer triplets.
@cite_31: Human declarative memory involves a systematic organization of information that supports generalizations and inferences from acquired knowledge. This kind of memory depends on the hippocampal region in humans, but the extent to which animals also have declarative memory, and whether inferential expression of memory depends on the hippocampus in animals, remains a major challenge in cognitive neuroscience. To examine these issues, we used a test of transitive inference pioneered by Piaget to assess capacities for systematic organization of knowledge and logical inference in children. In our adaptation of the test, rats were trained on a set of four overlapping odor discrimination problems that could be encoded either separately or as a single representation of orderly relations among the odor stimuli. Normal rats learned the problems and demonstrated the relational memory organization through appropriate transitive inferences about items not presented together during training. By contrast, after disconnection of the hippocampus from either its cortical or subcortical pathway, rats succeeded in acquiring the separate discrimination problems but did not demonstrate transitive inference, indicating that they had failed to develop or could not inferentially express the orderly organization of the stimulus elements. These findings strongly support the view that the hippocampus mediates a general declarative memory capacity in animals, as it does in humans.
@cite_12: Studies in rodents have demonstrated that the integration and flexible expression of memories, necessary for transitive inference, depend on an intact hippocampus. To test this hypothesis in humans, we studied brain activation during the discrimination of a series of overlap- ping and non-overlapping arbitrary visual stimulus pairs. We report that transitive inference about overlapping pairs is associated with right ante- rior hippocampal activation, whereas recognition of non-overlapping stimulus pairs is associated with bilateral medial temporal lobe activation centered in the anterior parahippocampal gyrus. We conclude that im- mediate access to simple stimulus-stimulus relationships is mediated via the parahippocampal gyrus, whereas the flexible representation of mem- ory requires the recruitment of the hippocampus. © 2004 Wiley-Liss, Inc.
@cite_23: Because of their superior ability to preserve sequence information over time, Long Short-Term Memory (LSTM) networks, a type of recurrent neural network with a more complex computational unit, have obtained strong results on a variety of sequence modeling tasks. The only underlying LSTM structure that has been explored so far is a linear chain. However, natural language exhibits syntactic properties that would naturally combine words to phrases. We introduce the Tree-LSTM, a generalization of LSTMs to tree-structured network topologies. Tree-LSTMs outperform all existing systems and strong LSTM baselines on two tasks: predicting the semantic relatedness of two sentences (SemEval 2014, Task 1) and sentiment classification (Stanford Sentiment Treebank).
The episodic memory in humans stores specific experiences in their spatial and temporal context. For instance, it might contain the first memory somebody has of flying a hang glider. Eichenbaum and Cohen have argued that episodic memories represent a form of relationship (i.e., relations between spatial, sensory and temporal information) and that the hippocampus is responsible for general relational learning @cite_23 . Interestingly, it also appears that the hippocampus is active during transitive inference @cite_12 , and disruption of the hippocampus impairs this ability @cite_31 .