Title: Automatic Rule Extraction from Long Short Term Memory Networks,
Abstract: Although deep learning models have proven effective at solving problems in
natural language processing, the mechanism by which they come to their
conclusions is often unclear. As a result, these models are generally treated
as black boxes, yielding no insight of the underlying learned patterns. In this
paper we consider Long Short Term Memory networks (LSTMs) and demonstrate a new
approach for tracking the importance of a given input to the LSTM for a given
output. By identifying consistently important patterns of words, we are able to
distill state of the art LSTMs on sentiment analysis and question answering
into a set of representative phrases. This representation is then
quantitatively validated by using the extracted phrases to construct a simple,
rule-based classifier which approximates the output of the LSTM.