Title: DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding,
Abstract: Recurrent neural nets (RNN) and convolutional neural nets (CNN) are widely
used on NLP tasks to capture the long-term and local dependencies,
respectively. Attention mechanisms have recently attracted enormous interest
due to their highly parallelizable computation, significantly less training
time, and flexibility in modeling dependencies. We propose a novel attention
mechanism in which the attention between elements from input sequence(s) is
directional and multi-dimensional (i.e., feature-wise). A light-weight neural
net, "Directional Self-Attention Network (DiSAN)", is then proposed to learn
sentence embedding, based solely on the proposed attention without any RNN/CNN
structure. DiSAN is only composed of a directional self-attention with temporal
order encoded, followed by a multi-dimensional attention that compresses the
sequence into a vector representation. Despite its simple form, DiSAN
outperforms complicated RNN models on both prediction quality and time
efficiency. It achieves the best test accuracy among all sentence encoding
methods and improves the most recent best result by 1.02% on the Stanford
Natural Language Inference (SNLI) dataset, and shows state-of-the-art test
accuracy on the Stanford Sentiment Treebank (SST), Multi-Genre natural language
inference (MultiNLI), Sentences Involving Compositional Knowledge (SICK),
Customer Review, MPQA, TREC question-type classification and Subjectivity
(SUBJ) datasets.