The RNN-Attention (RNN-Attention) model, (Fig. ), attaches a single self-attention head after the GRU of the RNN model to leverage the attention mechanism . The attention mechanism allows a model to store sequence wide connection weights between tokens in a sequence. Each token in the sequence is made to "attend" and give a weight to all other tokens in the sequence thus creating an attention map over all the tokens in the sequence. The single attention layer has been shown to increase interpretability and improve reconstructions over longer sequences .