Title: Multiscale sequence modeling with a learned dictionary,
Abstract: We propose a generalization of neural network sequence models. Instead of
predicting one symbol at a time, our multi-scale model makes predictions over
multiple, potentially overlapping multi-symbol tokens. A variation of the
byte-pair encoding (BPE) compression algorithm is used to learn the dictionary
of tokens that the model is trained with. When applied to language modelling,
our model has the flexibility of character-level models while maintaining many
of the performance benefits of word-level models. Our experiments show that
this model performs better than a regular LSTM on language modeling tasks,
especially for smaller models.