Optimal input setup for character-level text classification RNN
For character-level NLP, one-dimensional convolutions are often used to shrink the number of hidden states to a reasonable number.
For instance, ELMo first tokenizes the input into words and then uses a CNN with max-pooling to obtain character-based word embeddings that are used as an input to an LSTM.
In machine translation, there is an approach that shrinks the character-level input into pseudo-word hidden states. This is a one-dimensional CNN over the character input (here, without segmenting the input into words), the CNN is followed by max-pooling with window step 5 (which is the average word length) and a bunch of highway-layers. These pseudo-word states are then used again as an input to an LSTM.