XLNET how to deal with text with more than 512 tokens?
BERT also has the same limit of 512 tokens.
Normally, for longer sequences, you just truncate to 512 tokens.
The limit is derived from the positional embeddings in the Transformer architecture, for which a maximum length needs to be imposed. The magnitude of such a size is related to the amount of memory needed to handle texts: attention layers scale quadratically with the sequence length, which poses a problem with long texts.
There are new architectures that specifically deal with longer sequences, like Longformer, Reformer, Big Bird and Linformer.