Bert for QuestionAnswering input exceeds 512
The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not possible for the model to index the positional embedding for positions greater than the maximum.
This limitation, nevertheless, is not arbitrary, but has a deeper justification: in a vanilla transformer, the memory requirements are quadratic on the input length. Therefore, limiting such a length is necessary for the model not to need too much memory.
There are variants of the Transformer architecture that are designed to overcome the quadratic memory problem, like Reformer, Linformer, Longformer, BigBird. They, however, are not &quot;compatible&quot; with the Transformer weights, so someone would need to train one of those models in a masked language modeling task (potentially with multitask learning on the next sequence prediction task) in order to reuse their weights in your context.