What are good toy problems for testing Transformer architectures? I am testing various variants for Transformers and Transformer architectures. But training on full language tasks is a rather time consuming affair. What are good toy problems to test if a transformer (or alternative thereof) is working at all? I am looking for simple problems that can preferably be synthetically created and can be trained with really small setups (few layers, small embedding sizes etc.) in a short time. Ideally this should be problems that play to the strengths of transformers but would be hard to solve for, say, a fully connected feed-forward network. Tasks that can be applied to just an Attention Layer would be useful, too.
I would recommend the pre-processed datasets for language translation in the Stanford NLP group: https://nlp.stanford.edu/projects/nmt/

There are three datasets with sizes 0.13Mb, 4.5Mb and 15Mb - something for everybody  :)



Translation is something that is very difficult to solve for normal FF-networks and something that transformers dramatically improved due to attention mechanisms.

Also check out the related explanations of those ideas in The Annotated Transformer and the Illustrated Transformer