What are good toy problems for testing Transformer architectures?
I would recommend the pre-processed datasets for language translation in the Stanford NLP group: https://nlp.stanford.edu/projects/nmt/

There are three datasets with sizes 0.13Mb, 4.5Mb and 15Mb - something for everybody  :)



Translation is something that is very difficult to solve for normal FF-networks and something that transformers dramatically improved due to attention mechanisms.

Also check out the related explanations of those ideas in The Annotated Transformer and the Illustrated Transformer