Do transformers have success in other domains different than NLP?
When it talks to other domains such as image or music, using transformer will always face a problem of sequence length limitation. To the best of my knowledge, the bottleneck of self-attention which uses a $n^2$ matrix quite limits transformer being applied to other domains. For example, a 32x32 pixel image, means a sequence of 1024 tokens.
OpenAI did some related research, as the followings.
Generative Modeling with Sparse Transformers:
In the paper, transformers with sparse attention are applied to image and waveform.
ImageGPT:
A large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. (Abstract from the blog)