Transformer architecture not working on toy problem
I am not sure but maybe the excess zeroes are the culprit. I guess the results you are getting are just the accumulation of the biases. I would suggest converting your one-hot vectors to dense vectors using an Embedding Layer.