Why and how BERT can learn different attentions for each head?
After re-read the code, I am wrong. Each head is different part of the tensor's last dim.

Each head is to split the last dim of the tensor.

For example, head number is 12 and last dim of the tensor should be 12*N.

Each value of the tensor's the last dim represent a feature for the tensor.

But it seems the training could let different heads be different representations. This is amazing.

Think about CNN. CNN also has a lot of kernels. And the info of kernels may have some redundancy. As the init values of kernels are different, the kernels also learn some different info.

Then watch the figure in the link again, BERT's attentions also have some redundancy.