Problem when using Autograd with nn.Embedding in Pytorch
So, this is just half answer, I write it here to be able to format the text clearly. You are facing troubles because you are trying to do something that you shouldn't, which is applying gradient to indices instead of embeddings. When using embeddings (all kinds, not only BERT), before feeding them to a model, sentences must be represented with embedding indices, which are just number associated with specific embedding vectors. Those indices are just values mapping words to vectors, you will never apply any operation on them, especially you will never train them, cause they are not parameters. 

['Just an example...'] # text
       |
       | Words are turned into indices 
       v
[1., 2., 3., 4., 4., 4.] # indices, tensor type long, no sense in applying gradient here
       |
       | Indices are used to retrieve the real embedding vectors
       v
[0.3242, 0.2354, 0.8763, 0.4325, 0.4325, 0.4325] # embeddings, tensor type float, this is what you want to train


If you want to fine tune BERT for a specific task, I suggest you to take a look to this tutorial BERT-fine-tuning