Incrementally Train BERT with minimum QnA records - to get improved results We are using Google BERT for Question and Answering. We have fine tuned BERT with SQUAD QnA release train data set (https://github.com/google-research/bert , https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)   

It generated new checkpoints and BERT is giving good answers for most of questions we asked on our text documents. However, there are some questions which it is answering wrong, so we are trying to further fine tune with our Question and known answer on our text document. We further trained based on last generated checkpoint and got new checkpoint.   

With new checkpoint when we are asking the same question, the answer did not got corrected! Previously BERT was giving wrong answer with 99% confidence and now also giving same wrong answer with 95% confidence.  

Can someone suggest, if they have same or similar experience, and suggest please.
Following are questions in BERT github Issues, and are unanswered for quite some time:  


BERT accuracy reduced after providing custom training..The answer is
also not correct : https://github.com/google-research/bert/issues/492
Unable to incrementally train BERT with custom training: https://github.com/google-research/bert/issues/482
Little training has no impact:    https://github.com/google-research/bert/issues/481  
Custom Domain Training: https://github.com/google-research/bert/issues/498
Remember that BERT was first pre-trained using the concatenation of BooksCorpus (800M words) and English Wikipedia (2,500M words). Then the fine-tuning in your case uses the SQuAD dataset consisting of 100,000+ questions (based on Wikipedia articles) with a learning rate in the order of e-5. 

So when you add let's say 100 new domain-specific question/document/answer to your input during the fine-tuning, this will not have a major effect on the learned parameters especially if your new input has out-of-Bert-vocabulary words or uses different English style, as opposed to Wikipedia/Books style.


  Can someone suggest, if they have same or similar experience, and
  suggest please.


Following are some suggestions that might help:


Increase the amount of new input data to help the model converge to your domain-specific use case.
Before fine-tuning,  map recurrent out-of-Bert-vocabulary words of your new input to existing words in Bert vocabulary which have almost the same meaning.  
Increase the learning rate for your new input. But this is not very safe as the model may "forget" what he learned and not converge. But as the fine-tuning doesn't take much time you can try it out and see the results.