Incrementally Train BERT with minimum QnA records - to get improved results
Remember that BERT was first pre-trained using the concatenation of BooksCorpus (800M words) and English Wikipedia (2,500M words). Then the fine-tuning in your case uses the SQuAD dataset consisting of 100,000+ questions (based on Wikipedia articles) with a learning rate in the order of e-5. 

So when you add let's say 100 new domain-specific question/document/answer to your input during the fine-tuning, this will not have a major effect on the learned parameters especially if your new input has out-of-Bert-vocabulary words or uses different English style, as opposed to Wikipedia/Books style.


  Can someone suggest, if they have same or similar experience, and
  suggest please.


Following are some suggestions that might help:


Increase the amount of new input data to help the model converge to your domain-specific use case.
Before fine-tuning,  map recurrent out-of-Bert-vocabulary words of your new input to existing words in Bert vocabulary which have almost the same meaning.  
Increase the learning rate for your new input. But this is not very safe as the model may "forget" what he learned and not converge. But as the fine-tuning doesn't take much time you can try it out and see the results.