title: How to predict masked word in a sentence in BERT-base from Tensorflow checkpoint (ckpt) files?
body: <p>I have BERT-base model checkpoints which I trained from scratch in Tensorflow. How can I use those checkpoints to predict masked word in a given sentence?</p>
  
  <p>Like, let say sentence is,
  "[CLS] abc pqr [MASK] xyz [SEP]"
  And I want to predict word at [MASK] position.</p>
  
  <p>How can I do it?
  I searched a lot online but everyone is using BERT for their task specific classification tasks.
  Not using BERT to predict masked word.</p>
  
  <p>Please help me in solving this prediction problem. </p>
  
  <p>I created data using <code>create_pretraining_data.py</code> &amp; trained model from scratch using <code>run_pretraining.py</code> from official BERT repo (<a href="https://github.com/google-research/bert" rel="nofollow noreferrer">https://github.com/google-research/bert</a>)</p>
  
  <p>I have searched in issues in official bert repo. But didn't found any solution.</p>
  
  <p>Also looked at code in that repo. They're using Estimator which they are training not using from checkpoints weights.</p>
  
  <p>Didn't found any way to use way to use Tensorflow checkpoints of BERT-base model (trained from scratch) to predict word masked token (i.e. [MASK]).</p>