title: PyTorch Huggingface BERT-NLP for Named Entity Recognition
body: <p>I have been using the PyTorch implementation of Google's <a href="https://github.com/google-research/bert#fine-tuning-with-bert" rel="nofollow noreferrer">BERT</a> by <a href="https://github.com/huggingface/pytorch-pretrained-BERT" rel="nofollow noreferrer">HuggingFace</a> for the MADE 1.0 dataset for quite some time now. Up until last time (11-Feb), I had been using the library and getting an <strong>F-Score</strong> of <strong>0.81</strong> for my Named Entity Recognition task by Fine Tuning the model. But this week when I ran the exact same code which had compiled and run earlier, it threw an error when executing this statement:</p>
  
  <pre><code>input_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt) for txt in tokenized_texts], maxlen=MAX_LEN, dtype="long", truncating="post", padding="post")
  </code></pre>
  
  <blockquote>
    <p>ValueError: Token indices sequence length is longer than the specified
    maximum  sequence length for this BERT model (632 > 512). Running this
    sequence through BERT will result in indexing errors</p>
  </blockquote>
  
  <p>The full code is available in this <a href="https://colab.research.google.com/drive/1JxWdw1BjXZCFC2a8IwtZxvvq4rFGcxas" rel="nofollow noreferrer">colab notebook</a>.</p>
  
  <p>To get around this error I modified the above statement to the one below by taking the first 512 tokens of any sequence and made the necessary changes to add the index of [SEP] to the end of the truncated/padded sequence as required by BERT.</p>
  
  <pre><code>input_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt[:512]) for txt in tokenized_texts], maxlen=MAX_LEN, dtype="long", truncating="post", padding="post")
  </code></pre>
  
  <p>The result shouldn't have changed because I am only considering the first 512 tokens in the sequence and later truncating to 75 as my (MAX_LEN=75) but my <strong>F-Score</strong> has dropped to <strong>0.40</strong> and my <strong>precision</strong> to <strong>0.27</strong> while the <strong>Recall</strong> remains the same <strong>(0.85)</strong>. I am unable to share the dataset as I have signed a confidentiality clause but I can assure all the preprocessing as required by BERT has been done and all extended tokens like (Johanson --> Johan ##son) have been tagged with X and replaced later after the prediction as said in the <a href="https://arxiv.org/abs/1810.04805" rel="nofollow noreferrer">BERT Paper</a>.</p>
  
  <p>Has anyone else faced a similar issue or can elaborate on what might be the issue or what changes the PyTorch (Huggingface) people have done on their end recently?</p>