title: BERT Word Embedding for column of pandas data frame
body: <p>I m working on a NLP project using Tamil Universal Dependency dataset. I have preprocessed the data into a data frame, of which columns are tokens and its dependency tags. I would like to perform word embedding using mBERT model. Since the dataset is a pretrained model, it is already tokenized as seen in the attached Data frame. I m not sure how to proceed because, when tokens are converted to token id's are wrongly marked by the tokenizer.</p>
  <pre><code>b #List of tokens
  </code></pre>
  <p>Data Frame</p>
  <p><a href="https://i.stack.imgur.com/mIfhi.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mIfhi.png" alt="enter image description here" /></a></p>
  <p>Token ID Error</p>
  <p><a href="https://i.stack.imgur.com/3qVIU.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3qVIU.png" alt="enter image description here" /></a></p>