Transformers get named entity prediction for words instead of tokens
This is very basic question, but I spend hours struggling to find the answer. I built NER using Hugginface transformers.
Say I have input sentence
input = &quot;Damien Hirst oil in canvas&quot;

I tokenize it to get
tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased')
tokenized = tokenizer.encode(input) #[101, 12587, 7632, 12096, 3514, 1999, 10683, 102]

Feed tokenized sentence to the model to get predicted tags for the tokens
['B-ARTIST' 'B-ARTIST' 'I-ARTIST' 'I-ARTIST' 'B-MEDIUM' 'I-MEDIUM'
 'I-MEDIUM' 'B-ARTIST']

prediction comes as output from the model. It assigns tags to different tokens.
How can I recombine this data to obtain tags for words instead of tokens? So I would know that
&quot;Damien Hirst&quot; = ARTIST
&quot;Oil in canvas&quot; = MEDIUM


There are two questions here.
Annotating Token Classification
A common sequential tagging, especially in Named Entity Recognition, follows the scheme that a sequence to tokens with tag X at the beginning gets B-X and on reset of the labels it gets I-X.
The problem is that most annotated datasets are tokenized with space! For example:
[CSL]  O
Damien  B-ARTIST
Hirst  I-ARTIST
oil  B-MEDIUM
in  I-MEDIUM
canvas  I-MEDIUM
[SEP]  O

where O indicates that it is not a named-entity, B-ARTIST is the beginning of the sequence of tokens labelled as ARTIST and I-ARTIST is inside the sequence - similar pattern for MEDIUM.
At the moment I posted this answer, there is an example of NER in huggingface documentation here:
https://huggingface.co/transformers/usage.html#named-entity-recognition
The example doesn't exactly answer the question here, but it can add some clarification. The similar style of named entity labels in that example could be as follows:
label_list = [
    &quot;O&quot;, # not a named entity
    &quot;B-ARTIST&quot;, # beginning of an artist name
    &quot;I-ARTIST&quot;, # an artist name
    &quot;B-MEDIUM&quot;, # beginning of a medium name
    &quot;I-MEDIUM&quot;, # a medium name
]

Adapt Tokenizations
With all that said about annotation schema, BERT and several other models have different tokenization model. So, we have to adapt these two tokenizations.
In this case with bert-base-uncased, the expected outcome is like this:
damien  B-ARTIST
hi  I-ARTIST
##rst  I-ARTIST
oil  B-MEDIUM
in  I-MEDIUM
canvas  I-MEDIUM

In order to get this done, you can go through each token in original annotation, then tokenize it and add its label again:
tokens_old = ['Damien', 'Hirst', 'oil', 'in', 'canvas']
labels_old = [&quot;B-ARTIST&quot;, &quot;I-ARTIST&quot;, &quot;B-MEDIUM&quot;, &quot;I-MEDIUM&quot;, &quot;I-MEDIUM&quot;]
label2id = {label: idx for idx, label in enumerate(label_list)}

tokens, labels = zip(*[
   (token, label)
   for token_old, label in zip(tokens_old, labels_old)
   for token in tokenizer.tokenize(token_old)
])

When you add [CLS] and [SEP] in the tokens, their labels &quot;O&quot; must be added to labels.
With the code above, it is possible to get into a situation that a beginning tag like B-ARTIST get repeated when the beginning word splits into pieces. According to the description in huggingface documentation, you can encode these labels with -100 to be ignored:
https://huggingface.co/transformers/custom_datasets.html#token-classification-with-w-nut-emerging-entities
Something like this should work:
tokens, labels = zip(*[
   (token, label2id[label] if (label[:2] != &quot;B-&quot; or i == 0) else -100)
   for token_old, label in zip(tokens_old, labels_old)
   for i, token in enumerate(tokenizer.tokenize(token_old))
])

