how to get mentions in pytorch NER instead of toknes?
I am using PyTorch and a pre-trained model.
Here is my code:
class NER(object):
    def __init__(self, model_name_or_path, tokenizer_name_or_path):
        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path)
        self.model = AutoModelForTokenClassification.from_pretrained(
            model_name_or_path)
        self.nlp = pipeline(&quot;ner&quot;, model=self.model, tokenizer=self.tokenizer)

    def get_mention_entities(self, query):
        return self.nlp(query)


when I call get_mention_entities and print its output for &quot;اینجا دانشگاه صنعتی امیرکبیر است.&quot;
it gives:
[{'entity': 'B-FAC', 'score': 0.9454591, 'index': 2, 'word': 'دانشگاه', 'start': 6, 'end': 13}, {'entity': 'I-FAC', 'score': 0.9713519, 'index': 3, 'word': 'صنعتی', 'start': 14, 'end': 19}, {'entity': 'I-FAC', 'score': 0.9860724, 'index': 4, 'word': 'امیرکبیر', 'start': 20, 'end': 28}]

As you can see, it can recognize the university name, but there are three tokens in the list.
Is there any standard way to combine these tokens based on the &quot;entity&quot; attribute?
desired output is something like:
[{'entity': 'FAC', 'word': 'دانشگاه صنعتی امیرکبیر', 'start': 6, 'end': 28}]

Finally, I can write a function to iterate, compare, and merge the tokens based on the &quot;entity&quot; attribute, but I want a standard way like an internal PyTorch function or something like this.
my question is similar to this question.
PS: &quot;دانشگاه صنعتی امیرکبیر&quot; is a university name.

Huggingface's NER pipeline has an argument grouped_entities=True which will do exactly what you seek: group BI into unified entities.
Adding
self.nlp = pipeline(&quot;ner&quot;, model=self.model, tokenizer=self.tokenizer, grouped_entities=True)

should do the trick
