NER evaluation metric
A good starting point is to look at the evaluation measures used in the NER shared tasks: https://nlpprogress.com/english/named_entity_recognition.html.
Generally the F1-score can be used for one specific class, but there are different options regarding what is counted as an instance:

every occurrence of the full NE. In this case any difference between the predicted and the gold is counted as false, even if it's only one token difference.
every token in an entity. In this case a partially matched entity counts as &quot;partially correct&quot;: if a word is predicted outside instead of inside, it's a false negative and conversely.
Other variants:

count only unique entities, in order to observe the diversity of the entities recognized.
count only entities which didn't appear in the training set, to observe the generalization power.



(writing this from memory, I could miss something)