Do we have to use the IOB format on labels in the NER dataset? If so, why?
There's nothing stopping you from training a model with whatever tags you want.
Using what you describe as &quot;usual&quot; format means you would have approx half as many tags as using the IOB format. In theory this means your model will develop higher accuracy faster and with less training data. On the downside, you will need to do more work when interpreting the results in order to be confident where one named entity ends and another one begins.
I made a notebook to do some empirical tests on this which uses 17 tags in the IOB format and 9 in the &quot;usual&quot; format.
TL;DR using the &quot;usual&quot; format did not produce a noticeable improvement in model quality.