Metrics for Name Entity Recognition
As a first start I would recommend to use Precision and Recall.

You can also produce stats on the different types of errors:


entity not found (false negative, or Type I error). Also you can get the false negative rate by dividing by the total number of true entities. This would be 1-recall.
entity found where it shouldn't be (false positive or Type II error). Also you can get the false positive rate by dividing by the total number of returned entities. This would be 1-precision.
entity found but system got wrong number of tokens (e.g. York instead of New York). You can also get the "token boundary error rate" by dividing by the total number of true entities.


So the first two error rates I mentioned don't give any extra info other than precision and recall but the third allows you to check how accurately you are getting entity boundaries.

If you are doing entity disambiguation too, you can produce additional metrics by only counting a success if the entity was correctly resolved.

You can also use the F-score (thanks Erwan) which combines precision and recall. However you need to be careful because you have a large class imbalance of O-tokens vs entity tokens, and the F-score gives equal importance to precision and recall. You also have the options of weighted F-scores such as F2 score.

Other option: ROC curve and AUC

Finally I would add that if your model outputs a score or probability rather than a binary classification, it means you can tweak its sensitivity by changing the threshold (do you want to count all entities with score>0.5 as entities, or score>0.75, for example). This naturally leads to the possibility of calculating the false positive and false negative rates for every possible threshold choice. You can plot it on a graph and you have a ROC curve. This is a nice graphical way to see the performance of your model and exactly how it misclassifies instances. The area under the ROC curve is itself a nice metric called AUC. A classifier that is as good as random has AUC=0.5, a classifier that classifies everything wrong has AUC=0, and a perfect classifier has AUC=1. I normally plot the ROC curve as well as generating metrics, Sklearn and most stats packages have a function to do it.