What tokenizer is best?

#2
by turian - opened

What tokenizer is best to use with flair/pos-english?

It was not clear to me from the model card what tokenizer was used during training. And the default FLAIR tokenizer is relatively simple.

The tagger was trained on Ontonotes, so it reuses the tokenization from this dataset.

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment