Approach for training multilingual NER
First, some clarification: BPE does work with Chinese and Arabic.
The only problem with Chinese is that there are no blanks between words, and therefore there is no explicit word boundary. In order to address that problem, normally you would segment words before applying BPE. For that, the typical approach is to use Jieba or any of its multiple ports to other programming languages. Other languages without blanks, like Japanese, may have their own tools to perform word segmentation.
Now, the answer:
Subword vocabularies are the norm nowadays. The norm also consists of finetuning one of the many pre-trained neural models available (BERT, XLNet, RoBERTa, etc), and the tokenization is imposed by the model you choose.
BPE is, in general, a popular choice for tokenization, no matter the language. Lately, the unigram tokenization is becoming popular also.
I suggest you take a look at the recent tner library, which builds on top of Huggingface Transformers to make NER very easy.