Get the value of '[UNK]' in BERT
I have designed a model based on BERT to solve NER task. I am using transformers library with the "dccuchile/bert-base-spanish-wwm-cased" pre-trained model. The problem comes when my model detect an entity but the token is '[UNK]'. How could I know which is the string behind that token?

I know that an unknown token can't be reverted to the original one, but I would like to at least capture that values before passing the inputs to the model. 

The code is really simple:

    sentenceIds = tokenizer.encode(sentence,add_special_tokens = True)

    inputs = pad_sequences([sentenceIds], maxlen=256, dtype="long", 
                              value=0, truncating="post", padding="post")

    att_mask = torch.tensor([[int(token_id &gt; 0) for token_id in inputs[0]]]).to(device)
    inputs = torch.tensor(inputs).to(device)

    with torch.no_grad():        
        outputs = model(inputs, 
                          token_type_ids=None, 
                          attention_mask=att_mask)


As you see is really simple, just tokenize, padding or truncating, creating attentionMask and calling to the model.

I have tried using regex, trying to find the two tokens that are around it and things like that, but I can't solve it properly.

The tokenizer works in two steps. First, it does pre-tokenization, which is basically splitting on spaces and separating punctuation. Let's have a look at it on a random Czech sentence:

tokenizer.basic_tokenizer.tokenize("Kočka leze dírou.")


This gives you: ['kocka', 'leze', 'dirou', '.']

In the second step, it applies the word piece splitting algorithm, so you get:

tokenizer.tokenize("Kočka leze dírou.")


You get: ['[UNK]', 'le', '##ze', 'di', '##ro', '##u', '.']

If there is no way how to split the token into subwords, the whole word becomes [UNK]. Tokens starting with ## get appended to the previous ones, so this way you can find out where the [UNK] originally came from.

(And it seems weird to me that Spanish WordPiece tokenizer cannot parse a word that only consists of Latin characters.)
