code: def fix_blank_tokens(tokenizer, data):
      """Patch bert tokenizers with missing characters
  
      There is an issue that some tokenizers (so far the German ones identified above)
      tokenize soft hyphens or other unknown characters into nothing
      If an entire word is tokenized as a soft hyphen, this means the tokenizer
      simply vaporizes that word.  The result is we're missing an embedding for
      an entire word we wanted to use.
  
      The solution we take here is to look for any words which get vaporized
      in such a manner, eg `len(token) == 2`, and replace it with a regular "-"
  
      Actually, recently we have found that even the Bert / Electra tokenizer
      can do this in the case of "words" which are one special character long,
      so the easiest thing to do is just always run this function
      """
      new_data = []
      for sentence in data:
          tokenized = tokenizer(
              sentence, is_split_into_words=False
          ).input_ids
          new_sentence = [
              word if len(token) > 2 else "-"
              for word, token in zip(sentence, tokenized)
          ]
          new_data.append(new_sentence)
      return new_data
  
imports: import math
  import logging
  import numpy
  import torch
  import torch.nn
  import torch.nn.functional
  from torch.nn.utils.rnn import pad_packed_sequence, pack_padded_sequence, pack_sequence, PackedSequence
label_desc: Use image node_id 9 for calling a global function with example usage: fix_blank_tokens(tokenizer, data) and returns: new_data