class_docstr: Custom Dataset wrapping language modeling sequences.
  
  Each sample will be retrieved by indexing the list of token_ids and their corresponding lengths.
  
  Input:
  ------
      params: `NameSpace` parameters
      data: `List[np.array[int]]
docstr: Remove sequences with a (too) high level of unknown tokens.
docstr_tok: ["Remove","sequences","with","a","(","too",")","high","level","of","unknown","tokens","."]
code: def remove_unknown_sequences(self):
      """
      Remove sequences with a (too) high level of unknown tokens.
      """
      if "unk_token" not in self.params.special_tok_ids:
          return
      else:
          unk_token_id = self.params.special_tok_ids["unk_token"]
      init_size = len(self)
      unk_occs = np.array(
          [np.count_nonzero(a == unk_token_id) for a in self.token_ids]
      )
      indices = (unk_occs / self.lengths) < 0.5
      self.token_ids = self.token_ids[indices]
      self.lengths = self.lengths[indices]
      new_size = len(self)
      logger.info(
          f"Remove {init_size - new_size} sequences with a high level of unknown tokens (50%)."
      )
  
file_path: H2O/h2o_flexgen/benchmark/third_party/transformers/examples/research_projects/distillation/lm_seqs_dataset.py
imports: import numpy
  import torch
  from torch.utils.data import Dataset
  from utils import logger
label_desc: Use image node_id 7 for calling the LmSeqsDataset obj's underlying member method code with example usage: obj.remove_unknown_sequences() without return types