class_docstr: Custom Dataset wrapping language modeling sequences.
  
  Each sample will be retrieved by indexing the list of token_ids and their corresponding lengths.
  
  Input:
  ------
      params: `NameSpace` parameters
      data: `List[np.array[int]]
docstr: Sequences that are too long are split by chunk of max_model_input_size.
docstr_tok: ["Sequences","that","are","too","long","are","split","by","chunk","of","max_model_input_size","."]
code: def remove_long_sequences(self):
      """
      Sequences that are too long are split by chunk of max_model_input_size.
      """
      max_len = self.params.max_model_input_size
      indices = self.lengths > max_len
      logger.info(f"Splitting {sum(indices)} too long sequences.")
  
      def divide_chunks(l, n):
          return [l[i : i + n] for i in range(0, len(l), n)]
  
      new_tok_ids = []
      new_lengths = []
      if self.params.mlm:
          cls_id, sep_id = (
              self.params.special_tok_ids["cls_token"],
              self.params.special_tok_ids["sep_token"],
          )
      else:
          cls_id, sep_id = (
              self.params.special_tok_ids["bos_token"],
              self.params.special_tok_ids["eos_token"],
          )
  
      for seq_, len_ in zip(self.token_ids, self.lengths):
          assert (seq_[0] == cls_id) and (seq_[-1] == sep_id), seq_
          if len_ <= max_len:
              new_tok_ids.append(seq_)
              new_lengths.append(len_)
          else:
              sub_seqs = []
              for sub_s in divide_chunks(seq_, max_len - 2):
                  if sub_s[0] != cls_id:
                      sub_s = np.insert(sub_s, 0, cls_id)
                  if sub_s[-1] != sep_id:
                      sub_s = np.insert(sub_s, len(sub_s), sep_id)
                  assert len(sub_s) <= max_len
                  assert (sub_s[0] == cls_id) and (
                      sub_s[-1] == sep_id
                  ), sub_s
                  sub_seqs.append(sub_s)
  
              new_tok_ids.extend(sub_seqs)
              new_lengths.extend([len(l) for l in sub_seqs])
  
      self.token_ids = np.array(new_tok_ids)
      self.lengths = np.array(new_lengths)
  
file_path: H2O/h2o_flexgen/benchmark/third_party/transformers/examples/research_projects/distillation/lm_seqs_dataset.py
imports: import numpy
  import torch
  from torch.utils.data import Dataset
  from utils import logger
label_desc: Use image node_id 5 for calling the LmSeqsDataset obj's underlying member method code with example usage: obj.remove_long_sequences() and returns: unknown