class_docstr: Data collator used for T5 span-masked language modeling.
  It is made sure that after masking the inputs are of length `data_args.max_seq_length` and targets are also of fixed length.
  For more information on how T5 span-masked language modeling works, one can take a look
  at the `official paper <https://arxiv.org/pdf/1910.10683.pdf>`__
  or the `official code for preprocessing <https://github.com/google-research/text-to-text-transfer-transformer/blob/master/t5/data/preprocessors.py>`__ .
  Args:
      tokenizer (:class:`~transformers.PreTrainedTokenizer` or :class:`~transformers.PreTrainedTokenizerFast`):
          The tokenizer used for encoding the data.
      noise_density (:obj:`float`):
          The probability with which to (randomly) mask tokens in the input.
      mean_noise_span_length (:obj:`float`):
          The average span length of the masked tokens.
      input_length (:obj:`int`):
          The expected input length after masking.
      target_length (:obj:`int`):
          The expected target length after masking.
      pad_token_id: (:obj:`int`):
          The pad token id of the model
      decoder_start_token_id: (:obj:`int):
          The decoder start token id of the model
class_docstr_tok: ["Data","collator","used","for","T5","span-masked","language","modeling",".","It","is","made","sure","that","after","masking","the","inputs","are","of","length","`","data_args.max_seq_length","`","and","targets","are","also","of","fixed","length",".","For","more","information","on","how","T5","span-masked","language","modeling","works",",","one","can","take","a","look","at","the","`","official","paper","<","https",":","\/\/arxiv.org\/pdf\/1910.10683.pdf",">","`","__","or","the","`","official","code","for","preprocessing","<","https",":","\/\/github.com\/google-research\/text-to-text-transfer-transformer\/blob\/master\/t5\/data\/preprocessors.py",">","`","__",".","Args",":","tokenizer","(",":","class",":","`","~transformers.PreTrainedTokenizer","`","or",":","class",":","`","~transformers.PreTrainedTokenizerFast","`",")",":","The","tokenizer","used","for","encoding","the","data",".","noise_density","(",":","obj",":","`","float","`",")",":","The","probability","with","which","to","(","randomly",")","mask","tokens","in","the","input",".","mean_noise_span_length","(",":","obj",":","`","float","`",")",":","The","average","span","length","of","the","masked","tokens",".","input_length","(",":","obj",":","`","int","`",")",":","The","expected","input","length","after","masking",".","target_length","(",":","obj",":","`","int","`",")",":","The","expected","target","length","after","masking",".","pad_token_id",":","(",":","obj",":","`","int","`",")",":","The","pad","token","id","of","the","model","decoder_start_token_id",":","(",":","obj",":","`","int",")",":","The","decoder","start","token","id","of","the","model"]
docstr: Puts sentinel mask on `input_ids` and fuse consecutive mask tokens into a single mask token by deleting.
  This will reduce the sequence length from `expanded_inputs_length` to `input_length`.
docstr_tok: ["Puts","sentinel","mask","on","`","input_ids","`","and","fuse","consecutive","mask","tokens","into","a","single","mask","token","by","deleting",".","This","will","reduce","the","sequence","length","from","`","expanded_inputs_length","`","to","`","input_length","`","."]
code: def filter_input_ids(self, input_ids, sentinel_ids):
      """
      Puts sentinel mask on `input_ids` and fuse consecutive mask tokens into a single mask token by deleting.
      This will reduce the sequence length from `expanded_inputs_length` to `input_length`.
      """
      new_input_ids = list()
      for x, y in zip(input_ids, sentinel_ids):
          input_ids_full = np.where(y != 0, y, x)
          x = input_ids_full[input_ids_full > 0].tolist()[:-1]
          x = x + [self.tokenizer.eos_token_id]
          new_input_ids += [x]
      from itertools import zip_longest
  
      new_input_ids = np.array(
          list(zip_longest(*new_input_ids, fillvalue=0))
      ).T
      return new_input_ids
  
file_path: DeepKE/src/deepke/name_entity_re/cross/seq2seq/data_collator/t5mlm_data_collator.py
imports: import torch
  from dataclasses import dataclass
  from typing import Dict, Optional, Union
  import numpy
  from transformers import PreTrainedTokenizerBase, PreTrainedModel
  from transformers.file_utils import PaddingStrategy
label_desc: Use image node_id 3 for calling the DataCollatorForT5MLM obj's underlying member method code with example usage: obj.filter_input_ids(input_ids, sentinel_ids) and returns: new_input_ids