class_docstr: This masks out tokens according to the given tokenizer.
  
  The masked variables are
  
  output_type : "string" (default) or "token_ids"
docstr: Build a new Text masker given an optional passed tokenizer.
  
  Parameters
  ----------
  tokenizer : callable or None
      The tokenizer used to break apart strings during masking. The passed tokenizer must support a minimal
      subset of the HuggingFace Transformers PreTrainedTokenizerBase API. This minimal subset means the
      tokenizer must return a dictionary with 'input_ids' and then either include
      an 'offset_mapping' entry in the same dictionary or provide a .convert_ids_to_tokens or .decode method.
  
  mask_token : string, int, or None
      The sub-string or integer token id used to mask out portions of a string. If None it will use the
      tokenizer's .mask_token attribute, if defined, or "..." if the tokenizer does not have a .mask_token
      attribute.
  
  collapse_mask_token : True, False, or "auto"
      If True, when several consecutive tokens are masked only one mask token is used to replace the entire
      series of original tokens.
code: def __init__(
      self,
      tokenizer=None,
      mask_token=None,
      collapse_mask_token="auto",
      output_type="string",
  ):
      """Build a new Text masker given an optional passed tokenizer.
  
      Parameters
      ----------
      tokenizer : callable or None
          The tokenizer used to break apart strings during masking. The passed tokenizer must support a minimal
          subset of the HuggingFace Transformers PreTrainedTokenizerBase API. This minimal subset means the
          tokenizer must return a dictionary with 'input_ids' and then either include
          an 'offset_mapping' entry in the same dictionary or provide a .convert_ids_to_tokens or .decode method.
  
      mask_token : string, int, or None
          The sub-string or integer token id used to mask out portions of a string. If None it will use the
          tokenizer's .mask_token attribute, if defined, or "..." if the tokenizer does not have a .mask_token
          attribute.
  
      collapse_mask_token : True, False, or "auto"
          If True, when several consecutive tokens are masked only one mask token is used to replace the entire
          series of original tokens.
      """
  
      if tokenizer is None:
          self.tokenizer = SimpleTokenizer()
      elif callable(tokenizer):
          self.tokenizer = tokenizer
      else:
          try:
              self.tokenizer = SimpleTokenizer(tokenizer)
          except Exception:
              raise Exception(
                  "The passed tokenizer cannot be wrapped as a masker because it does not have a __call__ "
                  "method, not can it be interpreted as a splitting regexp!"
              )
  
      self.output_type = output_type
      self.collapse_mask_token = collapse_mask_token
      self.input_mask_token = mask_token
      self.mask_token = (
          mask_token  # could be recomputed later in this function
      )
      self.mask_token_id = (
          mask_token if isinstance(mask_token, int) else None
      )
      parsed_tokenizer_dict = parse_prefix_suffix_for_tokenizer(
          self.tokenizer
      )
  
      self.keep_prefix = parsed_tokenizer_dict["keep_prefix"]
      self.keep_suffix = parsed_tokenizer_dict["keep_suffix"]
      # self.prefix_strlen = parsed_tokenizer_dict['prefix_strlen']
      # self.suffix_strlen = parsed_tokenizer_dict['suffix_strlen']
      # null_tokens = parsed_tokenizer_dict['null_tokens']
  
      self.text_data = True
  
      if mask_token is None:
          if getattr_silent(self.tokenizer, "mask_token") is not None:
              self.mask_token = self.tokenizer.mask_token
              self.mask_token_id = getattr_silent(
                  self.tokenizer, "mask_token_id"
              )
              if self.collapse_mask_token == "auto":
                  self.collapse_mask_token = False
          else:
              self.mask_token = "..."
      else:
          self.mask_token = mask_token
  
      if self.mask_token_id is None:
          self.mask_token_id = self.tokenizer(self.mask_token)[
              "input_ids"
          ][self.keep_prefix]
  
      if self.collapse_mask_token == "auto":
          self.collapse_mask_token = True
  
      # assign mask token segment
      # if self.keep_suffix > 0:
      #     self.mask_token_segment = self.token_segments(self.mask_token)[self.keep_prefix:-self.keep_suffix]
      # else:
      #     self.mask_token_segment = self.token_segments(self.mask_token)[self.keep_prefix:]
  
      # note if this masker can use a different background for different samples
      self.fixed_background = self.mask_token_id is None
  
      self.default_batch_size = 5
  
      # cache variables
      self._s = None
      self._tokenized_s_full = None
      self._tokenized_s = None
      self._segments_s = None
  
      # flag that we return outputs that will not get changed by later masking calls
      self.immutable_outputs = True
  
imports: import math
  import re
  import numpy
  from .._serializable import Deserializer, Serializer
  from ..utils import safe_isinstance
  from ..utils.transformers import SENTENCEPIECE_TOKENIZERS, getattr_silent, parse_prefix_suffix_for_tokenizer
  from ._masker import Masker
label_desc: Use image node_id 1 to create a new Text object from inherited base classes: Masker with example: obj = Text(tokenizer, mask_token, collapse_mask_token, output_type)