class_docstr: Construct a SeamlessM4T tokenizer.
  
  Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on
  [SentencePiece](https://github.com/google/sentencepiece).
  
  The tokenization method is `<language code> <tokens> <eos>` for source language documents, and `<eos> <language
  code> <tokens> <eos>` for target language documents.
  
  Examples:
  
  ```python
  >>> from transformers import SeamlessM4TTokenizer
  
  >>> tokenizer = SeamlessM4TTokenizer.from_pretrained(
  ...     "facebook/hf-seamless-m4t-medium", src_lang="eng", tgt_lang="fra"
  ... )
  >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria"
  >>> expected_translation_french = "Le chef de l'ONU affirme qu'il n'y a pas de solution militaire en Syrie."
  >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_french, return_tensors="pt")
  ```
  
  Args:
      vocab_file (`str`):
          Path to the vocabulary file.
      bos_token (`str`, *optional*, defaults to `"<s>"`):
          The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the beginning of
          sequence. The token used is the `cls_token`.
  
          </Tip>
  
      eos_token (`str`, *optional*, defaults to `"</s>"`):
          The end of sequence token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the end of sequence.
          The token used is the `sep_token`.
  
          </Tip>
  
      sep_token (`str`, *optional*, defaults to `"</s>"`):
          The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
          sequence classification or for a text and a question for question answering. It is also used as the last
          token of a sequence built with special tokens.
      cls_token (`str`, *optional*, defaults to `"<s>"`):
          The classifier token which is used when doing sequence classification (classification of the whole sequence
          instead of per-token classification). It is the first token of the sequence when built with special tokens.
      unk_token (`str`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      pad_token (`str`, *optional*, defaults to `"<pad>"`):
          The token used for padding, for example when batching sequences of different lengths.
      tokenizer_file (`str`, *optional*):
          The path to a tokenizer file to use instead of the vocab file.
      src_lang (`str`, *optional*, defaults to `"eng"`):
          The language to use as source language for translation.
      tgt_lang (`str`, *optional*, defaults to `"fra"`):
          The language to use as target language for translation.
      sp_model_kwargs (`Dict[str, Any]`, *optional*):
          Additional keyword arguments to pass to the model initialization.
      additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*):
          A tuple or a list of additional special tokens. Can be used to specify the list of languages that will be
          supported by the tokenizer.
docstr: Args:
      text (`str`, `List[str]`, `List[List[str]]`, *optional*):
          The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
          (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
          `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
      text_pair (`str`, `List[str]`, `List[List[str]]`, *optional*):
          The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
          (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
          `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
      text_target (`str`, `List[str]`, `List[List[str]]`, *optional*):
          The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a
          list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized),
          you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
      text_pair_target (`str`, `List[str]`, `List[List[str]]`, *optional*):
          The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a
          list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized),
          you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
      padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`):
           Select a strategy to pad the returned sequences (according to the model's padding side and padding
           index) among:
  
          - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
            sequence if provided).
          - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
            acceptable input length for the model if that argument is not provided.
          - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
            lengths).
      pad_to_multiple_of (`int`, *optional*):
          If set will pad the sequence to a multiple of the provided value.
  
          This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
          `>= 7.5` (Volta).
      src_lang (`str`, *optional*):
          A string representing the source language. If not specified, the last `src_lang` specified (either
          during initialization or when calling this tokenizer) will be used.
      tgt_lang (`str`, *optional*):
          A string representing the target language. If not specified, the last `tgt_lang` specified (either
          during initialization or when calling this tokenizer) will be used.
      kwargs (*optional*):
          Remaining dictionary of keyword arguments that will be passed to [`PreTrainedTokenizer.__call__`].
docstr_tok: ["Args",":","text","(","`","str","`",",","`","List","[","str","]","`",",","`","List","[","List","[","str","]","]","`",",","*","optional","*",")",":","The","sequence","or","batch","of","sequences","to","be","encoded",".","Each","sequence","can","be","a","string","or","a","list","of","strings","(","pretokenized","string",")",".","If","the","sequences","are","provided","as","list","of","strings","(","pretokenized",")",",","you","must","set","`","is_split_into_words=True","`","(","to","lift","the","ambiguity","with","a","batch","of","sequences",")",".","text_pair","(","`","str","`",",","`","List","[","str","]","`",",","`","List","[","List","[","str","]","]","`",",","*","optional","*",")",":","The","sequence","or","batch","of","sequences","to","be","encoded",".","Each","sequence","can","be","a","string","or","a","list","of","strings","(","pretokenized","string",")",".","If","the","sequences","are","provided","as","list","of","strings","(","pretokenized",")",",","you","must","set","`","is_split_into_words=True","`","(","to","lift","the","ambiguity","with","a","batch","of","sequences",")",".","text_target","(","`","str","`",",","`","List","[","str","]","`",",","`","List","[","List","[","str","]","]","`",",","*","optional","*",")",":","The","sequence","or","batch","of","sequences","to","be","encoded","as","target","texts",".","Each","sequence","can","be","a","string","or","a","list","of","strings","(","pretokenized","string",")",".","If","the","sequences","are","provided","as","list","of","strings","(","pretokenized",")",",","you","must","set","`","is_split_into_words=True","`","(","to","lift","the","ambiguity","with","a","batch","of","sequences",")",".","text_pair_target","(","`","str","`",",","`","List","[","str","]","`",",","`","List","[","List","[","str","]","]","`",",","*","optional","*",")",":","The","sequence","or","batch","of","sequences","to","be","encoded","as","target","texts",".","Each","sequence","can","be","a","string","or","a","list","of","strings","(","pretokenized","string",")",".","If","the","sequences","are","provided","as","list","of","strings","(","pretokenized",")",",","you","must","set","`","is_split_into_words=True","`","(","to","lift","the","ambiguity","with","a","batch","of","sequences",")",".","padding","(","`","bool","`",",","`","str","`","or","[","`","~utils.PaddingStrategy","`","]",",","*","optional","*",",","defaults","to","`","True","`",")",":","Select","a","strategy","to","pad","the","returned","sequences","(","according","to","the","model","'s","padding","side","and","padding","index",")","among",":","-","`","True","`","or","`","'longest","'","`",":","Pad","to","the","longest","sequence","in","the","batch","(","or","no","padding","if","only","a","single","sequence","if","provided",")",".","-","`","'max_length","'","`",":","Pad","to","a","maximum","length","specified","with","the","argument","`","max_length","`","or","to","the","maximum","acceptable","input","length","for","the","model","if","that","argument","is","not","provided",".","-","`","False","`","or","`","'do_not_pad","'","`","(","default",")",":","No","padding","(","i.e.",",","can","output","a","batch","with","sequences","of","different","lengths",")",".","pad_to_multiple_of","(","`","int","`",",","*","optional","*",")",":","If","set","will","pad","the","sequence","to","a","multiple","of","the","provided","value",".","This","is","especially","useful","to","enable","the","use","of","Tensor","Cores","on","NVIDIA","hardware","with","compute","capability","`",">","=","7.5","`","(","Volta",")",".","src_lang","(","`","str","`",",","*","optional","*",")",":","A","string","representing","the","source","language",".","If","not","specified",",","the","last","`","src_lang","`","specified","(","either","during","initialization","or","when","calling","this","tokenizer",")","will","be","used",".","tgt_lang","(","`","str","`",",","*","optional","*",")",":","A","string","representing","the","target","language",".","If","not","specified",",","the","last","`","tgt_lang","`","specified","(","either","during","initialization","or","when","calling","this","tokenizer",")","will","be","used",".","kwargs","(","*","optional","*",")",":","Remaining","dictionary","of","keyword","arguments","that","will","be","passed","to","[","`","PreTrainedTokenizer.__call__","`","]","."]
code: def __call__(
      self,
      text: Union[
          TextInput,
          PreTokenizedInput,
          List[TextInput],
          List[PreTokenizedInput],
      ] = None,
      text_pair: Optional[
          Union[
              TextInput,
              PreTokenizedInput,
              List[TextInput],
              List[PreTokenizedInput],
          ]
      ] = None,
      text_target: Union[
          TextInput,
          PreTokenizedInput,
          List[TextInput],
          List[PreTokenizedInput],
      ] = None,
      text_pair_target: Optional[
          Union[
              TextInput,
              PreTokenizedInput,
              List[TextInput],
              List[PreTokenizedInput],
          ]
      ] = None,
      padding: Union[bool, str, PaddingStrategy] = True,
      pad_to_multiple_of: Optional[int] = 2,
      src_lang: Optional[str] = None,
      tgt_lang: Optional[str] = None,
      **kwargs,
  ):
      """
      Args:
          text (`str`, `List[str]`, `List[List[str]]`, *optional*):
              The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
              (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
              `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
          text_pair (`str`, `List[str]`, `List[List[str]]`, *optional*):
              The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
              (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
              `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
          text_target (`str`, `List[str]`, `List[List[str]]`, *optional*):
              The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a
              list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized),
              you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
          text_pair_target (`str`, `List[str]`, `List[List[str]]`, *optional*):
              The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a
              list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized),
              you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
          padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`):
               Select a strategy to pad the returned sequences (according to the model's padding side and padding
               index) among:
  
              - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
                sequence if provided).
              - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
                acceptable input length for the model if that argument is not provided.
              - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
                lengths).
          pad_to_multiple_of (`int`, *optional*):
              If set will pad the sequence to a multiple of the provided value.
  
              This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
              `>= 7.5` (Volta).
          src_lang (`str`, *optional*):
              A string representing the source language. If not specified, the last `src_lang` specified (either
              during initialization or when calling this tokenizer) will be used.
          tgt_lang (`str`, *optional*):
              A string representing the target language. If not specified, the last `tgt_lang` specified (either
              during initialization or when calling this tokenizer) will be used.
          kwargs (*optional*):
              Remaining dictionary of keyword arguments that will be passed to [`PreTrainedTokenizer.__call__`].
      """
      if src_lang is not None:
          self.src_lang = src_lang
      if tgt_lang is not None:
          self.tgt_lang = tgt_lang
  
      output = super().__call__(
          text=text,
          text_pair=text_pair,
          text_target=text_target,
          text_pair_target=text_pair_target,
          padding=padding,
          pad_to_multiple_of=pad_to_multiple_of,
          **kwargs,
      )
  
      return BatchEncoding(
          output, tensor_type=kwargs.get("return_tensors")
      )
  
imports: import os
  from shutil import copyfile
  from typing import Any, Dict, List, Optional, Tuple, Union
  import sentencepiece
  from ...convert_slow_tokenizer import import_protobuf
  from ...tokenization_utils import BatchEncoding, PreTokenizedInput, PreTrainedTokenizer, TextInput
  from ...tokenization_utils_base import AddedToken
  from ...utils import PaddingStrategy, logging
label_desc: Use image node_id 5 for calling the SeamlessM4TTokenizer obj's underlying member method code with example usage: obj.__call__(text, text_pair, text_target, text_pair_target, padding, pad_to_multiple_of, src_lang, tgt_lang) and returns: BatchEncoding