class_docstr: Construct a Marian tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
  
  This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
  this superclass for more information regarding those methods.
  
  Args:
      source_spm (`str`):
          [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that
          contains the vocabulary for the source language.
      target_spm (`str`):
          [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that
          contains the vocabulary for the target language.
      source_lang (`str`, *optional*):
          A string representing the source language.
      target_lang (`str`, *optional*):
          A string representing the target language.
      unk_token (`str`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      eos_token (`str`, *optional*, defaults to `"</s>"`):
          The end of sequence token.
      pad_token (`str`, *optional*, defaults to `"<pad>"`):
          The token used for padding, for example when batching sequences of different lengths.
      model_max_length (`int`, *optional*, defaults to 512):
          The maximum sentence length the model accepts.
      additional_special_tokens (`List[str]`, *optional*, defaults to `["<eop>", "<eod>"]`):
          Additional special tokens used by the tokenizer.
      sp_model_kwargs (`dict`, *optional*):
          Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
          SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
          to set:
  
          - `enable_sampling`: Enable subword regularization.
          - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
  
            - `nbest_size = {0,1}`: No sampling is performed.
            - `nbest_size > 1`: samples from the nbest_size results.
            - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
              using forward-filtering-and-backward-sampling algorithm.
  
          - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
            BPE-dropout.
  
  Examples:
  
  ```python
  >>> from transformers import MarianTokenizer
  
  >>> tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de")
  >>> src_texts = ["I am a small frog.", "Tom asked his teacher for advice."]
  >>> tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."]  # optional
  >>> inputs = tokenizer(src_texts, text_target=tgt_texts, return_tensors="pt", padding=True)
  # keys  [input_ids, attention_mask, labels].
  
  >>> outputs = model(**inputs)  # should work
  ```
docstr: Convert a list of lists of token ids into a list of strings by calling decode.
  
  Args:
      sequences (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`):
          List of tokenized input ids. Can be obtained using the `__call__` method.
      skip_special_tokens (`bool`, *optional*, defaults to `False`):
          Whether or not to remove special tokens in the decoding.
      clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`):
          Whether or not to clean up the tokenization spaces.
      use_source_tokenizer (`bool`, *optional*, defaults to `False`):
          Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence
          problems).
      kwargs (additional keyword arguments, *optional*):
          Will be passed to the underlying model specific decode method.
  
  Returns:
      `List[str]`: The list of decoded sentences.
docstr_tok: ["Convert","a","list","of","lists","of","token","ids","into","a","list","of","strings","by","calling","decode",".","Args",":","sequences","(","`","Union","[","List","[","int","]",",","List","[","List","[","int","]","]",",","np.ndarray",",","torch.Tensor",",","tf.Tensor","]","`",")",":","List","of","tokenized","input","ids",".","Can","be","obtained","using","the","`","__call__","`","method",".","skip_special_tokens","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","remove","special","tokens","in","the","decoding",".","clean_up_tokenization_spaces","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","clean","up","the","tokenization","spaces",".","use_source_tokenizer","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","use","the","source","tokenizer","to","decode","sequences","(","only","applicable","in","sequence-to-sequence","problems",")",".","kwargs","(","additional","keyword","arguments",",","*","optional","*",")",":","Will","be","passed","to","the","underlying","model","specific","decode","method",".","Returns",":","`","List","[","str","]","`",":","The","list","of","decoded","sentences","."]
code: def batch_decode(self, sequences, **kwargs):
      """
      Convert a list of lists of token ids into a list of strings by calling decode.
  
      Args:
          sequences (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`):
              List of tokenized input ids. Can be obtained using the `__call__` method.
          skip_special_tokens (`bool`, *optional*, defaults to `False`):
              Whether or not to remove special tokens in the decoding.
          clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`):
              Whether or not to clean up the tokenization spaces.
          use_source_tokenizer (`bool`, *optional*, defaults to `False`):
              Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence
              problems).
          kwargs (additional keyword arguments, *optional*):
              Will be passed to the underlying model specific decode method.
  
      Returns:
          `List[str]`: The list of decoded sentences.
      """
      return super().batch_decode(sequences, **kwargs)
  
imports: import json
  import os
  import re
  import warnings
  from pathlib import Path
  from shutil import copyfile
  from typing import Any, Dict, List, Optional, Tuple, Union
  import sentencepiece
  from ...tokenization_utils import PreTrainedTokenizer
  from ...utils import logging
label_desc: Use image node_id 8 for calling the MarianTokenizer obj's underlying member method code with example usage: obj.batch_decode(sequences) and returns: super