class_docstr: Constructs a Wav2Vec2CTC tokenizer.
  
  This tokenizer inherits from [`PreTrainedTokenizer`] which contains some of the main methods. Users should refer to
  the superclass for more information regarding such methods.
  
  Args:
      vocab_file (`str`):
          File containing the vocabulary.
      bos_token (`str`, *optional*, defaults to `"<s>"`):
          The beginning of sentence token.
      eos_token (`str`, *optional*, defaults to `"</s>"`):
          The end of sentence token.
      unk_token (`str`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      pad_token (`str`, *optional*, defaults to `"<pad>"`):
          The token used for padding, for example when batching sequences of different lengths.
      word_delimiter_token (`str`, *optional*, defaults to `"|"`):
          The token used for defining the end of a word.
      do_lower_case (`bool`, *optional*, defaults to `False`):
          Whether or not to accept lowercase input and lowercase the output when decoding.
  
      **kwargs
          Additional keyword arguments passed along to [`PreTrainedTokenizer`]
class_docstr_tok: ["Constructs","a","Wav2Vec2CTC","tokenizer",".","This","tokenizer","inherits","from","[","`","PreTrainedTokenizer","`","]","which","contains","some","of","the","main","methods",".","Users","should","refer","to","the","superclass","for","more","information","regarding","such","methods",".","Args",":","vocab_file","(","`","str","`",")",":","File","containing","the","vocabulary",".","bos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","s",">","''","`",")",":","The","beginning","of","sentence","token",".","eos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","\/s",">","''","`",")",":","The","end","of","sentence","token",".","unk_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","unk",">","''","`",")",":","The","unknown","token",".","A","token","that","is","not","in","the","vocabulary","can","not","be","converted","to","an","ID","and","is","set","to","be","this","token","instead",".","pad_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","pad",">","''","`",")",":","The","token","used","for","padding",",","for","example","when","batching","sequences","of","different","lengths",".","word_delimiter_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","|","''","`",")",":","The","token","used","for","defining","the","end","of","a","word",".","do_lower_case","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","accept","lowercase","input","and","lowercase","the","output","when","decoding",".","*","*","kwargs","Additional","keyword","arguments","passed","along","to","[","`","PreTrainedTokenizer","`","]"]
docstr: Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special
  tokens and clean up tokenization spaces.
  
  Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`.
  
  Args:
      token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`):
          List of tokenized input ids. Can be obtained using the `__call__` method.
      skip_special_tokens (`bool`, *optional*, defaults to `False`):
          Whether or not to remove special tokens in the decoding.
      clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`):
          Whether or not to clean up the tokenization spaces.
      output_char_offsets (`bool`, *optional*, defaults to `False`):
          Whether or not to output character offsets. Character offsets can be used in combination with the
          sampling rate and model downsampling rate to compute the time-stamps of transcribed characters.
  
          <Tip>
  
          Please take a look at the example below to better understand how to make use of `output_char_offsets`.
  
          </Tip>
  
      output_word_offsets (`bool`, *optional*, defaults to `False`):
          Whether or not to output word offsets. Word offsets can be used in combination with the sampling rate
          and model downsampling rate to compute the time-stamps of transcribed words.
  
          <Tip>
  
          Please take a look at the example below to better understand how to make use of `output_word_offsets`.
  
          </Tip>
  
      kwargs (additional keyword arguments, *optional*):
          Will be passed to the underlying model specific decode method.
  
  Returns:
      `str` or [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`]: The list of decoded
      sentences. Will be a [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`] when
      `output_char_offsets == True` or `output_word_offsets == True`.
  
  Example:
  
  ```python
  >>> # Let's see how to retrieve time steps for a model
  >>> from transformers import AutoTokenizer, AutoFeatureExtractor, AutoModelForCTC
  >>> from datasets import load_dataset
  >>> import datasets
  >>> import torch
  
  >>> # import model, feature extractor, tokenizer
  >>> model = AutoModelForCTC.from_pretrained("facebook/wav2vec2-base-960h")
  >>> tokenizer = AutoTokenizer.from_pretrained("facebook/wav2vec2-base-960h")
  >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base-960h")
  
  >>> # load first sample of English common_voice
  >>> dataset = load_dataset("common_voice", "en", split="train", streaming=True)
  >>> dataset = dataset.cast_column("audio", datasets.Audio(sampling_rate=16_000))
  >>> dataset_iter = iter(dataset)
  >>> sample = next(dataset_iter)
  
  >>> # forward sample through model to get greedily predicted transcription ids
  >>> input_values = feature_extractor(sample["audio"]["array"], return_tensors="pt").input_values
  >>> logits = model(input_values).logits[0]
  >>> pred_ids = torch.argmax(logits, axis=-1)
  
  >>> # retrieve word stamps (analogous commands for `output_char_offsets`)
  >>> outputs = tokenizer.decode(pred_ids, output_word_offsets=True)
  >>> # compute `time_offset` in seconds as product of downsampling ratio and sampling_rate
  >>> time_offset = model.config.inputs_to_logits_ratio / feature_extractor.sampling_rate
  
  >>> word_offsets = [
  ...     {
  ...         "word": d["word"],
  ...         "start_time": round(d["start_offset"] * time_offset, 2),
  ...         "end_time": round(d["end_offset"] * time_offset, 2),
  ...     }
  ...     for d in outputs.word_offsets
  ... ]
  >>> # compare word offsets with audio `common_voice_en_100038.mp3` online on the dataset viewer:
  >>> # https://huggingface.co/datasets/common_voice/viewer/en/train
  >>> word_offsets[:3]
  [{'word': 'WHY', 'start_time': 1.42, 'end_time': 1.54}, {'word': 'DOES', 'start_time': 1.64, 'end_time': 1.9}, {'word': 'MILISANDRA', 'start_time': 2.26, 'end_time': 2.9}]
  ```
docstr_tok: ["Converts","a","sequence","of","ids","in","a","string",",","using","the","tokenizer","and","vocabulary","with","options","to","remove","special","tokens","and","clean","up","tokenization","spaces",".","Similar","to","doing","`","self.convert_tokens_to_string","(","self.convert_ids_to_tokens","(","token_ids",")",")","`",".","Args",":","token_ids","(","`","Union","[","int",",","List","[","int","]",",","np.ndarray",",","torch.Tensor",",","tf.Tensor","]","`",")",":","List","of","tokenized","input","ids",".","Can","be","obtained","using","the","`","__call__","`","method",".","skip_special_tokens","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","remove","special","tokens","in","the","decoding",".","clean_up_tokenization_spaces","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","clean","up","the","tokenization","spaces",".","output_char_offsets","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","output","character","offsets",".","Character","offsets","can","be","used","in","combination","with","the","sampling","rate","and","model","downsampling","rate","to","compute","the","time-stamps","of","transcribed","characters",".","<","Tip",">","Please","take","a","look","at","the","example","below","to","better","understand","how","to","make","use","of","`","output_char_offsets","`",".","<","\/Tip",">","output_word_offsets","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","output","word","offsets",".","Word","offsets","can","be","used","in","combination","with","the","sampling","rate","and","model","downsampling","rate","to","compute","the","time-stamps","of","transcribed","words",".","<","Tip",">","Please","take","a","look","at","the","example","below","to","better","understand","how","to","make","use","of","`","output_word_offsets","`",".","<","\/Tip",">","kwargs","(","additional","keyword","arguments",",","*","optional","*",")",":","Will","be","passed","to","the","underlying","model","specific","decode","method",".","Returns",":","`","str","`","or","[","`","~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput","`","]",":","The","list","of","decoded","sentences",".","Will","be","a","[","`","~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput","`","]","when","`","output_char_offsets","==","True","`","or","`","output_word_offsets","==","True","`",".","Example",":","``","`","python",">",">",">","#","Let","'s","see","how","to","retrieve","time","steps","for","a","model",">",">",">","from","transformers","import","AutoTokenizer",",","AutoFeatureExtractor",",","AutoModelForCTC",">",">",">","from","datasets","import","load_dataset",">",">",">","import","datasets",">",">",">","import","torch",">",">",">","#","import","model",",","feature","extractor",",","tokenizer",">",">",">","model","=","AutoModelForCTC.from_pretrained","(","``","facebook\/wav2vec2-base-960h","''",")",">",">",">","tokenizer","=","AutoTokenizer.from_pretrained","(","``","facebook\/wav2vec2-base-960h","''",")",">",">",">","feature_extractor","=","AutoFeatureExtractor.from_pretrained","(","``","facebook\/wav2vec2-base-960h","''",")",">",">",">","#","load","first","sample","of","English","common_voice",">",">",">","dataset","=","load_dataset","(","``","common_voice","''",",","``","en","''",",","split=","''","train","''",",","streaming=True",")",">",">",">","dataset","=","dataset.cast_column","(","``","audio","''",",","datasets.Audio","(","sampling_rate=16_000",")",")",">",">",">","dataset_iter","=","iter","(","dataset",")",">",">",">","sample","=","next","(","dataset_iter",")",">",">",">","#","forward","sample","through","model","to","get","greedily","predicted","transcription","ids",">",">",">","input_values","=","feature_extractor","(","sample","[","``","audio","''","]","[","``","array","''","]",",","return_tensors=","''","pt","''",")",".input_values",">",">",">","logits","=","model","(","input_values",")",".logits","[","0","]",">",">",">","pred_ids","=","torch.argmax","(","logits",",","axis=-1",")",">",">",">","#","retrieve","word","stamps","(","analogous","commands","for","`","output_char_offsets","`",")",">",">",">","outputs","=","tokenizer.decode","(","pred_ids",",","output_word_offsets=True",")",">",">",">","#","compute","`","time_offset","`","in","seconds","as","product","of","downsampling","ratio","and","sampling_rate",">",">",">","time_offset","=","model.config.inputs_to_logits_ratio","\/","feature_extractor.sampling_rate",">",">",">","word_offsets","=","[","...","{","...","``","word","''",":","d","[","``","word","''","]",",","...","``","start_time","''",":","round","(","d","[","``","start_offset","''","]","*","time_offset",",","2",")",",","...","``","end_time","''",":","round","(","d","[","``","end_offset","''","]","*","time_offset",",","2",")",",","...","}","...","for","d","in","outputs.word_offsets","...","]",">",">",">","#","compare","word","offsets","with","audio","`","common_voice_en_100038.mp3","`","online","on","the","dataset","viewer",":",">",">",">","#","https",":","\/\/huggingface.co\/datasets\/common_voice\/viewer\/en\/train",">",">",">","word_offsets","[",":3","]","[","{","'word","'",":","'WHY","'",",","'start_time","'",":","1.42",",","'end_time","'",":","1.54","}",",","{","'word","'",":","'DOES","'",",","'start_time","'",":","1.64",",","'end_time","'",":","1.9","}",",","{","'word","'",":","'MILISANDRA","'",",","'start_time","'",":","2.26",",","'end_time","'",":","2.9","}","]","``","`"]
code: def decode(
      self,
      token_ids: Union[
          int, List[int], "np.ndarray", "torch.Tensor", "tf.Tensor"
      ],
      skip_special_tokens: bool = False,
      clean_up_tokenization_spaces: bool = True,
      output_char_offsets: bool = False,
      output_word_offsets: bool = False,
      **kwargs,
  ) -> str:
      """
      Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special
      tokens and clean up tokenization spaces.
  
      Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`.
  
      Args:
          token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`):
              List of tokenized input ids. Can be obtained using the `__call__` method.
          skip_special_tokens (`bool`, *optional*, defaults to `False`):
              Whether or not to remove special tokens in the decoding.
          clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`):
              Whether or not to clean up the tokenization spaces.
          output_char_offsets (`bool`, *optional*, defaults to `False`):
              Whether or not to output character offsets. Character offsets can be used in combination with the
              sampling rate and model downsampling rate to compute the time-stamps of transcribed characters.
  
              <Tip>
  
              Please take a look at the example below to better understand how to make use of `output_char_offsets`.
  
              </Tip>
  
          output_word_offsets (`bool`, *optional*, defaults to `False`):
              Whether or not to output word offsets. Word offsets can be used in combination with the sampling rate
              and model downsampling rate to compute the time-stamps of transcribed words.
  
              <Tip>
  
              Please take a look at the example below to better understand how to make use of `output_word_offsets`.
  
              </Tip>
  
          kwargs (additional keyword arguments, *optional*):
              Will be passed to the underlying model specific decode method.
  
      Returns:
          `str` or [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`]: The list of decoded
          sentences. Will be a [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`] when
          `output_char_offsets == True` or `output_word_offsets == True`.
  
      Example:
  
      ```python
      >>> # Let's see how to retrieve time steps for a model
      >>> from transformers import AutoTokenizer, AutoFeatureExtractor, AutoModelForCTC
      >>> from datasets import load_dataset
      >>> import datasets
      >>> import torch
  
      >>> # import model, feature extractor, tokenizer
      >>> model = AutoModelForCTC.from_pretrained("facebook/wav2vec2-base-960h")
      >>> tokenizer = AutoTokenizer.from_pretrained("facebook/wav2vec2-base-960h")
      >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base-960h")
  
      >>> # load first sample of English common_voice
      >>> dataset = load_dataset("common_voice", "en", split="train", streaming=True)
      >>> dataset = dataset.cast_column("audio", datasets.Audio(sampling_rate=16_000))
      >>> dataset_iter = iter(dataset)
      >>> sample = next(dataset_iter)
  
      >>> # forward sample through model to get greedily predicted transcription ids
      >>> input_values = feature_extractor(sample["audio"]["array"], return_tensors="pt").input_values
      >>> logits = model(input_values).logits[0]
      >>> pred_ids = torch.argmax(logits, axis=-1)
  
      >>> # retrieve word stamps (analogous commands for `output_char_offsets`)
      >>> outputs = tokenizer.decode(pred_ids, output_word_offsets=True)
      >>> # compute `time_offset` in seconds as product of downsampling ratio and sampling_rate
      >>> time_offset = model.config.inputs_to_logits_ratio / feature_extractor.sampling_rate
  
      >>> word_offsets = [
      ...     {
      ...         "word": d["word"],
      ...         "start_time": round(d["start_offset"] * time_offset, 2),
      ...         "end_time": round(d["end_offset"] * time_offset, 2),
      ...     }
      ...     for d in outputs.word_offsets
      ... ]
      >>> # compare word offsets with audio `common_voice_en_100038.mp3` online on the dataset viewer:
      >>> # https://huggingface.co/datasets/common_voice/viewer/en/train
      >>> word_offsets[:3]
      [{'word': 'WHY', 'start_time': 1.42, 'end_time': 1.54}, {'word': 'DOES', 'start_time': 1.64, 'end_time': 1.9}, {'word': 'MILISANDRA', 'start_time': 2.26, 'end_time': 2.9}]
      ```"""
      # Convert inputs to python lists
      token_ids = to_py_obj(token_ids)
  
      return self._decode(
          token_ids=token_ids,
          skip_special_tokens=skip_special_tokens,
          clean_up_tokenization_spaces=clean_up_tokenization_spaces,
          output_char_offsets=output_char_offsets,
          output_word_offsets=output_word_offsets,
          **kwargs,
      )
  
imports: import json
  import os
  import sys
  import warnings
  from dataclasses import dataclass
  from itertools import groupby
  from typing import TYPE_CHECKING, Dict, List, Optional, Tuple, Union
  import numpy
  from ...tokenization_utils import PreTrainedTokenizer, _insert_one_token_to_ordered_list
  from ...tokenization_utils_base import AddedToken, BatchEncoding
  from ...utils import ModelOutput, PaddingStrategy, TensorType, add_end_docstrings, is_flax_available, is_tf_available, is_torch_available, logging, to_py_obj
label_desc: Use image node_id 17 for calling the Wav2Vec2CTCTokenizer obj's underlying member method code with example usage: obj.decode(token_ids, skip_special_tokens, clean_up_tokenization_spaces, output_char_offsets, output_word_offsets) and returns: self