class_docstr: A similar Dataset to the VideoToCharDataset/VideoToBPEDataset, but which loads tarred video files.
  
  Accepts a single comma-separated JSON manifest file (in the same style as for the VideoToCharDataset/VideoToBPEDataset),
  as well as the path(s) to the tarball(s) containing the mp4 files. Each line of the manifest should
  contain the information for one video file, including at least the transcript and name of the audio
  file within the tarball.
  
  Valid formats for the audio_tar_filepaths argument include:
  (1) a single string that can be brace-expanded, e.g. 'path/to/audio.tar' or 'path/to/audio_{1..100}.tar.gz', or
  (2) a list of file paths that will not be brace-expanded, e.g. ['audio_1.tar', 'audio_2.tar', ...].
  
  Note: For brace expansion in (1), there may be cases where `{x..y}` syntax cannot be used due to shell interference.
  This occurs most commonly inside SLURM scripts. Therefore we provide a few equivalent replacements.
  Supported opening braces - { <=> (, [, < and the special tag _OP_.
  Supported closing braces - } <=> ), ], > and the special tag _CL_.
  For SLURM based tasks, we suggest the use of the special tags for ease of use.
  
  See the WebDataset documentation for more information about accepted data and input formats.
  
  If using multiple workers the number of shards should be divisible by world_size to ensure an
  even split among workers. If it is not divisible, logging will give a warning but training will proceed.
  In addition, if using mutiprocessing, each shard MUST HAVE THE SAME NUMBER OF ENTRIES after filtering
  is applied. We currently do not check for this, but your program may hang if the shards are uneven!
  
  Notice that a few arguments are different from the AudioToCharDataset; for example, shuffle (bool) has been
  replaced by shuffle_n (int).
  
  Additionally, please note that the len() of this DataLayer is assumed to be the length of the manifest
  after filtering. An incorrect manifest length may lead to some DataLoader issues down the line.
  
  Args:
      audio_tar_filepaths: Either a list of audio tarball filepaths, or a
          string (can be brace-expandable).
      manifest_filepath (str): Path to the manifest.
      parser (callable): A callable which is used to pre-process the text output.
      int_values (bool): If true, load samples as 32-bit integers. Defauts to False.
      shuffle_n (int): How many samples to look ahead and load to be shuffled.
          See WebDataset documentation for more details.
          Defaults to 0.
      min_duration (float): Dataset parameter.
          All training files which have a duration less than min_duration
          are dropped. Note: Duration is read from the manifest JSON.
          Defaults to 0.1.
      max_duration (float): Dataset parameter.
          All training files which have a duration more than max_duration
          are dropped. Note: Duration is read from the manifest JSON.
          Defaults to None.
      blank_index (int): Blank character index, defaults to -1.
      unk_index (int): Unknown character index, defaults to -1.
      normalize (bool): Dataset parameter.
          Whether to use automatic text cleaning.
          It is highly recommended to manually clean text for best results.
          Defaults to True.
      trim (bool): Whether to use trim silence from beginning and end
          of audio signal using librosa.effects.trim().
          Defaults to False.
      bos_id (id): Dataset parameter.
          Beginning of string symbol id used for seq2seq models.
          Defaults to None.
      eos_id (id): Dataset parameter.
          End of string symbol id used for seq2seq models.
          Defaults to None.
      pad_id (id): Token used to pad when collating samples in batches.
          If this is None, pads using 0s.
          Defaults to None.
      shard_strategy (str): Tarred dataset shard distribution strategy chosen as a str value during ddp.
          -   `scatter`: The default shard strategy applied by WebDataset, where each node gets
              a unique set of shards, which are permanently pre-allocated and never changed at runtime.
          -   `replicate`: Optional shard strategy, where each node gets all of the set of shards
              available in the tarred dataset, which are permanently pre-allocated and never changed at runtime.
              The benefit of replication is that it allows each node to sample data points from the entire
              dataset independently of other nodes, and reduces dependence on value of `shuffle_n`.
  
              .. warning::
                  Replicated strategy allows every node to sample the entire set of available tarfiles,
                  and therefore more than one node may sample the same tarfile, and even sample the same
                  data points! As such, there is no assured guarantee that all samples in the dataset will be
                  sampled at least once during 1 epoch. Scattered strategy, on the other hand, on specific
                  occasions (when the number of shards is not divisible with ``world_size``), will not sample
                  the entire dataset. For these reasons it is not advisable to use tarred datasets as validation
                  or test datasets.
      global_rank (int): Worker rank, used for partitioning shards. Defaults to 0.
      world_size (int): Total number of processes, used for partitioning shards. Defaults to 0.
      return_sample_id (bool): whether to return the sample_id as a part of each sample
class_docstr_tok: ["A","similar","Dataset","to","the","VideoToCharDataset\/VideoToBPEDataset",",","but","which","loads","tarred","video","files",".","Accepts","a","single","comma-separated","JSON","manifest","file","(","in","the","same","style","as","for","the","VideoToCharDataset\/VideoToBPEDataset",")",",","as","well","as","the","path","(","s",")","to","the","tarball","(","s",")","containing","the","mp4","files",".","Each","line","of","the","manifest","should","contain","the","information","for","one","video","file",",","including","at","least","the","transcript","and","name","of","the","audio","file","within","the","tarball",".","Valid","formats","for","the","audio_tar_filepaths","argument","include",":","(","1",")","a","single","string","that","can","be","brace-expanded",",","e.g",".","'path\/to\/audio.tar","'","or","'path\/to\/audio_","{","1","..","100","}",".tar.gz","'",",","or","(","2",")","a","list","of","file","paths","that","will","not","be","brace-expanded",",","e.g",".","[","'audio_1.tar","'",",","'audio_2.tar","'",",","...","]",".","Note",":","For","brace","expansion","in","(","1",")",",","there","may","be","cases","where","`","{","x","..","y","}","`","syntax","can","not","be","used","due","to","shell","interference",".","This","occurs","most","commonly","inside","SLURM","scripts",".","Therefore","we","provide","a","few","equivalent","replacements",".","Supported","opening","braces","-","{","<","=",">","(",",","[",",","<","and","the","special","tag","_OP_",".","Supported","closing","braces","-","}","<","=",">",")",",","]",",",">","and","the","special","tag","_CL_",".","For","SLURM","based","tasks",",","we","suggest","the","use","of","the","special","tags","for","ease","of","use",".","See","the","WebDataset","documentation","for","more","information","about","accepted","data","and","input","formats",".","If","using","multiple","workers","the","number","of","shards","should","be","divisible","by","world_size","to","ensure","an","even","split","among","workers",".","If","it","is","not","divisible",",","logging","will","give","a","warning","but","training","will","proceed",".","In","addition",",","if","using","mutiprocessing",",","each","shard","MUST","HAVE","THE","SAME","NUMBER","OF","ENTRIES","after","filtering","is","applied",".","We","currently","do","not","check","for","this",",","but","your","program","may","hang","if","the","shards","are","uneven","!","Notice","that","a","few","arguments","are","different","from","the","AudioToCharDataset",";","for","example",",","shuffle","(","bool",")","has","been","replaced","by","shuffle_n","(","int",")",".","Additionally",",","please","note","that","the","len","(",")","of","this","DataLayer","is","assumed","to","be","the","length","of","the","manifest","after","filtering",".","An","incorrect","manifest","length","may","lead","to","some","DataLoader","issues","down","the","line",".","Args",":","audio_tar_filepaths",":","Either","a","list","of","audio","tarball","filepaths",",","or","a","string","(","can","be","brace-expandable",")",".","manifest_filepath","(","str",")",":","Path","to","the","manifest",".","parser","(","callable",")",":","A","callable","which","is","used","to","pre-process","the","text","output",".","int_values","(","bool",")",":","If","true",",","load","samples","as","32-bit","integers",".","Defauts","to","False",".","shuffle_n","(","int",")",":","How","many","samples","to","look","ahead","and","load","to","be","shuffled",".","See","WebDataset","documentation","for","more","details",".","Defaults","to","0.","min_duration","(","float",")",":","Dataset","parameter",".","All","training","files","which","have","a","duration","less","than","min_duration","are","dropped",".","Note",":","Duration","is","read","from","the","manifest","JSON",".","Defaults","to","0.1.","max_duration","(","float",")",":","Dataset","parameter",".","All","training","files","which","have","a","duration","more","than","max_duration","are","dropped",".","Note",":","Duration","is","read","from","the","manifest","JSON",".","Defaults","to","None",".","blank_index","(","int",")",":","Blank","character","index",",","defaults","to","-1.","unk_index","(","int",")",":","Unknown","character","index",",","defaults","to","-1.","normalize","(","bool",")",":","Dataset","parameter",".","Whether","to","use","automatic","text","cleaning",".","It","is","highly","recommended","to","manually","clean","text","for","best","results",".","Defaults","to","True",".","trim","(","bool",")",":","Whether","to","use","trim","silence","from","beginning","and","end","of","audio","signal","using","librosa.effects.trim","(",")",".","Defaults","to","False",".","bos_id","(","id",")",":","Dataset","parameter",".","Beginning","of","string","symbol","id","used","for","seq2seq","models",".","Defaults","to","None",".","eos_id","(","id",")",":","Dataset","parameter",".","End","of","string","symbol","id","used","for","seq2seq","models",".","Defaults","to","None",".","pad_id","(","id",")",":","Token","used","to","pad","when","collating","samples","in","batches",".","If","this","is","None",",","pads","using","0s",".","Defaults","to","None",".","shard_strategy","(","str",")",":","Tarred","dataset","shard","distribution","strategy","chosen","as","a","str","value","during","ddp",".","-","`","scatter","`",":","The","default","shard","strategy","applied","by","WebDataset",",","where","each","node","gets","a","unique","set","of","shards",",","which","are","permanently","pre-allocated","and","never","changed","at","runtime",".","-","`","replicate","`",":","Optional","shard","strategy",",","where","each","node","gets","all","of","the","set","of","shards","available","in","the","tarred","dataset",",","which","are","permanently","pre-allocated","and","never","changed","at","runtime",".","The","benefit","of","replication","is","that","it","allows","each","node","to","sample","data","points","from","the","entire","dataset","independently","of","other","nodes",",","and","reduces","dependence","on","value","of","`","shuffle_n","`",".","..","warning",":",":","Replicated","strategy","allows","every","node","to","sample","the","entire","set","of","available","tarfiles",",","and","therefore","more","than","one","node","may","sample","the","same","tarfile",",","and","even","sample","the","same","data","points","!","As","such",",","there","is","no","assured","guarantee","that","all","samples","in","the","dataset","will","be","sampled","at","least","once","during","1","epoch",".","Scattered","strategy",",","on","the","other","hand",",","on","specific","occasions","(","when","the","number","of","shards","is","not","divisible","with","``","world_size","``",")",",","will","not","sample","the","entire","dataset",".","For","these","reasons","it","is","not","advisable","to","use","tarred","datasets","as","validation","or","test","datasets",".","global_rank","(","int",")",":","Worker","rank",",","used","for","partitioning","shards",".","Defaults","to","0.","world_size","(","int",")",":","Total","number","of","processes",",","used","for","partitioning","shards",".","Defaults","to","0.","return_sample_id","(","bool",")",":","whether","to","return","the","sample_id","as","a","part","of","each","sample"]
code: def __len__(self):
      return len(self.manifest_processor.collection)
  
imports: import os
  from typing import Callable, Dict, Iterable, List, Optional, Tuple, Union
  import torch
  import webdataset
  from nemo.collections.asr.data.audio_to_text import cache_datastore_manifests, expand_sharded_filepaths
  from nemo.collections.asr.parts.utils.audio_utils import ChannelSelectorType
  from nemo.collections.common import tokenizers
  from nemo.collections.common.parts.preprocessing import collections, parsers
  from nemo.collections.multimodal.speech_cv.parts.preprocessing.features import VideoFeaturizer
  from nemo.core.classes import Dataset, IterableDataset
  from nemo.core.neural_types import *
  from nemo.utils import logging
label_desc: Use image node_id 8 for calling the _TarredVideoToTextDataset obj's underlying member method code with example usage: obj.__len__() and returns: len